Abstract:
Disclosed herein is a method of analyzing examinee item response data comprising constructing a diagnosis model for reporting skill profiles of examinees, wherein the diagnosis model comprises at least a variable representing unobserved subpopulations, creating an item design matrix, distributing examinees across the unobserved subpopulations, iteratively estimating values for a plurality of variables within the diagnosis model, and reporting the estimated values to a user.

Description:
REFERENCE TO RELATED APPLICATIONS 
     This application claims priority to U.S. Application No. 60/824,837, filed Sep. 7, 2006, and entitled “Mixture General Diagnostic Model.” 
    
    
     BACKGROUND OF INVENTION 
     Item Response Theory (IRT) is a body of theory used in the field of psychometrics. In IRT, mathematical models are applied to analyze data from tests or questionnaires in order to measure abilities and attitudes studied in psychometrics. One branch of IRT is diagnostic models. Diagnostic models may be used to provide skill profiles, thereby offering additional information about the examinees. One central tenet behind diagnostic models is that different items tap into different sets of skills or examinee attributes and that experts can generate a matrix of relations between items and skills required to solve these items. A diagnostic model according to the prior art, the General Diagnostic Model (GDM), will hereinafter be described. 
     In order to define the GDM, several assumptions must first be presented. Assume an I-dimensional categorical random variable {right arrow over (x)}=(x 1 , . . . , x 1 ) with x i ε{0, . . . , m i ) for iε{1, . . . , I}, which may be referred to as a response vector. Further assume that there are N independent and identically distributed (i.i.d.) realizations {right arrow over (x)} 1 , . . . , {right arrow over (x)} N  of this random variable {right arrow over (x)}, so that x ni  denotes the i-th component of the n-th realization {right arrow over (x)} n . In addition, assume that there are N unobserved realizations of a K-dimensional categorical variable, {right arrow over (a)}=(a 1 , . . . , a k ), so that the vector
 
( {right arrow over (x)}   n   ,{right arrow over (a)}   n )=( x   n1   , . . . , x   n1   , a   n1   , . . . , a   nK )
 
exists for all nε{1, . . . , N} The data structure
 
( X,A )=(( {right arrow over (x)}   n   ,{right arrow over (a)}   n )) n=1, . . . , N  
 
may be referred to as the complete data, and ({right arrow over (x)} n )=(({right arrow over (x)} n ,{right arrow over (a)} n )) n=1, . . . , N  is referred to as the observed data matrix. Denote ({right arrow over (a)} n ) n=1, . . . , N  as the latent skill or attribute patterns, which is the unobserved target of inference.
 
     Let P({right arrow over (a)})=P({right arrow over (A)}=(a 1 , . . . , a K ))&gt;0 for all {right arrow over (a)} denote the nonvanishing discrete count density of {right arrow over (a)}. Assume that the conditional discrete count density P(x 1 , . . . , x 1 |{right arrow over (a)}) exists for all {right arrow over (a)}. Then the probability of a response vector {right arrow over (x)} can be written as 
     
       
         
           
             
               P 
               ⁡ 
               
                 ( 
                 
                   x 
                   ⇀ 
                 
                 ) 
               
             
             = 
             
               
                 ∑ 
                 
                   a 
                   ⇀ 
                 
               
               ⁢ 
               
                   
               
               ⁢ 
               
                 
                   P 
                   ⁡ 
                   
                     ( 
                     
                       a 
                       ⇀ 
                     
                     ) 
                   
                 
                 ⁢ 
                 
                   P 
                   ⁡ 
                   
                     ( 
                     
                       
                         x 
                         
                           1 
                           ⁢ 
                           
                               
                           
                         
                       
                       , 
                       … 
                       ⁢ 
                       
                           
                       
                       , 
                       
                         
                           x 
                           I 
                         
                         ❘ 
                         
                           a 
                           ⇀ 
                         
                       
                     
                     ) 
                   
                 
               
             
           
         
       
     
     Thus far, no assumptions have been made about the specific form of the conditional distribution of {right arrow over (x)} given {right arrow over (a)}, other than that P(x 1 , . . . , x 1 |{right arrow over (a)}) exists. For the GDM, local independence (LI) of the components {right arrow over (x)} given {right arrow over (a)} may be assumed, which yields 
               P   ⁡     (       x     1   ⁢               ,   …   ⁢           ,       x   I     ❘     a   ⇀         )       =       ∑     i   =   1     I     ⁢           ⁢       p   i     ⁡     (     x   =       x   i     ❘     a   ⇀         )               
so that the probability p i  (x=x i |{right arrow over (a)}) is the one component left to be specified to arrive at a model for P({right arrow over (x)}).
 
     Logistic models have secured a prominent position among models for categorical data. The GDM may also be specified as a model with a logistic link between an argument, which depends on the random variables involved and some real valued parameters, and the probability of the observed response. 
     Using the above definitions, the GDM may be defined as follows. Let
 
 Q =( q   ik ), i=1, . . . , I, k=1, . . . , K
 
be a binary I×K matrix, that is q ik ε{0,1}. Let
 
(γ ikx ), i=1, . . . , I, k=1, . . . , K, x=1, . . . , m i  
 
be a cube of real valued parameters, and let β ix  for i=1, . . . , I and xε{0, . . . , m i ) be real valued parameters. Then define
 
     
       
         
           
             
               
                 p 
                 i 
               
               ⁡ 
               
                 ( 
                 
                   x 
                   ❘ 
                   
                     a 
                     ⇀ 
                   
                 
                 ) 
               
             
             = 
             
               
                 exp 
                 ⁡ 
                 
                   ( 
                   
                     
                       β 
                       ix 
                     
                     + 
                     
                       
                         ∑ 
                         k 
                       
                       ⁢ 
                       
                         
                           γ 
                           ikx 
                         
                         ⁢ 
                         
                           h 
                           ⁡ 
                           
                             ( 
                             
                               
                                 q 
                                 ik 
                               
                               , 
                               
                                 a 
                                 k 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                   ) 
                 
               
               
                 1 
                 + 
                 
                   
                     ∑ 
                     
                       y 
                       = 
                       1 
                     
                     
                       m 
                       i 
                     
                   
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     exp 
                     ⁡ 
                     
                       ( 
                       
                         
                           β 
                           iy 
                         
                         + 
                         
                           
                             ∑ 
                             k 
                           
                           ⁢ 
                           
                             
                               γ 
                               iky 
                             
                             ⁢ 
                             
                               h 
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     q 
                                     ik 
                                   
                                   , 
                                   
                                     a 
                                     k 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       ) 
                     
                   
                 
               
             
           
         
       
     
     It may convenient to constrain the γ ikx  somewhat an to specify real valued function h(q ik , a k ) and the a k  in a way that allows emulation of models frequently used in educational measurements and psychometrics. It may be convenient to choose h(q ik , a k )=q ik a k , and γ ikx=x γ ik . 
     The GDM has some unfortunate limitations. Primarily, it is not equipped to handle unobserved partitions, or subpopulations, in the examinees. 
     Thus, there is a need for a diagnostic model that may be extended to handle unobserved subpopulations. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
       Aspects, features, benefits and advantages of the embodiments of the present invention will be apparent with regard to the following description, appended claims and accompanying drawings where: 
         FIG. 1  is a flow chart illustrating an exemplary method of applying diagnosis models to examinee data; 
         FIG. 2  is a diagram of an exemplary system upon which the above-referenced method may operate. 
     
    
    
     SUMMARY OF THE INVENTION 
     Disclosed herein is a method of analyzing examinee item response data comprising constructing a diagnosis model for reporting skill profiles of examinees, wherein the diagnosis model comprises at least a variable representing unobserved subpopulations, creating an item design matrix, distributing examinees across the unobserved subpopulations, iteratively estimating values for a plurality of variables within the diagnosis model, and reporting the estimated values to a user. 
     Also disclosed herein is method of analyzing examinee item response data comprising constructing a diagnosis model for reporting skill profiles of examinees, wherein the diagnosis model comprises at least a variable representing unobserved subpopulations and a cluster variable, creating an item design matrix, distributing examinees across the unobserved subpopulations, iteratively estimating values for a plurality of variables within the diagnosis model, and reporting the estimated values to a user. 
     DETAILED DESCRIPTION OF THE INVENTION 
     As mentioned above, the GDM has some limitations. In particular, the GDM does not take into account different behaviors among various subpopulations. That is, the probability of an observation {right arrow over (x)} may depend not only the unobserved latent trait, {right arrow over (a)}, but also on a subpopulation identifier g. The subpopulation identifier g may be observed, but is often unobserved. Mixture distribution models are useful because observations from different subpopulations may either differ in their distribution of skills or in their approach to the test items, or in both. A discrete mixture distribution in the setup of random variables as introduced above includes an unobserved grouping indicator g n  for n=1, . . . , N. The complete data for examinee n then becomes ({right arrow over (x)} n , {right arrow over (a)} n , g n ), of which only {right arrow over (x)} n  is observed in mixture distribution models. A mixture GDM, or MGDM, will hereinafter be disclosed. 
     In order accommodate different groups, the conditional independence assumption has to be modified, that is 
               P   ⁡     (         x   ⇀     ❘     a   ⇀       ,   g     )       =       P   ⁡     (       x     1   ⁢               ,   …   ⁢           ,       x   I     ❘     a   ⇀       ,   g     )       =       ∏     i   =   1     I     ⁢           ⁢       p   i     ⁡     (       x   =       x   i     ❘     a   ⇀         ,   g     )                 
Moreover, assume that the conditional probability of the components x i  of {right arrow over (x)} depends on nothing but {right arrow over (a)} and g, that is,
 
                     P   ⁡     (         x   ⇀     ❘     a   ⇀       ,   g   ,   z     )       =         ∏     i   =   1     I     ⁢           ⁢       p   i     ⁡     (       x   =       x   i     ❘     a   ⇀         ,   g     )         =     P   ⁡     (         x   ⇀     ❘     a   ⇀       ,   g     )                 Equation   ⁢           ⁢   1               
for any random variable z. In mixture models, when the g n  are not observed, the marginal probability of a response vector {right arrow over (x)} needs to be found, that is,
 
                     P   ⁡     (     x   ⇀     )       =       ∑   g     ⁢           ⁢       π   g     ⁡     (       x   ⇀     ❘   g     )                 Equation   ⁢           ⁢   2               
where P({right arrow over (x)}|g)=Σ {right arrow over (a)} p({right arrow over (a)}|g)P(x|{right arrow over (a)}, g). The π g =P(G=g) may be referred to as mixing proportions, or class sizes. The class-specific probability of a response vector {right arrow over (x)} given skill pattern {right arrow over (a)} the MGDM may then be defined as
 
                     P   ⁡     (         x   ⇀     ❘     a   ⇀       ,   g     )       =         ∏     i   =   1     I     ⁢           ⁢     P   ⁡     (         x   i     ❘     a   ⇀       ,   g     )         =       ∏     i   =   1     I     ⁢           ⁢     [       exp   ⁡     (       β   ixy     +       ∑   k     ⁢       x   i     ⁢     γ   ikg     ⁢     q   ik     ⁢     a   k           )         1   +       ∑   y     ⁢           ⁢     exp   ⁡     (       β   iyg     +       ∑   k     ⁢     y   ⁢           ⁢     γ   ikg     ⁢     q   ik     ⁢     a   k           )             ]                 Equation   ⁢           ⁢   3               
with class-specific item difficulties β ixg . The γ ikg  are the slope parameters relating skill k to item i in class g. This equation may be used to model, for instance, both polytomous and binary data.
 
     One special case of the MGDM is a model that assumes measurement invariance across populations, which is expressed in the equality of p({right arrow over (x)}|{right arrow over (a)}, g) across groups or, more formally
 
 P ( x   i   |{right arrow over (a)},g )= p ( x   i   |{right arrow over (a)},c ) for all iε{1, . . . , I} and all g, cε{1, . . . , G}
 
     Under this assumption, the MGDM equation may be rewritten without the group index g in the conditional response probabilities, so that 
                     P   ⁡     (     x   ⇀     )       =         ∑   g     ⁢           ⁢       π   g     ⁢     P   ⁡     (       x   ⇀     ❘   g     )           =       ∑   g     ⁢           ⁢       π   g     ⁢       ∑     a   ⇀       ⁢           ⁢       p   ⁡     (       a   ⇀     ❘   g     )       ⁢       ∏     i   =   1     I     ⁢     P   ⁡     (       x   i     ❘     a   ⇀       )                           Equation   ⁢           ⁢   4               
Note that the differences between groups are only present in the p({right arrow over (a)}|g), so that the skill distribution is the only component with a condition on g in the above equation.
 
     The MGDM may be expanded to introduce an additional structure, referred to as a cluster variable. This expanded model may be referred to as a Hierarchical GDM, or HGDM. This cluster variable may be used to account for correlations in the data. One example for clustered data is the responses to educational assessments sampled from students within schools or classrooms. For instance, it seems plausible to assume that students within schools are more similar than students across schools. In addition to the grouping variable g, the hierarchical extension of the GDM assumes that each observation n is characterized by an outcome s n  on a clustering variable s. The clusters identified by this outcome may be schools, classrooms, or other sampling units representing the hierarchical structure of the data collection. The (unobserved) group membership g, is thought of as an individual classification variable; for two examinees n≠m there may be two different group memberships, that is, both g n =g m  and g n ≠g m  are both permissible even if they belong to the same cluster (i.e., s n =s m ). 
     Moreover, it may be assumed that the skill distribution depends only on the group indicator g and no other variable, that is,
 
 P ( {right arrow over (a)}|g,z )= P ( {right arrow over (a)}|g )  Equation 5
 
for any random variable z. More specifically, for the clustering variable s,
 
               P   ⁡     (   g   )       =       ∑     s   =   1     S     ⁢           ⁢       p   ⁡     (   s   )       ⁢       P   ⁡     (     g   ❘   s     )       .               
With Equation 5,
 
                 P   ⁡     (     g   ❘   s     )       ⁢     P   ⁡     (       a   ⇀     ❘   g     )         =         p   ⁡     (     g   ❘   s     )       ⁢     P   ⁡     (         a   ⇀     ❘   g     ,   s     )         =     P   ⁡     (       a   ⇀     ,     g   ❘   s       )                   for               P   ⁡     (         x   ⇀     ❘   g     ,   s     )       =         ∑   a     ⁢           ⁢       P   ⁡     (         a   ⇀     ❘   g     ,   s     )       ⁢     P   ⁡     (         x   ⇀     ❘     a   ⇀       ,   g   ,   s     )           =         ∑   a     ⁢           ⁢       p   ⁡     (       a   ⇀     ❘   g     )       ⁢     P   ⁡     (         x   ⇀     ❘     a   ⇀       ,   g     )           =     P   ⁡     (       x   ⇀     ❘   g     )                 
with Equations 1 and 5. Then the marginal distribution of a response pattern {right arrow over (x)} in the HGDM is given by
 
                     P   ⁡     (     x   →     )       =       ∑   s     ⁢           ⁢       p   ⁡     (   s   )       ⁢       ∑   g     ⁢           ⁢       P   ⁡     (     g   ❘   s     )       ⁢       ∑   a     ⁢           ⁢       P   ⁡     (       a   →     ❘   g     )       ⁢     (     P   ⁡     (         x   →     ❘     a   →       ,   g     )                             Equation   ⁢           ⁢   6               
where, as above with respect to the MGDM, the p({right arrow over (a)}|g) denote the distribution of the skill patterns in group g, and the p({right arrow over (x)}|{right arrow over (a)}, g) denote the distribution of the response vector {right arrow over (x)} conditional on skill pattern {right arrow over (a)} and group g. An HGDM that assumes measurement invariance across clusters and across groups is defined by
 
                     P   ⁡     (     x   →     )       =       ∑   s     ⁢           ⁢       p   ⁡     (   s   )       ⁢       ∑   g     ⁢           ⁢       P   ⁡     (     g   ❘   s     )       ⁢       ∑   a     ⁢           ⁢       P   ⁡     (       a   ⇀     ❘   g     )       ⁢     (     P   ⁡     (       x   →     ❘     a   →       )                             Equation   ⁢           ⁢   7               
with conditional response probabilities p(x|{right arrow over (a)})=Π i p(x i |{right arrow over (a)}) that do not depend on cluster or group variables.
 
     The increase in complexity of HGDMs over, for instance, the MGDM, lies in the fact that the group distribution P(g|s) depends on the cluster variable s. This increases the number of group or class size parameters depending on the number of clusters # {s: sεS}. 
     Hereinafter, an exemplary method of applying the above models to examinee data will be described with respect to  FIG. 1 . First, data is defined  110 . More specifically, defining data may comprise steps of defining item response variables or defining mixture components. Then, a skill space is created  120 . More specifically, creating a skill space may comprise defining the number of skills and defining the assumed skill levels (e.g., how many, which numerical anchor) for each of the assumed skill variables. Then, an item design matrix (e.g., a Q-matrix) is created  130 , which may relate the item response variables to the assumed skill variables. Then, data for each examinee may be read  140 . For instance, this step may comprise reading item response variables and grouping variables. Then, the examinees may be randomly distributed  150  across groups. Then, initial skill distributions are calculated  160 . 
     Then, the MGDM or HGDM statistics are calculated  170 . In a preferred embodiment, this is performed using an expectation-maximization (EM) algorithm such as the one disclosed in “Multilevel latent class models,” by J. K Vermunt, published in Sociological Methodology 33, which is incorporated by reference herein. The EM algorithm will be described in more detail as it applies to the HGDM. One of ordinary skill will appreciate that this method is easily applicable to the MGDM as well. 
     Since the data are structured hierarchically, the first step is to define the complete data for the HGDM. Let S denote the number of clusters in the sample, and let N s  denote the number of examinees in cluster s, for s=1, . . . , S. Then, let x ins  denote the i-th response of the n-th examinee in cluster s and let {right arrow over (x)} ns  denote the complete observed response vector of examinee n in cluster s. Further, let a kns  denote the k-th skill of examinee n in cluster s and let {right arrow over (a)} ns  denote the skill pattern of examinee n in cluster s. Finally, let g ns  denote the group membership of examinee n in cluster s. Note that only the x ins  are observed, as are the cluster sizes N s  and the number of clusters S. The s kns  and g ns  are unobserved and have to be inferred by making model assumptions and calculating posterior probabilities such as P(g|s) and P({right arrow over (a)}, g|{right arrow over (x)}, s). 
     For the complete data (i.e., the observed data {right arrow over (x)} in conjunction with the unobserved skill profiles {right arrow over (a)} and group membership g), the marginal likelihood is 
             L   =       ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢           ⁢     P   ⁡     (         x   →     ns     ,       a   →     ns     ,         g   →     ns     ;   s       )                 
that is, a sum over cluster-specific distributions of the complete data. With the above assumptions,
 
             L   =       ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢           ⁢       P   ⁡     (           x   →     ns     ❘       a   →     ns       ,       g   →     ns       )       ⁢     p   ⁡     (         a   →     ns     ❘       g   →     ns       )       ⁢     p   ⁡     (       g   ns     ❘   s     )                   
which equals
 
             L   =       L     x   →       ×     L   a     ×     L     g   →                   with                 L     x   →       ×     L   a     ×     L     g   →         =       (       ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢           ⁢     P   ⁡     (           x   →     ns     ❘       a   →     ns       ,     g   ns       )           )     ⁢     (       ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢           ⁢     p   ⁡     (           a   →     ns     ❘       a   →     ns       ,     g   ns       )           )     ⁢     (       ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢           ⁢     p   ⁡     (       g   ns     ❘   s     )           )             
Note that these components may be rearranged and rewritten as
 
               L     x   →       =         ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢       ∏     i   =   1     I     ⁢     P   ⁡     (         x   ns     ❘       a   →     ns       ,     g   ns       )             =       ∏   g     ⁢       ∏     a   →       ⁢       ∏   i     ⁢       ∏   x     ⁢       P   ⁡     (         X     i   ⁢               =     x   ❘     a   →         ,   g     )           n   i     ⁡     (     x   ,     a   →     ,   g     )                         
with n(x i , i, {right arrow over (a)}, g)=Σ s n(x i , i, {right arrow over (a)}, g, s) is the frequency of category x i  responses on item I for examinees with skill pattern {right arrow over (a)} in group g. Also,
 
               L   g     =         ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢     p   ⁡     (         a   →     ns     ❘     g   ns       )           =       ∏     a   ⇀       ⁢       ∏   g     ⁢       p   ⁡     (       a   →     ❘   g     )         n   ⁡     (       a   →     ;   g     )                     
where n({right arrow over (a)}; g) is the frequency of skill pattern {right arrow over (a)} in group g. Finally,
 
               L   g     =         ∏     s   =   1     S     ⁢           ⁢       ∏     n   =   1       N   s       ⁢     p   ⁡     (       g   ns     ❘   s     )           =       ∏   s     ⁢       ∏   g     ⁢       p   ⁡     (     g   ❘   s     )         n   ⁡     (     g   ;   s     )                     
holds. The n(g; s) represents the frequency of group membership in g in cluster S.
 
     The EM algorithm cycles through the generation of expected values and the maximization of parameters given these preliminary expectations until convergence is reached. This process is well known in the art and is described in, for instance, The EM-algorithm and extensions by McLachlan et al., published by Wiley, which is incorporated by reference herein. For the HGDM, there are three different types of expected values to be generated in the E-step: First, {circumflex over (n)} i (x, {right arrow over (a)}, g)=Σ s Σ n 1{x ins =s}P({right arrow over (a)}, g|{right arrow over (x)} ns , s) is the expected frequency of response x to item I for examinees with skill pattern {right arrow over (a)} in group g, estimated across clusters and across examinees within clusters. Second, {circumflex over (n)}({right arrow over (a)}, g)=Σ s Σ n P({right arrow over (a)}, g|{right arrow over (x)} ns , s) is the expected frequency of skill pattern {right arrow over (a)} and group g, estimated across clusters and across examinees within clusters. Finally, {circumflex over (n)}(g,s)=Σ n P(g|{right arrow over (x)} ns , s) is the expected frequency of group g in cluster s, estimated across examinees in that cluster. For the first and second type of the required expected counts, this involves estimating 
               P   ⁡     (       a   →     ,     g   ❘     x   →       ,   s     )       =         P   ⁡     (       x   →     ,   s   ,     a   →     ,   g     )           ∑   g     ⁢           ⁢     P   ⁡     (       x   →     ,   s   ,   g     )           =         P   ⁡     (         x   →     ❘     a   →       ,   g     )       ⁢     p   ⁡     (       a   →     ❘   g     )       ⁢     p   ⁡     (     g   ❘   s     )             ∑   g     ⁢           ⁢     P   ⁡     (       x   →     ,   s   ,   g     )                       with               P   ⁡     (       a   →     ,     g   ❘     x   →       ,   s     )       =         P   ⁡     (       x   →     ,   s   ,     a   →     ,   g     )           ∑   g     ⁢           ⁢     P   ⁡     (       x   →     ,   s   ,   g     )           =         P   ⁡     (         x   →     ❘     a   →       ,   g     )       ⁢     p   ⁡     (       a   →     ❘   g     )       ⁢     p   ⁡     (     g   ❘   s     )             ∑   g     ⁢           ⁢     P   ⁡     (       x   →     ,   s   ,   g     )                   
for each response pattern {right arrow over (x)} ns , for s=1, . . . , S and n=1, . . . , N s . For the third type of expected count, use
 
               p   ⁡     (       g   ❘     x   →       ,   s     )       =       ∑   a     ⁢     P   ⁡     (       a   →     ,     g   ❘     x   →       ,   s     )               
which is equivalent to
 
               p   ⁡     (       g   ❘     x   -&gt;       ,   s     )       =         P   ⁡     (       x   -&gt;     ,   s   ,   g     )           ∑   g     ⁢     P   ⁡     (       x   -&gt;     ,   s   ,   g     )           =         ∑     a   -&gt;       ⁢       P   ⁡     (         x   -&gt;     ❘     a   -&gt;       ,   g     )       ⁢     p   ⁡     (       a   -&gt;     ❘   g     )       ⁢     p   ⁡     (     g   ❘   s     )               ∑   g     ⁢     [       ∑     a   -&gt;       ⁢       P   ⁡     (         x   -&gt;     ❘     a   -&gt;       ,   g     )       ⁢     p   ⁡     (       a   -&gt;     ❘   g     )       ⁢     p   ⁡     (     g   ❘   s     )           ]                 
This last probability then allows one to estimate the class membership g given both the observed responses {right arrow over (x)} and the known cluster membership s. The utility of the clustering variable may be evaluated in terms of increase of the maximum a postereori probabilities p(g|{right arrow over (x)}, s) over p(g|{right arrow over (x)}). If the clustering variable s is informative for the classification g, a noticeable increase of the maximum posterior probabilities should be observed. The improvement should also be seen in terms of the marginal log-likelihood if s is informative for g.
 
     Referring back to  FIG. 1 , once the statistics have converged, model fit measures, log-likelihoods, and related measures may be calculated  180 . Then, item fit measures are calculated  185 . Finally, person-based outcome statistics may be calculated  190 . Calculating person-based outcome statistics may include calculating most probable group membership and most probable skill level for each skill for each examinee. These statistics, item-fit measures, log-likelihoods, and model fit measures may be presented to a user in a human-readable fashion, such as a computer display, or printout. 
       FIG. 2  is a block diagram of exemplary internal hardware that may be used to contain or implement the program instructions of a system embodiment. Referring to  FIG. 2 , a bus  228  serves as the main information highway interconnecting the other illustrated components of the hardware. CPU  202  is the central processing unit of the system, performing calculations and logic operations required to execute a program. Read only memory (ROM)  218  and random access memory (RAM)  220  constitute exemplary memory devices. 
     A disk controller  204  interfaces with one or more optional disk drives to the system bus  228 . These disk drives may be external or internal floppy disk drives such as  210 , CD ROM drives  206 , or external or internal hard drives  208 . As indicated previously, these various disk drives and disk controllers are optional devices. 
     Program instructions may be stored in the ROM  218  and/or the RAM  220 . Optionally, program instructions may be stored on a computer readable medium such as a floppy disk or a digital disk or other recording medium, a communications signal or a carrier wave. 
     An optional display interface  222  may permit information from the bus  228  to be displayed on the display  224  in audio, graphic or alphanumeric format. Communication with external devices may optionally occur using various communication ports  226 . An exemplary communication port  226  may be attached to a communications network, such as the Internet or an intranet. 
     In addition to the standard computer-type components, the hardware may also include an interface  212  which allows for receipt of data from input devices such as a keyboard  214  or other input device  216  such as a remote control, pointer and/or joystick. 
     The diagnostic models described herein may be used in connection with, for instance and without limitation, English language testing, national large scale assessments, international assessments, or K-12 accountability testing. For instance, the MGDM and HGDM may be used in connection with Test of Engliesh as a Foreign Language (TOEFL) results. 
     While illustrative embodiments of the invention have been shown herein, it will be apparent to those skilled in the art that the invention may be embodied still otherwise without departing from the spirit and scope of the claimed invention.