Modeling Complex Tasks: An Individual Difference Approach

Modeling Complex Tasks: An Individual Difference Approach John Rehling (rehling@andrew.cmu.edu) Marsha Lovett (lovett@cmu.edu) Christian Lebiere (cl@cmu.edu) Lynne Reder (reder@cmu.edu) Baris Demiral (baris@andrew.cmu.edu) Carnegie Mellon University, 5000 Forbes Avenue Pittsburgh, PA 15213 USA Abstract model individual differences in working memory capacity (Lovett, Reder, & Lebiere, 1999). This was later demonstrated empirically by using individual performance in one simple memory task to measure the W value that best fit the individual's performance. The diagnostic memory task is called MODS, or the modified digit span task. In each MODS trial, subjects are presented strings of digits to be read aloud in synchrony with a metronome beat and are required to remember the final digit from each string for later recall. After a certain number of digit strings are thus presented, a recall prompt cues the subject to report the memory digits in the order they were presented. Each subject's MODS score was used to estimate their individual W value, which was then plugged into an ACT-R model of a separate working memory task, and the model output was used to predict individual data on that second task (Daily, Lovett, & Reder, Previous and concurrent work by other groups suggested a number of positive characteristics that might be combined into a single, more powerful methodology. ACT-R parameters had been manipulated (Taatgen, 2001) in a model of individual differences in learning, although the “individuals” in that work were simulated, and corresponded to types of individuals, not to actual subjects. In that work, performance in a complex task was related to individual difference parameters across the simulated individuals. Earlier work in modeling also accounted for relationships between ability in one task and ability in another, but making assessments on the group, not individual, level (Just, Carpenter, and Shell, 1990). A complementary approach measures individual performance on complex tasks, and utilizes statistical methods such as intercorrelation matrices, allowing predictions of individual performance on one task based upon measurements of performance on other tasks in the matrices, making no use of any particular theory of cognition (for example, Ackerman and Kanfer, 1993). All of this previous work, we felt, pointed towards a methodology that combined a number of positive features from these complementary approaches into a It is the usual case in cognitive modeling that a model's output is compared to the average of a number of subjects, in which case the enterprise of modeling is apparently to capture the behavior of the typical individual. Our approach is to administer two simple tasks to each subject, using performance on those tasks as measures of individual ability. Those measures are then used as the values for parameters in an ACT-R model of a more complex task, so that the model can predict individual performance on that task. Introduction Work in cognitive modeling, when it seeks validation in the performance of human subjects, is almost unanimously concerned with the average performance of many subjects. For many purposes, however, it is desirable to be able to model or predict individual performance. We present here the first work to use a fine-grained cognitive model to predict individual performance in a complex task. The ACT-R architecture, the basis of a great deal of work in cognitive modeling, has a detailed, well- developed theory of cognition – perception, learning, performance, and so on (Anderson and Lebiere, 1998). The architecture by necessity contains a number of parameters that can be used to fix levels of performance in, e.g., memory, to realistic levels. The ACT-R community has by custom sought universal values for these parameters wherever possible, finding values work across tasks, optimizing how well the model fits the data of the average subject. These parameters are each meaningful, each parameter determining the model's behavior in one specific way. For example, there is a parameter called W that determines the sum of the activations of all the pieces of information that may be retrieved at any point in time. It therefore controls the model's working memory capacity. Extensive empirical work in ACT- R modeling (again, of the kind where the model was meant to predict the average subject) found that a W value of 1.0 produces very good fits with subject data. It was later postulated, however, that the W parameter could be meaningfully varied in order to