Parameter Space Partitioning 1 Global Model Analysis by Parameter Space Partitioning

To model behavior, we need to know how models behave. This means learning what other behaviors a model can produce besides the one generated by participants in an experiment. This is a difficult problem because of the complexity of psychological models (e.g., their many parameters) and because the behavioral precision of models (e.g.,interval-scale performance) often mismatches their testable precision in experiments, where qualitative, ordinal predictions are the norm. Parameter space partitioning is a solution that evaluates model performance at a qualitative level. Given a definition of a qualitative data pattern, there exists a partition on the model's parameter space that divides it into regions that correspond to each data pattern. Markov-chain Monte Carlo methods are used to discover and define these regions. Three application examples, all using connectionist models, demonstrate its potential and versatility for studying the global behavior of psychological models. Among other things, one can easily assess how central and robust the empirical data pattern is to the model, as well as the range and characteristics of its other behaviors. Parameter Space Partitioning 3 The experimental method of scientific inquiry has proven to work quite well in psychology. Its unique blend of methodological control and statistical inference are effective for testing qualitative (i.e., ordinal) predictions derived from theories of behavior. Data collection and dissemination have become very efficient, so much so that far more may be known about a behavioral phenomenon than is reflected in its corresponding theory. That our knowledge extends beyond the reach of the theory may be a sign of productive science, and underscores the fact that theories are broad conceptualizations about behavior that cannot be expected to explain the minutia in data. Cognitive modeling is a research tool that can act as a counter-force to slow and fill this explanatory gap. It compensates for a theory’s limitations of precision in data synthesis, description, and prediction. Whether the model is an implementation of an existing theory, or a neurally inspired environment in which to study processing (e.g., information integration, representational specificity, probabilistic learning), models are rich sources of ideas and information on how to think about perception, cognition, and action. The pros and cons of various implementations can be evaluated. Inconsistencies and hidden assumptions can come to light during model creation and evaluation. In short, the modeler is forced to confront the complexity of what is being modeled, and in the process, can gain insight into the relationship between variables and the functionality of the model (see Shiffrin & Nobel, 1998 for a personal account of this process). Of course, the virtues of modeling are accompanied by vices. One of the more serious, often leveled against connectionist models (Dawson & Shamanski, 1994; McCloskey, 1991) but by no means restricted to them, is that model behavior can be mysterious and difficult to understand, which can defeat the purpose of modeling. A model should not be as (or more) complex than the data being described. Rather, models should offer a simpler and tractable description. The preceding observations are not so much a comment about models or modeling per se, but a comment about the need for methods for analyzing model behavior. The computational power of cognitive models requires correspondingly sophisticated tools to study them. The wide variety of models in the discipline makes the need for universal tools all the more pressing and challenging. In this paper, we introduce a versatile model analysis and comparison method. We begin by situating it in relation to existing methods. Methods of Model Analysis Model analysis methods can be differentiated along two dimensions, whether they measure a model’s local or global behavior, and whether model behavior is evaluated quantitatively or qualitatively. The position of many analysis methods along these, largely independent, dimensions is shown schematically in Figure 1. In the following discussion, quantitative methods are reviewed before qualitative ones. Figure 1 here Quantitative Model Analysis Most quantitative evaluations of models are local. That is, they consider the behavior of the model only at its best-fitting parameter values. In contrast, global methods aim to elucidate a model's behavior across the full range of its parameter values. Not surprisingly, the two approaches are complementary in what they tell us about model behavior. Local Quantitative Methods The most common form of local model analysis is data fitting, in which a model is tested by measuring how closely it approximates (i.e., fits) data generated in an experiment. Because Parameter Space Partitioning 4 data are a reflection of the psychological process under study, a good fit to the data is a necessary condition a model must satisfy to be taken seriously. A good fit determines how well a model passes the sufficiency test of mimicking human performance. It is especially useful in the early stages of model development as a quick and easy check on sufficiency. Quantitative measures of goodness-of-fit include percent variance accounted for, root mean square deviation, and maximum likelihood. Although a good fit makes a model a member of the class of possible contenders, the size of this class will depend on the amount and variety of evidence already accumulated. As more data are collected about an underlying cognitive process, the number of viable models will diminish. If two models that one is comparing fit all of the data similarly well, other analysis methods are needed to choose between them. Another local method that is useful for probing model behavior more deeply than goodness-of-fit is a sensitivity analysis, in which a model’s parameters are varied around its bestfitting values to learn how robust model behavior is to slight variations of those parameters. If a good fit reflects a fundamental property of the model, then this behavior should be stable across reasonable parameter variation. Another reason a model should satisfy this criterion is that human data are noisy. A model should not be so sensitive that its behavior changes noticeably when noise is encountered. Cross validation, in which a model is fit to the second of two data sets using the best fitting parameter values from fitting the first data set, is a fit-based approach to quantifying this sensitivity (Stone, 1974; Browne, 2000). Global Quantitative Methods A drawback of local analysis methods is the very fact that they are local. Each fit provides a view of the model's behavior at a particular point in its parameter space, but does not provide any information about how it behaves at other parameter settings. This can be particularly disconcerting if the behavior of the model is sensible only at a few settings. Furthermore, relying on purely local methods leaves one with a few “snapshots” of model performance that are difficult to piece together into a comprehensive understanding of the model. The task of comparing two models is even more arduous with local methods. For these reasons, researchers have begun developing global analysis techniques. They are intended to augment local methods, not replace them. From a global perspective, the goal is to learn something about the full range of behaviors that a model exhibits. By doing so, we can gain a deeper understanding of the model and how it compares to competing models. Two of the most popular quantitative global methods are Bayesian methods (e.g., Myung & Pitt 1997; Kass & Raftery 1995) and minimum description length (MDL; e.g., Rissanen 1996, 2001; Grünwald 1998; Grünwald, Myung, & Pitt, in press). In both, the focus is on predictions made by the model at all of its parameter values. This global perspective yields a natural measure of a model’s a priori data-fitting potential (i.e., the model’s flexibility or complexity in fitting future, unseen data sets; Myung 2000). Although both methods are statistically rigorous, technical requirements currently limit their application to the diverse range of models in psychology. More recently, Navarro et al (2004; Kim, Navarro, Pitt, & Myung, 2004; see also Wagenmakers, Ratcliff, Gomez, & Iverson, 2004) introduced a global quantitative method, called landscaping, that was developed with an eye toward increased versatility and informativeness about model discriminability. The essence of the technique involves determining how well two models fit each other's data. What is being measured is the extent to which two models mimic one another. The approach is attractive since landscapes are relatively easy to create, requiring only a comparison of fits to data sets. In addition, landscapes can be Parameter Space Partitioning 5 used to assess the informativeness of data in distinguishing pairs of models by inspecting where within these landscapes experimental data are located. In short, the landscape provides a global perspective from which to understand the relationship between two models and their fits to data. However, like MDL and Bayesian methods, landscaping requires that the models make quantitative predictions. A method free from this restriction would have much wider applicability. Qualitative Methods Implicit in the use of quantitative methods of model analysis is that the goal of modeling is to approximate closely empirical data. Although there are good reasons for doing so, one must be careful to avoid modeling the quantitative minutiae of a data set while missing theoretically important qualitative properties or trends in the data. In the words of Box (1976, p. 792) “Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad”. Thus, it is important not to lose sight of qualit