Decision Criteria for Model Comparison Using Cross-Fitting

A promising method for model evaluation and comparison that also takes into account model complexity is the cross-fitting method proposed by Wagenmakers, Ratcliff, Gomez, and Iverson (2004). A crucial part of this method involves assigning new empirical samples to one of two empirical distributions with the highest possible accuracy. Existing cognitive modeling research provides no systematic investigation of how to best achieve such an accurate assignment. This is disadvantageous for at least two reasons: First, no information is available regarding which methods give accurate assignments (and, thus, accurate model comparison) in which situations. Second, missing information on possible methods may prevent modelers from using the cross-fitting method instead of a misleading naı̈ve fitting approach. In this contribution we examine the properties of 9 assignment methods across a range of different pairs of distributions. Of all considered methods, a variant of the k-nearest neighbor rule was found to be most successful. A key property of any computational cognitive model is its ability to account for empirical data from the domain of human cognition that the model is supposed to address. If several candidate models are available for the same domain, a common aim in cognitive modeling is to identify that model that accounts best for the available data, that is, the model that provides the closest approximation to the processes that generated the empirical data. A naı̈ve approach to identify the best accounting model is to fit each model to the available data and choose the model that provides the tightest fit as indicated, for instance, by the models’ Root Mean Squared Error (RMSE). Such an approach is problematic, because it does not take into account the complexity of the compared models. As a result, there is a tendency for overfitting and for selecting more complex models even if simpler models provide the better approximation to the actual generating process (Pitt & Myung, 2002). A number of methods taking into account model complexity have been proposed to avoid the pitfalls of the naı̈ve approach (see Shiffrin, Lee, Kim, & Wagenmakers, 2008, for an overview). One of these methods is the datauninformed parametric bootstrap cross-fitting method proposed by Wagenmakers et al. (2004). We will refer to this method as the cross-fitting method (CM) henceforth. Given two models A and B and a set of empirical data, the CM proceeds as follows to decide which of the two models is better able to account for the available data: First, feasible ranges are determined for the values of all free parameters in both A and B. Second, a certain number N (say, N = 1000) of sets of parameter values are randomly drawn from the parameter value ranges for A. Third, using each set of parameters, A is used to generate N data sets for the same conditions that are present in the empirical data. Fourth, both A and B are fit to each of the N data sets generated from A yielding a goodness of fit value (GOF, e.g., RMSE) for each model and data set. Fifth, the difference in GOF, GOFA-GOFB , is computed for each data set such that one obtains N difference values. These N difference values can be conceived of as N samples from the distribution of GOF difference values when data has been generated by A. Put differently, the N samples constitute an empirical distribution of difference values when data has been generated by A. Steps two through five are repeated with B being the generating model resulting in a second empirical distribution. Now both models are fit to the empirical data and the difference between the fits of A and B is computed. Employing the two empirical distributions it is determined whether it is more likely that the observed difference on the empirical data is a sample from the distribution when A or when B is the generating model. If the difference is more likely to be from the A/B distribution, A/B is assumed to more closely approximate the true generating processes than B/A. Wagenmakers et al. (2004) propose to determine a decision boundary based on the empirical distributions and then use the decision boundary directly to decide which of the two models provides the better account: All GOF differences that fall to one side of the decision boundary are assumed to indicate that one model (say, A) provides the better account, while all GOF differences that fall to the other side of the boundary are assumed to indicate that the other model (say, B) provides the better account. A reasonable decision boundary (and, in fact, the type of decision boundary employed by Wagenmakers et al., 2004) is a boundary that minimizes the probability of incorrectly classifying samples from the A/B distribution as coming from the B/A distribution. Two features of the CM render it particularly instrumental for model evaluation and selection. First, the CM is applicable to any type of model, since it imposes no constraints on the modeling paradigm or the models’ structure. Second, compared to other methods of similar generality such as, for instance, the bootstrap (Efron & Tibshirani, 1997), its application is relatively straightforward and requires comparatively little technical effort. There is, however, one technical issue associated with the CM that may hamper frequent use in current modeling research. Obviously, the utility of the CM crucially hinges on making appropriate use of the empirical distributions (e.g., finding a suitable decision boundary) for selecting among the two competing models. Unfortunately, apart from one exception (Cohen, Rotello, & MacMillan, 2008) existing literature employing the CM (e.g., Cohen, Sanborn, & Shiffrin, 2008; Jang, Wixted, & Huber, 2011; Perea, Gomez, & Fraga, 2010; Wagenmakers et al., 2004) does not provide detailed information on how to best employ the empirical distributions for determining the better model. In particular, no systematic investigations exist regarding what kind of method is most appropriate in what kind of situations (i.e., different types of and relations between the distributions). In this contribution we provide a first systematic investigation of candidate methods across a wide range of situations. The results of this investigation foster model evaluation and comparison in at least two ways. First, differential strength and weaknesses of different methods are identified thus allowing to avoid suboptimal model selection accuracy and unnecessarily long runtimes when employing the CM. Second, since explicit information on possible methods is provided the CM becomes more accessible to cognitive modeling researchers. Both benefits, we hope, are conducive to increasing the frequency with which the CM instead of more naı̈ve methods will be employed for model evaluation and comparison. The remainder of this article is structured as follows. In Section 1, we describe the considered situations and the procedural details of the conducted simulations. Subsequently, all considered methods are listed and described (Section 2). Simulation results and their interpretation are presented in Section 3. Section 4 concludes our considerations and highlights topics for future work.