Abstract: Taking Into Account Sampling Variability of Model Selection Indices: A Parametric Bootstrap Approach

Taking Into Account Sampling Variability of Model Selection Indices: A Parametric Bootstrap Approach Sunthud Pornprasertmanit a , Wei Wu a & Todd D. Little a a University of Kansas Version of record first published: 29 Mar 2013. To cite this article: Sunthud Pornprasertmanit , Wei Wu & Todd D. Little (2013): Abstract: Taking Into Account Sampling Variability of Model Selection Indices: A Parametric Bootstrap Approach, Multivariate Behavioral Research, 48:1, 168-169 To link to this article: http://dx.doi. org/10.1080/00273171.2013. 752266 PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.tandfonline. com/page/termsand-conditions This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material. D ow nl oa de d by [ U ni ve rs ity o f N or th T ex as ] at 0 7: 48 2 6 A pr il 20 13 Multivariate Behavioral Research, 48:168–169, 2013 Copyright © Taylor & Francis Group, LLC ISSN: 0027-3171 print/1532-7906 online DOI: 10.1080/00273171.2013. 752266 Abstract: Taking Into Account Sampling Variability of Model Selection Indices: A Parametric Bootstrap Approach Taking Into Account Sampling Variability of Model Selection Indices: A Parametric Bootstrap Approach Sunthud Pornprasertmanit, Wei Wu, and Todd D. Little University of Kansas The rankings of the Akaike information criterion (AIC) or Bayesian information criterion (BIC) are often used for nonnestedmodel comparison.When the competing models do not include the population model, Preacher and Merkle (2012) demonstrated that the model selected based on the ranking of BIC varied over repeated sampling and the variability did not diminish with increased sample size. The current study proposes an approach to take into account the sampling variability of model selection through parametric bootstrap, which is a generalization of Millsap’s (2013) approach to model selection. Given 2 competing models (say A and B), the bootstrap approach creates 2 sampling distributions of the difference in a model selection index, say BIC (BIC), assuming 1 of the models is true. Note that the bootstrap samples are generated based on the parameter estimates from fitting the observed data to Model A or B, which will also yield an observedBIC. Model selection can be then made by comparing the observed BIC with the 2 sampling distributions. If the observed BIC lies only in the range of acceptance (given a priori alpha level) of one of the distributions, then either Model A or B is favored. On the other hand, if the observed BIC lies inside or outside the ranges of acceptance in both distributions, the 2 models are deemed indistinguishable (equally good or poor). A simulation study is conducted to evaluate the proposed approach following Preacher and Merkle (2012). Specifically, we created data from Model D and analyzed the data by Models A, B, and C as shown in Figure 1. We examined 6 sample sizes: 80, 120, 200, 500, 1,500, and 5,000. Using simple BIC and AIC rankings, the selected model significantly varied across the replications within any given sample size. The result from the proposed method revealed that the fit of the competing models was too similar to be differentiated (selection cannot be determined in a large proportion of replications and this proportion increased with increased sample size). Results support the conclusion that the parametric bootstrap method is superior to the ranking approach because it accounts for sampling variability and provides an option to discern between models. Millsap, R. E. (2013). A simulation paradigm for evaluating model fit. In M. Edwards & R. MacCallum (Eds.), Current issues in the theory and application of latent variable models (pp. 165–182). New York, NY: Routledge. Preacher, K. J., & Merkle, E. C. (2012). The problem of model selection uncertainty in structural equation modeling. Psychological Methods, 17, 1–14. This project was partially supported by Grant NSF 1053160 to Wei Wu and Todd D. Little. Correspondence concerning this abstract should be addressed to Sunthud Pornprasertmanit, Department of Psychology and Center for Research Methods and Data Analysis, University of Kansas, Lawrence, KS 66044. E-mail: psunthud@ku.edu 168 D ow nl oa de d by [ U ni ve rs ity o f N or th T ex as ] at 0 7: 48 2 6 A pr il 20 13 SMEP STUDENT PAPER COMPETITION ABSTRACTS 169 FIGURE 1 Models A, B, C were used in model comparison and Model D was used in data generation. This figure is modified from Figure 1 in Preacher and Merkle (2012). D ow nl oa de d by [ U ni ve rs ity o f N or th T ex as ] at 0 7: 48 2 6 A pr il 20 13