NECESSARY BUT INSUFFICIENT WHY MEASUREMENT INVARIANCE TESTS NEED ONLINE PROBING AS A COMPLEMENTARY TOOL

Cross-national data production in social science research has increased dramatically in recent decades. Assessing the comparability of data is necessary before drawing substantive conclusions that are based on cross-national data. Researchers assessing data comparability typically use either quantitative methods such as multigroup confirmatory factor analysis or qualitative methods such as online probing. Because both methods have complementary strengths and weaknesses, this study applies both multigroup confirmatory factor analysis and online probing in a mixed-methods approach to assess the comparability of constructive patriotism and nationalism, two important concepts in the study of national identity. Previous measurement invariance tests failed to achieve scalar measurement invariance, which prohibits a cross-national comparison of latent means (Davidov 2009). The arrival of the 2013 ISSP Module on National Identity has encouraged a reassessment of both constructs and a push to understand why scalar invariance cannot be achieved. Using the example of constructive patriotism and nationalism, this study demonstrates how the combination of multigroup confirmatory factor analysis and online probing can uncover and explain issues related to cross-national comparability. Katharina Meitinger is a researcher at GESIS Leibniz Institute for the Social Sciences, Mannheim, Germany, and a teaching associate at the University of Mannheim, Mannheim, Germany. The author thanks Michael Braun, Eldad Davidov, three anonymous reviewers, and the editors for their helpful comments on earlier versions of this manuscript, as well as Dorothée Behr, Lars Kaczmirek, and Wolfgang Bandilla for sharing their expertise in online probing. This research was funded by the German Research Foundation (DFG) as part of the project “Optimizing Probing Procedures for Cross-National Web Surveys” [BR 908/5-1 to Michael Braun, Wolfgang Bandilla, and Lars Kaczmirek]. An earlier version of this paper was presented at the 2016 Conference of the World Association for Public Opinion Research and won the Janet A. Harkness Student Paper Award and the NCHS Monroe Sirken Innovative Award for Young Scholars of Question Evaluation. *Address correspondence to Katharina Meitinger, GESIS Leibniz Institute for the Social Sciences, Post Office Box 122155, 68072 Mannheim, Germany; e-mail: katharina. meitinger@gesis.org. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http:// creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com Public Opinion Quarterly, Vol. 81, No. 2, Summer 2017, pp. 447–472 doi:10.1093/poq/nfx009 Advance Access publication May 19, 2017 With the proliferation of cross-national surveys, such as the International Social Survey Program (ISSP) and European Social Survey (ESS), access to cross-national datasets has been tremendously facilitated. One precondition for the analysis of such data is the assessment of their cross-national comparability. Two research traditions can be distinguished in this context. In the quantitative tradition, comparability is often assessed with measurement invariance tests that use multigroup confirmatory factor analysis (MGCFA) (Jöreskog 1971). This approach can test the cross-national comparability of numerous countries. The testing strategy is implemented using analysis software, such as Mplus, making it a handy control instrument for researchers analyzing secondary data. However, if these tests fail to establish cross-national comparability, this approach struggles to explain the existing noninvariance. By contrast, researchers in the qualitative tradition will most likely conduct cognitive interviews (CIs) (Miller et al. 2011) or online probing (OP) (Braun et al. 2014). These methods primarily seek to uncover the causes for the missing comparability, and they often reveal unexpected reasons. The drawbacks of these methods are the necessity of collecting data and the work-intensive analysis (Meitinger and Behr 2016), which limit the analysis to a small set of countries. Much can be learned through a combined approach of both perspectives. This study closes a research gap by simultaneously applying MGCFA and OP to assess the comparability of the constructs of constructive patriotism and nationalism. It will introduce MGCFA and OP, followed by a short discussion of constructive patriotism and nationalism. The constructs’ comparability will first be assessed with MGCFA and then with OP. Previous research has indicated that the items that measure constructive patriotism are particularly problematic (Latcheva 2011); as such, this study will focus on these items with regard to OP. Finally, the conclusions from both methods will be compared, and optimal research strategies that combine the two methods’ respective strengths will be presented. The Quantitative Approach: Tests of Measurement Invariance Most researchers conduct three tests of comparability when applying MGCFA: configural, metric, and scalar invariance tests (Vandenberg and Lance 2000). The three tests are nested, with configural invariance providing a test for the lowest and scalar invariance for the highest level of invariance. Configural invariance concerns whether all countries have the same factor structure (Horn and McArdle 1992). If configural invariance is established, the latent concept can be meaningfully discussed in all countries (Davidov et al. 2014). However, the respondents can still answer items differently because factor loadings may vary (Steenkamp and Baumgartner 1998). Meitinger 448