Re-Evaluating the Psychometric Properties of MicroFIN: A Multidimensional Measurement of Complex Problem Solving or a Unidimensional Reasoning Test?

Complex problem solving (CPS) skills are considered to be highly important in today's rapidly changing and increasingly complex world, (e.g., Funke, 1999; Kretzschmar & Sus, 2015; Neubert, Mainert, Kretzschmar, & Greiff, 2015; OECD, 2014) and hence, research about the assessment of CPS skills has been increasing in recent years (e.g., Greiff, Wustenberg, & Funke, 2012; Kroner, Plass, & Leutner, 2005; Sonnleitner et al., 2012). MicroFIN (Neubert, Kretzschmar, Wustenberg, & Greiff, 2015) is the latest development within the framework of multiple complex systems (MCS; Greiff et al., 2012), which was developed to overcome the psychometrical limitations of previous CPS tests and assessment frameworks (for an overview, see e.g., Greiff, Fischer, Stadler, & Wustenberg, 2014). Although the development of the MCS approach can be considered an important milestone in the CPS research field, the most prominent MCS-based tests MicroDYN (Greiff et al., 2012) and Genetics Lab (Sonnleitner et al., 2012) have been criticized as they only cover very selected characteristics of complex problems (e.g., Funke, 2010, 2014; Funke, Fischer, & Holt, 2017; Kretzschmar, 2017; Scherer, 2015; Schoppek & Fischer, 2015), particularly compared to more comprehensive CPS tests (a.k.a. microworlds) such as Tailorshop (Putz-Osterloh, 1981), FSYS (Wagener, 2001), or LEARN! (Grossler, Maier, & Milling, 2000). For example, MicroDYN and Genetics Lab tasks as applied in previous studies rely on only one specific strategy (vary-one-thing-at-a-time, VOTAT; see e.g., Chen & Klahr, 1999) or a very close adaptation of it (see Beckmann and Goode, 2014, who summarized the slightly different strategies as a vary-one-or-none-at-a-timestrategy, VONAT). Although the VOTAT (or VONAT) strategy is important in many contexts (see e.g., Wustenberg, Stadler, Hautamaki, & Greiff, 2014), it is obvious that one strategy is not sufficient to solve the variety of problems that can arise in complex problem solving research or even in daily life (e.g., Funke, 2014; Funke et al., 2017). Therefore, the psychometrically advantageous homogeneity of current versions of MicroDYN and Genetics Lab can be considered a threat to a content-valid operationalization of CPS (see, e.g., Neubert, Kretzschmar, et al., 2015; Scherer, 2015; Schoppek & Fischer, 2015). MicroFIN was developed to address the limitations of established MCS-based tests. Specifically, the development of MicroFIN was partially guided by the rationale to create tasks which are not solvable by solely applying the VOTAT strategy; instead, different problem solving strategies are required in each task (Kretzschmar, 2015; Neubert, Kretzschmar, et al., 2015). MicroFIN, therefore, has the potential to narrow the gap between highly reliable but homogeneous MCS-based assessment tools and the psychometrically less convincing but ecologically valid microworlds that have been applied in CPS research in recent decades (Kretzschmar, 2017). However, whether and to what extent MicroFIN fulfills this expectation still needs further investigation, although first evidence supports the view of MicroFIN as a heterogeneous CPS test (see Muller, Kretzschmar, & Greiff, 2013). The present study aims to answer a more fundamental research question about MicroFIN: its psychometric quality. Apart from in the initial study by Neubert et al. (2015), the psychometric properties of MicroFIN have not yet been addressed comprehensively. Due to the heterogeneity of the included tasks, the test is expected to face challenges in terms of reliability but more convincing findings in terms of construct validity (see attenuation paradox; Loevinger, 1954). As will be outlined below, empirical findings reveal an unclear pattern and the need for further research. Therefore, the purpose of the present study is to examine the psychometric properties of a further developed version of MicroFIN with regard to three important issues in CPS research: (1) factorial validity, (2) reliability, and (3) relation to reasoning (fluid intelligence; see McGrew, 2009). …