Reading Comprehension and Mental Model Development: A Cross Validation of Methods and Technologies to Assess Student Understanding of the Text
Venue: ICLS
Year: 2020

In this study, we developed a framework that conceptualizes two approaches to summary analysis (text based and model based) along three dimensions: Surface, structure, and semantic. We examined 47 cases in which students created an initial summary and a revision to explore how these two approaches track changes in students' summaries and mental models. The cross validation between the two approaches demonstrated the power of assessing multiple dimensions of language to provide a more nuanced account of how students understand the text. Future studies should further investigate the relations between the two approaches and their theoretical and pedagogical meanings.  Summary writing is essential because summaries act as an externalized re representation of a student's mental model of the text (Johnson Laird, 2005) Consequently, summaries can be used to assess students' understanding of the complex texts they encounter in their classes. In addition, writing a high quality summary of a text requires readers to identify main ideas as well as the connections between those ideas (Graesser, Singer, Trabasso, 1994; Kintsch, 1988) Thus, helping students to construct a high quality summary of the reading material is also a common strategy for increasing students' understanding of the text. Given the important roles that summaries play in comprehension and learning, researchers have been working to develop effective ways to assess studentwritten summaries. This body of research has, broadly, taken two different approaches: model based and text based. Modelbased approaches elicit a concept model from the summary as an externally represented mental model of the individual's mind (Kim, Gaul, Kim, Madathany, 2019, Kim, 2018) Text based analytic approach quantifies multiple dimensions of language used in the summary (Crossley, Kyle McNamara, 2016; Kyle, Crossley, Berger, 2018) Though there are theoretical similarities and differences in these approaches, there have been few empirical evaluations of the relations between the two approaches in terms of analytical dimensions and assessment methods. The Current Study Though driven by similar theories, text based and model based indices approach the analysis of student's mental models in different ways. The purpose of this study was to examine how text based indices are sensitive to changes in students' summaries from the initial and the final versions and to cross validate these indices with indices from the model based approach across dimensions. We focused on state of the art methods and technologies that analyze summaries to evaluate students' reading comprehension. To calculate model based indices, we used the Student Mental Model Analyzer for Research and Teaching (SMART; Kim et al. 2019) which uses natural language processing (NLP) to produce model based indices at the three levels: surface, structure, semantic. In previous work (Kim et al. 2019) these SMART indices captured critical changes in student mental models as they revised their summaries. Thus, we used SMART as a benchmark upon which to compare text based indices derived from NLP tools that assess various dimensions of language: Tool for the Automatic Assessment of Lexical Sophistication (TAALES, Kyle et al. 2018) and Tool for the Automatic Assessment of Cohesion (TAACO, Crossley et al. 2016) Cross Validation Framework Both model based and text based approaches are grounded in theories that assume that readers produce mental models of texts and that these mental models are comprised of multiple levels of dimensions of representation. Theories of mental models (e.g. Kim, 2015; Ifenthaler, 2010; Spector Koszalka, 2004) posit that mental models are comprised of three dimensions: surface, structure, and semantic dimensions. The surface dimension indicates the descriptive information of components of mental representation (e.g. number of concepts) The structural dimension relates to the way concepts are organized and connected in the mental model network (e.g. cohesiveness) The semantic dimension relates to the underlying ideas in the text, more specifically, individual concepts and propositional relations in a model. Similarly, theories of discourse comprehension (e.g. Graesser, Singer, Trabasso, 1994; Kintsch, 1988) generally agree that mental models are comprised of three levels of representation: surface code, text base, and situation model. The surface code includes the specific words in the text, conceptually corresponding to the surface dimension. Taken together with the text based level (understanding propositional content) the situation level, the activation of higher order mental representation of entities, characters, actions, and events, is associated with the structural and semantic characteristics of mental models. We summarized the theoretical dimensions, selected technologies, and measures across the two approaches (model based vs. text based) in Table 1, but describe them below. Technologies such as SMART (Kim et al. 2019) GISK (Kim, 2018) and AKOVIA (Ifenthaler, 2014) build on the model based framework. These technologies use a text input (i.e. summary) to elicit an underlying mental model in the form of a concept map from which various indices are generated. In this study, we used SMART analytics that provides the highest number of indices in the three dimensions (refer to the details of the SMART indices in Kim, 2015) Technologies associated with the text based approach, driven by theories of discourse comprehension, include the Tool for the Automatic Analysis of Lexical Sophistication (TAALES, Kyle et al. 2018) and the Tool for the Automatic Analysis of Cohesion (TAACO, Crossley et al. 2016) TAALES focuses on word based metrics such as lexical frequency, psycholinguistic norms, and n gram frequency. In contrast, TAACO evaluates discourse level cohesion that is, the degree to which information in connected across the text. TAACO calculates measures of overlap across sentence and paragraph, type token ratio, and source similarity. These tools derive hundreds of indices. We selected text based indices that were theoretically correspondent to dimensions of the model based approach. These indices are: Content and Lemma TTR: Type Token Ratio of content words (noun, verb, adjective, and adverb) and lemmas (e.g. measure is the lemma for measurement, measuring, and measurable) calculated by dividing the count of individual lexical items (types) by the total count of all words (Tokens) Overlap indices: Overlap between a given word (N, CN, or ARG) and its synonym set across an adjacent sentence (1S) or the next two sentences (2S)  Source similarity indices provide the overall similarity between the words in a source text and a target text, building on Latent semantic analysis (LSA) a statistical dimensionality reduction technique to locates extracted words in a multi dimensional and Word2vec (W2V) representing words in a vector space model, putting words that occur in the same context as closer and those that occur in dissimilar contexts as more different in the vector space.  Comparison of model based and text based approaches to summary analysis Model Based Approach Text Based Approach Dimension Technology Index* Dimension Technology Index