Across Performance Contexts: Using Automated Writing Evaluation to Explore Student Writing
Author: McCaffrey, Daniel F.
Date: 2022-00-00
Type: Reports - Research
Subjects: Persuasive Discourse,Academic Language,Writing (Composition),Academic Achievement,Writing Skills,Language Usage,Writing Evaluation,Automation,Natural Language Processing,College Students

Background: This exploratory writing analytics study uses argumentative writing samples from two performance contexts--standardized writing assessments and university English course writing assignments--to compare: (1) linguistic features in argumentative writing; and (2) relationships between linguistic characteristics and academic performance outcomes. Writing data from this study come from 180 students enrolled at five four-year universities in the United States. Automated writing evaluation (AWE) tools were used to generate linguistic features from students' writing. Literature Review: Few studies have been conducted that use AWE to examine postsecondary writing skill and relationships between writing and broader academic performance outcomes. To address this gap, our study draws upon research on standardized and coursework writing, construct modeling, and AWE feature design. We also draw on related work to demonstrate how AWE can provide insights about linguistic characteristics of students' writing and relationships of that writing to academic performance factors. Research Questions: We examine the feasibility of using AWE to examine writing samples from standardized assessment and coursework contexts to assess how the samples compare to each other and how they are related to measures of students' academic performance. We organize this study around two research questions (RQ): RQ1: What can AWE tell us about variation in student writing samples drawn from two specific performance contexts: standardized argumentative writing versus coursework argumentative writing? RQ2: What are the relationships between writing subconstructs (as captured by AWE) and other measures of students' academic performance? Methodology: As part of a larger program of research (n = 735), study participants (n = 180) were administered two standardized test instruments: the HEIghten Written Communication (WC) test and the HEIghten Critical Thinking test. The HEIghten tests are commercially available tests designed for assessing postsecondary students' skills. Students also submitted writing samples from their coursework writing assignments. We collected institutional data, including SAT total scores, SAT Reading scores, ACT total scores, high school GPAs, and college GPAs. AWE tools were used to generate the 36 writing features--aggregated to create six subconstruct composite scores--that were used to analyze both the HEIghten WC test and the coursework writing samples. To answer RQ1, we used multiple statistical analysis and data visualization methods. The distributions of the AWE subconstruct scores between the HEIghten WC and coursework responses were compared in histograms and boxplots. Paired t tests were conducted for each composite feature to examine whether the means of the composite scores were statistically the same (H0). To answer RQ2, we used the academic outcome variables from the institutional data noted previously. We also examined the correlations of AWE feature scores with the scores students received from the multiple-choice portion of the HEIghten WC test. Results: Findings showed that AWE feature distributions differed between the two performance contexts. This finding suggests that both AWE feature interpretation and the relationship of these features to academic performance factors are associated with the writing context. Findings also suggest that writing data from both contexts offers a complementary picture of students' writing achievement and relationships with performance outcomes. Discussion: Using AWE feature interpretation, this study establishes a foundation for variation in postsecondary student writing and demonstrates how a granular sense of variation can be leveraged in order to understand relationships between writing achievement and broader postsecondary academic performance factors. Conclusions: Study findings have practical implications for how institutions might use writing analytics to obtain detailed information about student writing and, in turn, relate that information to other performance measures to provide relevant support for students. However, study findings must be qualified in terms of study sample, genres examined, standardized criterion measures, and shifting educational pedagogies.