Contemporary Peer Review: Construct Modeling, Measurement Foundations, and the Future of Digital Learning

Background: In this article, we offer a study of peer review in a digital learning environment. Our analysis focuses on intrapersonal and interpersonal domains of the writing construct as they are enacted in the peer review process in terms of self-reflection and transaction. Our study is organized as a demonstration of the force of construct articulation, the usefulness of fairness as an integrative measurement framework, and the affordances of research in digital ecologies. Based on findings from our National Science Foundation funded research, we conclude with considerations for future peer review research.Literature Review: Despite the fact that most first-year composition programs utilize peer review, there is little writing studies research surrounding the practice of peer review (Haswell, 2005, p. 211). The studies that have addressed peer review generally find that peer review leads to positive outcomes (Moxley & Eubanks, 2015; Ross, Liberman, Ngo, & LeGrand, 2016). Notably, peer reviews appear to help both the reviewer and the reviewee (Dochy, Segers, & Sluijsmans, 1999). Our understanding of the revision process is rooted in Flower and Hayes’ (1981) social cognitive theory of writing. Combined with a need to expand models of the writing construct based on cognitive, interpersonal, and intrapersonal demands, our research seeks to fill the gap in acknowledging that the metacognitive nature of peer review is part of the construct of writing.Research Questions: Our research questions divide into three categories: the intrapersonal and interpersonal domain, forms of evidence, and digital learning affordances. We inquire into (1) the tone and quality of student self-reflection, as well as (2) the quality and tone of the peer review transaction. In the study of fairness evidence, we ask (3) what may be learned by investigating responses when student sub-groups are disaggregated according to gender, ethnicity, race, and English language learning. In the study of reliability evidence, we ask (4) what forms of evidence related to response consistency are useful in the analysis of peer review. In the study of validity evidence, we ask (5) how a precise definition of the writing construct lends precision to construct-related evidence. In terms of digital learning, we ask (6) what is the instrumental value of questions 1-5 in terms of demonstration of affordances to participate in the MyReviewers (MyR) peer review process.Research Methodology: Our research utilizes a sample of 837 students enrolled in first-year composition at a public research university, in particular their self-reflection ratings and the transaction ratings. These surveys were conducted voluntarily, presented to the students upon completing peer review (for reviewers) and the revision plan (for reviewees) as part of the MyR software.Results: The study shows that while self-reflection and transaction surveys received high to neutral ratings for helpfulness, politeness, and kindness, encouragement received only high or low ratings. In terms of fairness evidence for self-reflection, women believed their feedback was more polite and helpful. Similarly, Hispanic students believed their reviews were more helpful than non-Hispanic students did, and students who claimed proficiency in two or more languages felt their own reviews were more helpful than English-only speakers did. For fairness evidence for transactions, men were perceived as more encouraging than women in their feedback, while Hispanic students’ reviews were no more helpful than non-Hispanic students’ reviews. No statistically significant differences were found amongst English language learners and native English speakers. In relation to reliability evidence for self-reflection, for the most part reliability reaches statistically significant levels. In terms of digital learning affordances, students in groups typically associated with low writing performance thrived in the digital learning platform when the construct included domains beyond the cognitive.Discussion: Based on these findings, there are three areas of consideration worthy of extended pursuit: 1) consider the advantages of expanded notions of the writing construct; 2) consider information analysis in terms of opportunity to learn; and 3) consider digital ecologies as a way to advance writing instruction for all students.Conclusions: This study provides unique insight that writing program administrators (WPAs) might utilize to inform their programs. A natural next step to implement the findings of our study would be for WPAs to systematically examine how evidence reacted to gender, ethnicity, and race is manifested within the classroom at their own institutions.