Predicting the Quality of Revisions in Argumentative Writing
Year: 2023

Abstract:
The ability to revise in response to feedback is critical to students’ writing success. In the case of argument writing in specific, identifying whether an argument revision (AR) is successful or not is a complex problem because AR quality is dependent on the overall content of an argument. For example, adding the same evidence sentence could strengthen or weaken existing claims in different argument contexts (ACs). To address this issue we developed Chain-of-Thought prompts to facilitate ChatGPT-generated ACs for AR quality predictions. The experiments on two corpora, our annotated elementary essays and existing college essays benchmark, demonstrate the superiority of the proposed ACs over baselines.

heading: 1 Introduction
text: Argumentative Revision (AR) in response to feedback is important for improving the quality of students’ written work. Successful ARs1 usually include adding relevant evidence, deleting repeated evidence or reasoning, and elaborating relevant evidence examples to support claims (Afrin et al., 2020). Differentiating between successful versus unsuccessful ARs, however, is a complex endeavor. For example, making the same AR in distinct Argumentative Contexts (ACs) could differentially affect the quality of a student’s essay. Here the ACs are defined as pieces of sentences that present reasons, evidence, and claims supporting or opposing arguments in argumentative writing (see Sec. 4.2). For example, Figure 1 shows two pieces of ARs that added the same sentence “it was hard for them to concentrate though, as there was no midday meal” but caused opposite AR quality.
Recently developed Automated Writing Evaluation (AWE) systems have focused on assessing the content and structure of student essays to automatically provide students with formative feed-
1Afrin and Litman (2023) use the term desirable revisions.
back (Zhang et al., 2016; Writing Mentor, 2016; Wang et al., 2020; Beigman Klebanov and Madnani, 2020). Successful revisions (e.g., adding relevant evidence) improve an essay’s quality. Unsuccessful revisions, in contrast, lead to no improvement or can even weaken an essay’s argument (Afrin et al., 2020). As a result, assessing the success of ARs is important to assess the quality of ARs in line with provided feedback.
AR quality has previously been predicted by using long and short neighboring contexts of ARs (Afrin and Litman, 2023). This location-based approach for constructing ACs did not exploit any argumentative relationships between ARs and potential ACs. Another study (Zhang and Litman,
275
2016) incorporated AR contexts with cohesion blocks and employed sequence labeling to model AR interdependence across revisions. This work predicted AR purposes from discourse structures but did not further study AR quality or analyze AR quality from the perspective of ACs. To bridge these gaps, we address three research questions. RQ1: To what extent are ACs helpful for predicting AR quality? RQ2: What type of AC is the most helpful in AR quality predictions? RQ3: Can ChatGPT prompts be used to generate useful ACs? In studying the three RQs, we have made the following contributions:
• Our project is the first in the revision field to analyze the relationship between ACs and AR quality predictions.
• We are among the first to incorporate the stateof-the-art large language model ChatGPT in generating ACs in argumentative writing.
• Experiments using both elementary and college essay corpora show the superiority of the proposed ACs over existing location-based contexts for AR quality predictions.

heading: 2 Related Work

heading: 2.1 Argumentative Revision in NLP
text: Revision research has been conducted using multiple types of Natural Language Processing (NLP) corpora ranging from Wikipedia to argumentative essays. While argumentative writing research has analyzed argumentative roles and discourse elements in persuasive writing (Stab and Gurevych, 2014; Song et al., 2020; Putra et al., 2021) (e.g., by studying the stance towards some topic, backing up claims, or following argumentative and rhetorical considerations), such analyses have not typically been applied to revision research in this domain. Revision research, in contrast, has primarily focused on grammar correction, paraphrasing, semantic editing (Yang et al., 2017), and analyzing revision purposes (Zhang and Litman, 2015; Shibani et al., 2018; Afrin et al., 2020; Kashefi et al., 2022). Although revision research has sometimes leveraged contextualized features during classification, the contextual features have been locationbased (Zhang and Litman, 2016; Afrin and Litman, 2023). We instead extract contextual information from an essay based on argumentative essay analysis rather than on adjacency to a revision.

heading: 2.2 LLM in Argumentative Revision
text: Large Language Models (LLMs) have scaled up model sizes from a few million to hundreds of billions of parameters. Their strong capabilities of handling multiple downstream NLP tasks have made LLMs favorable in recent research (Chowdhery et al., 2022). Prior revision works, e.g., academic writing (Ito et al., 2019), debation assessment (Skitalinskaya et al., 2021), paraphrase generation (Mu and Lim, 2022), were mostly based on Transformer (Vaswani et al., 2017) and BERT (Devlin et al., 2019) models but not the cutting-edge LLMs, e.g., ChatGPT2. Pretrained LLMs have shown strong few-shot learning capabilities by way of developing prompts to guide LLMs in generating successful outputs (Brown et al., 2020; Liu et al., 2023). For example, Chain-of-Thought (CoT) prompts (Wei et al., 2022) enable pretrained LLMs to solve complex reasoning problems by decomposing the tasks into a series of intermediate steps. Kojima et al. (2022); Wang et al. (2022) investigated the effectiveness of CoT in multi-step reasoning, however, little work has used CoT for extracting and then generating tasks in the revision field. In this work, we leverage ChatGPT with CoT prompts to generate ACs in argumentative writing.

heading: 3 Corpora

heading: 3.1 Data Collection
text: AR corpora are rarely annotated in the revision community because of their expensive annotation costs. The publicly available college essay corpus for AR quality predictions (Afrin and Litman, 2023) contains paired drafts of argumentative essays written in response to an essay prompt (original drafts) and revised based on feedback (revised drafts). The corpus is comprised of 60 essays (N=60 college students), inclusive of both native and proficient non-native speakers of English, in response to an essay prompt about Technology Proliferation. Students received general feedback upon completion of their first drafts, asking them to add more examples in their second drafts. The second drafts then received non-textual feedback through the ArgRewrite system (Zhang et al., 2016) to help students write their third drafts. Afterward, the second and third drafts were collected as pairs of original and revised drafts.
We followed a similar protocol to collect 596 ele-
2https://openai.com/blog/chatgpt

heading: ID Original Draft Sentence Revised Draft Sentence Revision Revision
text: Type Revision Purpose Quality Label
1 According to the text. "The people in Sauri have made amazing progress in just eight years." According to the text, "The people in Sauri have made amazing progress in just eight years." Modify Surface N/A N/A
2 This tells me that the people of Sauri have made better living arrangements in eight years and it all did pay off. This tells me that the people of Sauri have made better living arrangements in eight years and it all did pay off, from how they used to live. Modify Content N/A N/A
3 The people of Sauri did do great progress. The people of Sauri did do great progress. N/A N/A N/A N/A

heading: 4 ... ... ... ... ... ...
text: 5 This lets me know that since there might be a few diseases that might affect anyone at ant time so the hospital has made medicine that can cure those diseases, so they gave that medicine to any one who needed it for free.
Delete Content Irrelevant Evidence Unsuccessful
6 This piece of text lets me know that the hospital, Yala Sub District, has free medicine for diseases that are most common around where they live. Add Content Paraphrase Reasoning Successful
7 In Sauri people had to pay a fee, which the people of Sauri couldn’t afford. Add Content not LCE Reasoning Unsuccessful

heading: 8 ... ... ... ... ... ...
text: 9 I can tell that the people of Sauri must of thought that children needed education money not so they didn’t ask people for the school fees, and the kids wouldn’t go hungry during school hours they served the children lunch.
Delete Content LCE Reasoning Successful

heading: 10 ... ... ... ... ... ...
text: mentary essays written by grade 5 to 6 students who were taking the Response to Text Assessment (Correnti et al., 2013). 296 students wrote an essay in
response to a prompt about the United Nation’s Millenium Villages Project (MVP). The students then revised their essays in response to formative feed-
back from an Automatic Writing Evaluation (AWE) system that used rubric-based algorithms to assess the quality of evidence use and reasoning (Zhang et al., 2019; Wang et al., 2020). The other 300 students did the same tasks for an essay prompt about Space Exploration (Space). We combined the collected essays from the two essay prompts because students shared similar argumentative writing skills and the scoring rubric and feedback messages were constant across prompts.

heading: 3.2 Preprocessing
text: We preprocessed collected elementary essays for annotations. First, sentences from original and revised drafts were aligned into pairs of original sentence (OS) and revised sentence (RS) using a sentence alignment tool Bertalign (Liu and Zhu, 2022). The aligned pairs were programmingly labeled with no change if OS and RS are the same, modifying if OS and RS are not empty but not same, adding if OS is empty but RS not, or deleting if RS is empty but OS not. The changed alignments were automatically classified into surface and content revisions by a pretrained classifier. Note that the sentence alignments and classification were first done by the system and then manually justified and corrected by annotators, and only aligned content revisions were used for annotations.

heading: 3.3 Annotations
text: We used the Revisions of Evidence use and Reasoning (RER) scheme (Afrin et al., 2020) to annotate revision purposes in elementary essays, which encodes the nature of students’ revision of evidence use and reasoning. Evidence use refers to the selection of relevant evidence from a given source article to support a claim, while reasoning means a reasoning process of connecting the evidence to the claim. Thus, the content revisions are annotated with claim-related, evidence, and reasoning revisions. The RER scheme only applies to evidence and reasoning, where evidence revisions were labeled with relevant, irrelevant, repeated evidence, non-text based and minimal, and reasoning revisions were labeled with linked claimevidence (LCE), not LCE, paraphrase evidence, generic, commentary, and minimal.
Furthermore, we followed the AR quality scheme (Afrin and Litman, 2023) to programmingly encode annotated RER labels (revision purposes) into successful and unsuccessful revisions. The relevant evidence was encoded as successful
while the repeated, non-text based, and minimal evidence were encoded as unsuccessful. The LCE and paraphrase reasoning were encoded as successful. The not LCE, paraphrase evidence, generic, commentary, and minimal reasoning were encoded as unsuccessful. Table 1 shows label distributions in elementary essays and college essays where elementary essays have almost an even number of reasoning and evidence annotations. The adding revisions are the most frequent ARs across two essays. Samples of annotations for elementary essays and college essays are shown in Table 2 and Table 6 (in Appendix A), respectively. In practice, the RER annotations were done by one expert annotator. We sampled about 20 percent of annotated essays about both Space and MVP prompts and asked another well-trained annotator to annotate the sampled essays. The two-annotator Kappa scores are shown in Table 3.

heading: 4 Methods

heading: 4.1 Preliminary
text: In this section, we introduce notations for the AR quality prediction task. We denote R1 and R2 as original and revised sentences in the original and revised drafts, respectively. In particular, R1 is always empty in adding ARs (e.g., row #6 in Table 2); R2 is always empty in deleting ARs (e.g., row #9 in Table 2); neither R1 nor R2 are empty in modifying ARs (e.g., row #1 in Table 2). Thus, we only use R1 in deleting and R2 in adding ARs. In terms of modifying ARs, we only use R2 because R2 is a revised version of R1 thus are very close to R1 (e.g., row #2 in Table 2). In addition, we denote ACs as a couple of sentences related to ARs in their corresponding drafts, where C1 represents the ACs of R1 in the original draft and C2 represents the ACs of R2 in the revised draft (details in Sec. 4.2), respectively. To this end, we formulate the task of predicting AR quality as classifying the ARAC pairs {Ri, Ci} into successful and unsuccessful labels, where i = 1, 2. Specifically, we use pair
{R1, C1} for deleting and {R2, C2} for adding and modifying ARs.

heading: 4.2 Argumentative Context
text: Consistent with long-established models of argumentation such as Toulmin’s model (Toulmin, 1958), well-developed arguments are characterized by the alignment of claim, evidence, and warrants (i.e., reasoning related to why the evidence supports the claim) (Reznitskaya et al., 2008). For example, the appropriateness of a piece of evidence for advancing an argument is context-dependent because that judgment is determined relative to an author’s prior claim(s) or reason(s). As a case in point, the unsuccessful AR #372 shown in Figure 1 would have been unobservable absent an understanding of the author’s claim or argument’s context. Recent work by Afrin and Litman (2023) has used short and long text segments immediately before and after the AR as context for predicting AR quality, however, the study has some significant drawbacks. First, the window size of the contexts is an unpredictable parameter because a reasoning sentence could refer to the evidence far ahead of the AR (e.g., reasoning in row #6 refers to the evidence in row #1 in Table 2). Second, location-based contexts did not explain why ACs make a difference to ARs from an argumentative perspective and thus fail to analyze the argumentative roles of ACs in AR quality predictions. As we noted above, the evaluation of a reasoning sentence as desirable depends on whether it appropriately references evidence or claims in the student’s essay, but this relationship has not been explored in prior revision research. Thus, in the current study, we define three ACs to study their relationship to AR quality: (1) ACClaim: the context containing essay claims or arguments; (2) AC-Reasoning: the context containing reasoning related to the claim or evidence in the essay; (3) AC-Evidence: the context containing evidence to support or oppose claims.

heading: 4.3 ChatGPT Prompts
text: Pretrained ChatGPT on a series of GPT3.5 models has shown promising results in solving information extraction (Li et al., 2023) and summarization (Yang et al., 2023) tasks in zero-shot settings, however, doing the two tasks at the same time has not been explored in generating ACs. Therefore, we developed two versions of ChatGPT prompts that generate useful ACs for predicting AR quality: (1) Single prompts that generate ACs in one pass
and (2) Chain-of-Thought prompts that generate ACs in two passes.

heading: 4.3.1 Single Prompts
text: In this section, we introduce Single prompts for AC generations. Basically, we need ChatGPT to generate useful ACs for AR quality predictions by reading the student essays. We limit the generation to a two-sentence length for two reasons. First, the generated ACs will be used in an AR-AC pair {Ri, Ci}, where Ri is normally one sentence, thus long ACs (Ci) paired with short ARs (Ri) will make the AR quality prediction model (introduced in Sec. 4.4) learn to attend to the context rather than the revisions. Second, the most intuitive location-based baseline (Base-Short in Sec. 5) uses the adjacent sentences before and after target ARs, which contain at most two sentences. Therefore we limit the generations to exact two sentences, which can be done with a single zero-shot prompt please summarize [X] in the essay [Y] in two sentences, where [X] slot is one of the claim, reasoning, and evidence, [Y] is an input essay.

heading: 4.3.2 Chain-of-Thought Prompts
text: In addition to Single prompts, Chain-of-Thought (CoT) prompts (Wei et al., 2022) are conceptually simple yet effective in multiple reasoning tasks. We adopt this idea and use zero-shot-CoT prompts to generate ACs, which run prompting in two passes but do not require step-by-step few-shot examples.
The first-pass CoT prompt to extract ACs. The first pass of the CoT prompts is to extract claim, evidence, and reasoning sentences from input essays. We aim to extract exact sentences from input essays without introducing any external knowledge in ChatGPT itself. The first-pass prompt is: please list [X] sentences in the essay [Y], where [X] slot is chosen from one of the claim, reasoning, and evidence, and [Y] is an input essay. The extracted ACs are formulated as a list of sentences from the input essays, where the length of the list ranges from one to dozens because claim, reasoning, and evidence have multiple instances in an essay. To make sure the extracted ACs are informative and not exceeding the word limits of DistilRoBERTa encoders (see Sec. 4.4), we perform summarization on the extracted sentences in the second-pass prompt.
The second-pass CoT prompt to summarize ACs. The second prompt is continued to the first
prompt, following an extraction-summarization CoT. The prompt is please summarize [X] in two sentences, where [X] slot is chosen from the claim, reasoning, and evidence sentences extracted in the first prompting pass, which ensures the outputs in a length of exact two sentences. Figure 2 and Figure 5 (in Appendix A) show examples of the zero-shot Single and CoT prompts that help ChatGPT generate ACs in elementary and college essays, respectively.

heading: 4.4 AR Quality Prediction
text: We define AR quality prediction as a binary classification of AR-AC pair {Ri, Ci} (see Sec. 4.1), where Ri is annotated and Ci is generated by ChatGPT. Prior works employed BERT-BiLSTM architecture to train revision classifiers (Anthonio and Roth, 2020; Afrin and Litman, 2023). Instead,
we use DistilRoBERTa (Sanh et al., 2019) as text encoders for both annotated ARs and ChatGPTgenerated ACs. The last hidden layers of the DistilRoBERTa encoders are fed to an average-pooling layer and then connected to a multi-layer perception classifier that contains a sequence of batch normalization layer, ReLU layer, dropout layer with a 0.5 rate, and Sigmoid layer. The overall framework is shown in Figure 3.

heading: 5 Experiments
text: To answer the RQs, we implemented locationbased contexts as baselines, and a series of ACs as comparable methods:
• Base-Short: We implement a standard revision prediction baseline that uses the adjacent sentences immediately before and after a revision as
contexts (Afrin and Litman, 2023).
• Base-Long: We implement a strong baseline that considers all the sentences that are revised around a target revision until an unchanged sentence is found (Afrin and Litman, 2023).
• AC-Claim: We use AC-Claim as the contexts that are generated by zero-shot Single and CoT prompts, respectively.
• AC-Reasoning: We use AC-Reasoning as contexts. The two versions use zero-shot Single and CoT prompts, respectively.
• AC-Evidence: We use AC-Evidence as contexts. The two versions are generated by zero-shot Single and CoT prompts, respectively.
In the implementation, we built the framework pipeline with PyTorch3 and generated two versions
3https://pytorch.org
of ACs using ChatGPT3.5-turbo API4. We used pretrained DistilRoBERTa-Base from Huggingface5 as text encoders, and optimized cross-entropy loss with Adam optimizer on a GeForce RTX 3090 GPU. We set the batch size as 16 and the learning rate as 5e-5 with 5% decays every 4 epochs. We conducted 10-fold cross-validation, where 80% of each 9-fold set was used for training, 20% for parameter tuning, and the rest 1-fold set for testing. Finally, we ran the ChatGPT generation and the experiment pipeline three times and reported 3-seedaverage macro Precision, Recall, and F1 on all the test sets. The implementation code is available at https://github.com/ZhexiongLiu/ Revision-Quality-Prediction.

heading: 6 Results and Discussion
text: Table 4 shows the experimental results for different sets of revisions from the elementary essay corpus: all reasoning and evidence revisions, just
4https://platform.openai.com 5https://huggingface.co
reasoning revisions, and just evidence revisions, respectively. We observed that both the proposed Single and CoT versions of ACs outperformed both baselines, with many of the CoT ACs significantly better than the strong (Base-Long) baseline. This answered RQ1 that ACs can help AR quality predictions. In reasoning ARs, excellent performance was yielded in using AC-Claim, AC-Reasoning, and AC-Evidence. This is because reasoning ARs might need claims to verify their usefulness and incorporate evidence and reasoning to check their relevance. Moreover, evidence ARs achieved the best with AC-Reasoning, which makes sense that identifying evidence AR requires related reasoning contexts that have information linking the evidence. Another interesting finding is that the Base-Long performed better than the Base-Short in evidence ARs but worse in reasoning ARs. This suggests that the longer context is not always helpful in the case that evidence contexts are usually sparsely distributed in the essay so the longer context will introduce more noise. It also suggests that reasoning sentences are mostly adjacent to other reasoning contexts and can be well captured by neighboring sentences. Furthermore, the observation that reasoning ARs results are generally better than evidence ARs indicates that reasoning ARs might be self-justifiable which means it might require fewer contexts than the evidence to identify AR quality. These observations answered RQ2 that reasoning contexts are mostly useful, and both reasoning, claim, and evidence contexts benefit AR quality predictions. In addition, CoT prompts are generally better than Single prompts in most reasoning and evidence ARs, which indicates that identifying AR quality requires some contexts that might not be generated with Single prompts. This answered RQ3 that CoT prompts are generally better than Single prompts.
We also evaluated the effectiveness of ACs on the college essay benchmark. Note that Afrin and Litman (2023) conducted data augmentation with a simple synonym replacement because they argued that it was impossible to obtain reasonable results without training on augmented data. We hypothesized that data augmentation will introduce noise but the limited data can yield reasonable results training with the DistilRoBERTa-based model. Therefore, we did not do data augmentation and compared AC-based methods to our implemented standard and strong baselines on reasoning revi-
sions (excluding the rare evidence revisions as shown in Table 1). Results in Table 5 show that the DistilRoBERTa model is able to learn from even small-size data without data augmentation. In addition, AC-based methods perform better than both the standard and strong baselines, where ACEvidence has significant improvement. This again suggests that ACs are generally useful for predicting AR quality and CoT prompts are generally better than Single prompts for generating useful ACs. Moreover, we observed that AC-Evidence generated by Single and CoT prompts is better than the other ACs. It is slightly different from the reasoning column in Table 4. This might suggest that revisions in college essays may focus on evidence revisions that match generated evidence ACs. However, claim and reasoning results have similar F1 scores across two versions of prompts, which might suggest the extracted AC-Claim and AC-Reasoning are similar in college essays (e.g., prompting outputs in Figure 5 in Appendix A), which might be because college essays have claim and reasoning sentences disentangled. In general, CoT prompts are somewhat better than Single prompts in AC-Claim and AC-Reasoning generation, and both Single and CoT prompts are promising in AC-Evidence generation.
As a case study, we examine the effectiveness of ACs in Revision #372 and #592 presented in Figure 1. The ChatGPT-generated AC-Claim is shown in Figure 4, where the red bold sentence “providing lunch to children, to argue that poverty can be reduced” is helpful to identify that the added sentence, “It was hard for them to concentrate though,
as there was no midday meal.” in Revision #372 is a already existed evidence, and thus it was an unsuccessful revision. However, AC-Claim in Revision #592 does not show repeated but relevant information, and thus the AR is regarded as successful.

heading: 7 Conclusion
text: This work studies the relationship between Argumentative Contexts (ACs) and Argumentative Revisions (ARs) in argumentative writing. In particular, we use zero-shot-CoT prompts to facilitate ChatGPT-generated ACs for AR quality predictions. The experiments on our elementary essays corpus and publicly available college essays benchmark demonstrate the superiority of the proposed ACs over existing location-based context baselines, which proposes a new direction for predicting AR quality. The analysis suggests that most evidence ARs need reasoning ACs, and reasoning ARs need a diverse set of claims, evidence, and reasoning ACs to predict their quality.

heading: 8 Limitations
text: Our experiments were built on perfect sentence alignments in the original and revised essay drafts, thus the performance could be lower in the real end-to-end Automated Writing Evaluation (AWE) system. In addition, our corpus is small due to expensive annotation processes, which makes it challenging to train or finetune large language models. Also, we only focus on revisions in argumentative writing, specifically, we focus on the evidence and reasoning revisions, however other revisions like claim revisions are not used. Furthermore, the revised drafts were done after providing feedback on the original drafts, which means the revised student essays are likely to follow the instructions in the feedback but we did not use this information for revision quality predictions, which will be used in our future work.
Our proposed Argumentative Contexts (ACs) are generated by ChatGPT which is not free for the whole community. Also, ChatGPT-generated ACs have small randomness, which is also the reason we did 3-seed runs in the experiments. In addition, the ACs are essay-level context which means different revisions in the same essay use the same context. It could be tailored to have sentence-level ACs where each sentence-level revision has slightly different revision purposes, but it would cost more
time and money. Moreover, our proposed zero-shotCoT prompts perform better than Single prompts by small margins in specific cases, which indicates that Chat-GPT is limited to conducting CoT extraction and summarization to handle complex wording and sentence structure. Therefore, we might need to redesign the prompts in our future work.

heading: 9 Ethics
text: Our corpus was collected under standard protocols that were approved by an institutional review board. Our annotated data is not publicly available which ensures the safety of private information of the students, and thus will not pose any ethical concerns because other researchers can not access our data and replicate our results. Our future work is to incorporate proposed methods in real AWE systems to evaluate student writings and provide informative feedback based on predictions. But there is a risk that the system might give poor advice based on incorrect AR quality predictions, given that the model may learn biases with small annotated data.

heading: Acknowledgments
text: The research was supported by the National Science Foundation Award #2202347 and a gift from CloudBank. The opinions expressed were those of the authors and did not represent the views of the institutes. We would like to thank anonymous reviewers and Pitt PETAL group for their valuable feedback on this work.

heading: 4 . . . . . . ... . . . . . . . . .

heading: Future Work
text: Our future work is to incorporate proposed methods in real AWE systems to evaluate student writings and provide informative feedback based on predictions. But there is a risk that the system might give poor advice based on incorrect AR quality predictions, given that the model may learn biases with small annotated data