Venue: NAACL 2024

# A School Student Essay Corpus for

Analyzing Interactions of Argumentative Structure and Quality

Maja Stahl1, Nadine Michel2, Sebastian Kilsbach2, Julian Schmidtke1, Sara Rezat2, and Henning Wachsmuth1

###### Abstract

Learning argumentative writing is challenging. Besides writing fundamentals such as syntax and grammar, learners must select and arrange argument components meaningfully to create high-quality essays. To support argumentative writing computationally, one step is to mine the argumentative structure. When combined with automatic essay scoring, interactions of the argumentative structure and quality scores can be exploited for comprehensive writing support. Although studies have shown the usefulness of using information about the argumentative structure for essay scoring, no argument mining corpus with ground-truth essay quality annotations has been published yet. Moreover, none of the existing corpora contain essays written by school students specifically. To fill this research gap, we present a German corpus of 1,320 essays from school students of two age groups. Each essay has been manually annotated for argumentative structure and quality on multiple levels of granularity. We propose baseline approaches to argument mining and essay scoring, and we analyze interactions between both tasks, thereby laying the ground for quality-oriented argumentative writing support.

1Leibniz University Hannover, Institute of Artificial Intelligence

2Paderborn University, Institute for German Language and Comparative Literature

{m.stahl, h.wachsmuth}@ai.uni-thannover.de, julian.schmidtke@stud.uni-hannover.de (nadine.michel, sebastian.kilsbach, sara.rezat)@uni-paderborn.de

## 1 Introduction

Writing argumentative texts, in particular argumentative essays, constitutes an essential part of school students' writing education. However, learning to write arguments of high quality can be challenging Zhu (2001); Ferretti et al. (2007); Peloghitis (2017); Alexander et al. (2023). It requires various skills, from writing fundamentals, such as syntax and grammar, to argumentation-specific skills, such as meaningfully organizing and structuring arguments and counter-considerations Rezat (2011). This takes time and effort to master Ka-kan-deep and Kaur (2014); Dang et al. (2020). Given teachers' limited time to give students feedback on their writing, automatic argumentative writing support could benefit students as it offers guidance at their own pace and convenience Wambsganss et al. (2022a).

Argumentative writing support systems employ argument mining to analyze input texts Stab (2017); Wambsganss and Niklaus (2022); Weber et al. (2023), that is, computational methods that identify argumentative components and their relations. Common components are _major claim_ (main standpoint of the text, also known as _thesis_), _claim_ (controversial statement), and _premise_ (reason for justifying or refuting the claim) along with their argumentative relations _support_ and _attack_. This knowledge enables the systems to give feedback on the structure of a text, e.g., by highlighting unwarranted claims Stab and Gurevych (2017), or by analyzing the number of argumentative components quantitatively Stab and Gurevych (2017); Wamsbganss and Niklaus (2022); Weber et al. (2023).

Unlike argument mining, automated essay scoring explicitly evaluates essay quality, either holis

Figure 1: Exemplary annotated school student essay on the use of school funding, taken from our corpus. The text is from the FD-LEX corpus Becker-Mrotzek and Grabowski (2018), translated from German for display.



tically (Uto et al., 2020; Yang et al., 2020; Wang et al., 2023) or specific linguistic aspects, such as coherence (Li et al., 2018; Farag et al., 2018), grammar (Ajit Tambe and Kulkarni, 2022), and organization (Persing et al., 2010; Rahimi et al., 2015). Combining argument mining with essay scoring may enable support systems to give students comprehensive feedback on their writing. In addition, it helps identify how different argumentative structures influence the overall essay quality and which structures are common for different levels of quality (Wachsmuth et al., 2016). However, student essay corpora for argument mining are scarce and do not include ground-truth essay quality annotations (Stab and Gurevych, 2017a). Moreover, no corpus with structure annotations for essays written by school students has been published yet.

To fill this research gap, we present a German corpus of 1,320 school student essays with manual annotations for argumentative structure and essay quality. The essays have been systematically selected from an existing corpus (Becker-Mrotzek and Grabowski, 2018), equally distributed over two age groups (fifth-graders and ninth-graders) and binary genders, three per student. We present an extensive annotation scheme focused on school student essays that covers argumentative structure on four levels of granularity as well as five essay quality aspects, as shown in Figure 1. To achieve consistent annotations, we developed annotation guidelines in close dialogue with our expert annotators from the field of language education. This led to high agreement between the annotations.

Our analyses of the corpus provide various insights into the correlation between the different levels of argumentative structure and essay quality, as well as the interaction between these two types of annotation. We experiment with fine-tuned transformers and adapters as baseline approaches to mining argumentative structure and scoring essay quality. Moreover, we demonstrate that the information on argumentative structure helps predicting the essay quality, which is in line with what previous studies showed on other corpora (Wachsmuth et al., 2016; Beigman Klebanov et al., 2016; Nguyen and Litman, 2018). This result underlines the usefulness of our corpus annotations for quality-oriented argumentative writing support.

More explicitly, this work aims to answer (i) how the argumentative structure and essay quality of school student essays can be modeled, (ii) how different levels of argumentative structure and essay quality correlate for school student essays, and (iii) how this correlation can be exploited to automatically score the essay quality.

Altogether, this paper's main contributions are:

* A corpus for studying argumentative structure and essay quality on school student essays
* Empirical insights into the interactions of argumentative structure and essay quality
* Baseline approaches to argument mining and essay scoring1 Footnote 1: Our corpus and experiment code can be found under: [https://github.com/webis-de/NAACL-24](https://github.com/webis-de/NAACL-24).

## 2 Related Work

Argumentative writing is a key capability that is taught in school across age groups and disciplines (Becker-Mrotzek et al., 2010; Rezat, 2011). A common educational form of argumentative text is the essay, where school students should introduce a thesis, to which they provide pro and con arguments, and finally conclude (Townsend et al., 1993; Schroter, 2021). The components of an argumentative text take on different roles (e.g., _claim_ or _premises_), and they may operationalize different actions (e.g., _conceding_ or _reasoning_) (Feilke, 2017). Learning to write argumentative text is complex and requires continuous and detailed feedback (Kellogg et al., 2010; Wambsganss et al., 2022b).

Analyzing the argumentative structure of texts computationally, also known as argumentation) mining, is a crucial and widely-studied step in providing automatic support for argumentative writing (Stede and Schneider, 2019). Student essays are a prominent domain for argument mining. A respective annotated corpus of 402 English student essays is available (Stab and Gurevych, 2017a), for which also quality issues such as insufficient claim support have been modeled (Stab and Gurevych, 2017b; Gurcke et al., 2021). Additionally, student corpora are available for more specific domains, such as argumentative legal texts (Weber et al., 2023), persuasive peer reviews on business models (Wambsganss et al., 2020), and business model pitches (Wambsganss and Niklaus, 2022).

Some research has used argumentative structure to assess essay quality. In consecutive works, Persing et al. (2010) and Persing and Ng (2013, 2014, 2015) graded different argumentation-related quality aspects for the well-known essay corpus ICLE 

(Granger et al., 2009), namely organization, thesis clarity, prompt adherence, and argument strength. In contrast, Horbach et al. (2017) targeted different quality aspects of argumentative writing at once. Some works further investigated the interaction between argumentative structure and essay quality. Wachsmuth et al. (2016) found that multiple argumentation-related essay scoring tasks benefit from argument mining, underlining the impact of argumentative structure on essay quality. The analyses by Beigman Klebanov et al. (2016) and Nguyen and Litman (2018) suggest that this finding also holds for predicting holistic essay scores, while Persing et al. (2010) observed similar for the quality aspect organization. However, these studies relied on automatically assigned quality scores only, due to the lack of ground-truth annotations.

In a related line of research, approaches have been proposed to suggest revisions for argumentative essays Afrin and Litman (2018), to assess the need for and the quality of revisions Skitalinskaya and Wachsmuth (2023); Liu et al. (2023), as well as to perform argument revisions computationally Skitalinskaya et al. (2023). Other works towards writing support for argumentative essays presented a prototypical system that gives simple feedback in terms of missed criteria Stab (2017), design principles for an adaptive learning tool Wamsganss and Rietsche (2019), visual feedback to the learner to prompt them to repair broken argument structures Wambsganss et al. (2022a), or point to ethylmematic gaps in arguments and make suggestions on how to fill these gaps Stahl et al. (2023). Most recently, Britner et al. (2023) proposed a tool that not only analyzes issues with argument quality but also generates an explanation for its prediction. Our corpus supports these steps towards support systems for argumentative writing by providing detailed ground-truth annotations for both the argumentative structure and the quality of essays.

However, all the works mentioned deal with texts written by university students, while our work targets argumentative essays written by school students, fifth-graders and ninth-graders specifically. To the best of our knowledge, the only other published corpora with school student essays is not openly available Correnti et al. (2013) and has been analyzed for essay-level quality aspects only, such as the integration of evidence Rahimi et al. (2014) and the essay's organization Rahimi et al. (2015). We recently came across another school student essay corpus in English with annotations for argumentative structure and quality, which has yet to be published.2 We go beyond in this work by assessing the quality of school student essays in terms of five aspects derived from language education literature while incorporating their interaction with annotated argumentative structures. This may foster the development of effective methods for helping school students improve their argumentative writing skills.

Footnote 2: The unpublished pre-print is available at [https://zenodo.org/records/8221504](https://zenodo.org/records/8221504).

## 3 School Student Essay Corpus

This section presents the source data and annotation of our corpus for analyzing the argumentative structure and quality of school student essays.

### Source Data

As the basis, we systematically selected 1,320 German school student essays from the FD-LEX corpus Becker-Mrotzek and Grabowski (2018).3 The authors instructed students to each write three argumentative essays on topics pertinent to school students: (a) a letter to school funding organization on possible use of funding, (b) a statement on how to deal with the misbehavior of a fellow student, (c) a statement on who is guilty in a bike accident.

Footnote 3: Essays and metadata (grade, school form, age, gender, language background, and age group) are available here: [https://fd-lex.uni-koeln.de](https://fd-lex.uni-koeln.de)

We seek to enable analyses of differences across different groups of school student essays on the corpus. Therefore, we pseudo-randomly chose 440 school students equally distributed across genders (only _male_ and _female_ exist in the corpus) and age groups (_fifth-graders_ and _ninth-graders_). Subsequently, we included all three essays written by each selected school student from the source data.

### Annotation Scheme

Our annotation scheme goes beyond existing corpora for argument mining, covering the macro and micro structure of argumentative essays on four levels in total. In addition, we evaluate the quality of the essays overall and in terms of four quality aspects. Figure 2 overviews our annotation scheme.

Argumentative StructureOn the broadest level of granularity for argumentative structure, we annotate _discourse functions_Persing et al. (2010): 

* _Introduction._ Initiates an essay by presenting the topic and possibly the context of an essay. This section is usually non-argumentative and placed at the beginning of an essay.
* _Body._ Core of the essay, containing the majority of argumentative components.
* _Conclusion._ Summary of main points, often with a final evaluation of the topic. This section is typically found at the end of the essay.

Next, we annotate _arguments_ that comprise one point in an argumentative text, following Walton et al. (2008). We differentiate them by stance towards the main standpoint (thesis) of an essay:

* _Argument._ Ideally a claim (conclusion) and premises (reasons) supporting the claim.
* _Counter-argument._ An argument that attacks the thesis of an essay.

For analyzing the micro structure, we annotate argumentative and non-argumentative _components_. As Stab and Gurevych (2017a), we also mark support and attack relations between them (see Figure 2):

* _Topic._ Non-argumentative component that describes the subject or purpose of the essay.
* _Thesis._ Main standpoint of the whole argumentative text towards the topic. Repetitions of the thesis are also annotated as such.
* _Antibresis._ Thesis contrary to the actual thesis.
* _Modified Thesis._ Modified version of the actual thesis (e.g., more detailed or resticted).
* _Claim._ Statement that conveys a stance towards the topic.
* _Premise._ Reason that is given to support or attack a claim or another premise.4 On the finest level of granularity, we annotate _discourse modes_Smith (2003) specific to school student essays. They are derived from language education literature, where they are used for developing and analyzing argumentative writing skills (Gatje et al., 2012; Rezat, 2018; Feilke and Rezat, 2021):

Footnote 4: The components in our annotation scheme are similar to those of Stab and Gurevych (2017a). We added _antihesis_ and _modified thesis_ to reflect the changes in position in the essays.

* _Comparing._ Contrasting supporting and attacking points to a statement.
* _Conceding._ Addressing a counter-consideration and refuting it to support the own stance.
* _Concluding._ Drawing logical inferences using consecutive or final clauses (_so that, if... then_).
* _Describing._ Providing additional information, such as facts, statistics, and background data.
* _Exemplifying._ Providing examples or reporting on experiences.
* _Instructing._ Providing explicit instructions that recommend a specific course of action.
* _Positioning._ Expressing the own standpoint.
* _Reasoning._ Providing causal links to support a claim/thesis using markers (_because_, _then_).
* _Referencing._ Mentioning statements made by others, for example, by authorities.
* _Qualifying._ Presenting a variation of the all-or-nothing standpoints.

Essay QualityAs Persing et al. (2010), we score essay quality on a 7-point scale from 1 (unsuccessful), 2 (rather unsuccessful), 3 (rather successful) to 4 (completely successful), with half points in between. We adapted the quality aspects of Kruse et al. (2012) for assessing school student essays in general to argumentative essays as follows:

* _Relevance._ The essay fits the prompt.
* _Content._ The selection of content helps to reach the essay's goal.

Figure 2: Proposed annotation scheme for argumentative school student essays: Four levels of argumentative macro and micro structure (discourse functions, arguments, components, discourse modes) and five essay quality aspects.

 

* _Structure._ The selected points are coherent and well-connected.
* _Style._ The use of language is adequate.
* _Overall._ The overall impression of the rater.

### Annotation Process

We underwent the following process for both argumentative structure and essay quality:

To test and refine our annotation guidelines, we conducted pilot studies in which all annotators worked on the same 30 texts. We then discussed their understanding of the guidelines and annotation differences. We integrated their feedback into the guidelines to then test the reliability of the annotations in an inter-annotator agreement (IAA) study, where all annotators independently worked on the same 120 texts. Finally, each annotator annotated a set of 1,200 essays in the main annotation study.

As annotators, we employed experts in German language education from our lab and started the pilot and IAA studies on argumentative structure with three annotators. For the main part, only the two annotators with the most reliable annotations proceeded. The same annotators then annotated the essay quality, since they had already been trained in argumentative texts and our general procedure. However, we acknowledge that the annotators may have been predisposed to view essays in a certain way after the first annotation.

To assemble the final corpus, we combined the 1200 essays from the main study with the 120 essays from the IAA study after solving annotation conflicts. For conflicts between the three structure annotations per IAA essay, we kept the annotations that had the highest agreement across all levels with the other two. For conflicts between essay quality scores, we used their mean as the final score.5

Footnote 5: We rounded down to the next valid quality score for low values (\(<2.5\)) and rounded up for high values (\(>2.5\)) to prevent an upward distortion of the distribution.

### Inter-Annotator Agreement

For the components, we follow Stab and Gurevych (2017a) in that we evaluate the agreement per essay at the token level, so the token labels are the unit of analysis. Thereby, overlaps of annotations are taken into account. For relations, we determined the component-level spans that at least two annotators agreed on with a relative overlap \(\geq\) 75%. For all pairs of these, we then compared the relation labels (no relation, support, or attack) between the annotators. The mean Krippendorff's \(\alpha\) scores over the 120 IAA assays are reported in Table 1. For essay quality, we computed the \(\alpha\)-value per quality aspect with essays as the unit of analysis.

The agreement is high for argumentative structure spans with values between 0.74 and 0.89. The agreement for relations is lower but reasonable, given that disagreement from the components annotations is propagated to the relations. The agreement for essay quality is also high, too, ranging from 0.77 to 0.95. Overall, we conclude that the annotations can mostly be seen as very reliable. Content and style quality annotations are very consistent between annotators, while assessing the relevance and structure seems slightly more subjective.

### Corpus Statistics

Table 2 gives insights into the label distribution for argumentation structure. _Body_ occurs most frequently among the discourse functions (1,335 times). With 56.75 tokens on average, bodies are also notably longer than _introductions_ and _conclusions_. On the argument level, we note that _counter-arguments_ are rather sparse in student essays. Among the components, _claims_ are most frequent in total and per essay, followed by _theses_. Furthermore, we notice that _modified__theses_ are usually longer than theses, which matches our expectation that students add more details or restrictions to the thesis there. The most used discourse modes are _positioning_, _describing_, and _reasoning_, while _referencing_, _comparing_, and _exemplifying_ occur rarely. Notable are also the differences in span length, e.g. _positioning_ spans have on average only about half as many tokens as _comparing_ spans.

Table 3 gives the frequency of annotated relations in our corpus. Most relations outgoing from _claims_ are directed towards _theses_ (92.6%), while most relations outgoing from _premises_ are directed towards _claims_ (85.8%). Overall, 96.2% of the re

\begin{table}
\begin{tabular}{l c l l} \hline \hline
**Argumentative Structure** & \(\alpha\) & **Essay Quality** & \(\alpha\) \\ \hline Discourse Functions & 0.89 & Relevance & 0.77 \\ Arguments & 0.86 & Content & 0.95 \\ Components & 0.81 & Structure & 0.84 \\ Discourse Modes & 0.74 & Style & 0.92 \\ Relations & 0.58 & Overall & 0.95 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Krippendorff’s \(\alpha\) agreement in the IAA study between three annotators for argumentative structure and two annotators for essay quality. The high values stress the high reliability of our annotations.

 

lations were labeled as _support_ and 3.8% as _attack_.

The distribution of quality scores is shown in Table 4. We can see that _relevance_, _structure_, and _style_ have a similar score distribution, while the distribution of _content_ scores is shifted towards the higher scores, with the highest mean (2.80). Overall quality has the lowest mean score (2.20) and was most often scored with the lowest score of 1.0. These results suggest that overall quality is not just the average of the other annotated quality aspects but that it emerges from the annotators' perception and possibly other aspects.

## 4 Analysis

This section reports on our corpus analysis of the interaction between the argumentative structure on micro vs. macro level and component vs. discourse mode level, and the different essay quality aspects.

### Macro vs. Micro structure

Figure 3(a) shows the overlap between macro structure (discourse functions and arguments) and micro structure (components and discourse mode) labels. The _introduction_ mainly includes the _topic_ (68%), while every second token in the _body_ is a _claim_ on average. The _thesis_ cooccurs with all three discourse functions. We see that the proportion of _claims_ and _premises_ differs for _arguments_ and _counter-arguments_. Counter-arguments contain more claim tokens than arguments, while fewer counter-argument tokens are part of a premise.

The usage of discourse modes differs between the macro-structure levels, too. In the introduction, students mostly _describe_ (68%) and _position_ (24%). The discourse modes are more diverse for the body, also including a notable portion of _reasoning_. As expected, in the _conclusion_, students focus more on _concluding_. While describing and reasoning are prevalent in arguments and counter-arguments, a notable portion of argument tokens is used for concluding. At the same time, more _conceding_ and

\begin{table}
\begin{tabular}{l r r r r r r r} \hline \multicolumn{1}{c}{**Quality Aspect**} & **1.0** & **1.5** & **2.0** & **2.5** & **3.0** & **3.5** & **4.0** & **Mean** \\ \hline Relevance & 64 & 131 & **548** & 358 & 190 & 23 & 6 & 2.22 \\ Content & 62 & 33 & 123 & 312 & **510** & **173** & **107** & **2.80** \\ Structure & 59 & 91 & 423 & 414 & 292 & 24 & 17 & 2.35 \\ Style & 75 & **159** & 396 & **436** & 233 & 12 & 9 & 2.25 \\ Overall & **90** & 142 & 478 & 390 & 195 & 23 & 2 & 2.20 \\ \hline \end{tabular}
\end{table}
Table 4: Distribution and mean of scores per quality aspect. The highest value per column is marked bold.

\begin{table}
\begin{tabular}{l r r r r} \hline \multicolumn{1}{c}{**From**_Claim_ **to**} & \multicolumn{1}{c}{**\#**} & \multicolumn{1}{c}{**\%**} & \multirow{2}{*}{**From**_Premise_ **to**} & \multicolumn{1}{c}{**\#**} & \multicolumn{1}{c}{**\%**} \\ \cline{1-1} \cline{5-5} Thesis & **2844** & **92.6** & & & & \\ Modified Thesis & 218 & 7.1 & Claim & **872** & **85.8** \\ Antithesis & 9 & 0.3 & Premise & 138 & 13.6 \\ \hline \end{tabular}
\end{table}
Table 3: Absolute and relative frequency of annotated relations outgoing from _claim_ (left) or _premise_ (right).

Figure 3: Cooccurrence matrices: Relative token-level overlap of (a) macro and micro structure and (b) component and discourse mode labels in percent. For example, 68% of all tokens labeled as _Introduction_ on the macro level are also labeled as _Topic_ on the micro level.

 

qualifying_ tokens occur in counter-arguments. This is expected, since especially in counter-arguments other points of view should be varied or refuted.

### Components vs. Discourse modes

The cooccurrences between components and actions can be seen in Figure 3(b). While the _topic_ is mostly _described_ (90%) and the _thesis_ consists primarily of _positioning_ (85%), the remaining components include more diverse discourse modes. In contrast to _theses_, _modified__theses_ also feature _describing_ and _qualifying_ tokens, while _antitheses_ additionally cover _conceding_ and _concluding_. _Claims_ and _premises_ mainly cooccur with _describing_, _reasoning_, and _concluding_. However, the proportions differ slightly. The cooccurrence matrix between all structure labels can be found in Appendix A.

### Essay Quality

To further assess the interaction between the quality aspects, Table 5 shows all pairwise Kendall's \(\tau\) correlations. All aspects correlate most with _overall quality_, most strongly _relevance_ (.75). The correlation between _content_ and _style_ is lowest (.41), which underlines their distinctive nature.

## 5 Experiments

This section presents baseline approaches to the two main tasks our corpus enables: Predicting the argumentative structure (argument mining) and the essay quality (essay scoring). Additionally, we investigate whether information about the argumentative structure helps to predict the essay quality.

### Argument Mining

We treat argument mining as a token classification task: Given a school student essay and a structure level, predict the label of each token on that structure level. The IOB2 format is used for the labels to separate adjacent spans of the same type. We performed 5-fold cross-validation for each structure level. For each folding, we used four folds (80%) for training and divided the fifth fold in half: one half (10%) for selecting the best-performing checkpoint in terms of macro-averaged F\({}_{1}\)-score, and the remaining half (10%) for testing.

ModelsWe used the multilingual model _mDeBERTaV3_He et al. (2023) (microsoft/mdebertav3-base) from Huggingface Wolf et al. (2020).6 Besides, we tested the effect of training _adapters_Houlsby et al. (2019), a set of task-specific parameters that are added to every transformer layer of mDeBERTaV3 and fine-tuned on the task while the model weights are fixed. To quantify the impact of learning, we compare against a random baseline that chooses a token label pseudo-randomly and a majority baseline that always predicts the majority token label from the training set. As upper bound, we report the human performance in terms of the average of each annotator in isolation on the 120 IAA texts annotated by all annotators.7

Footnote 6: Explorative experiments using instruction fine-tuned models such as Alpaça Taori et al. (2023) did not lead to promising results for our token classification task.

Experimental SetupWe train mDeBERTaV3 for 30 epochs (1,980 steps) using the suggested hyperparameter values: a learning rate of \(3e-5\), batch size 16, and 500 warmup steps. For mDeBERTaV3-adapter, we follow Pfeiffer et al. (2020) who recommend to use a higher learning rate of \(1e-4\) and train longer, here 50 epochs (3,300 steps).

ResultsTable 6 presents the token classification results for all levels of argumentative structure, averaged over all folds. Noteworthily, _mDeBERTaV3-adapter_ outperforms training the whole model (_mDeBERTaV3_) in all cases. Given that the F\({}_{1}\)-scores improve more than the accuracy, the

\begin{table}
\begin{tabular}{l r r r r r r} \hline \hline  & **D. Func.** & **Argumo.** & **Compon.** & **D. Mode** \\ \cline{2-7}
**Approach** & **Acc.** & **F\({}_{1}\)** & **Acc.** & **F\({}_{1}\)** & **Acc.** & **F\({}_{1}\)** \\ \hline Random & .14 & .00 & .20 & .00 & .08 & .00 & .05 & .00 \\ Majority & .86 & .52 & .56 & .00 & .41 & .00 & .24 & .00 \\ mDeBERTaV3 & .92 & .46 & .86 & .29 & .66 & .21 & .63 & .21 \\ -adapter & **.95** & **.68** & **.92** & **.52** & **.76** & **.49** & **.73** & **.46** \\ \hline Human & .98 & .94 & .96 & .85 & .93 & .89 & .89 & .84 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Argument mining results: Macro F\({}_{1}\)-score and accuracy of each approach in 5-fold cross-validation on all four argumentative structure dimensions. The best value per column is marked bold.

\begin{table}
\begin{tabular}{l c r r r} \hline \hline  & **Relevance** & **Content** & **Structure** & **Style** & **Overall** \\ \hline Relevance & & .53 & .61 & .47 & **.75** \\ Content & .53 & & .48 & .41 & .60 \\ Structure & .61 & .48 & & .51 & .71 \\ Style & .47 & .41 & .51 & & .61 \\ Overall & **.75** & **.60** & **.71** & **.61** & \\ \hline \hline \end{tabular}
\end{table}
Table 5: Kendall’s \(\tau\) correlation between the quality aspects. The highest value per column is marked bold.

 

adapters seem less prone to overfitting to the majority label. This learning success suggests the possibility of predicting all argumentative structure levels on our corpus. However, further improvements using more advanced approaches are expected.

### Essay Scoring

We treat predicting the essay quality as a text classification task: Given a school student essay and a quality aspect, predict the corresponding quality score. As before, we performed 5-fold cross-validation for each quality aspect using the same folds. We selected the best-performing checkpoint on the validation set using quadratic weighted kappa (QWK), the most widely adopted metric for automatic essay scoring Ke and Ng (2019).

ModelsWe adopted the previous approaches by changing the head to a text classification head. To analyze the interaction between argumentative structure and essay quality, we employed _AdapterFusion_ Pfeiffer et al. (2021), a multi-task learning framework that can be used to investigate relations between different dimensions by learning how to combine model weights with one or more adapters. We used the mDeBERTaV3-adapter trained on argumentative structure from the previous experiment. As the final adapter, we chose the one trained on the folding that performed most representative for all folds (\({\rm F}_{1}\)-score closest to the reported averaged \({\rm F}_{1}\)-score across folds). To measure the impact of each level of argumentative structure on the scoring performance, we used each adapter individually and a combination of all of them.

Experimental SetupThe experimental setup for mDeBERTaV3 and mDeBERTaV3-adapter was adopted from before. For training the AdapterFusion, we followed Pfeiffer et al. (2021) to use a learning rate of \(5e-5\) and trained shorter than the adapters, in our case for 20 epochs (1,320 steps).

ResultsTable 7 shows the scoring results. All models outperform the lower-bound baselines (_random_ and _majority_), suggesting that the quality scoring can be learned from our corpus. Furthermore, fusing all adapters trained on argumentative structure (_mDeBERTaV3-fusion-w/-all_) performs best for three out of five quality aspects, significantly beating mDeBERTaV3-adapter in predicting overall quality.8 This underlines the need for all four levels of argumentative structure together in order to improve scoring overall quality (0.686 vs. 0.648). In addition, using only the adapter trained on the component level (_mDeBERTaV3-fusion-w/-components_) helps to significantly improve over mDeBERTaV3-adapter in predicting relevance (0.600 vs. 0.564), indicating an interaction between the structure on component level and this quality aspect. QWK scores greater or equal to 0.6 suggest substantial agreement between the predicted and ground-truth quality scoring of essays.

Footnote 8: We use Wilcoxon signed-rank tests at \(p<.05\) for testing the significance.

AdaperFusion ActivationsAdapterFusion extracts information from adapters only if they benefit the target task. Similar to Falk and Lapesa (2023), we visualize the average activations of our model _mDeBERTaV3-fusion-w/-all_ over the layers in Figure 4 to investigate the influence of each level of argumentative structure on the quality scoring. All adapters are activated fairly evenly for all quality aspects, with slight deviations. This aligns with our previous results and underl

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Approach** & **Relevance** & **Content** & **Structure** & **Style** & **Overall** \\ \hline Random & \(-0.013\pm 0.084\) & \(-0.011\pm 0.071\) & \(-0.014\pm 0.073\) & \(0.017\pm 0.084\) & \(-0.004\pm 0.083\) \\ Majority & \(0.000\pm 0.000\) & \(0.000\pm 0.000\) & \(0.000\pm 0.000\) & \(0.000\pm 0.000\) & \(0.000\pm 0.000\) \\ mDeBERTaV3 & \(0.530\pm 0.069\) & \(0.295\pm 0.109\) & \(0.513\pm 0.044\) & \(0.492\pm 0.059\) & \(0.616\pm 0.040\) \\ -adapter & \(0.564\pm 0.018\) & \(0.431\pm 0.098\) & \(\textbf{0.575}\pm 0.038\) & \(0.579\pm 0.077\) & \(0.648\pm 0.054\) \\ -fusion-w/-discourse-functions & \(0.599\pm 0.043\) & \(0.381\pm 0.134\) & \(0.559\pm 0.036\) & \(0.569\pm 0.069\) & \(0.668\pm 0.049\) \\ -fusion-w/-arguments & \(0.593\pm 0.030\) & \(0.448\pm 0.105\) & \(\textbf{0.575}\pm 0.019\) & \(0.581\pm 0.054\) & \(0.668\pm 0.036\) \\ -fusion-w/-components & \(\textbf{`0.600}\pm 0.025\) & \(0.437\pm 0.137\) & \(0.543\pm 0.044\) & \(0.585\pm 0.053\) & \(0.663\pm 0.046\) \\ -fusion-w/-discourse-modes & \(0.544\pm 0.028\) & \(0.420\pm 0.118\) & \(0.535\pm 0.041\) & \(0.583\pm 0.064\) & \(0.645\pm 0.023\) \\ -fusion-w/-all & \(0.574\pm 0.039\) & \(\textbf{0.454}\pm 0.142\) & \(0.546\pm 0.013\) & \(\textbf{0.617}\pm 0.057\) & \(\textbf{`0.686}\pm 0.031\) \\ \hline Human & \(0.636\pm 0.055\) & \(0.632\pm 0.003\) & \(0.734\pm 0.007\) & \(0.766\pm 0.005\) & \(0.746\pm 0.003\) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Essay scoring results: QWK of each approach in 5-fold cross-validation on all five quality dimensions. The best value per column is marked bold. We mark significant gains over _mDeBERaV3-adapter_ at \(p<.05\) with †.

 

structure levels are helpful for quality scoring. The activations per layer can be found in Appendix B.

## 6 Conclusion

Argumentative writing support of school students presupposes that the quality of their arguments can be assessed. Until now, no argument mining corpus with school student essays has been published, let alone any essay corpus with both argument and quality annotations. With this work, we fill both research gaps with a new corpus of 1,320 German school student essays, annotated by experts for argumentative structure and essay quality.

Our corpus analysis has provided various insights into the correlation between the different levels of argumentative structure and essay quality. In our experiments with fine-tuned transformers and adapters for mining argumentative structure and scoring essay quality we have demonstrated that combining information on all four argumentative structure levels helps the prediction of essay quality. This shows the usefulness of our corpus for research on quality-oriented argumentative writing support, which we seek to enable with this paper.

We point out that our corpus contains various information yet to be explored, such as argumentative relations and school student metadata. It thus lays the ground for further analyses--like identifying unwarranted claims and studying differences across age groups and genders.

## 7 Limitations

Aside from the still-improvable performance of the presented baseline models for argument mining and essay scoring, we see two notable limitations of our work: the restriction to German texts, and the pending utilization of the corpus for quality-oriented argumentative writing support.

First, we point out the specific language background of our work. The essays were written by German school students, and the annotations were developed in close communication with German experts from the field of language education, while the discourse modes and essay quality aspects are, to a considerable extent, derived from work on German texts. This means that our findings may not perfectly align with argumentative writing in other countries or languages with different expectations for argumentative essays.

Second, while our analyses suggest that our corpus helps to enable quality-oriented argumentative writing support, the perceived usefulness of such a tool is still to be evaluated. We expect and encourage future work to utilize our corpus for such writing support tools, for example, by further analyzing which exact argumentative structures influence the essay quality and to what extent. Interpretable essay quality scoring based on the structure might generate helpful insights that can be used as writing feedback by school students.

## 8 Ethical Considerations

We see no apparent risk of the corpus or the methods presented in this paper being misused for ethically doubtful purposes. The authors of the FD-LEX corpus Becker-Mrotzek and Grabowski (2018) have already pseudonymized the author of each essay. Therefore, it is not possible to identify the individual school student from the provided data. However, we want to point out that one might find differences in the essays across gender or age groups that do not reflect reality but are rather due to an unintentional bias in the data selection.

### Acknowledgments

We would like to thank the participants of our study and the anonymous reviewers for the valuable feedback and their time. This work was partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) within the project ArgSchool, project number 453073654.

Figure 4: AdapterFusion activation on average over the layers for each _mDeBERTaV3-fusion-w/-all_ model per quality aspect. We average the activation for each fused adapter (discourse functions, arguments, components, discourse modes) over all instances in the most representative test set folding.

 



## References

* Afrin and Litman (2018) Tazin Arfin and Diane Litman. 2018. Annotation and classification of sentence-level revision improvement. In _Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 240-246, New Orleans, Louisiana. Association for Computational Linguistics.
* Tambe and Kulkarni (2022) Aniket Ajit Tambe and Manasi Kulkarni. 2022. Automated essay scoring system with grammar score analysis. In _2022 Smart Technologies, Communication and Robotics (STCR)_, pages 1-7.
* Alexander et al. (2023) Patricia A. Alexander, Jannah Fusenig, Eric C. Schoute, Anisha Singh, Yuting Sun, and Julianne E. van Meerten. 2023. Confronting the challenges of undergraduates' argumentation writing in a "learning how to learn" course. _Written Communication_, 40(2):482-517.
* Becker-Mrotzek and Grabowski (2018) Michael Becker-Mrotzek and Joachim Grabowski. 2018. Textkorpus Scriptoria. In Michael Becker-Mortozek and Joachim Grabowski, editors, _FDLEX (Forschungsdatenbank Lemeretexte)_. MercatorInstitut fur Sprachroderung und Deutsch alk Zweitsprache, Koln. Available at: [https://fd-lex.uni-koeln.de](https://fd-lex.uni-koeln.de).
* lehren und lernen. Vorschlage fur einen systematischen Kompetenzaufbau in den Stufen 5 bis 8.
* Klebanov et al. (2016) Beata Beigman Klebanov, Christian Stab, Jill Burstein, Yi Song, Binod Gyawali, and Iryna Gurevych. 2016. Argumentation: Content, structure, and relationship with essay quality. In _Proceedings of The Third Workshop on Argument Mining (ArgMining2016)_, pages 70-75, Berlin, Germany. Association for Computational Linguistics.
* Brinter et al. (2023) Sebastian Brinter, Lorik Dumani, and Ralf Schenkel. 2023. Aquaplane: The argument quality explainer app. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_, CIKM '23, page 5015-5020, New York, NY, USA. Association for Computing Machinery.
* Correnti et al. (2013) Richard Correnti, Lindsay Clare Matsumura, Laura Hamilton, and Elaine Wang. 2013. Assessing students' skills at writing analytically in response to texts. _The Elementary School Journal_, 114(2):142-177.
* Crossley et al. (2023a) Scott Crossley, Perpetual Baffour, Tian Yu, Alex Franklin, Meg Benner, and Ulrich Boser. 2023a. A large-scale corpus for assessing written argumentation: PERSUADE 2.0.
* Crossley et al. (2023b) Scott Crossley, Yu Tian, Perpetual Baffour, Alex Franklin, Youngmeen Kim, Wesley Morris, Meg Benner, Aginer Picou, and Ulrich Boser. 2023b. The english language learner insight, proficiency and skills evaluation (ellipse) corpus. _International Journal of Learner Corpus Research_, 9(2):248-269.
* Dang et al. (2020) Thi Hanh Dang, Thanh Hai Chau, and To Quyen Tra. 2020. A study on the difficulties in writing argumentative essays of english-majored sophomores at day do university, vietham. _European Journal of English Language Teaching_, 6(1).
* Falk and Lapessa (2023) Neele Falk and Gabriella Lapessa. 2023. Bridging argument quality and deliberative quality annotations with adapters. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 2469-2488, Dubrovnik, Croatia. Association for Computational Linguistics.
* Farag et al. (2018) Youma Farag, Helen Yannakoudakis, and Ted Briscoe. 2018. Neural automated essay scoring and coherence modeling for adversarially crafted input. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 263-271, New Orleans, Louisiana. Association for Computational Linguistics.
* Feilke (2017) Helmuth Feilke. 2017. Schreib- und Textprozeduren. In Jurgen Baumann, Clemens Kammler, and Astrid Muller, editors, _Handbuch Deutschunterricht. Theorie und Praxis des Lehrens und Lerness_, 1 edition, pages 51-57. Reine Pravis Deutsch.
* Feilke and Rezat (2021) Helmuth Feilke and Sara Rezat. 2021. Textprozeduren und der Erwerb literaler Kompetenz. In Nikolas Koch and Barbara Kozizkowski, editors, _Sprach(en)ververb_, pages 69-79. Der Deutschumtericht.
* Ferretti et al. (2007) Ralph P. Ferretti, Scott Andrews-Weckerly, and William E. Lewis. 2007. Improving the argumentative writing of students with learning disabilities: Descriptive and normative considerations. _Reading & Writing Quarterly_, 23(3):267-285.
* Granger et al. (2009) Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. _The International Corpus of Learmer English_. Presses universitaires de Louvain, Louvain-la-Neuve.
* Gurcke et al. (2021) Timon Gurcke, Milad Alshomyary, and Henning Wachsmuth. 2021. Assessing the sufficiency of arguments through conclusion generation. In _Proceedings of the 8th Workshop on Argument Mining_, pages 67-77, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Gate et al. (2012) Olaf Gate, Sara Rezat, and Torsten Steinhoff. 2012. Positionierung. Zur Entwicklung des Gebrauchs modalisierennder Prozeduren in argumentativen Texten von Schulern und Studenten. In Helmuth Feilke and Katrin Lehnen, editors, _Textroutinen. Theorie, Errorp und didaktisch-medialie Modellierung_, pages 125-153. Lang, Frankfurt/Main.
* He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTA using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In _The Eleventh International Conference on Learning Representations_

Andrea Horbach, Dirk Scholten-Akoun, Yuning Ding, and Torsten Zesch. 2017. Fine-grained essay scoring of a complex writing task for native speakers. In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_, pages 357-366, Copenhagen, Denmark. Association for Computational Linguistics.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 2790-2799. PMLR.
* Ka-kan-dee and Kaur (2014) Maleerat Ka-kan-dee and Sarjit Kaur. 2014. Argumentative writing difficulties of thai english major students. In _The 2014 WEI International Academic Conference Proceedings_, pages 193-207.
* Ke and Ng (2019) Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_, pages 6300-6308. International Joint Conferences on Artificial Intelligence Organization.
* Kellogg et al. (2010) Ronald T. Kellogg, Alison P. Whiteford, and Thomas Quinlan. 2010. Does automated feedback help students learn to write? _Journal of Educational Computing Research_, 42(2):173-196.
* Kruse et al. (2012) Norbert Kruse, Anke Reichardt, Maik Herrmann, Friederike Heinzel, and Frank Lipowsky. 2012. Zur qualitat von kindertexten. entwicklung eines bewertungsinstruments in der grundschule. _Didaktik Deutsch : Halbjahresschrift fur die Didadatik der deutschen Sprache und Literatur_, 17(32):87-110.
* Li et al. (2018) Xia Li, Minping Chen, Jianyun Nie, Zhenxing Liu, Ziheng Feng, and Yingdan Cai. 2018. Coherence-based automated essay scoring using self-attention. In _Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data_, pages 386-397, Cham. Springer International Publishing.
* Liu et al. (2023) Zhexiong Liu, Diane Litman, Elaine Wang, Lindsay Matsumura, and Richard Correnti. 2023. Predicting the quality of revisions in argumentative writing. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_, pages 275-287, Toronto, Canada. Association for Computational Linguistics.
* Nguyen and Litman (2018) Huy Nguyen and Diane Litman. 2018. Argument mining for improving the automated scoring of persuasive essays. _Proceedings of the AAAI Conference on Artificial Intelligence_, 32(1).
* Peloghitis (2017) John Peloghitis. 2017. Difficulties and strategies in argumentative writing: A qualitative analysis. _Transformation in language education. JALT_.
* Persinz et al. (2010) Isaac Persinz, Alan Davis, and Vincent Ng. 2010. Modeling organization in student essays. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 229-239, Cambridge, MA. Association for Computational Linguistics.
* Persing and Ng (2013) Isaac Persing and Vincent Ng. 2013. Modeling thesis clarity in student essays. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 260-269, Sofia, Bulgaria. Association for Computational Linguistics.
* Persing and Ng (2014) Isaac Persing and Vincent Ng. 2014. Modeling prompt adherence in student essays. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1534-1543, Baltimore, Maryland. Association for Computational Linguistics.
* Persing and Ng (2015) Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 543-552, Beijing, China. Association for Computational Linguistics.
* Pfeiffer et al. (2021) Jonas Pfeiffer, Aishway Kamath, Andreas Ruckle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 487-503, Online. Association for Computational Linguistics.
* Pfeiffer et al. (2020) Jonas Pfeiffer, Andreas Ruckle, Clifton Poth, Aishwayam Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 46-54, Online. Association for Computational Linguistics.
* Rahimi et al. (2015) Zahra Rahimi, Diane Litman, Elaine Wang, and Richard Correnti. 2015. Incorporating coherence of topics as a criterion in automatic response-to-text assessment of the organization of writing. In _Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 20-30, Denver, Colorado. Association for Computational Linguistics.
* Rahimi et al. (2014) Zahra Rahimi, Diane J. Litman, Richard Correnti, Lindsay Clare Matsumura, Elaine Wang, and Zahid Kisa. 2014. Automatic scoring of an analytical response-to-text assessment. In _Intelligent Tutoring Systems_, pages 601-610, Cham. Springer International Publishing.
* Rezat (2011) Sara Rezat. 2011. Schriffliches Argumentieren. Zur Ontogenese konzessiver Argumentationskompetenz.

 

Didakik Deutsch: Halbjahresschrift fur die Didakhidkider deutschen Sprache und Literatur_, 16(31):50-67.
* Rezat [2018] Sara Rezat. 2018. Argumentative Textprozeduren als Instrumente zur Anabahnung wissenschaftlicher Texttompetenz. In Sabine Schmolzer-Eibinger, Bora Bushati, Christopher Ebner, and Lisa Niederdorfer, editors, _Wissenschafthilches Schreihen lehren und lernen. Diagnose und Forderung wissenschafilicher Texttompetentz in Schulte und Universitat_, pages 125-146. Waxmann, Munster.
* Schroter [2021] Juliane Schroter. 2021. _Linguistiches Argumentationsanalyse_. Kurze Einfuhrungen in die germanistische Lingustik. Universitatsverlag Winter, Heidelberg.
* Skitalinskaya et al. [2023] Gabriella Skitalinskaya, Maximilian Spliethover, and Henning Wachsmuth. 2023. Claim optimization in computational argumentation. In _Proceedings of the 16th International Natural Language Generation Conference_, pages 134-152, Prague, Czechia. Association for Computational Linguistics.
* Sklatlinskaya and Wachsmuth [2023] Gabriella Skitalinskaya and Henning Wachsmuth. 2023. To revise or not to revise: Learning to detect improvable claims for argumentative writing support. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15799-15816, Toronto, Canada. Association for Computational Linguistics.
* Smith [2003] Carlota S. Smith. 2003. _Modes of Discourse: The Local Structure of Texts_. Cambridge Studies in Linguistics. Cambridge University Press.
* Stab [2017] Christian Stab. 2017. _Argumentative Writing Support by means of Natural Language Processing_. Ph.D. thesis, Technische Universitat Darmstadt, Darmstadt.
* Stab and Gurevych [2017a] Christian Stab and Iryna Gurevych. 2017a. Parsing argumentation structures in persuasive essays. _Computational Linguistics_, 43(3):619-659.
* Stab and Gurevych [2017b] Christian Stab and Iryna Gurevych. 2017b. Recognizing insufficiently supported arguments in argumentative essays. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 980-990, Valencia, Spain. Association for Computational Linguistics.
* Stahl et al. [2023] Maja Stahl, Nick Dusterhus, Mei-Hua Chen, and Henning Wachsmuth. 2023. Mind the gap: Automated corpus creation for enthymeme detection and reconstruction in learner arguments. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 4703-4717, Singapore. Association for Computational Linguistics.
* Stede and Schneider [2019] Manfred Stede and Jodi Schneider. 2019. _Argumentation Mining_. Springer International Publishing.
* Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Townsend et al. [1993] Michael A. R. Townsend, Lynley Hicks, Jacquilyn D. M. Thompson, Keri M. Wilton, Bryan F. Tuck, and Dennis W. Moore. 1993. Effects of introductions and conclusions in assessment of student essays. _Journal of Educational Psychology_, 85(4):670-678.
* Uto et al. [2020] Masaki Utto, Yukuan Xie, and Maomi Ueno. 2020. Neural automated essay scoring incorporating handcrafted features. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 6077-6088, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Wachsmuth et al. [2016] Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2016. Using argument mining to assess the argumentation quality of essays. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 1680-1691, Osaka, Japan. The COLING 2016 Organizing Committee.
* Walton et al. [2008] Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. _Argumentation Schemes_. Cambridge University Press.
* Wambsganss et al. [2022a] Thiemo Wambgsanss, Andrew Caines, and Paula Buttery. 2022a. ALEN app: Argumentative writing support to foster English language learning. In _Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)_, pages 134-140, Seattle, Washington. Association for Computational Linguistics.
* Wambsganss et al. [2022b] Thiemo Wambsganss, Andreas Janson, and Jan Marco Leimeister. 2022b. Enhancing argumentative writing with automated feedback and social comparison nudging. _Computers and Education_, 191:104644.
* Wambsganss and Niklaus [2022] Thiemo Wambsganss and Christina Niklaus. 2022. Modeling persuasive discourse to adaptively support students' argumentative writing. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8748-8760, Dublin, Ireland. Association for Computational Linguistics.
* Wambsganss et al. [2020] Thiemo Wambsganss, Christina Niklaus, Matthias Sommer, Siegfried Handschuh, and Jan Marco Leimeister. 2020. A corpus for argumentative writing support in German. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 856-869, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Wambsganss and Rietsche [2019] Thiemo Wambsganss and Roman Rietsche. 2019. Towards designing an adaptive argumentation learning tool. In _International Conference on Interaction Sciences_.
* Wang et al. [2023] Cong Wang, Zhiwei Jiang, Yafeng Yin, Zifeng Cheng, Shiping Ge, and Qing Gu. 2023. Aggregating multiple heuristic signals as supervision for unsupervised automated essay scoring. In _Proceedings of the 61st Annual Meeting of the Association for Computational 

Linguistics (Volume 1: Long Papers)_, pages 13999-14013, Toronto, Canada. Association for Computational Linguistics.
* Weber et al. (2023) Florian Weber, Thiemo Wambsganss, Seyed Parsa Neishaei, and Matthias Soellner. 2023. Structured persuasive writing support in legal education: A model and tool for German legal case solutions. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2296-2313, Toronto, Canada. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remin Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Yang et al. (2020) Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1560-1569, Online. Association for Computational Linguistics.
* Zhu (2001) Wei Zhu. 2001. Performing argumentative writing in english: Difficulties, processes, and strategies. _TESL Canada Journal_, 19(1):34-50.

## Appendix A Cooccurrence Matrix

The cooccurrence matrix between all argumentative structure levels (discourse functions, arguments, components, and discourse modes) is shown in Figure 5.

## Appendix B AdapterFusion Activations

Similar to Pfeiffer et al. (2021), we visualize the activations of our model _mDeBERTaV3-fusion-w/all_ per layer in Figure 6 to further investigate the influence of each level of argumentative structure on the quality scoring. The first activation layers show for all five quality aspects that all structure adapters are activated quite diversely. In contrast, the later layers have a clear tendency towards activating only one or two adapters. Notable is the similar activation pattern between relevance and overall quality, which could come from their value correlation 



## 6 Conclusion

Figure 5: Relative token-level overlap of all argumentative structure labels, seperated into the four levels of granularity. For example, 68% of all tokens labeled as _Introduction_ are also labeled _Topic_.

Figure 6: AdapterFusion activation per layer [1–12] and on average over the layers (_Avg_) for each _mDeBERTaV3-fusion-w/all_ model per quality aspect. We average the activation for each fused adapter (for discourse functions, arguments, components, or discourse modes) over all instances in the test set of the most representative folding.