Mind the Gap: Automated Corpus Creation for Enthymeme Detection and Reconstruction in Learner Arguments
Authors: Maja Stahl
Nick Düsterhus
Mei-Hua Chen
Henning Wachsmuth
Venue: EMNLP 2023 Findings (2023)
Keywords: dataset
argumentation
argument mining
enthymeme

Abstract: Writing strong arguments can be challenging for learners. It requires to select and arrange multiple argumentative discourse units (ADUs) in a logical and coherent way as well as to decide which ADUs to leave implicit, so called enthymemes. However, when important ADUs are missing, readers might not be able to follow the reasoning or understand the argument's main point. This paper introduces two new tasks for learner arguments: to identify gaps in arguments (enthymeme detection) and to fill such gaps (enthymeme reconstruction). Approaches to both tasks may help learners improve their argument quality. We study how corpora for these tasks can be created automatically by deleting ADUs from an argumentative text that are central to the argument and its quality, while maintaining the text's naturalness. Based on the ICLEv3 corpus of argumentative learner essays, we create 40,089 argument instances for enthymeme detection and reconstruction. Through manual studies, we provide evidence that the proposed corpus creation process leads to the desired quality reduction, and results in arguments that are similarly natural to those written by learners. Finally, first baseline approaches to enthymeme detection and reconstruction demonstrate the corpus' usefulness.

Mind the Gap: Automated Corpus Creation for Enthymeme Detection and Reconstruction in Learner Arguments
Maja Stahl
Nick Düsterhus
Leibniz University Hannover m. stahl@ai. uni moi-hannover.do de
Paderborn University
nduester@mail. upb. de
Mei-Hua Chen
Henning Wachsmuth
Tunghai University
Leibniz University Hannover h. wachsmuth@ai. uni -hannover. de
mhchen@thu. edu. tw
Abstract
Essay Topic: Discuss the advantages and disadvantages of banning smoking in restaurants.
Writing strong arguments can be challenging for learners. It requires to select and arrange multiple argumentative discourse units (ADUs) in a logical and coherent way as well to de- cide which ADUs to leave implicit, so called enthymemes. However, when important ADUs missing, readers might not be able fol- low the reasoning or understand the argument's main point. This paper introduces two new tasks for learner arguments: to identify gaps in arguments (enthymeme detection) and to fill such gaps (enthymeme reconstruction). Ap- proaches to both tasks may help learners im- prove their argument quality. We study how corpora for these tasks can be created automati- cally by deleting ADUs from an argumentative text that are central the argument and its qual- ity, while maintaining the text's naturalness. Based on the ICLEv3 corpus of argumentative learner essays, we create 40,089 argument in- stances for enthymeme detection and recon- struction. Through manual studies, we provide evidence that the proposed corpus creation pro- cess leads to the desired quality reduction, and results in arguments that are similarly natural to those written by learners. Finally, first baseline approaches to enthymeme detection and recon- struction demonstrate the corpus' usefulness.
To smoke used to he considered a fancy act, although nowadays the society has different point of view, smoking is wrong. [MASK] That is the reason why the goverument of many countries is banning smoking in private places, including restaurants. There are advantages and disadvantages of this particular prohibition.
Enthymematic Argument:
Enthymeme: Smoking is not healthy to anyone.
Table 1:
An enthymematic learner argument created automatically by our approach. The position of the re- moved ADU in the original learner argument is marked by [MASK]. The original learner argument was taken from the ICLEv3 corpus (Granger et al., 2020).
2006). Similarly, the quality of an argument will also often be limited if the conclusion is missing, as this easily renders unclear what the argument claims (Alshomary et al., 2020b).
Studies show that learners often fail to provide solid evidence for their claims (Ka-kan-dee and Kaur, 2014) as well as to compose their arguments in a coherent way (Dang et al., 2020). NLP may help learners improve the quality of their arguments by providing feedback on the arguments' logical quality. For example, the feedback could point enthymematic gaps in arguments and make sugges- tions on how to fill these gaps. Until now, research on assessing logical quality dimensions learner argumentation has focused on the amount of evi- dence an essay provides (Rahimi et al., 2014), and the sufficiency of its arguments, i.e., whether the premises together give enough reason to accept the conclusion (Stab and Gurevych, 2017b; Gur- cke al., 2021). To the best of our knowledge, no approach explicitly studies the adequacy of en- thymemes in learner arguments yet.
1 Introduction
Argumentative writing is an essential skill that can be challenging acquire (Ozfidan and Mitchell, 2020; Dang et al., 2020; Saputra et al., 2021). To write a logically strong or cogent argument, a writer has to present sufficient evidence (i.e., the argument'ss ' premises) that justifies the argument's s claim (i.e., its conclusion). In addition, arguments draw premises that are left implicit, so called enthymemes (Feng and Hirst, 2011), 1), which may be key to match the expectations of the audience (Habernal et al., 2018). However, if relevant premise is missing, the argument may be insuffi- cient to accept its conclusion (Johnson and Blair,
To initiate research in this direction, we intro- duce two new tasks on learner arguments: en- thymeme detection and enthymeme reconstruction. We define an enthymeme broadly here as any miss-

---

ing argumentative discourse unit (ADU) that would complete the logic of written argument. The goal of the first task is to assess whether there is a pos- sibly problematic enthymematic gap at a specified position of an argument. Given an argument with such a gap, the goal of the second task is then to generate new ADU that fills the gap. Approaches to both tasks may help learners improve the qual- ity of their arguments by pointing to locations of enthymemes and making suggestions on how to reconstruct the enthymemes if desired.
An automated approach for creating corpora of enthymematic arguments in learner essays
A corpus for studying two new NLP tasks on learner arguments: enthymeme detection and enthymeme reconstruction
Baseline approaches detecting and recon- structing enthymemes computationally
Related Work
Originally, Aristotle (ca. 350 BCE / translated 2007) referred enthymeme to a specific presump- tive argument scheme in which certain premises are left unstated intentionally. Nowadays, the term is used both for arguments with one or more missing or implicit premises (Walton al., 2008) and for the implicit premises themselves (Feng and Hirst, 201 I). We adopt the latter use here and also cover implicit conclusions, which are often left unstated for rhetorical reasons (Al Khatib et al., 2016).
Data for studying enthymemes in learner argu- ments is neither available so far, nor is it straightfor- ward write respective arguments in a natural way. Therefore, we study how to automatically create a corpus for the two tasks from existing argument corpora. In particular, we propose a self-supervised process that carefully removes ADUs from high- quality arguments while maintaining the text's nat- uralness. We rank candidate ADUs for removal by both their contribution to argument's quality and their centrality within argument, combining es- say scoring methods (Wachsmuth al., 2016) with a PageRank approach (Page al., 1999). Further- more, we ensure the naturalness of the remaining argument (with an enthymematic gap) by utilizing BERT's capabilities to predict whether a sentence follows another sentence (Devlin et al., 2019). We create enthymemes only that cannot be detected by a BERT model. Table 1 shows example of an enthymeme created by our approach.
The detection of enthymemes has been studied in empirical analyses (Boltužić and Snajder, 201 60 and with computational methods (Rajendran et al., 2016; Saadat- Yazdi et al., 2023). Also, their recon- struction with generation methods has come into focus recently (Alshomary et al., 2020b; Oluwa- toyin et al., 2020; Chakrabarty et al., 2021). For specific type of enthymemes, namely warrants which explain the connection between premises and conclusions, Habernal et al. (2018) created a dataset with 1,970 pairs of correct and incorrect in- stances to be used for comprehending debate portal arguments. In contrast, we target learner arguments in this work, particularly to enable research on the question how to identify inadequate enthymemes, that is, those that should not be left implicit.
Based on the third version of the ICLE corpus of urgumentative learner essays (Granger et al., 2020), we automatically create 40,089 argument instances for enthymeme detection well as 17,762 argu- ment instances for enthymeme reconstruction. An evaluation of the created corpus suggests that our enthymeme creation approach leads to the desired quality reduction in arguments. In addition, manual annotators of the argument's naturalness deemed our modified arguments similarly natural to the original ones hand-written by the learners. Finally, we develop baseline approaches to enthymeme de- tection and enthymeme reconstruction based on DeBERTa (He et al., 2020) and BART (Lewis et al., 2020), respectively. Experimental results provide evidence for the usefulness of the corpus for study- ing the two presented tasks.
Argumentative writing poses many challenges for learners, particularly in the organization and development of arguments (Dang et al., 2020; Oz- fidan and Mitchell, 2020). Learners commonly struggle with unclear, irrelevant, or missing thesis statements (Ozfidan and Mitchell, 2020) and face difficulties providing substantial evidence as argu- ment premises to support their opinions (Ka-kan- dee and Kaur, 2014; Ozfidan and Mitchell, 2020). In some cases, learners address the claims but fail to adequately support them by omitting certain rea- sons (Ozfidan and Mitchell, 2020). Formulating clear conclusions. also known the claims of arguments, is another area where learners often falter (Lee and Deakin, 2016; Skitalinskaya and Wachsmuth, 2023). These findings indicate that
Altogether, this paper's 's main contributions are:
'The code to reproduce the data and results can be found under: https: / //t/ thub. cwm/ httn/webis-de de/EMNLP-23

---

inadequate enthymemes are problem in learner arguments, which, so far, has been extensively explored using computational methods.
Corpus
Argument mining, the automated process of ex- tracting argumentative discourse units (ADUs) and their relationships, has demonstrated its effective- ness in analyzing argumentative essays written by learners (Stab and Gurevych, 2014, 2017a). We do not study argument mining, but we leverage it to filter candidate enthymemes in our corpus creation approach. In line with Chen et al. (2022), we con- sider the ADU types premise, claim, major claim, and aronenggumetttive.
(a)
candidate arguments
Essay quality filtering
Argument length filtering
(b)
Find candidate enthymemes
Moreover, we employ argument quality assess- ment in the corpus creation process (Wachsmuth et al., 2016). Essay argumentation has been eval- uated across various quality dimensions such as an essay's organization (Persing et al., 2010), the clarity of its thesis (Persing and Ng, 2013), the strength of its arguments altogether (Persing and Ng. 2015), and the sufficiency of each individual argument (Stab and Gurevych, 2017b). This has also facilitated the development approaches to generate missing argumentative ADUs, specifically argument conclusions (Gureke et al., 2021). a related line of research, approaches have been pro- posed to suggest specific revisions of argumentative essays (Afrin and Litman, 2018), to assess the qual- ity of revisions (Liu et al., 2023), and to perform argument revisions computationally (Skitalinskaya al., 2023). Our proposed enthymeme-related tasks complement these attempts, having the same ultimate goal, namely to learn support systems for argumentative writing (Wambsganss et al., 2020). In conclusion, the computational study of en- thymemes in learner arguments is fairly unex- plored research area, and the automatic detection and reconstruction of inadequate enthymemes in this domain have not been addressed as of now. Developing corpora specifically designed to ana- lyze inadequate enthymemes learner arguments may provide valuable insights into common pitfalls and enable the development of effective methods for helping learners to improve their argumentative writing skills in this regard.
Naturalness
Argumentativeness ADU filtering
ADU filtering
(c)
Rank enthymemes
Enthymeme ranking
Quality reduction ranking 2
Essays with enthymemes
Figure 1: Proposed corpus creation process: (a) Argu- ments of a defined minimum length from high-quality essays are candidates enthymeme creation. (b) gumentative ADUs whose removal maintains natural- ness are candidates for enthymemes. (c) The candidate ADUs are ranked by centrality and quality reduction determine the final enthymeme.
ing argumentative discourse units (ADUs) from high-quality arguments in these texts. In the fol- lowing, we detail the three parts that constitute our approach: (a) finding candidate arguments, (b) find- ing candidate enthymemes these arguments, and (c) ranking the candidates to remove the most suit- able one per argument. Figure illustrates the corpus creation process.
3.1 Finding Candidate Arguments
3 Automatic Corpus Creation
We consider only arguments (i) from essays of high quality that (ii) have specified minimum length. With these filtering steps, we aim minimize the risk already existing, unknown enthymemes within an argument.
This section presents our approach automatic corpus creation. Given base corpus of argumen- tative learner texts as input, we propose to create enthymematic arguments automatically by remov-

---

Organization
Clarity
Strength
only the essays with an overall score above 3.0, they should be of decent quality.
Approach
mae
isse
mae
mis
mae
mise
Our approach
.300
.165
.488
.395
.300
.217
Argument Length Filtering As a simple heuris- tic, we expect essay paragraphs with at least four sentences to be arguments, since we observed that shorter paragraphs are unlikely to be substantial enough to contain a full argument. Furthermore, removing ADU from argument with three or fewer sentences may likely corrupt the argument beyond creating an inadequate enthymeme."
Persing et al.
.323
.175
483
.369
.392
.244
Wachsmuth et al.
.314
.164
5501
.25
.378
.226
Table 2: Performance of our essay scoring models for organization, thesis clarity and argument strength in terms of mean absolute error (mac) and mean squared error (mse). Lower values better. Our models close to or better than the state-of-the-an approaches.
3.2 Finding Candidate Enthymemes
Essay Quality Filtering We model three essay- level quality dimensions that have been well stud- ied previous work: organization, thesis clarity, and argument strength (Persing et al., 2010; Pers- ing and Ng. 2013, 2015; Wachsmuth et al., 6; Chen ct al., 2022). To assess these dimensions for the base essays, we train one LightGBM regres- sion model (Ke et al., 2017) per quality dimension, using a combination of new features and reimple- mented features from Wachsmuth et al. (2016):2
From the candidate arguments, we select ADUs as candidate enthymemes in two ways: (i) We en- sure that removing the ADU does not destroy the argument's naturalness, that is, the arguments still read like they were written by human; and (ii) we require an ADU to be argumentative in order be selected as a candidate enthymeme. Otherwise, it might not reduce the logical quality of an argument.
Naturalness ADU Filtering We aim to create enthymemes that are likely to appear in human- written arguments. To prevent the remaining argu- check whether
ADU n-grams, with n E (1,2,3}
Sentiment flow (Wachsmuth et al., 2015)
ment from being unnatural, candidate enthymeme can be detected by BERT's next sentence prediction (Devlin et al., 2019): We retain only those candidates for which BERT model predicts that the sentences before and after it could be neighbors, that is, the enthymeme does not disrupt the text's 's naturalness. The text naturalness is evaluated manually in Section 5.2.
Discourse function flow (Persing et al., 2010)
Prompt similarity flow
Token and POS n-grams
Length statistics
Frequency of linguistic errors (LanguageTool)
Distribution of named entity types
Distribution of metadiscourse type markers (Hyland, 2018)
A Aggumentativeness ADU Filtering Removing non-argumentative ADU from an argument does not necessarily create an inadequate enthymeme. Thus, we predict the ADU type of argument sen- tences adopting the approach of Chen et al. (2022), which distinguishes between premise, claim, major claim, and arn-anggueettevve. Additionally, we consider only ADUs with more than five tokens significant enough argument to create an in- adequate enthymeme based manual inspection.
We performed ablation tests to determine the best feature combination per quality dimension in terms of mean squared error (mse). Details on all features and the best combinations can be found in Appendix A. Table 2 shows the performance of essay scoring models with the best-performing feature combinations. Our models on par with the above-mee mentioned state of the art approaches, even being best in predicting argument strength. For filtering, we average the three scores per essay to an overall score between 1.0 (very bad) to 4.0 (very good) (Persing and Ng, 2015) and keep
3.3 Ranking Candidate Enthymemes
To find the most suitable candidate enthymemes, we use a PageRank- approach (Page et al., 1999) that ranks ADUs according to two criteria:
'Initial experiments on predicting the essay quality dimen- sions using Longformer (Beltagy et al., 2020) led to lower per- formance compared to the Scature-based approaches, prompt- ing us to stop further investigations in this direction.
We also only create enthymematic arguments of at most 500 tokens so that together with added special tokens, they fit the maximum input length of most common language models.

---

(i) their centrality within an essay and (ii) their con- tribution to the essay's quality. We hypothesize that this enables us to choose high-quality ADUs for re- moval that are critical to an argument and therefore leave a quality-reducing. inadequate enthymeme. We take the whole essay into consideration to have much context possible for estimating the cen- trality and quality contribution of the ADUs. This way, we also reuse our essay scoring models. After ranking, we determine the highest-ranked candidate enthymeme on argument level.
U/j :- cj. To align the value range of the quality scores with the centrality scores, we use min-max scaling before applying the softmax function:
B¹ min (B)) (3) Bi softmar max (B:) min B))
PageRank-based Ranking To rank the tences by both criteria, their centrality and their quality contribution, we combine the matrixes A and B into one transition matrix M:
M = 0.5 A 0.5
(4)
Enthymeme Centrality Ranking Previous work has modeled document as a graph where each node corresponds to sentence, and the weights are derived from sentence similarity. This allows esti- mating the centrality of the sentences by calculating the sentence rank (Erkan and Radev, 2004; Mihal- cea and Tarau, 2004; Alshomary al., 2020a). We adapt this idea by constructing a graph that repre- sents an essay's ' sentences and title by encoding each sentence 8; as a vector using sBERT em- beddings (Reimers and Gurevych, 2019). From this, construct a similarity matrix A' with
The final ranking is obtained by applying the power iteration method. Accordingly, the sentences with the highest rank are expected to be those that bal- ance high centrality within the essay and great con- tribution to the quality of the essay.
Corpus
This section summarizes the corpora that we use for model training and for automatic corpus creation. Then, we give details about the composition and statistics of the resulting created corpus.
de cos(s;,
S Sj
(I)
4.1
Base Corpora
st/l
||,
To train essay scoring models, we needed essay quality data. For this, reused the 1,003 argu- mentative learner essays rated for organization by (Persing et al., 2010), 830 essays rated for thesis clarity (Persing and Ng, 2013), and 1,000 essays rated for argument strength (Persing and Ng, 2015). These essays come from the second version of the International Corpus of Learner English, ICLEv2 (Granger et al., 2009).
Since cosine similarity lies in the interval [-1, 1]. we apply softmax on each row A₁ to obtain a stochastic matrix A, with A; == softmar(A.)
Quality Reduction Ranking We aim to remove sentences that contribute to the quality of ar- gument in the sense that the created enthymemes harm the quality. Since there is, to our knowledge, no research on the quality contributions of individ- ual sentences the context an essay, we esti- mate by removing the sentences and observing the change in essay scores. In more detail, given the essay score f(1.....x] of an essay with n sen- tences, we estimate the quality contribution of the i-th sentence to be:
The basis for automatically corpus is the third version of the same corpus, ICLEv3 (Granger al., 2020). contains 8,965 argumen- tative essays written by learners with 25 different language backgrounds.
4.2 Corpus Composition and Statistics
G [[ I(1..... 1,11 m....
(2)
Each filtering step described above resulted in the number of essays, arguments, and candidate en- thymemes shown in Table 3. The steps applied fil- ters to essays, arguments, or ADUs directly, which mostly resulted in indirect effects other levels.
We use our essay scoring models to determine the quality scores for each candidate enthymeme. For the other sentences, we the average quality score of all sentences in the essay.
With probability of 80%, we removed the highest-ranked ADU of an argument with least one candidate enthymeme create two instances: (a) One positive example with a separator token marking the correct enthymeme position and (b)
As for the sentence centrality, we construct quality graph, represented as matrix B', where each sentence s; is linked to each other sentence sj with the weight being the quality contribution of 8j. i.e.

---

Data
Essays # Arguments # Candidates
Our Corpus
Random Corpus
Base Corpus, ICLEv3
8,965
63,811
289,779
3k
After filtering by
essay quality
5,541
36,835
176,580
argument length
5,464
22,526
146,567
2k
naturalness 5,464
22,525
137,855
argumentativeness 5,464
22,331
110,702
1k-
Table Results of the four filtering steps of our corpus creation approach to the ICLEv3 corpus: Number of essays, arguments candidate ADUs for enthymeme creation after each filtering step.
Position
0
5 10
15
20
Figure 2: Histogram of the chosen enthymeme positions our corpus and in baseline corpus with enthymemes at random positions. Our approach more often chooses the first ADU to create the enthymeme, but the rest of the distribution is comparable the random corpus.
one negative example with the separator token at a random, incorrect position. In the remaining cases, the original argument without deletion and with a separator token at random position is added as negative example improve model robustness. This results in 17,762 positive and 22.327 negative examples, with 40,089 in total. Within the positive examples, 63.28% of the enthymemes are premise removals, 30.97% claim removals, and 5.75% ma- jor claim removals, according to the ADU type classifiers based on Chen et al. (2022)'s work. We randomly split the corpus into training, validation, and test set using ratio of 7:1:2.
rationally worthy of drawing its conclusion (John- son and Blair, 2006), which captures our intended goal of having only adequate enthymemes well. We relax the notion of sufficiency here also account for missing argument conclusions, and deem an argument sufficient (1) its premises together give enough reason to accept the conclusion and (2) its conclusion is clear. Given this definition, we evaluate the change sufficiency resulting from the created enthymemes, that is, whether it is more, equally, or less sufficient than the original argu- ment. Recall that the goal of our approach is to create insufficient arguments.
To provide insights into the behavior of our ap- proach, Figure visualizes the enthymeme po- sitions in our corpus compared to randomly- generated corpus created by removing random argu- mentative ADUs with more than five tokens (argu- mentativeness ADU filtering only). The distribution of enthymeme positions shows slight preference for the first ADU, which often introduces the topic and stance and is central to the argument. The po- sitional bias can largely be attributed to differences in argument length.
For evaluation, we randomly chose 200 original arguments from the given corpus and their modified versions created with four different methods:
(a) Our full approach
(b) Our approach without centrality ranking (i.c..
M instead of Equation
(c) Our approach without quality reduction rank- ing (i.c., M := A)
5 Corpus Analysis
This section reports on our evaluation of the ef- fectiveness of our automatic approach for creating inadequate enthymemes that are harmful to the quality of arguments while not making their texts seem unnatural. To this end, we manually assessed (a) the induced reduction in logical argument qual- ity in terms of sufficiency and (b) the naturalness of the enthymematic arguments in comparison to the original arguments written by learners.
(d) A baseline that randomly removes any argu- mentative ADU with at least five tokens*
We hired five native or bilingual English speakers via Upwork for this task and paid 13$ per hour of work. The annotation guidelines can be found in Appendix B. To model each annotator's reliability, MACE for combining the non-expert anno- tators' labels into final labels (Hovy et al., 2013). Table 4 shows the distribution of MACE labels.
5.1 Manual Sufficiency Evaluation
The results suggest that all compared approaches
argumentation theory, sufficiency captures whether an argument's 's premises together make
"The haseline equals the modified arguments from the randomly created corpus, (Random Corpus).

---

Sufficiency
Naturalness Scores
Arguments modified by
more
equal
less f
Arguments
2
3
Mean
Full approach
10.0%
29.0%
61.0%
Original Modified
18% (9) 36% (18) 46% (23)
2.28
centrality ranking
13.5%
39.0%
47.5%
20% (10) 38% (19) 42% (21)
2.22
w/o quality reduction ranking
10.5%
36.0%
53.5%
Random baseline
10.5%
36.0%
53.5%
Table 5: Manual naturalness evaluation: Distribution of majority scores and their mean for the original argu- ments written by learners and arguments modified by our approach. Naturalness is largely maintained.
Table Manual sufficiency evaluation: Percentage of arguments that are more/equally/les sufficient after cre- ating enthymemes in arguments with our full approach, variations of our approach, and random argumenta- tive ADU removal. The best value per column is marked in bold. Our approach succeeds in reducing sufficiency.
The results of the manual naturalness evaluation are shown in Table 5. Both original and modified arguments achieve a mean score slightly above par- tial naturalness. This indicates that the arguments written by learners already have some naturalness issues, Additionally, creating enthymematic gaps using our approach only slightly reduces the ar- guments' naturalness. The average pairwise inter- annotator agreement in terms of Kendall's 7 is 0.27.
reduce the arguments' sufficiency in most cases, which is expected from the definition of sufficiency. However, our full approach works best in decreas- ing the original argument's is sufficiency, outperform- ing its two variations and the random baseline. Ad- ditionally, our full approach to enthymeme creation is least likely to cause increase in sufficiency. The average pairwise inter-annotator agreement in terms of Kendall's is 0.25.
Experiments
Our automatically-crcated corpus is meant en- able studying two new tasks on learner arguments: Enthymeme detection and enthymeme reconstruc- tion. This section presents baselines for the tasks order to demonstrate the usefulness of the corpus.
5.2 Manual Naturalness Evaluation
To assess the impact of our naturalness ADU fil- tering (Section 3.2), we manually evaluated the naturalness of the arguments with enthymematic gaps in comparison the original learner argu- ments. For this, three authors of this paper manu- ally inspect random sample of 100 arguments: 50 original learner arguments and 50 arguments mod- ified by our full approach. We scored naturalness on a 3-point scale:
6.1 Enthymeme Detection
We treat enthymeme detection binary classifica- tion task: Given learner argument and a position within the argument, predict whether there an inadequate enthymeme (say, a logical gap) at the given position. We mark the positions in question using a separator token, [SEP]. Furthermore, we use the token type IDs to differentiate between the sequences before and after the separator token.
1. Artificial. part of the argument is unrelated to the preceding or succeeding text. For ex- ample, reference cannot be resolved (cohe- sion), the text abruptly changes its theme or does not have any continuous theme (coher- ence), some definitely required information is missing (clarity).
Models For classification, we fine-tune the lan- guage model DeBERTa (He et al., 2020) with I 134M parameters, 12 layers, and a hidden size of 768 (miccrooof/ddeeerta-baaa provided by Huggingface (Wolf et al., 2020). To quantify its learning success, we also report the results of random classifier that makes random predictions and a majority classifier that always predicts the majority training class.
2. Partly Natural. The text is mainly coherent, its use of cohesive devices does not affect con- tinuity or comprehensibility, and the informa- tion provided suffices to get the author's 's point.
Experimental Setup We tune the training hyper- parameters of DeBERTa in grid-search manner ing Raytune (Liaw et al., 2018). We explored num- bers of epochs in {8, 16, 24} with learning rates in {10 6 6,5- 10 6 10 $) and batch size 16. The
3. Natural. The text is coherent, its use of co- hesive devices correct, and the information provided allows to clearly understand the au- thor's point.

---

Approach
Accuracy
Precision
Recall
Fi-Score
Approach
BERTSc.
ROU.-1
ROU.-2 ROU.-L
Random
0.51
0.48
0.51
0.50
Gurcke et al. (2021)
0.247
21.50
4.62
17.10
Majority
0.53
0.00
0.00
0.00
BART-augmented
0.252
21.83
4.64
17.28
DeRERTa 0.72
0.73
0.64
0.68
Table Enthymeme reconstruction results: Rescaled BERTScore F₁ and BOUGE-1/-2/44 for Gurcke al. (2021) and the extended approach, BART-augmented.
Table 6: Enthymeme detection results: Classification performance of fine-tuned DeBERTa classifier com- pared to random baseline and majority baseline.
Data
Source
best Fi -ccre on the validation set was achieved by training for 24 epochs with a learning rate of 10-5
[no enthymeme position]
DeBERTa
One argument that the students the credit card unwisely.
Human learner
Results Table 6 shows the classification results.
[enthymeme position]
DeBERTa
The learning success of the DeBERTa model sug- gests the possibility of automating the task of en- thymeme detection in learner arguments on our cor- pus. However, further improvements using more advanced approaches are expected and encouraged.
2 Accorling to <R> the students need pay off the debts in short period of time.
BART-augmented
3 They need do part-time-jobs to earn money to settle the debts.
Human learner
[no enthymeme position]
DeBERTa
6.2 Enthymeme Reconstruction
4 The students invest much of time to do part-time jobs therefore waste lot of time on their studies.
Human learner
We treat enthymeme reconstruction as mask- infilling task: Given learner argument and known enthymeme position within the argument, generate the enthymeme. We mark the enthymeme position with special token, [MASK]. For this task, corpus instances without enthymemes are excluded since they lack target text.
[no enthymeme position]
DeBERTa
5 The part-time johs which is lower payment to the students.
Human learner
[no enthymeme position]
DeBERTa
6 The student should manage time ef- fectively to develop a balanced and health life style.
Human learner
[enthymeme position]
DeBERTa
Models For mask-infilling, we fine-tune BART models (Lewis et al., 2020) with 400M parameters, 24 layers, and hidden size of 1024 (facebook/bart- large) provided by Huggingface (Wolf et al., 2020). We compare two different approaches: First, we adapt the approach Gurcke et al. (2021) to ar- gument conclusion generation to our task by pro- viding the model with the argument and the mask token at the enthymeme position as input. To eval- uate the importance of contextual information, the second approach, BART-augmented, extends the approach of Gureke et al. (2021) by prepending the essay topic and the predicted ADU type of the enthymeme the input.5
7 addition, the students need to learn how to manage their financaal affairs in responsible way.
BART-augmented
Table 8: Illustration of the output from detection model (DeBERIa) and our reconstruction model (BART- augmentedy: Application each potential position example learner argument from the ICLEv3 (Granger et al., 2020) about the use of credit cards by students. The result is an argument with seven sentences, of which two are reconstructed enthymemes (42 and #7).
with other hyperparameter values did not lead further improvements.
Experimental Setup Using Huggingface's (Wolf training
Results Table 7 reports the automatic evaluation results for enthymeme reconstruction. We only include test instances for which both models gener- ated text. In line with Gurcke et al. (2021), we re- port the recall-oriented ROUGE scores (Lin, 2004), since see it is worse to omit parts of the tar- get text than to generate text that goes beyond the ground-truth text for this task. The results sug- gest that the additional information on the topic and/or target ADU type can help with enthymeme
et al., 2020) default sequence-to-sequeee parameters and learning rate 2 10 we fine-tune the BART models for five epochs. For this task we did not perform extensive hyperparameter op- timization since the models' performance on the validation set was already comparable to related work (Gurcke et al., 2021) and sample experiments
ADU types are predicted with the classifiers that we trained hased on the work by Chen et al. (2022).

---

reconstruction, although the gain is small on aver- age. Moreover, a manual investigation of 30 argu- ments, as detailed Appendix D, suggested that language models can learn to automatically recon- struct enthymemes in learner arguments using our corpus. We expect more advanced approaches can and should further increase effectiveness.
we see three notable limitations of the research presented in this paper. We discuss each of them in the following.
First, due to the lack of learner argument corpora with annotated inadequate enthymemes, we cannot compare the automatically-create enthymemes to human-created ones. Creating such enthymemes intentionally in arguments is not a straightforward task for humans, which is why we aimed to do this automatically in the first place. However, it should be noted that cannot guarantee that our created enthymemes consistently align with the traditional understanding of enthymemes studied in argumentation theory.
To provide further insights into how our models work on full arguments we applied our enthymeme detection model to each potential position in an ex- ample learner argument from the ICLEv3 (Granger et al., 2020). On positions that were classified as enthymematic, we applied our enthymeme recon- struction model to generate the missing component. The results are displayed in Table 8. Our models identified two enthymeme positions and generated corresponding texts, that make the connection be- tween credit card usage and depts and the implicit claim of the argument towards the topic explicit.
Second, despite applying quality filters to the original learner essays, there is a possibility that the arguments contain unknown inadequate en- thymemes. This can harm the effectiveness of the corpus annotations for both presented tasks.
Lastly, we evaluate our corpus creation approach only on a single corpus of argumentative learner es- says. This particular corpus may not represent the whole diversity of argumentative texts or learner proficiency levels. Consequently, the generalizabil- ity of our findings and the applicability of the pro- posed approaches to wider array of argumentative writing remains be explored.
7 Conclusion
Until now, data for studying enthymemes in learner arguments is neither available nor is it straightfor- ward to write respective arguments in a natural way. Therefore, this paper has studied how to automati- cally create corpora of enthymematic learner argu- ments using self- process that carefully removes ADUs from argumentative texts of high- quality while maintaining the texts' naturalness. To be able to point to possibly problematic gaps in learner arguments and make suggestions on how to fill these gaps, we present the tasks of enthymeme detection and reconstruction for learner arguments that can be studied using the created corpus.
Ethical Considerations
We do not see any apparent risks of the approach we developed being misused for ethically doubtful purposes. However, it is important to acknowledge that automatically-created artificial data could be mistaken for real or authentic data, leading to false understandings, biased conclusions, misguided decision-making. This is not our intended use case for the created corpus, which is meant for method development. If texts from the corpus are directly displayed to learners, we suggest to mark that they have been created automatically.
Our analyses provided evidence that our corpus creation approach leads to the desired reduction in logical argument quality while resulting in ar- guments that are similarly natural to those written by learners. Hence, we conclude that, mostly, in- adequate enthymemes are created in the sense of missing ADUS that would complete the logic of a written argument. Moreover, first baselines demon- strate that large language models can learn to de- tect and fill these enthymematic gaps when trained on our corpus. Such models may help learners im- prove the logical quality of their written arguments.
Acknowledgments
This project has been partially funded by the Ger- man Research Foundation (DFG) within the project ArgSchool, project number 453073654. We would like to thank the participants of our study and the anonymous reviewers for the feedback and their time.
8 Limitations
Aside from the still-improvable performance of the presented baseline models and the limitations of the models that are used as part of our approach,

---

References
Wei-Fan Chen, Mei-Hua Chen, Garima Mudgal, and Henning Wachsmuth. 2022. Analyzing culture- specific argument structures in learner essays. In Proceedings of the 9th Workshop on Argument Min- ing, pages 51-61, Online and in Gyeongju, Republic of Korea. International Conference on Computational Linguisties.
Tazin Afrin and Diane Litman. 2018. Annotation and classification of sentence-level revision improvement. In Proceedings of the Thirteenth Workshop In- novative Use of NLP for Building Educational Ap- plications, pages 240-246, New Orleans, Louisiana. Association for Computational Linguistics.
Thi Hanh Dang. Thanh Hai Chau, and To Quyen Tra. 2020. A study on the difficulties in writing argumen- tative essays of english-majored sophomores at tay do university, vietnam. Eumpean Journal of English Language Teaching, 6(1).
Khalid Al Khatib, Henning Wachsmuth, Johannes Kiesel, Matthias Hagen, and Benno Stein. 2016. news editorial corpus for mining argumentation strategies. In Proceedings of COLING 2016, the 26th International Conference on Computational Lin- guisticy: Technical Papers, pages 3433-3443. The COLING 2016 Organizing Committee.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Milad Alshomary, Nick Düsterhus, and Henning Wachsmuth, 2020a. Extractive snippet generation for arguments. In Proceedings of the 43rd Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 1969-1972, New York, NY, USA. Association for Computing Machinery.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality salience in text sum- marization. J. Artif. Int. Res., 22(1):457-479.
Milad Alshomary, Shahbaz Syed, Martin Potthast, and Henning Wachsmuth. 2020b. Target inference in ar- gument conclusion generation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4334-4345, Online, Asso- ciation for Computational Linguistics.
Vanessa Wei Feng and Graeme Hirst. 2011. Classifying arguments by scheme. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 987-996, Portland, Oregon, USA. Association for Computational Linguistics.
Aristotle. ca. 350 BCE translated 2007. On Rhetoric: A Theory of Civic Discourse. Oxford Univer- sity Press, Oxford, UK. Translated by George A. Kennedy.
Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. The International Cor- pus of Learner English. Presses universitaires de Louvain, Louvain-la-Neuve.
Francesco Barbieri, Jose Camacho-Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online. Association for Computational Linguistics.
Sylviane Granger, Maïté Dupont, Fanny Meunier, Hu- bert Naets, and Magali Paquot. 2020. The Interna- tional Corpus of Learner English. Version 3. Presses universitaires de Louvain, Louvain-la-Neure
Timon Gurcke, Milad Alshomary, and Henning Wachsmuth. 2021. Assessing the sufficiency of ar- guments through conclusion generation. Proceed- ings of the 8th Workshop on Argument Mining, pages 67-77, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Filip Boltužié and Jan Snajder. 2016. Fill the gap! Analyzing implicit premises between claims from online debates. In Proceedings the Third Workshop on Argument Mining ArrgMiiin20166 pages 124- 133, Berlin, Germany. Association for Computational Linguistics.
Ivan Habernal, Henning Wachsmuth, Tryna Gurevych, and Benno Stein. 2018 The argument reasoning comprehension task: Identification and reconstruc- tion of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1930-1940, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Tuhin Chakrabarty, Aadit Trivedi, and Smaranda Muresan. 2021. Implicit premise generation with discourse-aware commonsense knowledge models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6247-6252 Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding-
enhanced BERT with disentangled attention. CoRR, abs/2006.0365..

---

Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spacy: Industrial- strength natural language processing in python.
Rada Mihalcea and Paul Tarau. 2004, TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain, Asso- ciation for Computational Linguistics.
Dirk Hovy, Taylor Berg- Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 1120-1 130, Atlanta, Georgia. Association for Computational Linguistics.
Enikuomehin Oluwatoyin, Odunowo Adebisi T., and Mustapha Oluwatoyin S. 2020. An algorithm for the reconstruction of enthymemes for effective ma- chine translation. International Journal of Applied Information Systems, 12(31):1-7.
Ken Hyland. 2018. Metadiscourse: Exploring Interac- tion in Writing. Bloomsbury Classies in Linguistics. Bloomsbury Academic, London, England.
Burhan Ozfidan and Connie Mitchell. 2020. Detected difficulties in argumentative writing: The case of culturally and linguisticallly saudi backgrounded stu- dents. Journal of Ethnic and Cultural Studies, 7(2):pp. 15-29.
Ralph H. Johnson and J. Anthony Blair. 2006. Logical sclf-defense.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation rank- ing: Bringing order to the web. Technical Report 1999-66. Stanford InfoLab. Previous Comber==SIDD WP-1999..
Maleerat Ka-kan-dee and Sarjit Kaur. 2014. Argumenta- tive writing difficulties of thai english major students. The 2014 WEI International Academic Conference Proceedings, pages 193-207.
Guolin Ke, Qi Meng, Thomas Finley. Taifeng Wang, Wei Chen, Weidong Ma. Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boost- ing decision tree. In Proceedings of the 31 Interna- tional Conference on Neural Information Processing Systems, NIPS' 17. page 3149-3157, Red Hook, NY. USA. Curran Associates Inc.
Isaac Persing, Alan Davis, and Vincent Ng. 2010. Mod- eling organization in student essays. In Proceedings of the 2010 Conference Empirical Methods in Natural Language Processing, pages 229-239, Cam- bridge, MA. Association for Computational Linguis- tics.
Joseph Lee and Lydia Deakin. 2016. Interactions in I1 and 12 undergraduate student writing: Interactional metadiscourse in successful and luss-successful ar- gumentative essays. Journal of second language writing, 33:21-34.
Isaac Persing and Vincent Ng. 2013. Modeling thesis clarity in student essays. In Proceedings of the S1 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 260- 269, Sofia, Bulgaria. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training BART: Denoising sequence-to-sequee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Isaac Persing and Vincent Ng. 2015. Modeling argu- ment strength in student essays. In Proceedings of the 53rd Annual Meeting of the Association for Com- putational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- une 1: Long Popers), pages 543-552, Beijing, China. Association for Computational Linguisties.
Zahra Rahimi, Diane J. Litman, Richard Correnti, Lind- say Clare Matsumura, Elaine Wang, and Zahid Kisa. 2014. Automatic scoring of an analytical response- to-text assessment, In Intelligent Tutoring Systems, pages 601-610, Cham. Springer International Pub- lishing.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv.1807.05/1..
Chin- Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Pavithra Rajendran, Danushka Bollegala, and Simon Parsons. 2016. Contextual stance classification of opinions: A step towards enthymeme reconstruction in online reviews. Proceedings of the Third Work- shop on Argument Mining (ArgMining2016), pages 31-39, Berlin, Germany, Association for Computa- tional Linguistics.
Zhexiong Liu, Diane Litman, Elaine Wang, Lindsay Matsumura, and Richard Correnti. 2023. Predicting the quality of revisions in argumentative writing. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 275-287, Toronto, Canada. Association for Computational Linguistics.
Nils Reimers and Tryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing

---

and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP- (MMNLP- pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Henning Wachsmuth, Johannes Kiesel, and Benno Stein. 2015. Sentiment flow - A general model of web review argumentation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 601-611, Lisbon, Portugal. Association for Computational Linguisties.
Ameer Saadat- Yazdi, Jeff Z. Pan, and Nadin Kokciyan. 2023. Uncovering implicit inferences for improved relational argument mining. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2484- 2495, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberd- ingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pa- pers, pages 176-187, Valencia, Spain, Association for Computational Linguistics.
Agustinus Bambang Balinga Saputra, Jumariati, and Emma Rosana Febriyanti. 2021. Efi students' prob- lems in writing argumentative essays. In Proceed- ings of the 2nd International Conference Educa- tion, Language, Literature, and Arts (ICELLA 2021).
Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008 Argumentation Schemes. Cam- bridge University Press.
pages 12. Atlantis Press.
Gabriella Skitalinskaya, Maximilian SplicthAfwer and Henning Wachsmuth. 2023. Claim optimization in computational argumentation. In Proceedings of the 16th International Natural Language Genera- tion Conference, pages 13-1122. Prague, Czechia Association for Computational Linguistics.
Thiemo Wambsganss, Christina Niklaus, Matthias Cetto, Matthias Söllner, Siegfried Handschuh, and Jan Marco Leimeister. 2020. AL: An adaptive learn- ing support system for argumentation skills. In Pro- ceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20 page 1-14, New York, NY, USA. Association for Computing Machinery.
Gabriella Skitalinskaya and Henning Wachsmuth. 2023. To revise or not to revise: Learning to detect improv- able claims for argumentative writing support. In Proceedings of the ōl Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 15799-15816, Toronto, Canada. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: tffte--f-theaatt natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Regina L. Smalley, Mary K. Ruetten, and Joann Rishel Kozyrev. 2001. Refining Composition Skills: Rhetoric and grammar. Heinle and Heinle Boston.
Christian Stab and Tryna Gurevych. 2014, Annotating argument components and relations in persuasive es- says. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguis- tics: Technical Papers. pages 1501-1510, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
A Essay Scoring Models
This section gives additional details on the features and the model performance using different combi- nations of features used as input for the essay scor- ing models predicting organization, thesis clarity, and argument strength. We also report the feature combinations that led to the lowest mean squared error (mse) for each quality dimension.
Christian Stab and Iryna Gurevych. 2017a. Parsing argumentation structures in persuasive essays. Com- putational Linguistics, 43(3):619-659,
Christian Stab and Iryna Gurevych. 2017b. Recognizing insufficiently supported arguments in argumentative essays. In Proceedings of the I 15th Conference of the European Chapler of the Association for Compu- lational Linguistics: Volume 1, Long Papers, pages 980-990, Valencia, Spain. Association for Computa- tional Linguistics.
A.1 Details on Features
ADU n-grams: ADU n-grams, with n E (1,2,3); frequencies of ADU type combi- nations and sequences of ADU types within paragraphs.
Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2016. Using argument mining to assess the argumentation quality of essays. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1680-1691, Osaka, Japan. The COLING 2016 Orga- nizing Committee.
Sentiment flow: Sequence of the paragraph's sentiments, using Barbieri et al. (2020)'s sen- timent classifier.

---

Sequences of dis- course functions: introduction, body, and con- clusion (Persing et al., 2010).
Discourse function flow:
B Annotation Guidelines: Sufficiency
B.1 Task Description
In this task, you will be presented with arguments from argumentative learner essays (original argu- ments) and different variations of them (modified arguments). We want to evaluate the argument quality of the variations in relation to the origi- nal argument. The task consists of reading 200 arguments and up to four argument variations each. You are asked to assess for each modified argu- ment whether the quality of the modified argument is higher, equal, or lower than the quality of the original argument. We measure argument quality in terms of sufficiency defined below. The esti- mated time for this task is 3-5 minutes evaluate all variations of one original argument, approxi- mately 15 hours of work.
Max., min. and mean cosine similarity of the paragraph embeddings to the prompt; Similarity flow, i.e., sequence of the paragraph's 's similarities to prompt using sBERT embeddings (Reimers and Gurevych, 2019).
Prompi similarity flow:
Token and POS n-grams: Token n-grams (n E {1, 2, that occur in more tham 1% of the training data; POS m-grams (m c {1,..,5) 5)) that occur in more than 5% of the training data.
Length statistics: Number of paragraphs, sen- tences, tokens; maximum, minimum and aver- age number of sentences per paragraph.
B.2 Data
You are asked to assess the sufficiency the mod- ified arguments in relation to the corresponding original argument. The modified arguments are variations of the original argument in which one of the original sentences is removed. We replaced the removed sentence with "<mask>" to indicate the deletion position. When row has no original argument, the previous original argument is the cor- responding original argument. The same holds for the topic.
Frequency of linguistic errors: Number of linguistic emrors given by LanguageTool.
Distribution of named entiry types: Occur- rences per named entity type using spaCy (Honnibal et al., 2020).
Distribution of metadiscourse type markers: Occurrences per metadiscourse marker cate- gory, as proposed by Hyland (2018).
B.3 Sufficiency Definition
A.2 Comparison of Feature Combinations
An argument is sufficient when (1) the premises to- gether give enough reason to accept the conclusion, (2) and it is clear what the argument's conclusion is.
Table 9 compares the model performances for dif- ferent combinations of input features: all features, all feature combinations in which one feature is left out and the feature combination that performed best in terms of mse for each of the three quality dimensions.
B.4
Annotation Scale
Sufficiency increased: +1
The combinations of features that led the best model performance it terms of mse, reported in Table 9, are:
Sufficiency unchanged: 0
Sufficiency decreased: +1
Organization. Linguistic errors, token and pos n-grams, length statisties, named entity types, prompt similarity flow.
Annotation Guidelines: Naturalness
C.1 Task Description
For this task, you will be asked to evaluate what we defined "naturalness" of paragraph. Therefore you will rate the provided paragraphs on ordi- nal scale from 1-3, A rating of points represents the highest grade and the lowest. To come a conclusion, we would like you to pay attention to the three main criteria of coherence, cohesion,
Thesis Clarity. Linguistic errors, sentiment flow, discourse function flow, named entity types, prompt similarity flow.
Argument Strength. ADU n n-grams, token and pos n-grams, prompt similarity flow.

---

Organization
Thesis Clarity
Argument Strength
Features
mac
mac
mae
mac
mse
All features
.325
.171
.521
.419
.375
.222
w o ADU n-grams
.324
.171
.515
.433
.374
.224
W O sentiment flow
.344
.172
.519
413
.368
.20
w o discourse function flow
.318
.166
.520
.412
.378
.227
prompt similarity flow
.324
.174
.551
.466
.379
.227
W token and pos n-grams
.322
.122
.510
.422
.398
.246
w named entity types
.323
.172
.526
.424
.378
.227
w length statistics
.336
.185
.519
.413
.376
.222
W o linguistic errors
.35
.173
.516
.47
.377
.23
o metadiscourse lype markers
.35
.173
.517
.466
.377
.224
Feature combination with lowest mse per dimension
.320
.165
.498
.395
.370
.217
Table Features essay scoring: Model performances different combinations of input features: All features, all features but one, and the feature combination that performed best in terms of mse for each of the three quality dimensions organization, thesis clarity and argument strength. The best values per column are marked in bold.
and clarity. The paragraph should be awarded points if the topic not abruptly changed, cohesive devices can be resolved and the paragraph does not lack the information we do not expect elsewhere. We award points if we see stronger flaws cohe- sion or the use of cohesive devices or clarity, which are not out of the ordinary for student essays. If we observe strong discontinuity between parts of the paragraph, we award I point. We do not expect the essay written by English learners to be perfectly coherent, cohesive and clear. However, if you notice strong incoherences or cohesive ties that cannot be resolved, the paragraph would be seen as artificial. Intuitively, read the paragraph sentence per sentence, and examine the continu- ity/transition between the sequence of sentences. If you consider sentence not dependent on the pre- ceding text, the text can be seen unnatural and artificial. The evaluation scheme and scoring crite- ria are presented below. Concrete representations of natural or artificial are difficult precisely de- fine, criteria should help the annotators to judge the "naturalness" based on their feeling for the En- glish language and knowledge about the level of student's ss writing.
we would have expected based on the given text cannot be found (clarity)
Partially Natural: The text is largely coher- ent (no strong incoherences), the use of co- hesive devices does not damage the continu- ity/compprhensiiility of the text, and the pro- vided information is enough to get his point across.
3. Natural: The text is at least not incoherent, co- hesive devices are correctly used, and the pro- vided information allows one to understand his argument clearly.
D
Manual Investigation: Enthymeme Reconstruction
As manual investigation of the generated recon- structed enthymemes, we asked three annotators to evaluate 30 arguments along the three dimen- sions clarity, argument strength and coherence on 5-point scales as defined below, with being the best score. The annotators were given arguments with a [MASK] token the enthymeme position as context and three different ADUs for the masked position: (i) the original ADU, (ii) the ADU gen- erated by Gurcke et al. (2021)'s approach, and (ii) the ADU generated by the approach BART- augmented.
C.2 Annotation Scale
1. Artificial: A part of the paragraph (e.g. a sen- tence) seems unrelated to the preceding suc- ceeding text. For example, a reference cannot be resolved. (cohesion). Or the text abruptly changes the overall thematic scheme or does not have an underlying theme. (coherence). Or a substantial amount of information that
D.1 Definitions and Scales
Clarity The argument has a high clarity if "it uses correct and widely unambiguous language as well as it avoids unnecessary complexity and deviation from the issue" (Wachsmuth et al., 2017).

---

Clarity
Argument Strength
Coherence
Approach
Mean
Agreement
Mean
Agreement
Mean
Agreement
Original
3.79
57%
3.64
70%
3.84
90%
Gurcke et al. (2021)
3.17
73%
2.86
60%
2.99
60%
BART-augmented
3.39
40%
3.13
60%
3.47
63%
Table 10: Manual enthymeme reconstruction results: Mean evaluation scores and majority agreement ratios for the original ADUs and ADUs generated by Gurcke et al. 22021)'s approach and the extended approach, BART- augmented. Best values are marked bold.
1.
The addition of the ADU makes the argument less clear. The ADU itself uses incorrect lan- guage and does not align with the topic of the argument.
Coherence In line with (Smalley et al., 2001); define that coherent paragraph or argument
should contain sentences that are arranged logically and have smooth flow.
2. The addition of the ADU makes the argument less clear. The ADU itself is not fully under- standable and only fits partly the topic of the argument.
The addition of the ADU results in an illogical argument, which hard to follow. The topic the ADU does not align with the rest of the argument.
3.
The addition of the ADU does not improve the clarity of the argument. The ADU itself is in terms of language in line with the rest of the argument.
2. The addition of the ADU results in an argu- ment that is less logical than before and the argument is more difficult to follow. The topic of the ADU does not fully fit into the rest of the argument.
4.
The addition of the ADU slightly improves the clarity of the argument. The ADU itself uses mostly correct, unambiguous language and fits the topic.
3.
The addition of the ADU is in line with the flow and logical arrangement of the argument. The ADU fits the underlying topic.
5. The addition of the ADU improves the clarity of the argument. The ADU itself uses correct, unambiguous language and fits the topic.
4.
The addition of the ADU improves the coher- ence of the argument a little. The ADU adds the logical arrangement and makes it easier follow.
Argument Strength Following (Persing and Ng. 2015), we define the strength of an argument by its effectiveness in convincing a majority of a (reason- able) audience.
5.
The addition of the ADU improves the coher- ence of the argument a lot. The ADU adds the logical arrangement and makes it much easier to follow.
1. The addition of the ADU does not fit the argu- ment, it contradicts makes the main stand- point unclear.
D.2 Results
Table 10 shows the results of the manual en- thymeme reconstruction evaluation. As expected, the original ADUs lead to the best average score in all three dimensions, followed by the extended approach BERT-augmented. The generated ADUs reach medium quality in terms of clarity, argument strength and coherence in most cases. Overall, the quality of the generated ADUs seems to be very comparable with the original ADUs written by learners, which suggests that language models can learn automatically reconstruct enthymemes in learner arguments using our corpus.
2. The addition of the ADU weakens the effec- tiveness of the argument.
3. The addition of the ADU leads neither an improvement in strength nor does it make the argument less convincing.
4. The addition the ADU has a positive effect on the effectiveness of the argument.
5. The addition of the ADU strengthens the ar- gument, it becomes more convincing.


Reviews:
[{"review_id": "Hk9qaB6Zba", "reviewer": "Reviewer_7rF4", "summary": "", "questions": "Question A: It being 2023, I'm afraid I have to ask: have you tried any of the publicly available LLMs on your task?\n\nQuestion B: What would be the (per-document) baseline/trained model accuracy if you evaluate the binary classification task at each possible position (i.e. between two sentences) in a document? In other words, how does the positive/negative performance translate to per-document ethymeme identification accuracy?\n\nQuestion 3: Why are the agreement scores so low?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive judgment of the novelty, motivation, and thoroughness of our work.\n\n**Reason to reject #1:** We agree that a manual analysis of applying our model on real arguments can provide further insights and will gladly add it, when we revise the paper. As a first illustration what our models detect and reconstruct, we applied them to the following example argument about about the use of credit cards by students:\n\n- `no enthymeme position`\n- One argument that the students use the credit card unwisely. \n- `enthymeme position, reconstruction:` *According to \\<R\\> the students need to pay off the debts in a short period of time.*\n- They need to do part-time-jobs to earn money to settle the debts. \n- `no enthymeme position`\n- The students invest much of time to do part-time jobs therefore waste a lot of time on their studies. \n- `no enthymeme position`\n- The part-time jobs which is lower payment to the students. \n- `no enthymeme position`\n- The student should manage time effectively to develop a balanced and health life style .\n- `enthymeme position, reconstruction:` *In addition, the students need to learn how to manage their financial affairs in a responsible way.*\n\nOur models identified two enthymeme positions and generated corresponding texts, that makes the connection between credit card usage and depts and the implicit claim of the argument towards the topic explicit. \n\n**Reason to reject #2 + Question B:** The classification task exactly serves to deal with the challenge you raise: Through the formulation we uses, we obtained a balanced dataset with diverse negative instances for training and evaluation. In practice, however, the classifier should be applied to *all* positions. This can lead to multiple enthymeme positions, including the most probable one, which we use for our evaluation. To ensure that this does not negatively effect our results, we will add the performance for including all negative instances to the final version of the paper. Thank you for the suggestion. We expect that the randomness in our training data sufficiently approximates this setting.\n\n**Reason to reject #3 + Question 3:** The agreement scores of the annotators are indeed rather low, as often in crowdsourcing, but similar to related work such as Gurcke et al. (2021). That is why we apply MACE (Hovy et al., 2013) to model each annotator’s confidence in order to obtain more reliable final labels. \n\n**Question A:** As indicated in the paper, the goal of our experiments was to provide baselines rather than to present the find the approach for the task. Hence, we refrained from testing the latest LLMs, but placed value on comparability with previous work that used BART (Gurcke et al., 2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is creative, the task seems novel and the automated construction of the dataset is well motivated, documented and evaluated. Overall, the paper is well written and organised. The task and dataset in itself seems well-rounded.", "reasons_to_reject": "While intrinsically, the paper is coherent, I am not convinced that the proposed proxy-task really represents what the paper claims it represents. \n\nBeyond the human quality judgements, how do we know that applying models trained on these data on learner essays \"in the wild\" will yield the desired improvements? I understand that a full-scale evaluation of the tool as a writing assistant is out of scope of this paper, but I expected to see at least a sampled manual analysis of the trained model on some real ethymeme examples. What would the classification model detect? What would the reconstruction model generate?\n\nFurthermore, I am not entirely sure why the classification task was formulated as it was. Isn't detecting the position of the inappropriate ethymeme part of the challenge? Instead the task is simplified by providing \"True\" and \"Fake\" ethymeme positions and the model essentially only learns to predict whether a sentence is missing or not.\n\nFinally, the agreement scores of the annotators seem rather low, indicating only a slight correlation of the annotators choices.", "questions_to_authors": "Question A: It being 2023, I'm afraid I have to ask: have you tried any of the publicly available LLMs on your task?\n\nQuestion B: What would be the (per-document) baseline/trained model accuracy if you evaluate the binary classification task at each possible position (i.e. between two sentences) in a document? In other words, how does the positive/negative performance translate to per-document ethymeme identification accuracy?\n\nQuestion 3: Why are the agreement scores so low?", "paper_topic_and_main_contributions": "This paper presents an automatically constructed corpus for the task of Ethymeme (i.e. implicit argumentative discourse unit, ADU) detection and reconstruction. The paper proposes a pipeline of automated steps to automatically remove central ADUs from argumentative texts ranked high in quality. The pipeline is validated by manual annotation. The paper conducts baseline experiments on the constructed corpus to investigate the feasibility of automatically detecting and reconstructing the removed sentences.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "I think overall, the paper is very well written and clear."}, "extra_text": {}}, {"review_id": "n3muF5naxc", "reviewer": "Reviewer_432F", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the contribution and practical importance of our work. To the best of our knowledge, we indeed introduce the first enthymeme corpus for learner arguments, which is why a comparison to other learner enthymeme datasets is not possible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and easily understandable. Given the authors' claim is right, this corpus may be the first of his kind and therefore be a legitimate contribution. I am not enough acquainted with this specific topic for any deeper evaluation.", "reasons_to_reject": "The only thing I could mention is a quote from the authors themselves in the limitation section, saying that the work cannot be compared well since there is no other data available that would permit a comparison.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper deals with detection and reconstruction of enthymemes (missing argumentative discourse units), but central is the\nconstruction of a respective corpus that permits the previously mentioned tasks.\n\nThe authors propose a regression model for essay quality filtering, over features such as ADU n-grams, sentiment flow, token as POS n-grams, distribution of NEs and metadiscourse markers. Via ablation studies the authors determined the best set of features.\nThey also take argument length into account.\nNaturalness is ranked by BERTs next sentence prediction.\n\nFor ranking candidate enthymemes, they use a PageRank-based apporach.\n\nAccording to the rank, enthymeme candidates are removed from the data in order to produce training data for detection and reconstruction.\n\nThis data is then used for enthymeme detection, which is treated as a binary classification task.\nThe authors fine-tune deBERTa for the classification task.\n\nEnthymeme reconstruction is treated as a mask-infilling task. This time the authors fine-tune BART for the task.\n\n Main contributions: The authors claim to have produced one of the first (the first?) corpus for systematically exploring enthymeme detection and reconstruction.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uHDoOSrrnU", "reviewer": "Reviewer_P3fJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for pointing out the importance and originality of both the task and our method.\n\n**Reason to reject #1:** We acknowledge that the approach sections are quite dense. We will use the additional space in the camera-ready version to give more details on candidate enthymeme selection (Section 3.2) and ranking (Section 3.3) in more detail. As part of this, we plan to add a table exemplifying what candidates are filtered out (or not), as well as figure visualizing the PageRank idea.\n\n**Reason to reject #2:** Regarding the overall evaluation, we point out that the manual evaluation in Sections 5.1 and 5.2 of our approach and its components measures the quality of our corpus in terms of creating inadequate enthymemes: By asking the annotators to compare the modified arguments to the original ones, we find that they are of less sufficiency but near-identical naturalness. We will include excerpts of the guidelines in Appendices B and C to clarify this point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses an interesting and original task such as enthymeme identification and reconstruction. Further development in these area can be helpful in the automatic analysis of argumentation since enthymemes are frequently used in argumentative dialogues and speeches. Furthermore, with the proposed method for the automatic generation of an enthymeme corpus, it will be possible to ease the creation of additional corpora for addressing these tasks.", "reasons_to_reject": "The paper is quite dense and some aspects are not thoroughly described in the current version of the work. I am missing some additional discussion and analysis in sections 3.2 and 3.3. Furthermore, when relying exclusively on probabilistic models to generate a corpus some risks are assumed as pointed out in lines 227-228. A human evaluation of the whole automatic method is missing, and it would be the only way to determine the real quality of the automatically generated corpus instead of independently evaluating the probabilistic models used to previous work approaches.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In the paper titled \"Mind the Gap: Automated Corpus Creation for Enthymeme Detection and Reconstruction in Learner Arguments\", the authors propose a methodology based on deep learning algorithms and NLP to automatically create a corpus for enthymeme detection and enthymeme reconstruction. Furthermore, they provide initial baselines to both of these tasks. This way, the main contributions of this paper are: the proposed methodology, the corpus of natural language enthymemes, and the reported baselines in these tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed on the importance of this novel resources, automatically constructed, for the enthymeme detection and reconstruction tasks. However, they also highlighted some drawbacks to be addressed (i.e., simplification of the task, lack of deeper discussion on the selection and ranking of the enthymemes). The reviewers appreciated the author rebuttal where some issues have been clarified.
Decision: Accept-Findings