Grammatical Error Correction for Sentence-level Assessment in Language Learning
Year: 2023

Abstract:
The paper presents experiments on using a Grammatical Error Correction (GEC) model to assess the correctness of answers that language learners give to grammar exercises. We empirically check the hypothesis that the GEC model corrects only errors and leaves correct answers unchanged. We perform a test on assessing learner answers in a real but constrained language-learning setup: the learners answer only fill-in-the-blank and multiple-choice exercises. For this purpose, we use ReLCo, a publicly available manually annotated learner dataset in Russian (Katinskaia et al., 2022). In this experiment, we fine-tune a large-scale T5 language model for the GEC task and estimate its performance on the RULEC-GEC dataset (Rozovskaya and Roth, 2019) to compare with top-performing models. We also release an updated version of the RULEC-GEC test set, manually checked by native speakers. Our analysis shows that the GEC model performs reasonably well in detecting erroneous answers to grammar exercises, and potentially can be used in a real learning setting for the best-performing error types. However, it struggles to assess answers which were tagged by human annotators as alternative-correct using the aforementioned hypothesis. This is in large part due to a still low recall in correcting errors, and the fact that the GEC model may modify even correct words—it may generate plausible alternatives, which are hard to evaluate against the gold-standard reference.

heading: 1 Introduction
text: Grammatical error correction (GEC) is the task of automatically detecting and correcting grammatical errors in text. Given the recent advancements in Transformer-based GEC models, which have the ability to suggest fluent and grammatically accurate corrections for input sentences, our focus lies in examining their application in language learning settings. One potential application is to check essays written by learners and provide suggestions for
corrections—this can be a useful tool for secondlanguage (L2) learners to improve their writing. We are interested in incorporating GEC into an intelligent computer-aided language learning (CALL) system, but in a more constrained scenario: our objective is to evaluate whether a GEC model can be used for automatic assessment of the learner’s answers to fill-in-the-blank (“cloze”) and multiplechoice (MC) grammar exercises. We assume that this task is comparatively easier than correcting free-text essays, since the number of possible errors in each input sentence is constrained by the number of exercises, these exercises do not change the word order, and our focus is only on grammar. We empirically test the hypothesis: the GEC model can be employed to assess the grammatical correctness of learner answers to grammar exercises, because in an input sentence containing learner answers, the GEC model will fix only tokens with errors—for each erroneous answer it will suggest a correction, and will leave all correct answers unchanged.
In our setting, exercises are generated by Revita, a language learning system, which is used by several hundred L2 learners. These exercises are automatically generated based on a text selected for practice (Katinskaia et al., 2017, 2018). The system has one particular expected answer for each exercise— the one found in the original text. When doing an exercise, the learner may insert the expected answer, an error, or an alternative-correct answer, which is not expected, but fits the context. The problem can be stated as follows: an unexpected but suitable answer should be recognized as alternative correct, since providing incorrect feedback for valid answers can discourage learners (Katinskaia and Ivanova, 2019; Katinskaia and Yangarber, 2021). For example, in certain sentences, using the present or past tense can be equally acceptable. However, few corpora provide this type of annotation, therefore GEC models are predominantly trained and evaluated using only one reference per instance (Rozovskaya
488
and Roth, 2021; Bryant et al., 2022).
We use a freely available dataset ReLCo, collected from Revita over several years (Katinskaia et al., 2022). This dataset contains short paragraphs with answers from learners of Russian. The paragraphs include multiple answers provided to the same grammar exercises, which were manually checked and tagged as acceptable or erroneous. To the best of our knowledge, this is the only freely available dataset of this type. As a GEC model, we fine-tune a pre-trained monolingual T5 language model (Raffel et al., 2020).
The contributions of this paper are: (1) We show that a GEC model can achieve reasonable performance in assessing erroneous answers for fill-in-theblank and MC grammar exercises, if we use several top correction hypotheses. We empirically confirm the intuition that a Transformer-based GEC model cannot be used for assessing alternative-correct answers since top correction hypotheses can include corrections even for valid words. The lower-ranked hypotheses change the input sentence more freely: include more lexical changes, and more word removals or insertions. (2) We release a new version of the manually corrected RULEC-GEC test set, which, we believe, can improve the evaluation of GEC models in the future. (3) We present the first experiment with ReLCo (Katinskaia et al., 2022), the semi-automatically collected learner data, to train a GEC model. Using ReLCo shows an improvement in GEC performance. (4) We extensively evaluate the performance of our model on the RULEC-GEC (henceforth—RULEC) test set automatically and manually, including an evaluation of several top hypotheses, and show an improvement of F0.5 score over the existing state-of-the-art results for Russian. Prior work showed that evaluating GEC output only by automatically comparing it with a single gold-standard reference per sentence results in under-estimating the performance (Rozovskaya and Roth, 2021).
The paper is organized as follows. Section 2 covers prior work on the GEC task. Section 3 describes the problem and our approach. Section 4 presents the data for training the GEC model, the training procedure, and the evaluation. Section 5 presents the experiments on assessing learner answers using the trained GEC model. It includes a discussion of results and error analyses. Section 6 presents the conclusions and future work.

heading: 2 Related Work
text: Most current approaches treat GEC as a natural language generation task. It can be formulated as a monolingual translation from incorrect to correct language using various architectures (Yuan and Briscoe, 2016; Junczys-Dowmunt et al., 2018; Chollampatt and Ng, 2018; Yuan et al., 2019; Náplava and Straka, 2019; Grundkiewicz et al., 2019; Zhao et al., 2019; Kaneko et al., 2020). Due to the paucity of annotated training data for GEC, it has become standard practice to generate synthetic data, using various ways of creating erroneous sentences—by back-translation or random token-level transformations (Kiyono et al., 2019), using the history of Wikipedia edits (Lichtarge et al., 2019), confusion sets suggested by spell-checkers (Grundkiewicz et al., 2019; Náplava and Straka, 2019), real error patterns (Choe et al., 2019; Takahashi et al., 2020; Li and He, 2021; Stahlberg and Kumar, 2021), or applying noise to a latent representation of an errorfree sentence (Wan et al., 2020). A comparative study of methods of generating synthetic data is presented in (White and Rozovskaya, 2020). Another approach is text editing—generating a sequence of edits to apply to the incorrect input sentence (Malmi et al., 2019; Stahlberg and Kumar, 2020; Tarnavskyi et al., 2022). In GECTOR (Omelianchuk et al., 2020), the authors develop a set of custom token-level transformations to recover the target text from the source. Editing is faster than generating the whole corrected sentence, but requires constructing many language-specific transformations. More on GEC and existing approaches to the problems and evaluation is reviewed in (Bryant et al., 2022). A number of papers focus on the actual use of GEC models by language learners. Homma and Komachi (2020) approach the problem of GEC usability as a part of a writing-support system for Japanese, with a focus on inference speed and working with incomplete sentences. Zomer and Frankenberg-Garcia (2021) present a writing-improvement model, which is adapted to the writer’s first language (L1). The model’s output was evaluated on grammaticality, acceptability, and lexical and syntactic diversity. An Example-Based GEC with a focus on interpretability is introduced in (Kaneko et al., 2022): the model presents to the learners correction results and examples as a base for correction. Takahashi et al. (2022) explore the learners’ proficiency-wise evaluation for Quality
Estimation (QE) of GEC. Several papers on GEC focus on low-resource languages, including Russian (Rozovskaya and Roth, 2019; Katsumata and Komachi, 2020). Náplava and Straka (2019) adapted the approach of Grundkiewicz et al. (2019) for Russian, German, and Czech. Their results on Russian outperformed those of Rozovskaya and Roth (2019) by more than 100% on the F0.5 score, but still performed quite poorly compared with other languages. GEC for Russian is shown to be a challenging task, which is explained by the small size of the RULEC corpus. In (Rothe et al., 2021), the biggest multilingual T5 model which was pre-trained on synthetic data and fine-tuned on real data achieved the best performance on Russian among other approaches. Performance was improved by adding to the GEC pipeline a Transformer model for re-ranking the suggested correction edits (Sorokin, 2022).

heading: 3 Problem Setup
text: Our task is to evaluate and provide feedback on the grammatical correctness of answers given by the learner to all grammar exercises (cloze and MC) generated by a CALL system in a sentence. For example, in the sentence in Figure 1, “Вероятно, такие приборы уже изобретены.” (“Probably, such gadgets have already been invented.”), the learner receives one MC exercise (приборы vs. преборы, “gadgets”) and a cloze exercise with lemma “изобрести” (“to invent”). The MC has only one correct answer, if the exercise is well-designed. In the cloze, the student’s an-
Dataset Training Develop Test
RULEC 4 980 2 500 5 000 cLang-8 (Ru) 44 830 — — ReLCo 8 560 — 7 017

heading: 4 GEC Experiments

heading: 4.1 Data
text: To train the GEC model, we use the datasets: RULEC (Rozovskaya and Roth, 2019), Russian cLang-8 (Rothe et al., 2021), and ReLCo (Katinskaia et al., 2022). Dataset statistics are in Table 1. Incorporating the Lang-8 (Tajiri et al., 2012) corpus did not yield a significant improvement, see Table 3. Similar results were shown by Trinh and Rozovskaya (2021), where adding RU-Lang8 to the training data did not improve the results on the
RULEC test either, although their experiments were conducted using a different model. Therefore, we included the Russian part of the cLang-8 dataset, which is a cleaned version of Lang-8. cLang8 was used only for training. For tuning parameters and analysis of model outputs, we use only the RULEC validation set. The RULEC test set was used for evaluation and comparison with other GEC models. We split the manually annotated ReLCo into a training and test set. ReLCo consists of short paragraphs, which include learner answers given to grammar exercises. Exercises in the same paragraph can vary depending on the learner’s proficiency. The same paragraphs can be practiced by different students or by the same student multiple times, resulting in numerous repeating sentences in the corpus. We ensured that the same sentence never occurs in different data splits. Since we are interested in GEC performance on sentences with multiple acceptable corrections—henceforth, alternative correct, or AC—we placed more of such sentences into the test set, see the number of erroneous and AC answers in each data split in Table 2. We also do not want the GEC model to be forced to replace AC answers with expected answers during fine-tuning.

heading: 4.2 GEC Model
text: We use the Text-to-Text Transfer Transformer (T5) model, an encoder-decoder multi-task model that was pre-trained on unsupervised and supervised tasks, with converting each task into a text-to-text format. Rather than the multilingual T5 as in Rothe et al. (2021), we fine-tuned a monolingual Russian T5 model (Raffel et al., 2020).
Rothe et al. (2021) showed that bigger T5 models perform GEC better for all tested languages. We chose a large-size configuration (over 700M parameters), since we cannot run T5 xl or T5 xxl with available resources. The T5 model is instructed to perform a par-
ticular generation task by adding a prefix at the beginning of an input sequence. We conditioned each input sentence by adding the task definition “improve_grammar”.1 First, we tried to directly fine-tune the T5 model on the real data, see the results of tuning with several combinations of learner corpora in Table 3. The combination of the RULEC train partition, cLang8, and the ReLCo train partition yields the best F-score and, therefore, it was used in all the following experiments. Since Rothe et al. (2021) report that the bestperforming setup for the T5 model used GEC pretraining on synthetic data, we also (1) pre-train the T5 model on a synthetic dataset until convergence,2 followed by (2) fine-tuning on the three mentioned datasets.3 The synthetic data was generated from WMT News Crawl monolingual training data (Bojar et al., 2017) using Aspell confusion sets following (Grundkiewicz et al., 2019). We generated 10M sentences using the same parameters as presented in (Náplava and Straka, 2019). To choose parameters for the fine-tuning on original data, we run hyper-parameter search4 using Population Based Training (PBT) optimization algorithm (Jaderberg et al., 2017). We set the dropout rate of the T5 model at 0.2, which was found to give the biggest gain in F-score on a validation set. Higher dropout may teach the model to trust the source sentence less and introduce more corrections, as noted previously by Junczys-Dowmunt et al. (2018).

heading: 4.3 GEC Evaluation
text: Given an original sentence with errors (a source sentence), the GEC system generates a ranked list of suggested corrections (hypotheses). The perfor-
1Our implementation is based on Hugging Face. 23 GPU V100, pre-training for 1.48M steps with batch size = 6, weight decay = 0, learning rate = 5e-5. 3The fine-tuned model is available at RuT5_GEC 4The best performance was obtained with the following parameters: number of epochs = 2, weight decay = 0.180335, learning rate = 3.83229e-05.
mance that we discuss next is calculated for the top-1 hypotheses.
Evaluation with M2 Scorer: Evaluation of all GEC models was done on the RULEC test set using the MaxMatch (M2) scorer (Dahlmeier and Ng, 2012). It computes GEC performance in terms of phrase-level edits. The results of the evaluation and effect of pre-training and tuning hyper-parameters are shown in Table 5; all reported scores are averaged over 3 runs. A simple pre-processing improvement of the data5 gives a performance gain, see “data preprocessing” in Table 5. We also have detected some formatting issues and word repetitions in the hypotheses generated by the model for the validation set. Therefore, we run post-processing for the model output, see details in Appendix A. The results of the evaluation after post-processing are in Table 4 and Table 5, and they are on par with the current state of the art.
Evaluation with ERRANT. ERRANT (Bryant et al., 2017; Felice et al., 2016) is a reference-based scorer which measures performance in terms of an edit-based F-score. Unlike the M2 scorer, it is also able to calculate error type scores at different granularity, e.g., Replacement edit or Replacement:Noun:Case edit. We use an extension6 of ERRANT for Russian (Katinskaia et al., 2022). Evaluation of GEC performance using ERRANT was not reported for Russian in the previously published papers. We measured performance with ERRANT on
5Inspection of the GEC model’s output on the validation set showed that during inference, the pre-trained and finetuned T5 model inserts white spaces into tokens containing the characters [´] (“stress”), or [ё].
Filtering out stress characters and replacing [ё] with [е] is a trivial fix that does not alter the meaning of the text.
6RuERRANT
the post-processed output of the best GEC model which we trained, see Table 6. ERRANT’s F0.5 score for correction is lower (52.1) than F0.5 calculated byM2 scorer (52.9). The same discrepancy between these scores was reported in (Kiyono et al., 2019) for English. The T5-based GEC model is performing significantly better for replacement errors than for insertion or deletion errors. One possible reason for that can be related to the distribution of error types in the training data: syntactic data was generated mostly by replacing tokens; and in the real learner data, errors were corrected following the principle of minimum correction needed to fix the source sentence, which mostly involves replacing separate words rather than removing or inserting words. ReLCo includes only replacement errors collected from cloze and MC exercises.
Manual Evaluation. Of the total 5 000 top-1 GEC hypotheses generated for the test set, 1 199 were found to be different from both the source sentences and the corresponding gold-standard references. These hypotheses were manually evaluated
by a native-speaking annotator with a degree in teaching Russian and prior annotation experience. The task was to mark whether a sentence is acceptable grammatically. The results showed that 285 of the checked hypotheses can be considered grammatically acceptable. In some cases, the corresponding gold references include typos or uncorrected errors, while in others, GEC hypotheses and the gold reference both present alternative corrections of the source sentence. In addition, 52 hypotheses differ from their gold references only by capitalization, e.g., the first word is not capitalized in a reference, but it is capitalized in the generated hypothesis. The remaining 862 sentences were annotated as indeed ungrammatical. As a final result, the manual evaluation showed that 62.4% of all 5 000 suggested top-1 hypotheses are correct.
Other Hypotheses. We generated 7 hypotheses7 for each source sentence with beam search decoding. Comparing hypotheses with the references shows that in some cases the GEC model produces a correction which is the same as the reference sentence, but it is not chosen as the top-1 hypothesis. Figure 2 shows the percentage of hypotheses equal to the reference sentences by the rank of the hypotheses. Ranked top 3 include 65.5% of hypotheses equal to the references. More on the manual evaluation of the top-3 hypotheses is in Appendix B.

heading: 4.4 RULEC Test Cleaning
text: Testing various models on the RULEC test set showed that it contains uncorrected errors, ungrammatical corrections, and mistakes in indexing of proposed corrections. Since this impedes assessing the true performance of the models, we undertook a re-annotatation of the data. At this stage, we do
7This number of hypotheses is the maximum we can generate with resources available to us.
not claim that all errors and inconsistencies in the RULEC test set have been fixed. Annotation was done by three native speakers: two Master’s students in linguistics and one expert in teaching Russian. Source sentences were randomly split into two subsets and presented to two annotators, 2.5K sentences each. The annotators could see the original erroneous sentence and its correction (gold reference) proposed in RULEC. The task was to fix the gold reference only if needed, following the minimal-edits principle that results in a grammatically correct reference sentence. The third annotator checked all 5K source sentences and the proposed corrections. Due to limited resources, we could not involve more annotators to correct source sentences without seeing the gold references, or to get more corrections per source sentence. We measured the agreement between the last annotator and the two annotators in the first phase of correction: average agreement is 87%. Most disagreements relate to punctuation, and were resolved by the final annotator. We calculated our GEC model performance (with output post-processing) on the corrected RULEC test set, see the last row in Table 5. TheF0.5 score increases to 55.4, which is above the current state-of-the-art results for Russian. A re-annotated test set allowed us to evaluate more realistically the corrections which were attempted by the model, though many errors are still left uncorrected. The updated test set is released inM2 format.8

heading: 5 GEC for Evaluating Learner Answers
text: The following task is to evaluate whether a GEC model can be directly used for assessing learners’ answers in a CALL system.
Evaluation. We generated 7 hypotheses for each sentence in the ReLCo test set. The source sentences did not need pre-processing. We applied a post-processing step to filter out 874 hypotheses containing word repetitions, following the method used for the RULEC test set: a filtered-out hypothesis is replaced with its source sentence. Next, we describe the procedure for checking learner answers based on the suggested corrections. We define the word inserted by the learner as an answer to an exercise as the target word. Firstly, we align the suggested GEC hypotheses with the corresponding source sentences. Then, for each
8RULEC-GEC test updated
target word, we follow the steps:
1. Check whether a target word was corrected by the majority of the suggested hypotheses.
2. If corrected by majority, the target word is classified as a grammatical error.
3. Otherwise, check whether the target word was left unchanged in all top 3 hypotheses.
4. If not corrected in all top 3 hypotheses, it is potentially an alternative correct answer.
5. Else it is classified as an error.
We chose to evaluate the top-3 hypotheses because previous testing on RULEC showed that they had the highest quality among all generated hypotheses. The results of evaluating grammatical correctness of answers using this algorithm are presented in Table 7, see the third column marked “top-3”. BesidesF0.5, we report theF1 score: for language learning, it is important not only to provide valid corrections (low false positives) but also not to silently miss errors (low false negatives). Examining multiple hypotheses allows us to improve the precision of detecting AC and the recall of detecting errors. We experimented with modifying steps (3) and (4) by requiring the target word to remain unchanged in all seven suggested hypotheses (see column “all” in Table 7). Furthermore, we compared performance on AC answers with the highest disagreement rate among annotators, referred to as “Hard AC” in Table 7. The table presents the performance measured in accuracy, which indicates how many Hard AC answers are recognised as correct.
Re-ranking. One of the problems with using all hypotheses, or only the top-N , is that some of the hypotheses can include more uncorrected errors or can differ significantly from the source sentence lexically and syntactically. For this reason, we experiment with several methods for scoring and re-ranking hypotheses, e.g., using LM
scores, the number of errors detected by a GED model, VERNet (Liu et al., 2021), Discriminative re-ranking (Lee et al., 2021), OpenAI’s GPT-3.5 model9 as re-ranker, etc. We test them on RULEC and choose COMET as the best-performing score. Different methods allow increasing precision or recall which, depending on the use case, can be beneficial. COMET metric10 for MT evaluation (Stewart et al., 2020; Rei et al., 2020) exploits information from both the source sentence and the reference in order to evaluate the quality of an MT hypothesis. Unlike re-ranking methods which are not using any information about references, COMET allowed to get significant improvement, see performance of a GEC model evaluated after re-ranking with COMET in Table 5. This is the first application of this metric to GEC. Table 7 (column “top-3 & re-ranked”) shows results of assessing learner answers after re-ranking hypotheses according to their COMET score. Using top-3 hypotheses and re-ranking shows the best scores for assessing learners answers overall.

heading: 5.1 Error Analysis
text: We separately analyzed the GEC model’s performance in assessing alternative correct answers and erroneous answers.
Alternative Correct (AC). Table 8 shows the accuracy of the GEC model on 14 different types of AC answers in the test data, which were annotated manually. The notation “Tense: past/present” means that the expected answer was in the past tense, the learner’s answer was in the present, and both forms are acceptable in the context. Performance significantly varies across different types, which should be considered when utilizing GEC
9https://platform.openai.com/docs/models/gpt-3-5 10The model used is Unbabel/wmt22-comet-da
for assessing learner answers. We found that sometimes the GEC model proposes to correct AC answers by words with similar spelling but different meaning that are not relevant in the context, e.g., “бесплотны” (“ethereal”) is corrected as “бесплaтны” (“free of charge”); “от вора” (“from a thief”) is corrected as “от ворот” (“from the gate”). It especially relates to rare words, e.g., “калорифер” (“heater”) replaced with “калории” (“calories”). In many cases, the GEC model indeed does not change an AC answer, but it frequently proposes the expected correct answer as a correction, e.g., the top-2 suggestions (Output 1 and 2) in Table 9 include both “смотри” and “посмотри”.11 For more examples, see Appendix D. Potentially, GEC may be used only for the bestperforming types, while for other types, we might need to train separate models. We could provide a learner with 2-3 top corrections suggested by the model and, if it is possible, involve a teacher in a final assessment step.
Errors. One of the detected problems relates to a mismatch between annotation and evaluation conditions: learners’ answers in ReLCo were annotated within the context of a paragraph, while we have run GEC evaluation of separate sentences. Therefore, some answers, which are erroneous within a paragraph but not a sentence, were not detected as errors by the model. We run a preliminary evaluation by providing the model with whole paragraphs as input, instead of sentences. Some longer paragraphs have to be pruned to 100 tokens.12 Performance drops in terms of recall for error detection, though precision increases, especially for a setting
11“Look” in imperfect and perfect aspect, respectively. 12Due to technical limitations, the input sentence length for
beam search cannot exceed 100 tokens.
with re-ranking. See more details in Appendix C. Paragraph-level assessment needs more investigation in future work. Another issue is that the GEC model is not informed which word is a target of an exercise. In the first example in Table 9, only the underlined word “привокзальных” (“near railway station”) was provided as an answer which is an incorrect plural form in the context. However, the model corrected the noun “площади” (“square”) from singular to a plural form. The second example shows issues with reverted word order: the model detects local syntactic relations between the preposition “из-за” (“because of”) and the following noun phrase “этих документов” (“these documents”), so it puts the noun phrase in genitive case. However, it failed to detect government relations with a head verb “отвечает” (“is responsible”), which requires the preposition “за” (“for”), not “из-за”. The last example shows an issue with checking whether an answer was corrected by the majority of the hypotheses: instead of correcting a preposition “в” (“in”), the model rephrases the whole time expression. Figure 3 shows the evaluation of detection and correction performance for several error types using ERRANT, on the RULEC and ReLCo test sets. Performance on the two test sets differs drastically on some error types: e.g., spelling, verb aspect, and
tense errors. Adverbs and pronouns have low performance in both test sets. All scores for ReLCo are higher, likely because it includes only replacement errors that are better handled by the GEC model. This indicates that the model can potentially be used for detecting errors in cloze exercises with best-performing error types, without providing suggested corrections, since correction performance is lower.

heading: 6 Conclusions and Future Work
text: We present experiments on using Transformerbased GEC models to evaluate the correctness of answers provided by language learners to grammar exercises. To the best of our knowledge, it is the first attempt to directly employ a GEC model for this task. We find that the top-performing GEC model demonstrates the potential to detect and correct errors in user answers provided to fill-in-theblank and multiple-choice grammar exercises, if we use multiple top hypotheses generated with beam search. However, this approach is less effective for assessing alternative-correct answers. Given the current low recall of the GEC model, there is a high chance of labeling erroneous answers as acceptable. Furthermore, the number of possible alternative corrections proposed by more advanced GEC models can be high, meaning that when the GEC model corrects an answer, it does not necessarily indicate the presence of an error. The problem of evaluating alternative correct answers is equivalent to the problem of multiple possible corrections for a given error span in GEC. This issue is particularly challenging because GEC
models are primarily trained and evaluated using a single reference for each sentence, as discussed in (Rozovskaya and Roth, 2021; Bryant et al., 2022). In our future work, we aim to focus on developing methods for evaluating the suggested corrections by combining reference-based and reference-free scoring approaches.
While GEC is typically approached as a task involving isolated sentences, there have been studies addressing document-level GEC as well (Chollampatt et al., 2019; Yuan and Bryant, 2021). In our experiments, we also focused on assessing grammatical correctness at the sentence level. However, in future work, we plan to investigate the assessment of learner answers within a paragraph. We intend to conduct further research on leveraging large language models to evaluate the acceptability of answers and explore the combination of various re-ranking methods.

heading: Acknowledgements
text: This work was supported in part by the Academy of Finland, Helsinki Institute for Information Technology (HIIT), BusinessFinland (Grant “Revita”, 42560/31/2020), and Tulevaisuusrahasto, the Future Development Fund, Faculty of Arts, University of Helsinki.

heading: 7 Ethical Considerations
text: We use only publicly available resources for all conducted experiments. All annotators were volunteer students who performed the tasks as a part of their studies and received credits for it.

heading: 8 Limitations
text: The current work has a number of limitations to consider.
(A) The paper’s experimental design was limited to a single language because we are not aware of any other learner corpora with multiple answers provided to the same exercises.
(B) The described approach to assessing the correctness of learner answers is limited by its design. First, the number of GEC hypotheses to check depends on the GEC model’s performance and, potentially, on the language. Second, if a word was not corrected, it can be a false negative error instead of a correct answer. Third, the GEC model can suggest corrections (valid and not valid) even to a correct answer depending on the data it was trained on. (C) Our approach focuses only on grammatical errors and it does not take into account semantic or pragmatic errors.
(D) Due to limited resources, we were unable to involve more people with prior annotation experience in the re-annotation of the RULEC test set, as well as in the manual verification of hypotheses generated by the GEC model. We acknowledge that the annotation performed by our annotators may not be entirely error-free: the annotators were free to work at their own pace and therefore could potentially rush and make errors themselves. Hence, we do not claim that the re-annotated RULEC test set does not include any inconsistency anymore. We believe that the existing datasets should be thoroughly checked, given the small amount of learner data available for languages other than English, before utilizing them to train and evaluate new models. (E) Considering the practical use of our GEC model as a component of a CALL system, we find that it can potentially be used in a limited context, i.e., for checking answers provided to cloze and multiple-choice exercises, only for best-performing error types. As for alternative correct answers, even for best-performing categories of answers, a human teacher should verify proposed corrections. We have to underline that learner errors in RULECGEC (and especially in any synthetic dataset) can significantly differ from errors made by learners with various backgrounds, native languages, and proficiency levels. We also find that low recall of state-of-the-art GEC models impedes their usage in language learning settings. At the moment, learner answers should be verified by a human teacher.

heading: A Cleaning Model’s Output
text: We have discovered two issues in the hypotheses generated by the GEC model for the validation set. One is extra white spaces in front of hyphenated suffixes added to numbers, e.g., “25 -го апреля” (“on the 25th of April”) instead of “25-го апреля”. These extra spaces were removed. Another issue relates to corrections of some short sentences (1-5 words): the generated hypotheses have repeating tokens. It is especially relevant to incomplete sentences in the test set which end with a semicolon, e.g., “Рай :” (“Heaven :”). The model is either trying to continue these sentences or just repeating the same word. We have detected all hypotheses for which source sentences were shorter than 6 words (without punctuation) and which include repetitions and replaced them with the source sentences as if they were not corrected by the model at all, in total 44 sentences.

heading: B Manual Evaluation of Top Hypotheses
text: We have picked top-3 hypotheses for 100 randomly sampled source sentences from the RULEC test set. These hypotheses were manually evaluated by a native speaker on the following aspect: whether the second-ranked and the third-ranked hypotheses improve the corrections suggested in the top-1 hypothesis or whether the quality of corrections degrades. Manual evaluation has shown that for 58% of checked sentences, the quality only improves with more hypotheses.

heading: C Paragraph Correction
text: Due to technical limitations, the GEC model input length cannot exceed 100 tokens. Therefore, to run a preliminary evaluation with whole paragraphs as input, instead of sentences, we had to prune the longest paragraphs to 100 tokens. This leads to losing 107 learner answers. Regarding assessing and detecting grammatical errors, recall drops and precision increases, especially for a setting with re-ranking, see Table ??. As a result, this leads to lower precision for AC answers, since more errors are not corrected. We find several reasons for this decrease in error recall. First of all, the GEC model was pre-trained and fine-tuned on sentences. For example, it corrects erroneous “уже из конца недели” to “уже в конце недели” (“already at the end of the week”) only if this error is in a separate sentence. However, the error is not corrected if the model
gets as input a three-sentence paragraph, with this error in the second sentence. Another issue relates to pruning paragraphs which leads to incomplete sentences and broken syntactic relations between words. Paragraph-level assessment requires more research and training GEC models on a wider context, though there are few available datasets for this task.

heading: D Examples
text: Table 10 presents more examples where the GEC model generates multiple valid corrections in the same context.

heading: Future Work
text: This work was supported in part by the Academy of Finland, Helsinki Institute for Information Technology (HIIT), BusinessFinland (Grant “Revita”, 42560/31/2020), and Tulevaisuusrahasto, the Future Development Fund, Faculty of Arts, University of Helsinki