Low-Resource Grammatical Error Correction: Selective Data Augmentation with Round-Trip Machine Translation
Authors: Frank Palma Gomez, Alla Rozovskaya
Accepted at: acl25

Low-Resource Grammatical Error Correction: Selective Data Augmentation with Round-Trip Machine Translation

Frank Palma Gomez
Boston University
frankpalma12@gmail.com

Alla Rozovskaya
City University of New York
arozovskaya@qc.cuny.edu

Abstract

Supervised state-of-the-art methods for grammatical error correction require large amounts of parallel data for training. Due to lack of gold-labeled data, techniques that create synthetic training data have become popular. We show that models trained on synthetic data tend to correct a limited range of grammar and spelling mistakes that involve character-level changes, but perform poorly on (more complex) phenomena that require word-level changes. We propose to address the performance gap on such errors by generating synthetic data through selective data augmentation via round-trip machine translation. We show that the proposed technique, SeLex-RT, is capable of generating mistakes that are similar to those observed with language learners. Using the approach with two types of state-of-the-art learning frameworks and two low-resource languages (Russian and Ukrainian), we achieve substantial improvements, compared to training on synthetic data produced with standard techniques. Analysis of the output reveals that models trained on data noisified with the SeLex-RT approach are capable of making word-level changes and correct lexical errors common with language learners.1

1 Introduction

Grammatical Error Correction (GEC) is the task of detecting and correcting mistakes in text. Supervised state-of-the-art approaches to GEC require large amounts of training data in the form of sentence examples with errors and their corrected counterparts. Because hand-labeled data is expensive to obtain, it is common practice to use monolingual data with synthetic noise for pre-training. The models are further finetuned on gold learner data. The use of synthetic data is essential to obtaining good performance, especially when the language is low-resource and has a limited amount of gold-labeled training data (Flachs et al., 2021). However, existing methods do not generate sufficiently diverse synthetic errors (Stahlberg and Kumar, 2024).

We show that low-resource state-of-the-art strategies for generating synthetic data mainly produce errors in spelling and inflectional morphology, and models trained on such synthetic noise fail to address errors related to overall fluency that are known to pose challenges even to high-proficiency learners (Yasunaga et al., 2021; Choshen and Abend, 2018b). Fluency is used to refer to edits that make the original text native-sounding; these edits typically go beyond grammaticality and may include a change for a more preferred word order or a better lexical choice (Napoles et al., 2017).

In this work, we focus on correcting lexical errors. Many of these errors are known to arise from first-language interference (Leacock et al., 2010; Dahlmeier and Ng, 2011; Rozovskaya et al., 2017). Moreover, studies in second language acquisition reveal that learners may generate a sentence in the second language by translating it from their native language (Derakhshan and Karimi, 2015), thereby incorrectly transferring structures and expressions into the second language. A lexical error may occur when a learner picks an incorrect but semantically-related translation for an ambiguous word with several related meanings in their native language.

Using this observation, we hypothesize that lexical challenges, such as choosing an incorrect word in the second language, will also manifest themselves in the machine translation output, as back-translated words that are semantically close to the target. We propose to address such errors using select substitutions obtained from round-trip machine translation. Indeed, we show that the approach with select substitutions is capable of generating a wide range of mistakes that non-native speakers make, and is particularly good at producing confusions similar to lexical errors common among non-native speakers. We refer to this approach as SeLex-RT.

We start with a monolingual corpus of language l for which we wish to build a GEC model. The sen-

1The data is available at https://github.com/arozovskaya/Low-Resource-GEC-SeLex-RT
tences are translated into another language (pivot), and then back into \( l \). The original sentence is token-aligned with its round-trip translation, and the alignments are used to create confusions resulting from imperfect translations. In contrast to other approaches that employ round-trip translation (Lichtarge et al., 2019; Kementchedjhieva and Søgaard., 2023), we (1) do not make use of the entire resulting back-translated sentences, but only generate *targeted confusion sets* of relevant errors that are used to corrupt the data; and (2) generate multiple translation hypotheses in each direction. We demonstrate that both of these innovations are crucial for obtaining a diverse set of high-quality synthetic errors (see Section 6.1).

We present experiments on two low-resource languages for GEC, Russian and Ukrainian, with additional analyses on Russian. We show that only 5%-12% of the lexical errors observed in gold learner data are represented in the data produced with the standard synthetic methods. In contrast, SeLex-RT is capable of generating synthetic data that represents over 50% of learner lexical errors. Experimental results demonstrate substantial performance gains on correcting lexical errors, compared to training with standard synthetic data.

The paper makes the following contributions: (1) Using two Russian benchmarks, we show that the majority of mistakes identified by state-of-the-art supervised GEC models are in spelling and inflectional morphology; (2) We propose *SeLex-RT*, a novel method that uses select substitutions from round-trip translations and generate synthetic errors that involve replacements of unrelated but semantically similar words and complements existing approaches; (3) We use *SeLex-RT* to generate synthetic data for two low-resource languages, and substantially improve over standard methods; additional analysis on Russian reveals that *SeLex-RT* is particularly beneficial for correcting lexical errors.

2 Background

Supervised approaches to GEC can be broken down into sequence-to-sequence (seq2seq) (Chollampatt and Ng, 2018; Yuan and Briscoe, 2016; Grundkiewicz et al., 2019; Grundkiewicz and Junczys-Dowmunt, 2019; Kiyono et al., 2019; Zhao et al., 2019; Ji et al., 2017; Katsumata and Komachi., 2019; Xie et al., 2018; Qorib et al., 2024), and sequence-to-editing (Omelianchuk et al., 2020; Awasthi et al., 2019; Tarnavskyi et al., 2022). Both achieve state-of-the-art performance on English GEC. Recent works also achieve strong results with edit ranking (Sorokin, 2022) or ensembling (Omelianchuk et al., 2024; Qorib and Ng, 2023). Other works have made additional advancements on English GEC, but they rely on large amounts of hand-labeled data that is not available for other languages (Sun and Wang, 2022; Lai et al., 2022; Bout et al., 2023).

The edit-based framework (Stahlberg and Kumar, 2020; Omelianchuk et al., 2020) was shown to be competitive on English, however, for other languages proved to be less successful (Syvokon and Romanyshyn, 2023), due to the fact that the approach requires language-specific knowledge to develop rules (Bryant et al., 2023). (It is important to note that almost all non-English GEC can be viewed as low-resource, due to scarcity of hand-labeled training data). The seq2seq framework, on the other hand, has shown state-of-the-art performance in non-English GEC (Rothe et al., 2021), and we thus adopt this approach in our work. In the seq2seq approach, GEC is cast as a machine translation task with the erroneous sentences treated as the source and corrected sentences treated as the target. Sentences with real learner errors and their manually-corrected counterparts or sentences from a monolingual corpus with added synthetic noise can be used for training GEC models. Pretrained language models (PLMs) can be used as a starting point (Kaneko et al., 2020; Malmi et al., 2019; Omelianchuk et al., 2020; Katsumata and Komachi., 2019). We follow Rothe et al. (2021) that successfully applied the approach in multilingual settings, making use of mT5.

**Low-resource GEC** Compared to English, Russian and Ukrainian are low-resource for GEC, due to the limited amounts of labeled training data (see Table 1). In terms of monolingual data, the mC4 corpus used to pre-train mT5, can be considered as high resource for Russian (3.6 TB of data) but low-resource for Ukrainian (196 GB of data).

3 Learner Datasets

**Russian and Ukrainian gold data** We use two datasets of Russian learner data manually corrected for errors\footnote{Manually corrected learner data is referred to as *gold data*.} RULEC-GEC (Alsfuieva et al., 2012; Rozovskaya and Roth, 2019) (henceforth RULEC) and RU-Lang8 (Trinh and Rozovskaya, 2021). Both datasets were originally annotated with a single gold reference each (a corrected version produced by an expert), however, the annotations have been en-
<table>
  <tr>
    <th>Dataset</th>
    <th colspan="3">Partition (sents.)</th>
  </tr>
  <tr>
    <th></th>
    <th>Train</th>
    <th>Dev.</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>RULEC</td>
    <td>4,980</td>
    <td>2,500</td>
    <td>5,000</td>
  </tr>
  <tr>
    <td>RU-Lang8</td>
    <td>-</td>
    <td>1,968</td>
    <td>2,444</td>
  </tr>
  <tr>
    <td>UA-GEC</td>
    <td>32,734</td>
    <td>1,506</td>
    <td>2,644</td>
  </tr>
</table>

Table 1: Russian and Ukrainian gold datasets.

<table>
  <tr>
    <th>Error group</th>
    <th colspan="2">Percentage (%)</th>
  </tr>
  <tr>
    <th></th>
    <th>RULEC</th>
    <th>RU-Lang8</th>
  </tr>
  <tr>
    <td>Grammar</td>
    <td>39.3</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>Orth.</td>
    <td>32.7</td>
    <td>33.0</td>
  </tr>
  <tr>
    <td>Lex./morph.</td>
    <td>14.0</td>
    <td>13.5</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>14.0</td>
    <td>13.5</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>5283</td>
    <td>3382</td>
  </tr>
</table>

Table 2: Learner error distributions in Russian by coarse category. Other includes word deletion/insertion, word order, phrase replacement.

riched with two additional references, for a total of three references per sentence (Palma Gomez and Rozovskaya, 2024). We use the enriched benchmarks. For Ukrainian, we use the UA-GEC dataset (Syvokon and Romanyshyn, 2023). We report results on the Fluency track (that includes both grammar and fluency edits). Dataset sizes are in Table 1.

Error distributions in learner Russian To understand the distribution of errors in learner data and to perform evaluations by error type (Section 5), we classify gold errors in the Russian data using a tool for Russian (Rozovskaya, 2022).3 The tool is similar in spirit to ERRANT, developed for English (Bryant et al., 2017), and classifies edits into 24 categories in spelling, punctuation, morphology, and lexical errors. We further group all errors into four broad categories: grammar, orthography, lexico-morphology, and other (Table 2). Mistakes in the grammar category involve 1-2 character modifications in inflections or a small confusion set (errors on closed-class words). The orthography category includes spelling, punctuation, and capitalization mistakes. The lexico-morphology category includes lexical mistakes and errors in derivational morphology, i.e. those that involve word formation and go beyond single character misuse. This category is the main focus of our work. Sample learner errors are illustrated in Appendix Table B3.

4 Synthetic Data Methods

Ye et al. (2023) states the following: “the lack of high-quality publicly available data remains a challenge in low resource settings.” In this section, we first review existing approaches to generating synthetic data, including the baseline methods used in this work. Next, we introduce the SeLex-RT approach, describe how the errors are generated with each method. We then compare the synthetic errors produced with each method with errors found in the learner texts.

4.1 Existing Synthetic Data Methods

Overview of existing methods used as baselines
Synthetic data generation methods lie along a continuum from knowledge-lean to knowledge-intensive, depending on the language-specific knowledge and resources required. As baselines, we implement data generation techniques that are general enough to be used across a variety of languages and do not require extensive language-specific knowledge.

Random character-level perturbations (Char) This approach creates synthetic errors in monolingual data, by probabilistically inserting, deleting, or perturbing characters (Kiyono et al., 2019; Grundkiewicz et al., 2019; Flachs et al., 2021; Stahlberg et al., 2019). We refer to it as Char.

Spell-based transformations (Spell) It combines character-level perturbations of the Char method and token-level perturbations generated from confusions from an open-source spellchecker such as Aspell (Grundkiewicz and Junczys-Dowmunt, 2019). This approach showed state-of-the-art performance in English (Bryant et al., 2019; Grundkiewicz and Junczys-Dowmunt, 2019; Grundkiewicz et al., 2019) and in other languages (Náplava and Straka, 2019).

Morphology-based transformations (Morph) The third baseline method is based on the approach in Choe et al. (2019), and derives token-level confusions from morphological transformations. Originally applied in English, the approach also showed good results on other languages with rich morphology (Flachs et al., 2021). The original method of Choe et al. (2019) combined morphological transformations with patterns mined from the development data for English. Flachs et al. (2021) implement a similar approach, but only use morphological transformations, without the patterns, in four languages, including Russian. We follow Flachs et al. (2021) and only use morphological transformations.

We use the three synthetic methods above – Char, Spell, and Morph – as the baselines in our experi-

3We are not aware of a similar tool for Ukrainian.
ments (see Section 5).

Other synthetic methods not used in this work
Several other methods of generating synthetic data have been proposed in the literature, however, these approaches typically require more hand-labeled data (or language-specific knowledge). In particular, the tagged corruption method (Stahlberg and Kumar, 2024) mines likely confusions from gold data that are used to corrupt monolingual data. This approach relies on gold training data for mining the error patterns and a tagger model that can tag the errors. This approach performed well on English, but did not perform well in a multilingual setting (Stahlberg and Kumar, 2024). Notably, the results on Russian (RULEC) for a similar model to ours are not competitive with the results we obtain in this work (see Section 6.1).

Several works employ round-trip machine translation where full-sentence translations are used (Lichtarge et al., 2019; Kementchedjhieva and Sogaard., 2023; Zhou et al., 2020). We compare with this approach in Section 6.1.

Xie et al. (2018) and Kiyono et al. (2019) use back-translation, where a machine translation model is trained in reverse direction (from well-formed to ungrammatical sentences). The approach is applied to English GEC, where over 1M gold sentence pairs are used to train the translation model for English. In contrast, we only have 5K and 32K gold sentence pairs for Russian and Ukrainian, respectively (please also see our further discussion in the Limitations).

4.2 Our Approach (SeLex-RT)
In the SeLex-RT approach, we generate confusion sets by translating a native corpus \( T \) in language \( l \) (e.g. Russian) into another language (\( pivot \)) and then back into \( l \). We token-align the original sentence with its back-translated version using the alignment model of Sabet et al. (2020). A target word \( t \) that occurs in \( T \) in a diverse set of contexts, is expected to have different back-translations determined by each unique context. A confusion set for \( t \) will include all unique back-translated words aligned to \( t \), for all occurrences of \( t \) in \( T \). Our expectation is that because translations are imperfect, the back-translated sentences will not be identical to the original sentence. Furthermore, because word sense nuances can be challenging for non-native speakers due to the differences in contextual word usage of ambiguous words in their native language and in the second language, on the lexical level, the differences in back-translation will be similar to mistakes made by language learners. The intuition that this approach of round-trip translation with token-based alignments will generate errors similar to learner mistakes is based on a recent study that used this method for generating vocabulary exercises for English language learners (Panda et al., 2022; Palma Gomez et al., 2023). Indeed, we show below that the method is able to replicate well real learner mistakes.

We stress that, in contrast to the full-sentence translation approach (Section 6.1), we only extract aligned word pairs used to corrupt the data. Furthermore, we use multiple translation hypotheses, thereby generating multiple back-translations for each token in a single sentence.

4.3 Generating Synthetic Errors with SeLex-RT and the Baseline Methods
We now compare the SeLex-RT method and the baseline methods Spell and Morph with respect to their ability to generate errors that mimic real learner errors.\footnote{We do not include the Char method here, as it only performs character-level perturbations.} The three methods all produce token-level confusion sets that are used to corrupt monolingual data. Examples of confusion sets (in English) for the word walk are {walk,talk,wall}, {walk,walking,walked}, and {walk,go,stroll}, generated with the Spell, Morph, and SeLex-RT method, respectively. If walk occurs in a sentence and is selected to be corrupted, it is replaced with another candidate from its corresponding confusion set (generated with the corresponding corruption method). Appendix Table C4 illustrates sample confusion sets in Russian.

We use these methods to generate the erroneous side of a sentence from a monolingual corpus, by introducing errors into it. We iterate over the tokens in the sentence and select tokens to be replaced with another token from the confusion set of that token.\footnote{The tokens to be replaced are selected with prob. 15%.} An example of a sentence (in Russian) produced with each synthetic data generation method can be found in Appendix Table C5.

The errors are generated in a 15M monolingual Russian corpus (Sorokin, 2017), used to train GEC models (see next Section). SeLex-RT confusion sets are generated, by taking a sample of 100K sentences from the 15M corpus. The 100K sentence sample is translated into English and then back into Russian.\footnote{We only use 100K sentences to generate confusion sets, which are then used to corrupt the entire 15M corpus.} We use the translation systems of Tiedemann and
<table>
  <tr>
    <th>Synth. method</th>
    <th colspan="2">RULEC</th>
    <th colspan="2">RU-Lang8</th>
  </tr>
  <tr>
    <th></th>
    <th>Gram.</th>
    <th>Lex.</th>
    <th>Gram.</th>
    <th>Lex.</th>
  </tr>
  <tr>
    <td>Spell</td>
    <td>41.9</td>
    <td>12.0</td>
    <td>34.4</td>
    <td>9.8</td>
  </tr>
  <tr>
    <td>Morph</td>
    <td><b>75.9</b></td>
    <td>5.0</td>
    <td><b>72.4</b></td>
    <td>5.6</td>
  </tr>
  <tr>
    <td>SeLex-RT</td>
    <td>58.8</td>
    <td><b>46.7</b></td>
    <td>65.1</td>
    <td><b>51.5</b></td>
  </tr>
</table>

Table 3: Percentage of unique errors (represented as source-target pairs) in Russian learner test data that have been produced with each synthetic method. Results broken down by error group. Best result is in bold. Confusion set size is determined based on the average number of corruption options for a target token.

Thottingal (2020), with English as the pivot. We choose English as there are typically high-quality translation systems available to and from English for a variety of languages (but also see Limitations). For each sentence, 10 forward translations, and 10 backward translations for each forward translation are produced, for a total of 100 back-translations per sentence. In 6.1, we discuss the computational costs of the approach.

How well do synthetic errors created with each method replicate learner errors? To answer this question, we compute the percentage of unique errors (represented as source-target pairs) in the Russian learner datasets that are found in the synthetic data produced with each method (target refers to the correct word, and source refers to the potentially erroneous word in learner data or the replacement used to corrupt the data with a synthetic method). Table 3 demonstrates that the SeLex-RT method generates about half of learner errors in the lexico-morph. category, while the baseline methods produce only a small fraction, confirming that the SeLex-RT-based confusion sets are able to better mimic real learner errors, compared to the other methods.

5 Experimental Setup

Model architecture Drawing on the methods that showed superior performance in multilingual GEC (Rothe et al., 2021; Palma Gomez et al., 2023), we chose the seq2seq framework (see Section 2). We implement two types of Transformer seq2seq models – a smaller one trained from scratch, and a larger model that uses mT5 as a starting point.

Model 1 Our (smaller) model is pre-trained on synthetic data and finetuned on the gold data. It has 275M parameters.

Model 2 We adopt the approach of Rothe et al. (2021), making use of mT5 (Xue et al., 2021). Rothe et al. (2021) finetune mT5 on GEC gold data only, but Palma Gomez et al. (2023) show the importance of a 2-step finetuning, which we adopt here: in the first step, mT5 is finetuned on the synthetic data, and in the second step we finetune on the gold training data. Due to computational constraints, we use mT5-Base that has 580M parameters. Larger models (large, xl, xxl) lead to stronger results both with 1-step finetuning on gold data only (Rothe et al., 2021) and with a 2-step finetuning on Russian (Palma Gomez and Rozovskaya, 2024) and Ukrainian (Palma Gomez et al., 2023). Appendix A lists the training details and the hyperparameters.

Baseline synthetic methods We implement four data generation techniques: (1) character-level transformations (Char); (2) spell-based transformations (Spell); and (3) morphology-based transformations (Morph), (4) and a combination Morph+Spell.7 Appendix C provides more detail on how we generate errors for the two languages.

Synthetic data The synthetic data is created by corrupting monolingual data from the Yandex corpus for Russian (Sorokin, 2017), and from the CC-100 dataset for Ukrainian (Wenzek et al., 2020). Each training example in the synthetic data consists of a pair of sentences: a corrupted sentence and its correct counterpart from the monolingual corpus. Example is shown in Appendix Table C5.

Evaluation To compare with published work, we adopt the MaxMatch scorer (Dahlmeier and Ng, 2012) for Russian, and ERRANT for Ukrainian (Bryant et al., 2017). All results are on the test partitions. In Russian, we apply a spellchecker (Rozovskaya, 2022) to the data as pre-processing.

6 Key Results

Baselines We use the smaller Model 1 to compare the baseline methods. Model 1 is trained on 15M synthetic sentence pairs, while the mT5-models are trained on 2M synthetic sentence pairs (due to computational constraints). Gold training data (in Table 1) is used to finetune the models.

Key results on Russian and Ukrainian Key results on Russian and Ukrainian are shown in Tables 4 and 5, respectively. Detailed performance that includes precision and recall is shown in Appendix Tables D6 and D7.

As the tables show, the Char method is the weakest baseline, whereas Morph outperforms Spell

7In Morph+Spell, the Morph method is applied to each sentence first, followed by Spell.
<table>
  <tr>
    <th>Synth. method</th>
    <th colspan="2">F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th></th>
    <th>RULEC</th>
    <th>RU-Lang8</th>
  </tr>
  <tr>
    <th colspan="3">Model 1 (Baselines)</th>
  </tr>
  <tr>
    <td>Char</td>
    <td>40.0</td>
    <td>40.2</td>
  </tr>
  <tr>
    <td>Spell</td>
    <td>54.0</td>
    <td>52.6</td>
  </tr>
  <tr>
    <td>Morph</td>
    <td>57.5</td>
    <td>56.0</td>
  </tr>
  <tr>
    <td>M+S</td>
    <td><b>60.7</b></td>
    <td><b>61.6</b></td>
  </tr>
  <tr>
    <td>SeLex-RT</td>
    <td>49.1</td>
    <td>47.7</td>
  </tr>
  <tr>
    <th colspan="3">Model 1 (Best baselines with SeLex-RT)</th>
  </tr>
  <tr>
    <td>M+Spell-RT</td>
    <td><b>60.7*</b></td>
    <td><b>59.9*</b></td>
  </tr>
  <tr>
    <td>M+S+Spell-RT</td>
    <td><b>62.1*</b></td>
    <td><b>62.6*</b></td>
  </tr>
  <tr>
    <th colspan="3">Model 2</th>
  </tr>
  <tr>
    <td>M+S</td>
    <td>62.0</td>
    <td>59.9</td>
  </tr>
  <tr>
    <td>M+S+Spell-RT</td>
    <td><b>62.9*</b></td>
    <td><b>60.9*</b></td>
  </tr>
</table>

Table 4: Key results on Russian. Best \( F_{0.5} \) for each dataset and model are in bold. <i>Model 1</i> denotes models trained from scratch, while <i>Model 2</i> refers to models that finetune mT5-Base. <i>M+S</i> stands for <i>Morph+Spell</i>. Results marked with a \( * \) are statistically significant at the .03 level, compared to the respective baseline.

<table>
  <tr>
    <th>Synth. method</th>
    <th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th colspan="2">Model 1 (Baselines)</th>
  </tr>
  <tr>
    <td>Char</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>Spell</td>
    <td>62.8</td>
  </tr>
  <tr>
    <td>Morph</td>
    <td>43.8</td>
  </tr>
  <tr>
    <td>M+S</td>
    <td><b>63.3</b></td>
  </tr>
  <tr>
    <td>SeLex-RT</td>
    <td>57.6</td>
  </tr>
  <tr>
    <th colspan="2">Model 1 (Best baselines+SeLex-RT)</th>
  </tr>
  <tr>
    <td>Spell+SeLex-RT</td>
    <td><b>64.7</b></td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td>
    <td>64.3</td>
  </tr>
  <tr>
    <th colspan="2">Model 2</th>
  </tr>
  <tr>
    <td>Spell</td>
    <td>64.1</td>
  </tr>
  <tr>
    <td>Spell+SeLex-RT</td>
    <td><b>66.1</b></td>
  </tr>
</table>

Table 5: Key results on the Ukrainian benchmark UA-GEC. Best \( F_{0.5} \) for each model is in bold. <i>Model 1</i> denotes models trained from scratch, while <i>Model 2</i> refers to models that finetune mT5. <i>M+S</i> stands for <i>Morph+Spell</i>.

for Russian, while <i>Spell</i> outperforms <i>Morph</i> for Ukrainian. The combination <i>Morph+Spell</i> produces a stronger model for both languages. We also show the SeLex-RT performance on its own. The middle part of the tables shows the results of adding SeLex-RT to the best baselines. SeLex-RT errors are added to the text after adding errors produced with the baseline methods.\footnote{Note that when combining Spell and Morph, Morph and SeLex-RT, and Morph with Spell and SeLex-RT, the same number of training sentences is used in all cases. SeLex-RT errors are generated with probability 0.1, set experimentally.} The bottom part of the tables compares Model 2 types using the best baseline data generation strategy.

<b>Findings</b> <i>The improvements from adding SeLex-RT are consistent across all models and both languages.</i> The differences between the best models with the SeLex-RT component and their respective baselines are all statistically significant at the .03 level on both Russian benchmarks.\footnote{We use a two-sided approximate randomization test (Graham et al., 2014). We used the implementation of Alhafni et al. (2023).} We do not perform statistical significance testing on UA-GEC since the gold references are not publicly available. On Russian, improvements are larger for Model 1 than for Model 2. By contrast, improvements in Ukrainian are slightly larger for Model 2. We believe this may signal that using a pre-trained language model (mT5) is more beneficial for Russian, as the mC4 dataset on which mT5 is pre-trained contains 18 times less Ukrainian data compared to Russian, thus the SeLex-RT component could have a larger impact on Ukrainian. This finding is promising for low-resource GEC scenarios.

Comparing the two model types, Model 2 outperforms Model 1, with the exception for RU-Lang8. We hypothesize this may be due to overfitting (we are finetuning on RULEC data, which is really small), and thus a model with more parameters is more likely to overfit, resulting in a less optimal performance on out-of-domain RU-Lang8 data.

6.1 Comparison to Previous Work

<b>Comparison to other translation methods</b> In Table 6 we compare SeLex-RT with the round-trip machine translation method (henceforth MT) in Lichtarge et al. (2019) that uses full-sentence translations to generate the source side of the parallel data (detailed performance with precision and recall scores is shown in Appendix Table D8). We use English as the pivot and translate the 15M target side of the monolingual data. The round-trip translations are generated using the MT systems of Tiedemann and Thottingal (2020), the same system used to generate round-trip translations for SeLex-RT. The difference is that with SeLex-RT we only use confusion sets obtained from the round-trip translations, whereas for the round-trip MT method, following Lichtarge et al. (2019), we use full-sentence round-trip translations as synthetic training sentence pairs.

The original round-trip MT method (Lichtarge et al., 2019) (listed as <i>MT only (orig)</i> in the table) exhibits low precision. Upon further inspection, we conclude that this is due to the MT method introduc-
<table>
  <tr>
    <th>Synth. method</th>
    <th colspan="2">F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th></th>
    <th>RULEC</th>
    <th>RU-Lang8</th>
  </tr>
  <tr>
    <td>3 configurations of the round-trip MT method</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MT only (orig)</td>
    <td>40.1</td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>Morph+MT</td>
    <td>56.1</td>
    <td>55.7</td>
  </tr>
  <tr>
    <td>M+S+MT</td>
    <td>57.8</td>
    <td>60.7</td>
  </tr>
  <tr>
    <td>This work</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td>
    <td><b>62.1*</b></td>
    <td><b>62.6*</b></td>
  </tr>
</table>

Table 6: Comparison with full-sentence round-trip translation method in Lichtarge et al. (2019). Results marked with a * are statistically significant at the .05 level, compared to the best MT configuration above.

ing many changes (including low-quality changes). To address the issue, we combine MT with our two strongest baselines – Morph and Morph+Spell, as follows: (1) for half of the training sentences, we apply the baseline methods on top of the full-sentence round-trip MT translation, while (2) for the other 50% of the data, we apply the baseline synthetic method only, without the round-trip MT. 1 or 2 is chosen uniformly at random for each sentence. The resulting models are shown as Morph+MT and M+S+MT in Table 6.10 These combinations improve precision, but hurt recall. Our conclusion is thus that <i>our method (SeLex-RT) introduces more high-quality changes</i>, which yield higher recall and do not hurt precision. As the table shows, our approach substantially outperforms all the models that use full-sentence round-trip MT.11

<i>Computational costs of SeLex-RT:</i> MT systems are known to incur high computational costs. It is important to emphasize that with SeLex-RT (in contrast to Lichtarge et al. (2019)) we do not translate the entire synthetic training set, but only a small subset – 100K sentences. To elaborate further: using 10 translations in each direction would amount to 10M translations, which is less expensive than translating 15M sentences used to train Model 1. Further, as we show in 7.3, using even 5 translations in each direction (for a total of 2.5M translations) already results in substantial improvements that are almost as good as using 10 translations. Finally, with SeLex-RT confusions, we can generate an unlimited number of synthetic sentence pairs.

10We tried several ways of combining MT with Morph and Spell (with different probabilities) and show the best ones.
11We note that the SeLex-RT candidates also contain noise; filtered naturally in the corruption stage: a replacement candidate is chosen based on the relative frequency of their alignment (i.e. how many times a round-trip translation was observed with the specific target token in our data). This way, noisy candidates that typically occur less frequently compared to the high-quality translations, are chosen less frequently.

<table>
  <tr>
    <th>Model</th>
    <th colspan="2">F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th></th>
    <th>RULEC</th>
    <th>RU-Lang8</th>
  </tr>
  <tr>
    <td>P&R’24 (Model 1)</td>
    <td>57.6</td>
    <td>56.0</td>
  </tr>
  <tr>
    <td>This work (Model 1)</td>
    <td>62.1</td>
    <td>62.6</td>
  </tr>
  <tr>
    <td>P&R’24</td>
    <td>64.8</td>
    <td>62.1</td>
  </tr>
  <tr>
    <td>This work (Model 2)</td>
    <td>62.9</td>
    <td>60.9</td>
  </tr>
</table>

Table 7: Comparison with previous work for RULEC and RU-Lang8 using enriched references. P&R’24 is mT5-Large trained on 10M sentence pairs in Palma Gomez and Rozovskaya (2024). <i>Model 1</i> is pre-trained on 15M sentence pairs. <i>Model 2</i> denotes mT5-Base pre-trained on 2M synthetic sentence pairs. Both models use Morph+Spell+SeLex-RT errors.

<table>
  <tr>
    <th>Model</th>
    <th colspan="2">F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th></th>
    <th>RULEC</th>
    <th>RU-Lang8</th>
  </tr>
  <tr>
    <td colspan="3">This work (M+S+SeLex-RT)</td>
  </tr>
  <tr>
    <td>Model 1 (15M)</td>
    <td>49.2</td>
    <td>51.8</td>
  </tr>
  <tr>
    <td>Model 2 (2M)</td>
    <td>51.2</td>
    <td>52.3</td>
  </tr>
  <tr>
    <td>Previous work (similar model arch. to ours)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>P&R (15M)</td>
    <td>47.4</td>
    <td>47.7</td>
  </tr>
  <tr>
    <td>P&R mT5-B (10M)</td>
    <td>51.0</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td>P&R mT5-L (10M)</td>
    <td>53.2</td>
    <td>54.5</td>
  </tr>
  <tr>
    <td>S&K mT5-B (2.5M)</td>
    <td>26.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td>S&K mT5-XXL (2.5M)</td>
    <td>44.3</td>
    <td>-</td>
  </tr>
</table>

Table 8: Comparison with previous work for Russian, using original references. The top segment shows models trained in this work. The second segment shows previous work with similar model architectures to ours. S&K stands for Stahlberg and Kumar (2024). P&R stands for Palma Gomez and Rozovskaya (2024). mT5-B and mT5-L stand for mT5-Base and mT5-Large, respectively. In the parentheses, we show the number of synthetic training examples.

<b>Comparison to other GEC methods</b> We compare to the results in Palma Gomez and Rozovskaya (2024) with enriched Russian benchmarks (Table 7). They implement similar architectures to our Model 1 and 2, but use <i>Morph</i> transformations. Their Model 2 is mT5-Large pre-trained on 10M sentence pairs, whereas we use mT5-Base and pre-train on 2M sentence pairs. On Model 1, we outperform Palma Gomez and Rozovskaya (2024) by 5 points on RULEC and 7 points on RU-Lang8. On Model 2, we are 2 points below their results, however, since we use similar architectures, we expect that repeating our experiments with mT5-Large will at least match their results.

<b>Comparison on Russian using original references</b> We compare with prior work using the orig-
<table>
  <tr>
    <th>Model</th>
    <th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td>This work (Model 1 best, 15M)</td>
    <td>64.7</td>
  </tr>
  <tr>
    <td>This work (Model 2 best, 2M)</td>
    <td>66.1</td>
  </tr>
  <tr>
    <td>P. et al.(Model 1, 35M)</td>
    <td>66.0</td>
  </tr>
  <tr>
    <td>P. et al. (mT5-Large, 10M)</td>
    <td>68.1</td>
  </tr>
  <tr>
    <td><b>Bondarenko et al. (2023)</b></td>
    <td>68.2</td>
  </tr>
</table>

Table 9: Comparison with previous work for Ukrainian. P. et al denotes Palma Gomez et al. (2023).

inal single-reference annotations in Table 8. The top segment of the table shows models trained in this work. The second segment shows models that are directly comparable to ours due to similar architectures and/or model parameters. We outperform all models in segment 2 (with the exception of a much more powerful mT5-Large pre-trained on 10M sentence pairs), often by a large margin, even when we use less synthetic data. Note that S&K is a recent multilingual implementation of tagged corruption models (Stahlberg and Kumar, 2024) with the same architecture and similar amounts of synthetic data used to ours, but only obtains a result of 26.4 compared to our result of 51.2 on RULEC. This suggests the importance of high-quality synthetic data even when using pre-trained models in GEC. Appendix D.3 provides an expanded version of Table 8, where we compare a variety of models from previous works. We show that our models are competitive with or better than state-of-the-art.

Comparison to previous work on Ukrainian This is shown in Table 9. SOTA on Ukrainian use larger models and more data. Our best result of 66.1 is 2 points below the 2 systems that placed first in the shared task. However, these models use more knowledge and data (Bondarenko et al., 2023; Palma Gomez et al., 2023; Syvokon and Romanyshyn, 2023). Palma Gomez et al. (2023) is the same architecture as our Model 2 but uses mT5-Large and 10M synthetic sentence pairs.

7 Additional Analyses on Russian

7.1 SeLex-RT and Performance by Error Type
Since our focus is on lexical errors, we evaluate performance by coarse error type for Russian. To this end, both gold errors and errors flagged by the systems are classified using an ERRANT-style tool for Russian (Rozovskaya, 2022) (see Table 2). Results (\( F_{0.5} \)) on RU-Lang8 (for grammar and lexico-morph. errors) are shown in Table 10.

Adding SeLex-RT improves performance on lexico-morph. errors. The performance on gram-

<table>
  <tr>
    <th rowspan="2">Synth. method</th>
    <th colspan="2">F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th>Gram.</th>
    <th>Lex., morph.</th>
  </tr>
  <tr>
    <th colspan="3">Model 1 (Baselines)</th>
  </tr>
  <tr>
    <td>Spell</td>
    <td>56.4</td>
    <td>13.8</td>
  </tr>
  <tr>
    <td>Morph</td>
    <td>60.5</td>
    <td>6.9</td>
  </tr>
  <tr>
    <td>M+S</td>
    <td>69.8</td>
    <td>15.4</td>
  </tr>
  <tr>
    <td>SeLex-RT</td>
    <td>51.2</td>
    <td>20.3</td>
  </tr>
  <tr>
    <th colspan="3">Model 1 (M+S+SeLex-RT)</th>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td>
    <td>71.1</td>
    <td>29.9</td>
  </tr>
  <tr>
    <th colspan="3">Model 2</th>
  </tr>
  <tr>
    <td>M+S</td>
    <td>67.2</td>
    <td>22.8</td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td>
    <td>69.4</td>
    <td>27.0</td>
  </tr>
</table>

Table 10: Results by coarse error group on RU-Lang8.

<table>
  <tr>
    <th rowspan="2">Synth. method</th>
    <th rowspan="2">Ref. set</th>
    <th colspan="2">Performance (F<sub>0.5</sub>)</th>
    <th rowspan="2">Total</th>
  </tr>
  <tr>
    <th>Gram.</th>
    <th>Lex., morph.</th>
  </tr>
  <tr>
    <th colspan="5">Model 1 (Baselines)</th>
  </tr>
  <tr>
    <td rowspan="2">Spell</td>
    <td>RG</td>
    <td>63.3</td>
    <td>13.3</td>
    <td>60.8</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>70.5</td>
    <td>16.1</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td rowspan="2">Morph</td>
    <td>RG</td>
    <td>59.1</td>
    <td>12.5</td>
    <td>60.8</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>74.6</td>
    <td>30.7</td>
    <td>75.3</td>
  </tr>
  <tr>
    <td rowspan="2">M+S</td>
    <td>RG</td>
    <td>66.7</td>
    <td>22.3</td>
    <td>65.0</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>78.1</td>
    <td>33.7</td>
    <td>78.3</td>
  </tr>
  <tr>
    <th colspan="5">Model 1 (M+S+SeLex-RT)</th>
  </tr>
  <tr>
    <td>RG</td>
    <td></td>
    <td>66.8</td>
    <td>32.2</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td>CG</td>
    <td></td>
    <td>79.6</td>
    <td>59.6</td>
    <td>79.7</td>
  </tr>
</table>

Table 11: Evaluation with three standard references (RG) and closest golds (CGs) on a 500-sentence subset (Model 1) on RULCE. Best results are in bold.

mar errors improves as well, indicating that the SeLex-RT errors complement the baseline methods in that category. Tables D10 and D9 in Appendix show results on 4 coarse categories on both Russian benchmarks, revealing similar trends on RULEC.

7.2 Evaluation with Closest Gold References
Evaluation in GEC is known to be notoriously difficult (Choshen and Abend, 2018b; Bryant and Ng, 2015; Felice and Briscoe, 2015; Napoles et al., 2015). This is because the space of valid corrections for a given sentence is large (Choshen and Abend, 2018b,a), but the set of human references in evaluation is limited. The prevalent use of too few references is known to underestimate model performance, especially on lexical errors (Choshen and Abend, 2018b), which explains low precision on these errors in Section 7.1.

To evaluate performance more accurately, we generate closest gold (CGs) references, i.e. cor-
<table>
  <tr>
    <th rowspan="2">Hypo pool size</th>
    <th colspan="4">Performance (F<sub>0.5</sub>)</th>
  </tr>
  <tr>
    <th colspan="2">RULEC</th>
    <th colspan="2">RU-Lang8</th>
  </tr>
  <tr>
    <th></th>
    <th>P</th>
    <th>R</th>
    <th>F<sub>0.5</sub></th>
    <th>P</th>
    <th>R</th>
    <th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td colspan="7"><b>Baseline (Morph+Spell)</b></td>
  </tr>
  <tr>
    <td>NA</td>
    <td>68.8</td>
    <td>41.2</td>
    <td>60.7</td>
    <td>69.5</td>
    <td>41.4</td>
    <td>61.2</td>
  </tr>
  <tr>
    <td colspan="7"><b>Morph+Spell+SeLex-RT</b></td>
  </tr>
  <tr>
    <td>1</td>
    <td>69.9</td>
    <td>40.9</td>
    <td>61.2</td>
    <td>69.3</td>
    <td>42.6</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>25</td>
    <td>69.7</td>
    <td>42.7</td>
    <td>61.9</td>
    <td>69.6</td>
    <td>45.0</td>
    <td>62.7</td>
  </tr>
  <tr>
    <td>100</td>
    <td>71.0</td>
    <td>42.0</td>
    <td><b>62.3</b></td>
    <td>69.9</td>
    <td>44.2</td>
    <td>62.6</td>
  </tr>
  <tr>
    <td>256</td>
    <td>70.5</td>
    <td>42.6</td>
    <td><b>62.3</b></td>
    <td>70.0</td>
    <td>44.7</td>
    <td><b>62.9</b></td>
  </tr>
</table>

Table 12: Size of translation pool. Evaluation on Russian using 3 references with smaller models (Model 1 type). Best results are in bold.

rected versions constructed relative to the system output instead of the source sentence, akin to post-evaluation for machine translation (Rozovskaya and Roth, 2021). Due to the cost of generating CGs (a separate set of references needs to be generated for each system output), we perform this evaluation for a subset of 500 sentences for Model 1. Appendix D.2 provides detail on the annotation.
Results Table 11 shows evaluation on RULEC. Appendix Tables D12 and D13 show detailed results on both datasets. The key findings are: (1) Precision on lexico-morph. errors is as high as on grammar errors, confirming that, indeed, performance is more severely underestimated on lexical errors; (2) The SeLex-RT component improves recall on lexical errors by more than 20 points, whereas precision numbers remain just as high. Overall, evaluation with CGs demonstrates that the SeLex-RT component significantly improves performance on lexical errors, compared to the baselines.

7.3 Varying the Number of Translations
Thus far, we have used 100 round-trip translations per sentence, to generate confusion candidates. We refer to the 100 translations as the hypothesis pool. To evaluate the effect of the pool size on performance, we vary the number of hypotheses in each direction, and use 1, 5, 10, and 16, for a total number of translations for a sentence being 1, 25, 100, and 256. The average confusion set size increases from 3 to 120, as the pool increases from 1 to 256.

Table 12 shows evaluation with different pool sizes. While every setting outperforms the baseline, there is no significant difference in performance, as the translation pool is expanded beyond 25 hypotheses, suggesting that going further down the list of back-translations is not beneficial, perhaps due to the degrading quality of the translations.

1. значит?/означает means/?signifies
2. следовать?/соблюдать follow?/observe
3. обучения?/знаний learning?/knowledge
4. норм?/правил norms?/rules
5. несколько?/некоторые some?/several
6. есть?/находится exist?/are

Table 13: Examples of lexical errors (in Russian, with the English translations) missed by models trained on the synthetic data generated methods but corrected by models that include the SeLex-RT component.

7.4 Error Analysis
We analyze model output and identify interesting examples of errors corrected by the model with the SeLex-RT component (these errors are missed by the models trained on standard synthetic data). Table 13 illustrates some of the errors. In Appendix Table D15 we show these examples within sentences. Observe that the model corrects mistakes that do not share the root/stem (examples 2, 3, 6), as well as those that share the root (example 1) but have diverging derivations. Example 5 illustrates an interesting and challenging learner error in the use of indefinite pronouns. Acquisition of subtle semantic nuances of indefinite pronouns pose a challenge to English learners (Rabinovich et al., 2019), and we also observe this in Russian data.

8 Conclusion
We present a novel approach to creating synthetic data for correcting language learner lexical errors. We hypothesized, drawing on evidence from second language acquisition research, that challenges facing language learners will also manifest in the machine translation output, as many lexical errors result from deviations in the second language from the native usage. Our approach creates controlled confusions from token-aligned sentences and their translations. Crucially, we do not use full-sentence back-translation and generate multiple translation hypotheses, which allows us to create diverse and high-quality synthetic errors. Extensive experiments on two low-resource languages – Russian and Ukrainian – and three benchmarks, and additional analyses on Russian demonstrate the effectiveness of our approach. Future work will focus on the application of this method to other languages.

Limitations

Availability of MT systems for low-resource languages One limitation of our approach is that it requires the use of machine translation systems to
translate the target language data into a pivot language and back. While for many languages, good MT systems are available today (typically to and from English), for some low-resource languages, MT systems are not available or are extremely weak. Our expectation is that the method will require a relatively good MT system in both directions (forward and backward). In fact, as we use 256 round-trip translations for Russian, we do not observe the performance improvement (see Section 7.3), suggesting that the degrading quality of translations affect the quality of generated confusion sets. It is worth noting that we only used a single pivot – English – due to the fact that there are typically high-quality translation systems available to and from English for a variety of languages.

Analyzing the impact of the translation systems employed In this work, we have used one pivot language (English) and a single MT system for each target language. Using multiple pivot languages is outside the scope of the paper, but we hope to extend this work in the future and investigate this interesting question. It should be noted that this experiment might conflate the impact of a pivot with the quality of an MT system.

Another question concerns the quality of an MT system used when using the same pivot. While we only used one type of MT system for each language, our experiments with hypothesis pool in Section 7.3 show that going down the list of translations is not beneficial (beyond 10 top-scoring translations) is not beneficial. We hypothesize that this indicates that higher-quality translation at the top of the hypothesis list are preferable for generating confusion sets for our task.

Coverage of diverse error types We focus on single-word lexical errors, and do not address other multi-token replacements and less-preferred word order. Extending the approach to multi-token replacements is planned for future work. We believe that the SeLex-RT approach can be fit to address these types of errors, as well.

It should also be pointed out that SeLex-RT operates on the assumption of correcting lexical errors resulting from first language interference and the similarities between learner and translation errors. However, in terms of first language backgrounds, while in RULEC, all learners are native English speakers, in RU-Lang8 and in the UA-GEC dataset there is a variety of first language backgrounds. We have not explored the potential of utilizing various pivot languages to fit the language background of the learners, and we leave it for future work.

Approaches appropriate for low-resource GEC There are several definitions on low-resource settings in natural language processing. In this work, we follow the definition of low-resource as languages that only have a small amount of hand-labeled data available. However, the proposed approach, SeLex-RT, assumes the availability of high-quality machine translation systems, as well as resources required to generate synthetic errors (a spellchecker and a part-of-speech tagger). We also acknowledge the limitations of our experiments, specifically, the Morph baseline that we have implemented only uses morphological transformations, without the patterns, as in (Choe et al., 2019). One reason for this is that their approach was found effective for correcting grammar errors but not for lexical errors (White and Rozovskaya, 2020). We further hypothesize that the amount of hand-labeled data available to us is not sufficient to extract high-quality patterns, however, we leave this as future work to investigate whether adding patterns mined from hand-labeled (training or development) data is effective for correcting lexical errors.

We have selected several baselines that we consider to be appropriate for multilingual settings that require several resources, but do not require large amounts of hand-labeled data. We discuss several other methods in Section 4.1 to be more resource- or knowledge-intensive, such as the back-translation approach (Xie et al., 2018). That said, recent work on low-resource machine translation (Tan and Zhu, 2024) can train models with under 50K training examples, which is still significantly more than what is available for languages other than English as hand-labeled data. Nevertheless, we believe this is an intriguing research question that could be explored in the context of GEC, but is orthogonal to the solution we propose.

Use of reference-based evaluations for performance on lexical errors Finally, we also note that reference-based evaluation is challenging when lexical errors are concerned, and evaluation with closest golds (CGs) is expensive and not feasible on a large scale. This problem arises in several recent works that attempted to gauge the performance of large language models (LLMs) on GEC, including ChatGPT (Fang et al., 2023; Katinskaia and Yargarber, 2024). We believe this is the same issue we encountered in this work when trying to evaluate the performance on lexical mistakes. We leave this as a direction for future work.
Acknowledgments

The authors thank the annotators for their annotation work. The authors are grateful to the anonymous ARR reviewers for their insightful comments.

References

Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar Habash. 2023. Advancements in Arabic grammatical error detection and correction: An empirical investigation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6430–6448. Singapore. Association for Computational Linguistics.

Anna Alsufieva, Olsya Yatsenko Kisseliev, and Sandra G. Freels. 2012. Results 2012: Using flagship data to develop a russian learner corpus of academic writing. Russian Language Journal, 62:79–105.

Abhiject Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Parallel iterative edit models for local sequence transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics.

Maksym Bondarenko, Artem Yushko, Andrii Shportko, and Andrii Fedorych. 2023. Comparative study of models trained on synthetic data for Ukrainian grammatical error correction. In Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP). Association for Computational Linguistics.

Andrey Bout, Alexander Podolskiy, Sergey Nikolenko, and Irina Piontkovskaya. 2023. Efficient grammatical error correction via multi-task training and optimized training schedule. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Christopher Bryant, Mariano Felice, Óistein E. Andersen, and Ted Briscoe. 2019. The BEA-19 shared task on grammatical error correction. In Proceedings of the ACL Workshop on Innovative Use of NLP for Building Educational Applications (BEA-19).

Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.

Christopher Bryant and Hwee Tou Ng. 2015. How far are we from fully automatic high quality grammatical error correction? In ACL.

Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical error correction: A survey of the state of the art. Computational Linguistics, pages 643–701.

Yo Joong Choe, Jiyeon Ham, Kyubyong Park, and Yeol Yoon. 2019. A neural grammatical error correction system built on better pre-training and sequential transfer learning. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics.

Shamil Chollampatt and Hwee Tou Ng. 2018. A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction . In Proceedings of the AAAI. Association for the Advancement of Artificial Intelligence.

Leshem Choshen and Omri Abend. 2018a. Automatic metric validation for grammatical error correction. In ACL.

Leshem Choshen and Omri Abend. 2018b. Inherent biases in reference-based evaluation for grammatical error correction and text simplification. In ACL.

Daniel Dahlmeier and Hwee Tou Ng. 2011. Grammatical error correction with alternating structure optimization. In Proceedings of ACL.

Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of NAACL.

Ali Derakhshan and Elham Karimi. 2015. The interference of first language and second language acquisition. Theory and Practice in Language Studies, 5(10):2112–2117.

Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is ChatGPT a highly fluent grammatical error correction system? In arXiv preprint arXiv:2304.01746.

Mariano Felice and Ted Briscoe. 2015. Towards a standard evaluation method for grammatical error detection and correction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.

Simon Flachs, Felix Stahlberg, and Shankar Kumar. 2021. Data strategies for low-resource grammatical error correction. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 117–122. Online. Association for Computational Linguistics.

Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation. Association for Computational Linguistics.
Roman Grundkiewicz and Marcin Junczys-Dowmunt. 2019. Minimally-augmented grammatical error correction. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT).

Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In Proceedings of the ACL Workshop on Innovative Use of NLP for Building Educational Applications (BEA-19).

Jianshu Ji, Qinlong Wang, Kristina Toutanova, Yongen Gong, Steven Truong, , and Jianfeng Gao. 2017. A nested attention neural hybrid model for grammatical error correction. In ACL.

Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248–4254, Online. Association for Computational Linguistics.

Anisia Katinskaia and Roman Yangarber. 2024. GPT-3.5 for grammatical error correction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). ELRA and ICCL.

Satoru Katsumata and Mamoru Komachi. 2019. (almost) unsupervised grammatical error correction using synthetic comparable corpus. In Proceedings of the ACL Workshop on Innovative Use of NLP for Building Educational Applications (BEA-19).

Satoru Katsumata and Mamoru Komachi. 2020. Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.

Yova Kementchedjhieva and Anders Søgaard. 2023. Grammatical error correction through round-trip machine translation. In Findings of the Association for Computational Linguistics: EACL 2023. Association for Computational Linguistics.

Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1236–1242, Hong Kong, China. Association for Computational Linguistics.

Shaopeng Lai, Qingyu Zhou, Jiali Zeng, Zhongli Li, Chao Li, Yunbo Cao, and Jinsong Su. 2022. Type-driven multi-turn corrections for grammatical error correction. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland. Association for Computational Linguistics.

Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel Tetreault. 2010. Automated Grammatical Error Detection for Language Learners. Morgan and Claypool Publishers.

Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019. Corpora generation for grammatical error correction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3291–3301, Minneapolis, Minnesota. Association for Computational Linguistics.

Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag, realize: High-precision text editing. In EMNLP-IJCNLP.

Masato Mita, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, and Kentaro Inui. 2019. Cross-corpora evaluation and analysis of grammatical error correction models – is single-corpus evaluation enough? In NAACL.

Jakub Náplava and Milan Straka. 2019. Grammatical error correction in low-resource scenarios. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 346–356, Hong Kong, China. Association for Computational Linguistics.

Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In ACLIJCNLP.

Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229–234, Valencia, Spain. Association for Computational Linguistics.

Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The conll-2014 shared task on grammatical error correction. In Proceedings of CoNLL: Shared Task.

Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR ? Grammatical Error Correction: Tag, Not Rewrite . In Building Educational Applications Workshop (BEA).

Kostiantyn Omelianchuk, Andrii Liubonko, Oleksandr Skurzhanskyi, Artem Chernodub, Oleksandr Kornienko, and Igor Samokhin. 2024. Pillars of grammatical error correction: Comprehensive inspection of contemporary approaches in the era of large language models. In Proceedings of the 19th Workshop
on Innovative Use of NLP for Building Educational Applications (BEA 2024). Association for Computational Linguistics.

Frank Palma Gomez, Subhadarshi Panda, Michael Flor, and Alla Rozovskaya. 2023. Using neural machine translation for generating diverse challenging exercises for language learner. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.

Frank Palma Gomez and Alla Rozovskaya. 2024. Multi-reference benchmarks for Russian grammatical error correction. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1253–1270, St. Julian’s, Malta. Association for Computational Linguistics.

Frank Palma Gomez, Alla Rozovskaya, and Dan Roth. 2023. A low-resource approach to the grammatical error correction of ukrainian. In Proceedings of the Second Ukrainian Natural Language Processing Workshop in conjunction with EACL.

Subhadarshi Panda, Frank Palma Gomez, Michael Flor, and Alla Rozovskaya. 2022. Automatic generation of distractors for fill-in-the-blank exercises with round-trip neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. Association for Computational Linguistics.

Muhammad Reza Qorib, Alham Fikri Aji, and Hwee Tou Ng. 2024. Efficient and interpretable grammatical error correction with mixture of experts. In Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics.

Muhammad Reza Qorib and Hwee Tou Ng. 2023. System combination via quality estimation for grammatical error correction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Ella Rabinovich, Julia Watson, Barend Beekhuizen, and Suzanne Stevenson. 2019. Say anything: Automatic semantic infelicity detection in L2 english indefinite pronouns. In Proceedings CoNLL.

Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In ACL.

Alla Rozovskaya. 2022. Automatic Classification of Russian Learner Errors. In LREC.

Alla Rozovskaya and Dan Roth. 2019. Grammar error correction in morphologically-rich languages: The case of russian. In Transactions of ACL.

Alla Rozovskaya and Dan Roth. 2021. How good (really) are grammatical error correction systems? In EACL.

Alla Rozovskaya, Dan Roth, and Mark Sammons. 2017. Adapting to learner errors with minimal supervision. Computational Linguistics. To appear.

Masoud Jalili Sabet, Philipp Duffer, and Hinrich Schütze. 2020. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728.

Alexey Sorokin. 2017. Spelling correction for morphologically rich language: a case study of russian. In Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing.

Alexey Sorokin. 2022. Improved grammatical error correction by ranking elementary edits. In EMNLP.

Alexey Sorokin, Alexey Baytin, Irina Galinskaya, Elena Rykunova, and Tatiana Shavrina. 2016. SpellRuEval: the first competition on automatic spelling correction for russian. In Proceedings of the International Conference "Dialogue 2016".

Felix Stahlberg, Christopher Bryant, and Bill Byrne. 2019. Neural grammatical error correction with finite state transducers . In NAACL.

Felix Stahlberg and Shankar Kumar. 2020. Seq2Edits: Sequence transduction using span-level edit operations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.

Felix Stahlberg and Shankar Kumar. 2024. Synthetic data generation for low-resource grammatical error correction with tagged corruption models. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Mexico City, Mexico. Association for Computational Linguistics.

Xin Sun and Houfeng Wang. 2022. Adjusting the precision-recall trade-off with align-and-predict decoding for grammatical error correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics.

Oleksiy Syvokon and Mariana Romanyszyn. 2023. The UNLP 2023 shared task on grammatical error correction for Ukrainian. In Proceedings of the Second Ukrainian Natural Language Processing Workshop in conjunction with EACL.

William Tan and Kevin Zhu. 2024. Nusamt-7b: Machine translation for low-resource indonesian languages with large language models.

Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022. Ensembling and knowledge distilling of large sequence taggers for grammatical error
correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT), Lisbon, Portugal.

Viet Anh Trinh and Alla Rozovskaya. 2021. New dataset and strong baselines for the grammatical error correction of russian. In ACL Findings.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference.

Maxwell White and Alla Rozovskaya. 2020. A comparative study of synthetic data generation methods for grammatical error correction. In BEA.

Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018. Noising and denoising natural language: Diverse backtranslation for grammar correction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 619–628, New Orleans, Louisiana. Association for Computational Linguistics.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. . In NAACL.

Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. LM-Critic: Language Models for Unsupervised Grammatical Error Correction . In EMNLP.

Jingheng Ye, Yinghui Li, Yangning Li, and Hai-Tao Zheng. 2023. Mixedit: Revisiting data augmentation and beyond for grammatical error correction. In Findings of EMNLP.

Zheng Yuan and Ted Briscoe. 2016. Grammatical error correction using neural machine translation. In NAACL.

Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In NAACL.

Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, and Ming Zhou. 2020. Improving grammatical error correction with machine translation pairs. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 318–328, Online. Association for Computational Linguistics.
<table>
  <tr>
    <th>Hyperparam.</th>
    <th>Value</th>
  </tr>
  <tr>
    <td colspan="2">Model 1</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>\(5 \times 10^{-5}\)</td>
  </tr>
  <tr>
    <td>Min. learning rate</td>
    <td>\(1 \times 10^{-9}\)</td>
  </tr>
  <tr>
    <td>Init. learning rate</td>
    <td>\(1 \times 10^{-7}\)</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>Adam (0.9, 0.98)</td>
  </tr>
  <tr>
    <td>Max epochs</td>
    <td>25</td>
  </tr>
  <tr>
    <td>Label smoothing</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Max. tokens</td>
    <td>13,000</td>
  </tr>
  <tr>
    <td>Seeds</td>
    <td>1,2</td>
  </tr>
  <tr>
    <td colspan="2">Model 2</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>\(1 \times 10^{-4}\)</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>Adam</td>
  </tr>
  <tr>
    <td>Max epochs</td>
    <td>5 (20)</td>
  </tr>
  <tr>
    <td>Input/output lengths</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Seeds</td>
    <td>42 (1,42)</td>
  </tr>
</table>

Table A1: Hyperparameter settings for Model 1 and Model 2. The seeds and the number of epochs are shown separately for the pre-training and the finetuning stages (in parentheses).

<table>
  <tr>
    <th>GPU</th>
    <th>Model type</th>
  </tr>
  <tr>
    <td>A100 × 4model 1</td>
    <td>3hrs</td>
  </tr>
  <tr>
    <td>A100 × 4model 2</td>
    <td>12hrs</td>
  </tr>
</table>

Table A2: Training times per epoch for the pre-training stage (on synthetic data).

A  Training Details and Hyperparameters

Hyperparameters  Experiments are performed on four A100 32GB GPUs. Hyperparameters for model 1 (we use the Transformer (Vaswani et al., 2017), with the “Transformer (big)” settings and the parameters in Kiyono et al. (2019) for Pretrain setting) and for Model 2 are shown in Table A1. Table A2 shows training times per model and per epoch on the synthetic data (15M and 2M sentence pairs for model 1 and model 2, respectively). Finetuning on gold data is fast due to the small sizes of the finetuning sets. We use 2 seeds with each model, and report results averaged over two runs.

B  Examples of Russian Learner Errors

Table B3 shows examples of some common Russian learner errors. The top part of the table shows errors on closed-class words (prepositions, conjunctions) and mistakes in inflectional morphology. The bottom part of the table shows lexical mistakes and errors in derivational morphology.

C  Synthetic Data Generation Methods

C.1 Baseline Synthetic Data Generation Methods

Random character-level perturbations (Char)
This is a simple approach that performs insert/delete/replace operations at the character level. It was used in several GEC works (e.g., Kiyono et al. (2019); Xie et al. (2018)).

Spell-based transformations (Spell) This approach of generating synthetic data showed state-of-the-art performance in English (Bryant et al., 2019; Grundkiewicz and Junczys-Downmunt, 2019; Grundkiewicz et al., 2019), and other languages (Náplava and Straka, 2019; Flachs et al., 2021). Spell-based confusions include highly confusable words based on edit distance obtained from a dictionary available in a spellchecker. For example, for the target word “there”, the confusion set would include words such as their, there’re, here. The replacement for the target token is then chosen uniformly at random from its confusion set. Because Aspell is an open-source spellchecker, it is common to use Aspell for this purpose. In line with Grundkiewicz and Junczys-Downmunt (2019), the word error rate used (percentage of tokens perturbed) is that of 15%, and characters are perturbed in 10% of the word tokens to account for spelling mistakes. We follow Náplava and Straka (2019) for the parameter values for token replacement, deletions, and insertions.

Morphology-based transformations (Morph)
In the Morph approach (Choe et al., 2019; Flachs et al., 2021), confusion sets are formed by including all variants that belong to the morphological paradigm of the same base form. An example of a morphological paradigm in English is {“walk”, “walking”, walked”, “walks”} for the base form “walk”. In a highly-inflectional language such as Russian, inflectional paradigms exist for nouns, verbs, adjectives, pronouns, and numerals. The paradigms are quite complex, for example, a noun paradigm includes up to 12 wordforms; adjectival paradigms contain 24–26 wordforms, while verb paradigms may include up to 200 forms.

We generate confusion sets based on the output of a morphological analyzer for Russian (Sorokin et al., 2016), applied to large corpus of text that we use to generate synthetic data (see Section 5).12

12Similar to the spell-based method, 15% of tokens are modified in this approach.
<table>
  <tr>
    <th>Error type</th>
    <th>Example</th>
  </tr>
  <tr>
    <td colspan="2">Mistakes in grammar on closed-class words</td>
  </tr>
  <tr>
    <td>Prep. (ins.,del.,repl.)</td>
    <td>в “ин” → из “from, out of”</td>
  </tr>
  <tr>
    <td colspan="2">Mistakes in grammar (inflectional morphology)</td>
  </tr>
  <tr>
    <td>Noun:case</td>
    <td>специалист-ы “experts” (pl.,nom) → специалист-ам (pl.,dat.)</td>
  </tr>
  <tr>
    <td>Verb:number/gender</td>
    <td>жив-ут “live” (3rd person pl.) → жив-ет (3rd person sg.)</td>
  </tr>
  <tr>
    <td>Verb:aspect</td>
    <td>чувствовала “feel” (past, imperf.) → по-чувствовала (past, perf.)</td>
  </tr>
  <tr>
    <td>Verb:voice</td>
    <td>продолжала “continue” (past, active) → продолжала-сь (past, reflexive)</td>
  </tr>
  <tr>
    <td colspan="2">Mistakes in lexico-morphology category</td>
  </tr>
  <tr>
    <td>Deriv. morph.</td>
    <td>вдохнов-ленным “inspired” → вдохнов-енной “inspiring”</td>
  </tr>
  <tr>
    <td>Lexical choice</td>
    <td>предлагает “proposes” → утверждает “claims”</td>
  </tr>
</table>

Table B3: Some common error types in Russian learner data. Partial changes on a word are shown with a hyphen.

For Ukrainian, we follow Flachs et al. (2021) and use Unimorph.

Morph+Spell In this approach, we combine errors generated with the Morph method and those generated with the Spell method. Flachs et al. (2021) apply this combination approach by combining Aspell and Unimorph, where each method is selected with equal likelihood. Since the two methods do not perform at the same level, we found that it is best not to use equal probabilities. Instead, we use a 15% error rate followed by a 3% error rate for Morph and Spell (for Russian), and for Ukrainian this was reversed. We found these probabilities optimal for the two languages, where the stronger method is assigned a higher value.

D Experiments on Russian

D.1 Additional Results

Key results Tables D6 and D7 show detail Precision, Recall, and \( F_{0.5} \) of models trained with and without the SeLex-RT method, for Russian and Ukrainian, respectively.

Effect of SeLex-RT on performance by error type Table D10 shows key results by coarse error type in the Russian RULEC benchmark for the two models.

D.2 Evaluation with Closest Golds

Reference-based evaluation and low-coverage bias The use of too few references in GEC evaluation is known to underestimate system performance and to bias evaluation is known as low-coverage bias (Choshen and Abend, 2018b). The standard approach of evaluating GEC systems is to make use of reference-based measures, where system output is compared against a reference created by a human expert. A system is rewarded for proposing corrections that are in the reference, and penalized for proposing corrections not found in the reference. For this reason, reference-based evaluation measures tend to severely underestimate system performance (Choshen and Abend, 2018b; Mita et al., 2019; Felice and Briscoe, 2015; Bryant and Ng, 2015). When more than a single reference is available, system output is compared independently against each reference, and the best score is selected for each sentence. As a result, scores tend to increase with the number of references used (Ng et al., 2014). Using multiple references thus provides a less biased evaluation of system performance, although does not eliminate the issue entirely.

Bryant and Ng (2015) find that using more than 3 references tends to provide diminishing returns, suggesting that the use of 3 references may provide a more realistic idea of system performance (Bryant and Ng, 2015).

Evaluation with Closest Golds Although using multiple references provides a more realistic performance evaluation, the coverage bias is not entirely eliminated. Moreover, performance bias is also error-specific, as errors that allow for a larger set of possible corrections, specifically, lexical mistakes, suffer more than errors such as verb agreement where the number of correction options is limited, as the chance of matching the gold correction for a system is smaller. Indeed, Rozovskaya and Roth (2021) showed that while the performance of GEC systems is severely underestimated with standard evaluations, performance on mistakes that have more correction options (lexical and fluency) is underestimated more severely than performance on spelling and grammar errors. They introduce the concept of evaluations with closest golds (CGs), i.e. human references constructed relative to the system-corrected output, akin to post-evaluation for
<table>
  <tr>
    <th>Target token</th>
    <th>Synth. method</th>
    <th>Conf. size</th>
    <th>Confusion set candidates</th>
  </tr>
  <tr>
    <td rowspan="3">продлить “to extend”</td>
    <td>Spell</td>
    <td>10</td>
    <td>продлинить “(you) will extend”; продлит “(he/she) will extend”; пролить “to spill”; подлить “to add (fluid)”; продлённой “extended (past, partic., sg., fem., instr.)”; продлит “(it/he/she) will extend; продлил “(we) will extend”; продлит “(sg., masc.) extended”; продлены “(they) were extended”; продлен “(it/he/she) was extended”</td>
  </tr>
  <tr>
    <td>Morph</td>
    <td>48</td>
    <td></td>
  </tr>
  <tr>
    <td>SeLex-RT</td>
    <td>21</td>
    <td>продолжительность “duration”; продлит “(he/she) will extend”; увеличивать “to increase”; повышать “to raise”; продолжают “(they) are extending”; задержать “to delay”</td>
  </tr>
  <tr>
    <td rowspan="3">огромного “huge” (sg., masc., gen.)</td>
    <td>Spell</td>
    <td>6</td>
    <td>погромный “pogromous” (sg., masc., nom.); огромный “huge” (sg., masc., nom.); погромного “pogromous” (sg., masc., gen.)</td>
  </tr>
  <tr>
    <td>Morph</td>
    <td>38</td>
    <td>огромных “huge” (pl., gen.); огромным “huge” (sg., masc., instr.); огромной “huge” (sg., fem., gen.); огромнейший “very huge” (sg., fem., gen.); огромнейшим “very huge” (sg., masc., instr.); огромные “huge” (pl., masc., nom.); огромная “huge” (sg., fem., nom.)</td>
  </tr>
  <tr>
    <td>SeLex-RT</td>
    <td>115</td>
    <td>огромном “huge” (sg., masc., instr.); богатейшего “rich”; громадного “enormous”; многочисленных “numerous”; обширными “extensive”; широкого “wide”; крупными “large”; неограниченных “unlimited”; большой “big”; значительных “significant”; большинство “majority”; серьезных “serious”</td>
  </tr>
  <tr>
    <td rowspan="3">деньги “money” (pl., gen.)</td>
    <td>Spell</td>
    <td>5</td>
    <td>деньге “money” (sg., fem., dat.); деньгу “money” (sg., fem., acc.); деньги “money” (sg., fem., nom.); деньги “days” (nom.)</td>
  </tr>
  <tr>
    <td>Morph</td>
    <td>10</td>
    <td>деньгах “money” (pl., masc., prepos.); деньгам “money” (pl., masc., dat.); деньгами “money” (pl., masc., instr.); денег “money” (pl., masc., gen.)</td>
  </tr>
  <tr>
    <td>SeLex-RT</td>
    <td>596</td>
    <td>оплату “payment” (acc.); взносы “fees” (nom.); капитала “capital” (gen.); стоимости “cost” (gen.); средств “resources” (gen.); состоянии “fortune” (gen.); сумму “sum” (acc.); заработки “earnings” (nom.); долларов “dollars” (gen.); наличности “cash” (gen.)</td>
  </tr>
</table>

Table C4: Sample confusion sets generated with each synthetic method. The tokens in the confusion set are used to corrupt the native data by replacing occurrences of the target word with one of the tokens in the corresponding confusion set.

<table>
  <tr>
    <th>Original sentence</th>
    <th>Как только уходит звание , работа или деньги , уходит и сила . As soon as your title, job or money leaves, your strength leaves, as well .</th>
  </tr>
  <tr>
    <th>Char</th>
    <td>Как только уходит знание , работа или деньги , уход ит и сила .</td>
  </tr>
  <tr>
    <th>Spell</th>
    <td>Как только уходит знание , работа или деньги , уходят и сила .</td>
  </tr>
  <tr>
    <th>Morph</th>
    <td>Как только уходит знание , работа или деньги , уходящим и сила .</td>
  </tr>
  <tr>
    <th>SeLex-RT</th>
    <td>Как только уходит знание , труд или средств , уходит и мощь .</td>
  </tr>
</table>

Table C5: An example of an original well-formed sentence (top) from a monolingual Russian corpus and artificially generated erroneous sentences (bottom), using each synthetic data generation method. Modified or replaced tokens are underlined; deleted tokens are crossed out.
<table>
  <tr>
    <th rowspan="2">Synth. method</th>
    <th colspan="3">RULEC</th>
    <th colspan="3">RU-Lang8</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td colspan="7"><b>Model 1 (Baselines)</b></td>
  </tr>
  <tr>
    <td>Char</td><td>50.6</td><td>21.7</td><td>40.0</td><td>50.5</td><td>22.2</td><td>40.2</td>
  </tr>
  <tr>
    <td>Spell</td><td>66.0</td><td>31.3</td><td>54.0</td><td>63.0</td><td>31.8</td><td>52.6</td>
  </tr>
  <tr>
    <td>Morph</td><td>67.9</td><td>35.7</td><td>57.5</td><td>65.3</td><td>35.5</td><td>56.0</td>
  </tr>
  <tr>
    <td>M+S</td><td><b>68.8</b></td><td><b>41.2</b></td><td><b>60.7</b></td><td><b>68.9</b></td><td><b>43.1</b></td><td><b>61.6</b></td>
  </tr>
  <tr>
    <td>SeLex-RT</td><td>65.5</td><td>24.5</td><td>49.1</td><td>59.9</td><td>26.2</td><td>47.7</td>
  </tr>
  <tr>
    <td colspan="7"><b>Model 1 (Best baselines with SeLex-RT)</b></td>
  </tr>
  <tr>
    <td>M+SeLex-RT</td><td>68.7</td><td>41.4</td><td>60.7</td><td>66.7</td><td>42.7</td><td>59.9</td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td><td><b>70.6</b></td><td><b>41.9</b></td><td><b>62.1</b></td><td><b>69.9</b></td><td><b>44.2</b></td><td><b>62.6</b></td>
  </tr>
  <tr>
    <td colspan="7"><b>Model 2</b></td>
  </tr>
  <tr>
    <td>M+S</td><td>75.1</td><td>36.5</td><td>62.0</td><td>71.4</td><td>36.5</td><td>59.9</td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td><td><b>75.6</b></td><td><b>37.6</b></td><td><b>62.9</b></td><td><b>71.9</b></td><td><b>37.9</b></td><td><b>60.9</b></td>
  </tr>
</table>

Table D6: Key results on Russian. Best \( F_{0.5} \) and recall for each dataset and model are in bold. *Model 1* denotes models trained from scratch, while *Model 2* refers to models that finetune mT5-Base. *M+S* stands for *Morph+Spell*.

<table>
  <tr>
    <th rowspan="2">Synth. method</th>
    <th colspan="3">UA-GEC</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td colspan="4"><b>Model 1 (baselines)</b></td>
  </tr>
  <tr>
    <td>Char</td><td>57.7</td><td>47.0</td><td>55.2</td>
  </tr>
  <tr>
    <td>Spell</td><td>66.9</td><td>50.3</td><td>62.8</td>
  </tr>
  <tr>
    <td>Morph</td><td>57.5</td><td>22.4</td><td>43.8</td>
  </tr>
  <tr>
    <td>M+S</td><td><b>67.0</b></td><td><b>51.8</b></td><td><b>63.3</b></td>
  </tr>
  <tr>
    <td>SeLex-RT</td><td>61.9</td><td>45.0</td><td>57.6</td>
  </tr>
  <tr>
    <td colspan="4"><b>Model 1 (Best baselines+SeLex-RT)</b></td>
  </tr>
  <tr>
    <td>Spell+SeLex-RT</td><td><b>68.3</b></td><td><b>53.5</b></td><td><b>64.7</b></td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td><td>67.8</td><td>53.2</td><td>64.3</td>
  </tr>
  <tr>
    <td colspan="4"><b>Model 2</b></td>
  </tr>
  <tr>
    <td>Spell</td><td>73.1</td><td>42.9</td><td>64.1</td>
  </tr>
  <tr>
    <td>Spell+SeLex-RT</td><td><b>74.6</b></td><td><b>45.4</b></td><td><b>66.1</b></td>
  </tr>
</table>

Table D7: Key results on Ukrainian. Best \( F_{0.5} \) and recall for each model are in bold. *Model 1* denotes models trained from scratch, while *Model 2* refers to models that finetune mT5. *M+S* stands for *Morph+Spell*.

<table>
  <tr>
    <th rowspan="2">Synth. method</th>
    <th colspan="3">RULEC</th>
    <th colspan="3">RU-Lang8</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td colspan="7">3 configurations of the round-trip MT method</td>
  </tr>
  <tr>
    <td>MT only (orig)</td><td>37.4</td><td>56.0</td><td>40.1</td><td>40.5</td><td>56.5</td><td>42.9</td>
  </tr>
  <tr>
    <td>Morph+MT</td><td>65.4</td><td>35.8</td><td>56.1</td><td>62.4</td><td>39.0</td><td>55.7</td>
  </tr>
  <tr>
    <td>M+S+MT</td><td>65.7</td><td>39.1</td><td>57.8</td><td>68.2</td><td>42.1</td><td>60.7</td>
  </tr>
  <tr>
    <td colspan="7">This work</td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td><td><b>70.6</b></td><td><b>41.9</b></td><td><b>62.1</b></td><td><b>69.9</b></td><td><b>44.2</b></td><td><b>62.6</b></td>
  </tr>
</table>

Table D8: Comparison with full-sentence round-trip translation method in Lichtarge et al. (2019).
<table>
  <tr>
    <th rowspan="2">Synthetic method</th>
    <th colspan="3">Grammar</th>
    <th colspan="3">Orth.</th>
    <th colspan="3">Lex., morph.</th>
    <th colspan="3">Other</th>
    <th colspan="3">Total</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td colspan="15"><b>Model 1 (baselines)</b></td>
  </tr>
  <tr>
    <td>Spell</td><td>74.8</td><td>28.5</td><td>56.4</td><td>69.3</td><td>50.4</td><td>64.5</td><td>44.0</td><td>3.7</td><td>13.8</td><td>26.7</td><td>15.2</td><td>23.2</td><td>63.0</td><td>31.8</td><td>52.6</td>
  </tr>
  <tr>
    <td>Morph</td><td>70.2</td><td>39.0</td><td>60.5</td><td>71.6</td><td>53.9</td><td>67.2</td><td>27.5</td><td>1.7</td><td>6.9</td><td>27.3</td><td>10.1</td><td>20.3</td><td>65.3</td><td>35.5</td><td>56.0</td>
  </tr>
  <tr>
    <td>M+S</td><td>78.8</td><td>48.0</td><td>69.8</td><td>75.9</td><td>62.0</td><td>72.6</td><td>38.9</td><td>4.5</td><td>15.4</td><td>27.9</td><td>15.1</td><td>23.9</td><td>69.5</td><td>41.4</td><td>61.2</td>
  </tr>
  <tr>
    <td colspan="15"><b>Model 1 (baselines+SeLex-RT)</b></td>
  </tr>
  <tr>
    <td>SeLex-RT</td><td>69.0</td><td>25.2</td><td>51.2</td><td>67.2</td><td>41.3</td><td>59.7</td><td>26.6</td><td>10.5</td><td>20.3</td><td>25.2</td><td>7.5</td><td>17.1</td><td>59.9</td><td>26.2</td><td>47.7</td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td><td>80.6</td><td>48.2</td><td>71.1</td><td>76.5</td><td>62.4</td><td>73.2</td><td>41.3</td><td>14.2</td><td>29.9</td><td>30.3</td><td>14.5</td><td>24.9</td><td>70.3</td><td>44.3</td><td>62.9</td>
  </tr>
  <tr>
    <td colspan="15"><b>Model 2 (best baseline+SeLex-RT)</b></td>
  </tr>
  <tr>
    <td>M+S</td><td>79.6</td><td>41.5</td><td>67.2</td><td>73.9</td><td>52.1</td><td>68.2</td><td>58.3</td><td>6.6</td><td>22.8</td><td>33.3</td><td>10.9</td><td>23.7</td><td>71.3</td><td>36.5</td><td>59.9</td>
  </tr>
  <tr>
    <td>M+S+SeLex-RT</td><td>81.4</td><td>43.6</td><td>69.4</td><td>74.7</td><td>53.0</td><td>69.0</td><td>47.7</td><td>9.9</td><td>27.0</td><td>32.1</td><td>10.0</td><td>22.3</td><td>71.8</td><td>37.9</td><td>60.9</td>
  </tr>
</table>

Table D9: Key results by coarse error group on Russian (RU-Lang8).

<table>
  <tr>
    <th rowspan="2">Synthetic method</th>
    <th colspan="3">Grammar</th>
    <th colspan="3">Orth.</th>
    <th colspan="3">Lex., morph.</th>
    <th colspan="3">Other</th>
    <th colspan="3">Total</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td colspan="15"><b>Model 1 (baselines)</b></td>
  </tr>
  <tr>
    <td>Spell</td><td>79.1</td><td>31.3</td><td>60.6</td><td>70.4</td><td>49.9</td><td>65.1</td><td>45.2</td><td>2.4</td><td>9.9</td><td>20.8</td><td>9.1</td><td>16.6</td><td>66.0</td><td>31.3</td><td>54.0</td>
  </tr>
  <tr>
    <td>Morph</td><td>69.2</td><td>39.7</td><td>60.3</td><td>74.3</td><td>55.4</td><td>69.6</td><td>29.8</td><td>2.3</td><td>8.9</td><td>18.7</td><td>4.3</td><td>11.1</td><td>67.9</td><td>35.7</td><td>57.5</td>
  </tr>
  <tr>
    <td>M+S</td><td>76.3</td><td>47.4</td><td>68.0</td><td>73.9</td><td>60.8</td><td>70.9</td><td>40.4</td><td>3.4</td><td>12.6</td><td>15.9</td><td>6.1</td><td>12.1</td><td>68.8</td><td>41.2</td><td>60.7</td>
  </tr>
  <tr>
    <td colspan="15"><b>Model 1 (baselines+SeLex-RT)</b></td>
  </tr>
  <tr>
    <td>SeLex-RT</td><td>74.5</td><td>23.7</td><td>52.1</td><td>72.8</td><td>41.0</td><td>63.0</td><td>31.8</td><td>8.3</td><td>20.3</td><td>14.2</td><td>2.8</td><td>7.9</td><td>65.5</td><td>24.5</td><td>49.1</td>
  </tr>
  <tr>
    <td>M+S+LexRT</td><td>78.7</td><td>47.9</td><td>69.8</td><td>76.6</td><td>60.6</td><td>72.8</td><td>36.2</td><td>10.0</td><td>23.8</td><td>19.5</td><td>6.3</td><td>13.7</td><td>71.0</td><td>42.0</td><td>62.3</td>
  </tr>
  <tr>
    <td colspan="15"><b>Model 2 (best baseline+SeLex-RT)</b></td>
  </tr>
  <tr>
    <td>M+S</td><td>81.3</td><td>45.4</td><td>70.2</td><td>78.1</td><td>50.2</td><td>70.3</td><td>49.3</td><td>5.9</td><td>19.8</td><td>28.6</td><td>6.5</td><td>16.9</td><td>75.1</td><td>36.5</td><td>62.0</td>
  </tr>
  <tr>
    <td>M+S+LexRT</td><td>81.6</td><td>45.6</td><td>70.5</td><td>79.4</td><td>52.0</td><td>71.9</td><td>50.0</td><td>8.1</td><td>24.7</td><td>29.3</td><td>6.9</td><td>17.7</td><td>75.6</td><td>37.6</td><td>62.9</td>
  </tr>
</table>

Table D10: Key results by coarse error group on Russian (RULEC).
machine translation.

Similarly, for GEC, Rozovskaya and Roth (2021) generate references relative to the system outputs and not the original texts and show that automatic evaluation against reference golds (RGs), that is corrections generated relative to the original sentences, severely underestimates model performance. This is because the set of possible corrections for a given source sentence is extremely large and possibly infinite. We expect that evaluating against closest golds, i.e. corrections produced relative to the system hypothesis, would give us the most realistic evaluation of the system quality. To provide a more accurate evaluation of the SeLex-RT component, we apply this method to evaluate the contribution of the SeLex-RT approach to handle lexical mistakes. We generate CGs for the same subset of 500 sentences for 4 system outputs – Morph, Spell, and Morph+Spell, and Morph+Spell+SeLex-RT, following the same approach outlined in Rozovskaya and Roth (2021). The annotations are generated by one of the raters who contributed to the original annotation of RULEC. The annotators were compensated $25 per hour for the work. Complete results are shown in Tables D12 and D13 for RULEC and RU-Lang8, respectively. The CG annotations will be released upon paper publication.

Example comparing evaluation using reference golds and closest golds In Table D11 we show an example (in English) that shows two references (ref. 1 and ref. 2). Ref. 1 is the original reference (created relative to the source sentence) and ref. 2 is generated relative to system hypothesis.

Note that system scores (P.R., \( F_{0.5} \)) depend on the amount of overlap between a reference and system hypothesis. Evaluation wit closest golds attempts to generate a reference that is as close as possible to the system hypothesis (by producing a reference relative to system output, and not relative to the original sentence). Thus, evaluation with CGs provides a more realistic evaluation of system performance. Note that the amount of overlap (number of correct edits) is larger for ref. 2, and thus the F0.5 score against ref. 2 is higher (91.0) vs. F0.5 score against ref. 1 (50.0):

D.3 Comparison to Related Work

We compare our models with prior work using the original single-reference annotations for RULEC and RU-Lang8 in Table D14. The top segment of the table shows models trained in this work. The second segment shows models that are directly comparable to ours due to similar architectures and/or model parameters. We outperform all models in segment 2 (with the exception of a much more powerful mT5-Large pre-trained on 10M sentence pairs), often by a large margin, even when we use less synthetic data. Interestingly, a recent implementation of tagged corruption models (Stahlberg and Kumar, 2024) (S&K’24) present a similar model to ours but only obtains a result of 26.4 compared to 51.2 on RULEC. This suggests the importance of high-quality synthetic data even when using pre-trained models as a starting point in GEC.

The remaining three segments show results of previous work broken down by the amount of gold data used in training and fine-tuning. The special symbols next to each model indicate the type and amount of gold data used (explained in the table caption). Our mT5-base result is comparable to gT5 xxl (13B parameters, last table section); with mT5-large, we obtain a 2-point improvement. Our smaller seq2seq model outperforms all models of similar sizes (section 2 in the table) that also use RULEC training data. Sorokin (2022) uses ruGPT-3 and RoBERTa-large. Their model is comparable to mT5-large, in terms of parameters, but is trained on Russian data, whereas mT5 is multilingual.

E Analysis of the SeLex-RT Method

Table D15 shows examples of errors missed by the baseline models but corrected by models that use the BT component.
<table>
  <tr>
    <th>Source</th>
    <td>The settings are very realistic and the actors had a great performance .</td>
  </tr>
  <tr>
    <th>System hypo</th>
    <td>The settings are very realistic and the actors had great performance .</td>
  </tr>
  <tr>
    <th>System edits</th>
    <td>realistic → realistic; had a great → had great</td>
  </tr>
  <tr>
    <th colspan="2">Evaluation against original gold (OG)</th>
  </tr>
  <tr>
    <th>Ref. 1 (OG)</th>
    <td>The settings are very realistic and the actors gave a great performance .</td>
  </tr>
  <tr>
    <th>Gold edits (OG)</th>
    <td>(1) realistic → realistic; (2) had → gave</td>
  </tr>
  <tr>
    <th>Correct edits (OG)</th>
    <td>(1) realistic → realistic</td>
  </tr>
  <tr>
    <th>Performance against OG</th>
    <td>\( P = 50.0; R = 50.0; F_{0.5} = 50.0 \)</td>
  </tr>
  <tr>
    <th colspan="2">Evaluation against closest gold (CG)</th>
  </tr>
  <tr>
    <th>Ref. 2 (CG)</th>
    <td>The settings are very realistic and the actors had great performances .</td>
  </tr>
  <tr>
    <th>Gold edits (CG)</th>
    <td>(1) realistic → realistic; (2) had great → had a great; (3) performance → performances</td>
  </tr>
  <tr>
    <th>Performance against CG</th>
    <td>\( P = 100.0; R = 66.0; F_{0.5} = 91.0 \)</td>
  </tr>
</table>

Table D11: Evaluation with original reference (OG) and closest gold reference (CG).

<table>
  <tr>
    <th rowspan="2">Synth. data</th>
    <th rowspan="2">Ref. set</th>
    <th colspan="4">Grammar</th>
    <th colspan="4">Performance Lex.,morph.</th>
    <th colspan="4">Total</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th colspan="10">Model 1 (Baselines)</th>
  </tr>
  <tr>
    <td rowspan="2">Spell</td>
    <td>3 RGs</td>
    <td>82.0</td><td>33.1</td><td>63.3</td>
    <td>50.0</td><td>3.4</td><td>13.3</td>
    <td>75.9</td><td>33.9</td><td>60.8</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>87.1</td><td>40.0</td><td>70.5</td>
    <td>66.7</td><td>4.0</td><td>16.1</td>
    <td>85.7</td><td>43.3</td><td>71.6</td>
  </tr>
  <tr>
    <td rowspan="2">Morph</td>
    <td>3 RGs</td>
    <td>68.6</td><td>38.0</td><td>59.1</td>
    <td>36.4</td><td>3.4</td><td>12.5</td>
    <td>71.0</td><td>38.6</td><td>60.8</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>83.8</td><td>51.8</td><td>74.6</td>
    <td>90.9</td><td>8.4</td><td>30.7</td>
    <td>85.6</td><td>50.8</td><td>75.3</td>
  </tr>
  <tr>
    <td rowspan="2">M+S</td>
    <td>3 RGs</td>
    <td>75.8</td><td>45.1</td><td>66.7</td>
    <td>53.3</td><td>6.7</td><td>22.3</td>
    <td>74.0</td><td>43.9</td><td>65.0</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>85.9</td><td>57.4</td><td>78.1</td>
    <td>75.0</td><td>10.5</td><td>33.7</td>
    <td>86.5</td><td>56.9</td><td>78.3</td>
  </tr>
  <tr>
    <th colspan="10">Model 1 (M+S+SeLex-RT)</th>
  </tr>
  <tr>
    <td rowspan="2">M+S+SeLex-RT</td>
    <td>3 RGs</td>
    <td>74.9</td><td>46.6</td><td>66.8</td>
    <td>44.4</td><td>15.3</td><td>32.2</td>
    <td>72.3</td><td>44.7</td><td>64.3</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>86.6</td><td>60.2</td><td>79.6</td>
    <td>80.4</td><td>29.3</td><td>59.6</td>
    <td>87.6</td><td>58.7</td><td>79.7</td>
  </tr>
</table>

Table D12: Evaluation on RULEC with three references (RGs) and closest golds (CGs) on a 500-sentence subset for baselines (Model 1) and the model that includes the SeLex-RT component. Best overall recall and \( F_{0.5} \), as well as best values on the lexico-morph. errors are in bold.

<table>
  <tr>
    <th rowspan="2">Synth. data</th>
    <th rowspan="2">Ref. set</th>
    <th colspan="4">Grammar</th>
    <th colspan="4">Performance Lex.,morph.</th>
    <th colspan="4">Total</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
    <th>P</th><th>R</th><th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th colspan="10">Model 1 (Baselines)</th>
  </tr>
  <tr>
    <td rowspan="2">Spell</td>
    <td>3 RGs</td>
    <td>87.9</td><td>26.3</td><td>59.9</td>
    <td>57.1</td><td>4.3</td><td>16.4</td>
    <td>66.0</td><td>28.5</td><td>52.3</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>91.1</td><td>30.5</td><td>65.2</td>
    <td>66.7</td><td>5.6</td><td>20.8</td>
    <td>76.5</td><td>35.2</td><td>61.9</td>
  </tr>
  <tr>
    <td rowspan="2">Morph</td>
    <td>3 RGs</td>
    <td>73.2</td><td>35.2</td><td>60.2</td>
    <td>77.8</td><td>4.0</td><td>16.7</td>
    <td>67.8</td><td>32.8</td><td>55.9</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>82.7</td><td>43.8</td><td>70.2</td>
    <td>100.0</td><td>5.5</td><td>22.5</td>
    <td>78.9</td><td>41.5</td><td>66.8</td>
  </tr>
  <tr>
    <td rowspan="2">M+S</td>
    <td>3 RGs</td>
    <td>84.9</td><td>48.7</td><td>73.9</td>
    <td>64.3</td><td>5.0</td><td>19.0</td>
    <td>73.7</td><td>40.5</td><td>63.3</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>91.0</td><td>57.4</td><td>81.5</td>
    <td>84.6</td><td>6.7</td><td>25.5</td>
    <td>82.4</td><td>49.3</td><td>72.6</td>
  </tr>
  <tr>
    <th colspan="10">Model 1 (M+S+SeLex-RT)</th>
  </tr>
  <tr>
    <td rowspan="2">M+S+SeLex-RT</td>
    <td>3 RGs</td>
    <td>84.7</td><td>47.9</td><td>73.4</td>
    <td>42.4</td><td>15.3</td><td>31.3</td>
    <td>72.0</td><td>42.1</td><td>63.0</td>
  </tr>
  <tr>
    <td>CG</td>
    <td>91.7</td><td>56.5</td><td>81.5</td>
    <td>74.6</td><td>28.3</td><td>56.3</td>
    <td>83.4</td><td>52.4</td><td>74.6</td>
  </tr>
</table>

Table D13: Evaluation on RU-Lang8 with three references (RGs) and closest golds (CGs) on a 500-sentence subset for baselines (Model 1) and the model that includes the SeLex-RT component. Best overall recall and \( F_{0.5} \), as well as best values on the lexico-morph. errors are in bold.
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">F<sub>0.5</sub></th>
  </tr>
  <tr>
    <th></th>
    <th>RULEC</th>
    <th>RU-Lang8</th>
  </tr>
  <tr>
    <th colspan="3">This work (M+S+SeLex-RT)</th>
  </tr>
  <tr>
    <td>Model 1 (15M synth.) ★</td>
    <td>49.2</td>
    <td>51.8</td>
  </tr>
  <tr>
    <td>Model 2 (2M synth.)★</td>
    <td>51.2</td>
    <td>52.3</td>
  </tr>
  <tr>
    <th colspan="3">Previous work (similar model arch. to ours)</th>
  </tr>
  <tr>
    <td>P&R (15M synth., cf. 1) ★</td>
    <td>47.4</td>
    <td>47.7</td>
  </tr>
  <tr>
    <td>P&R (10M, mT5-Base) ★</td>
    <td>51.0</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td>P&R (10M, mT5-Large) ★</td>
    <td>53.2</td>
    <td>54.5</td>
  </tr>
  <tr>
    <td>S&K mT5-Base, 2.5M ★</td>
    <td>26.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td>S&K mT5-XXL 2.5M ★</td>
    <td>44.3</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Rothe et al. (2021) gT5 base ★</b></td>
    <td>26.2</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Náplava and Straka (2019) ★</b></td>
    <td>47.2</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Flachs et al. (2021) ★</b></td>
    <td>44.7</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Katsumata and Komachi (2020) ★</b></td>
    <td>44.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Náplava and Straka (2019) ★</b></td>
    <td>50.2</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Rothe et al. (2021) gT5 xxl ♦</b></td>
    <td>51.6</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Sorokin (2022) 'scorer-only' ♦</b></td>
    <td>53.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Sorokin (2022) 'combined' ♦</b></td>
    <td>55.0</td>
    <td>-</td>
  </tr>
</table>

Table D14: Comparison with previous work for Russian, using original references. The top segment shows models trained in this work. The second segment shows previous work with similar model architectures to ours. The remaining segments show results obtained in previous work, broken down by the amount of gold data used. Extra large models are grouped in the bottom segment. ★ refers to models that use RULEC training data for fine-tuning. * denotes models that use RULEC training and dev data for fine-tuning; ♦ denotes extra large models in terms of parameters and native data used that also use RULEC training data. S&K stands for Stahlberg and Kumar (2024). P&R stands for Palma Gomez and Rozovskaya (2024).

<table>
  <tr>
    <td>Технологическая граница значит?/означает границу сегодняшнего развития технологии.<br>“Technological boundary means?/signifies the limit of today technological development.”</td>
  </tr>
  <tr>
    <td>Мне также надо быть внимательной и почтительной и следовать?/соблюдать важные обычаи.<br>“I also need to pay attention, be respectful, and follow?/observe important traditions.”</td>
  </tr>
  <tr>
    <td>Я провела два года в качестве учителя английского языка в Казахстане , ...но я чувствую факт , что у меня мало обучения?/знаний быть преподавателем .<br>“I spent two years as a teacher of English in Kazakhstan but I feel that I do not have enough learning?/knowledge to be a teacher.”</td>
  </tr>
  <tr>
    <td>Но сегодня я понтересовалась разницей норм?/правил эвакуации между двумя авариями в Фукусиме и Чернобыле . “But today I asked about the difference in the evacuation norms?/rules for two accidents – Fukushima and Chernobyl.”</td>
  </tr>
  <tr>
    <td>Несколько?/Некоторые из их стали диверсифицировать в другие продукты...<br>“Some?/Several of them started changing into a different type of products...”</td>
  </tr>
  <tr>
    <td>В такой ситуации можно верить только простым честным людям , которые сейчас есть?/находятся рядом , а ни в коем случае не правительству...<br>“In such a situation, one can only trust simple honest people who exist?/are there for you, and neve the government...”</td>
  </tr>
</table>

Table D15: Examples of Russian lexical errors missed by standard models but corrected by models trained on data that includes the SeLex-RT-based confusions.

Reviewer paper summaries: The paper proposes a novel method for synthetic data augmentation in grammatical error correction (GEC). In addition to grammar correction, GEC also involves improving lexical fluency, which is the main focus of this paper. The authors focus on two languages: Russian (considered high-resource) and Ukrainian (considered low-resource).

The proposed method is inspired by back translation. The core idea is to use back translation to identify potentially incorrect translations for each word. This creates a mapping where each word is associated with possible incorrect translations, which is then used to introduce noise to 15% of the words in a sentence.

The authors carry out a set of experiments on Russian and Ukrainian GEC, finding that their data augmentation approach outperforms previous methods based on character and/or word corruption.
Summary of strengths: 1. The paper proposes a novel method to improve GEC, especially the lexical fluency for low-resource languages, such as Ukrainian.

2. The paper is well-motivated by the fact that current GEC augmentation methods only cover part of the issues and are more focused on grammar rather than lexical fluency.

3. The results are quite convincing. The proposed method, Selex-RT, is still beneficial when used in combination with other standard methods.

4. The paper is easy to read and follow (but the tables could be formatted much better).
Summary of weaknesses: Here I propose again the weaknesses raised in the past reviewing cycle. The authors suggested strong motivations for raised weaknesses.
I propose again for the sake of continuation of my previous review, so look at them as minor weaknesses.

1. I could not find a discussion on the computational cost of the proposed method. It looks like it is very intensive, as each sentence is  translated 10 times and each back-translation is translated 10 times (hence, 100 translations per sentence).

2. Did the authors analyze the impact of the translation systems employed to generate the mapping?

3. Did the authors analyze the usage of different pivot languages during back translation, finding out that the english is the best out of others?
Comments, suggestions and typos: I didn't have much time to take this reviewing process with the deserved focus. I appreciated the modification added by the authors, on this second review process.

Personally, I find the visualization of the tables can be improved  With respect to the previous submission now them are less dense and more readable.
Overall assessment scores: 3
Metareview score: 3
Metareview: The paper introduces SeLex-RT, a synthetic data augmentation method for Grammatical Error Correction (GEC), with a focus on low-resource languages. It leverages round-trip translation (RTT) but differs from prior RTT-based methods in that it extracts targeted lexical confusion sets rather than using full-sentence back-translations. This allows for better control over the kind of changes introduced. 
The method is evaluated on Russian (RULEC dataset) and Ukrainian (UA-GEC dataset) using an mT5-Base model, following a two-step fine-tuning procedure (synthetic data followed by natural data). The authors compare their approach against traditional synthetic data generation techniques (e.g., spelling and morphological error generation) and show competitive results.
Summary of reasons to publish: The method can be easily adapted to other languages.

The results show a consistent improvement in standard GEC metrics. 

The method proves beneficial even in combination with other earlier methods.