Training for Grammatical Error Correction Without Human-Annotated L2 Learners’ Corpora
Year: 2023

Abstract:
Grammatical error correction (GEC) is a challenging task for non-native second language (L2) learners and learning machines. Datadriven GEC learning requires as much humanannotated genuine training data as possible. However, it is difficult to produce larger-scale human-annotated data, and synthetically generated large-scale parallel training data is valuable for GEC systems. In this paper, we propose a method for rebuilding a corpus of synthetic parallel data using target sentences predicted by a GEC model to improve performance. Experimental results show that our proposed pre-training outperforms that on the original synthetic datasets. Moreover, it is also shown that our proposed training without human-annotated L2 learners’ corpora is as practical as conventional full pipeline training with both synthetic datasets and L2 learners’ corpora in terms of accuracy.

heading: 1 Introduction
text: Grammatical error correction (GEC) is one of the essential processes needed to produce sentences in a grammar-based language, and it is a challenging task for non-native second language (L2) learners and learning machines as well. Each language has its own grammar, however, data-driven language learning by a machine does not use the grammar, but corpora, more preferably, large-scale corpora. While classifiers that predict some token from candidates for a certain position in a sentence have been developed in the past (Li et al., 2019), sequence-to-sequence models have become more popular for GEC because the task is regarded as a sequence-to-sequence one and the models are flexible in editing sentences and covering various error types.
In sequence-to-sequence models, Felice et al. (2014) and Junczys-Dowmunt and Grundkiewicz (2014) treat the task as a statistical machine translation (SMT) problem and produce state-of-the-art
performance on the CoNLL2014 shared task. Neural machine translation models (Sutskever et al., 2014), which consist of an encoder and a decoder, also have been investigated to improve their capabilities. In particular, the Transformer (Vaswani et al., 2017), which is an encoder-decoder model incorporating a self-attention mechanism, has become popular and various improved versions have been investigated. One of its alternative architectures is the Copy-Augmented Transformer, which has become popular for GEC (Hotate et al., 2020).
Another modification to the Transformer architecture is altering the encoder-decoder attention mechanism in the decoder to accept and make use of additional context. For example, Kaneko et al. (2019) use the BERT representation of the input sentence as additional context for GEC. GECToR (Omelianchuk et al., 2020) employs a BERT-like pre-trained encoder stacked with a linear layer with the softmax activation function, and treats the GEC task as a token labeling problem. Addressing training data for GEC models, Kiyono et al. (2019), Grundkiewicz et al. (2019) and Choe et al. (2019) employ synthetically generated pseudo data for pre-training of GEC systems prior to fine-tuning on human-annotated corpora for the Building Educational Applications (BEA) 2019 shared task (Bryant et al., 2019).
This paper addresses the effectiveness of synthetic parallel data, which is generally used as a consequence of the insufficiency of humanannotated L2 learners’ corpora. We propose a method of substituting target sentences in synthetic parallel data with alternatives and rebuilding synthetic datasets to boost GEC training. Experiments demonstrate that pre-training on synthetic datasets rebuilt by the proposed method outperforms pretraining on the original synthetic datasets. Moreover, our synthetic datasets can be effectively employed not only to pre-train, but also to fine-tune GEC models, that is, training on synthetic data only
455
all through the pipeline. The GEC model’s training without L2 learners’ corpora is as practical as conventional training with both synthetic datasets and L2 learners’ corpora in terms of accuracy.

heading: 2 Synthetic parallel training data

heading: 2.1 Generating synthetic training data
text: Supervised machine learning requires as much genuine training data as possible, and the same is true for GEC. Training data or corpora for GEC may be created with annotations by trained native speakers of the language or by grammarians. This fact makes it difficult for us to produce larger-scale genuine data, so researchers are compelled to use limited resources to train their learning models (Bryant et al., 2019). Therefore, synthetically generated large-scale parallel training data contributes to GEC systems along with the human-annotated data.
Synthetic parallel training data consist of erroneous sentences generated by corruption models from error-free sentences. In general, the corruption models can generate unlimited versions of erroneous sentences from a given error-free one, with the ability to vary the versions in the number of errors, error types, etc. Back-translation (Sennrich et al., 2016) provides monolingual training data with synthetic source sentences that are obtained from automatically translating the target sentence into the source language for NMT. Kiyono et al. (2019) apply back-translation to GEC and achieves state-of-the-art performance on the CoNLL2014 and BEA2019 test datasets.
PIE synthetic data (Awasthi et al., 2019) is often used in state-of-the-art GEC models proposed by Omelianchuk et al. (2020); Sorokin (2022), etc. Seq2Edits (Stahlberg and Kumar, 2020) is a sequence-to-sequence transducer which consists of a Transformer encoder and decoders, and can predict span-based edit operation probabilities for GEC. Stahlberg and Kumar (2021), furthermore, propose tagged corruption models using both Seq2Edits and a finite state transducer to match the observed error type distribution of the BEA2019 dev dataset, and generate synthetic data for pretraining GEC models.

heading: 2.2 Problems in synthetic training data
text: Given some noise to an error-free (grammatically correct) sentence, a system can generate a different version of the sentence which is generally regarded
as a grammatically incorrect sentence. However, it does not always become an incorrect sentence. Table 1 shows some examples of inappropriate edits on the PIE-9M1 and the C4-200M2 synthetic datasets. The PIE model (Awasthi et al., 2019) and the tagged corruption model (Stahlberg and Kumar, 2021) each applies deletion to the source sentence, removing an adverb. In the PIE-9M synthetic dataset, the system removes the word also from the source sentence y1 to generate the erroneous sentence (Corrupted), and the edit to correct the sentence is missing also to recover from the error. However, the removed word is not necessarily required for the sentence x1 because it is an additive adverb, so the corrupted sentence x1 itself is an error-free sentence whose edit should be nooperation. The table also shows the same case in the C4-200M synthetic dataset. Note that Source is a target sentence to be outputted from a GEC model and Corrupted is a source sentence inputted to the model. The examples are cases where the original error-free sentences (Source) are inappropriate for the target sentences.
Large-scale synthetic parallel training datasets are often used to pre-train a GEC model prior to its fine-tuning on small-scale genuine datasets. The genuine datasets for the fine-tuning are annotated by trained native speakers of the language with respect to L2 learners’ mistakes because the GEC model is expected to correct L2 learners’ mistakes in text. Synthetic data for pre-training, therefore, should also match the data characteristics of L2 learners’ grammatical mistakes as shown in humanannotated datasets to be employed in the final training. The corruption mechanism produces unexpected inappropriate edits on synthetic data that differ from human errors. Finally, synthetic data, itself, is one of the key resources for building better GEC systems.

heading: 3 Erroneous synthetic data rebuilt by GEC models
text: In this section, we further examine the problem described in the previous section and propose to rebuild conventional synthetic datasets, which are often employed by researchers, in order to create effective synthetic parallel training datasets for pre-training. A trained GEC model can be
1https://github.com/awasthiabhijeet/PIE/ 2https://github.com/google-research-datasets/C4_200M-
synthetic-dataset-for-grammatical-error-correction/
represented by g(xi), where xi(= (xi1, · · · , xin)) is the ith erroneous input sentence with tokens xij(1 ≤ j ≤ n), g(xi) is the ith predicted output sentence: ỹi = (ỹi1, · · · , ỹim). We train the model g with given datasets of incorrect and correct sentence pairs: D = {(xi,yi)|i = 1, · · · , N}, where the size of D is N , so as to decrease the difference (loss) of yi between ỹi.

heading: 3.1 Process of generating synthetic data
text: Fig.1 shows a general process for generating synthetic parallel data consisting of an incorrect and correct sentence pair. The sentence yi is an errorfree sentence from a large-scale corpus such as Wikipedia, BookCorpus (Zhu et al., 2015) and the Colossal Clean Crawled Corpus (C4) (Raffel et al., 2020), and a corruption model produces some grammatical errors in the sentence yi resulting in an erroneous sentence xi. The sentences xi and yi are the input sentence to a GEC model and the sentence that should be inferred by the model, respectively. The arrow from yi to xi is a noising process to add the errors, and the reverse dotted arrow is a de-noising process to restore the erroneous sentence to the correct form. In some cases, however, the target sentence of the noisy or erroneous sentence xi should not be the unedited sentence yi, but another sentence ŷi.
The noising and de-noising processes of the corruption models, therefore, often have irreversibility, and the hypothetically correct sentence ŷi does not always match the unedited error-free sentence yi. On the other hand, the process of generating a correct sentence ŷi from the erroneous sentence xi by human annotators on genuine parallel data matches the correction process, and can create a
dataset D̂ = {(xi, ŷi)|i = 1, · · · , N} which is significantly reliable as long as the annotators do not make mistakes. Even in human-annotated data, there can be plural candidates for the correct sentence ŷi, but, the dataset D̂ is still reliable (Bryant et al., 2019).

heading: 3.2 Proposed method for rebuilding synthetic data
text: We address synthetic data for GEC models and propose a modification where hypothetical target sentences are not original unedited sentences yi, but sentences predicted from corrupted ones by a conventional GEC model. In other words, we rebuild the synthetic data D̃ = {(xi, ỹi)|i = 1, · · · , N} from D = {(xi,yi)|i = 1, · · · , N} which are usually used in pre-training of GEC models. This idea is similar to Rothe et al. (2021).
We employ a conventional GEC model g(xi) to generate hypothetical target sentences ỹi. One would expect that the predicted sentences ỹi from corrupted sentences xi by a GEC model would match the corrected sentences ŷi: ỹi ≃ ŷi for xi,
and build an appropriate synthetic dataset : D̃ ≃ D̂. The conventional GEC model we employ in this paper is GECToR (Omelianchuk et al., 2020), where the number of labels is 5, 004. GECToR has achieved state-of-the-art results on GEC, however, the version of the model we employ achieves F0.5 scores of 64.0 and 71.8 on the CoNLL 2014 and BEA 2019 test datasets, respectively. As the GEC systems, of course, are still under development by researchers, we have to compromise on the quality of synthetic data rebuilt by our proposed method. Table 1 also shows examples of hypothetical target sentences ỹi, which contain grammatical errors, generated by the GEC model.

heading: 3.3 Synthetic data rebuilt by the GEC model
text: To predict ỹi from xi we employ a newer version of the trained GECToR model3 which has a RoBERTa encoder based on the results of Omelianchuk et al. (2020) and the inference hyperparameters, confidence bias and minimum probability threshold, are set to 0.2 and 0.5, respectively. As synthetic data to be examined, we use the above-mentioned PIE9M and C4-200M in the experiments; the former is widely used for pre-training GEC models and the latter is generated by attempting to match the error type frequency distribution to the development dataset. Note that the C4-200M dataset is downsized to 9M sentences to match the size of the PIE-9M in the experiments.
Table 2 shows the fundamental statistics of the synthetic datasets rebuilt by the proposed method, compared to the original ones. The average numbers of tokens per sentence in the rebuilt datasets D̃s are not significantly different from those of the original datasets Ds. To compare statistical relationships between sentences xi and yi, we generate m2 formatted information using the ERRor ANnotation Toolkit (ERRANT)4(Bryant et al., 2017) and calculate the average number of edits per sentence. Applying the proposed method to the PIE-9M and C4-200M train datasets, the procedure reduces the average number of edits (corruptions) per sentence, resulting in about 0.8 and 2.8 fewer than the original datasets, respectively. We also indicate the dataset Ď, which has a comparable average number of edits with the dataset D̃. The erroneous sentences x̌i are generated from the corrupted sentences xi in the PIE-9M dataset by recovering edits
3https://github.com/grammarly/gector/ 4https://github.com/chrisjbryant/errant/
partly to adjust its average of edits to that of the dataset D̃. The dataset Ď is used in the experiments in the next section to prove that the effectiveness of our method does not depend on the number of edits per sentence empirically.
Stahlberg and Kumar (2021) have tried to match their synthetic data characteristics to L2 learners’ error characteristics with respect to the frequency of occurrence of the error types for the reason that the trained model is mainly expected to correct L2 learners’ sentences. We further examine whether our method can regulate the frequency of occurrence with respect to grammatical error types in the synthetic datasets to match the L2 learners’. Fig.2 shows the frequency distribution of occurrence with respect to grammatical error types in our rebuilt synthetic datasets D̃s, comparing the original synthetic datasets Ds, PIE-9M and C4-200M, and L2 learners’ corpus, the Cambridge English Write & Improve (W&I+LOCNESS) v2.15(Bryant et al., 2019; Granger, 1998). The proposed method changes the frequency of error occurrence, and we expect that the frequency distribution of D could approach that of the L2 learners’ corpus by the proposed method. Note that the L2 learners’ corpus for comparison is employed in stage III training of GEC models, which is the final fine-tuning stage in the experiments, and the corpus for the final stage of training is of utmost importance.
To investigate the similarity between two frequency distributions, we calculate KullbackLeibler (KL) divergence, which is a measure of how different two probability distributions are from each other, defined as
DKL(P ||Q) = ∑
x∈χ P (x) log
( P (x)
Q(x)
) , (1)
where P and Q are discrete probability distributions and χ is the sample space. We consider the frequency distributions as the probability distributions, and the sample space χ is 24 error types defined by ERRANT. Table 2 also shows the average level of information, i.e., entropy. The entropy measures uncertainty of the types of grammatical errors that will occur in a sentence.
Comparing each entropy value of the proposed synthetic datasets D̃s with that of their original ones Ds, the proposed method approaches the entropy of the PIE-9M synthetic data and that of the W&I LOCNESS dataset DWI , while there is no
5https://www.cl.cam.ac.uk/research/nl/bea2019st/
significant difference from the C4-200M dataset. In the PIE-9M synthetic dataset, the proposed method also approaches the frequency distribution of the types of grammatical errors to that of DWI . Regarding the C4-200M dataset, on the other hand, the proposed method moves the frequency distribution away from that of DWI , however, the two datasets rebuilt by the proposed method, D̃s, have almost the same value of KL divergence from DWI . The table also refers to the values of KL divergence from the CoNLL2014 dataset for evaluating the GEC models. Note that the CoNLL2014 dataset is small-sized and consists of 1,312 sentences.

heading: 4 Experiments
text: To empirically investigate the effectiveness of the proposed method and the capabilities of a GEC model trained on synthetic data rebuilt by the method, we train the GEC model choosing the hyperparameters described below. The GEC model is fundamentally trained through the three stage pipeline adopted in Choe et al. (2019), Omelianchuk et al. (2020), Stahlberg and Kumar (2021), etc.: stage I is a pre-training stage on a synthetic dataset, stage II is a training stage on a human-annotated dataset and stage III is a finetuning stage on a smaller human-annotated dataset more consistent with the target domain of GEC.

heading: 4.1 Training model and datasets
text: In the experiments, we employ RoBERTa (Liu et al., 2019)(roberta-base6) and train the model on the datasets indicated below. Hyperparameters in the training stage are set to the same values as on the website7 (Omelianchuk et al., 2020), and choosing a set of labels to be predicted by the model is done in the same manner as described there. We also employ three different PIE-9M and three different C-200M datasets.
Stage I (Pre-training) Either the PIE-9M or the C-200M is used in stage I as a conventional method. Each dataset consists of 9M sentence pairs, which we randomly split into two sets: 95% train and 5% dev datasets. The data splitting creates 8.42M sentence-pair synthetic parallel datasets Ds. We apply the proposed method to the above datasets Ds to create the proposed synthetic parallel datasets D̃s. We also create the dataset Ď which has a similar average number of edits per sentence by recovering some edits randomly and partially to adjust to the statistics of the proposed datasets. The statistical information for all the synthetic parallel datasets is shown in Table 2. Note that all text in the C-200M dataset is tokenized using spaCy and the en_core_web_sm model8.
6https://huggingface.co/models/ 7https://github.com/grammarly/gector/ 8https://spacy.io/
Stage II (Training) We employ L2 learners’ human-annotated corpora used in the BEA2019 shared task. The corpora consist of W&I+LOCNESS v2.1, the First Certificate in English (FCE) v.2.1 (Yannakoudakis et al., 2011), the National University of Singapore Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013) and the Lang-8 Corpus of Learner English (Lang-8) (Mizumoto et al., 2011; Tajiri et al., 2012) shown in Table 3. We split the corpora into 98% train and 2% dev datasets because they are small-sized and train data of a larger size is preferable. Table 3 shows the characteristics of each corpus and the overall corpus for stages II and III.
Stage III (Fine-tuning) We choose W&I+LOCNESS, one of the corpora in stage II, as an L2 learners’ corpus consistent with the target domain of GEC. This selection is based on Choe et al. (2019) for the restricted track and Omelianchuk et al. (2020). In addition to the L2 learners’ corpus, the synthetic dataset rebuilt by the proposed method is downsized to 34K sentence pairs for fine-tuning of the models pre-trained on the same synthetic data. The sentence pairs of the downsized synthetic dataset are chosen randomly from the 9M sentence pairs.

heading: 4.2 Results
text: We trained the GEC models on either of the original PIE-9M, C4-200M or our rebuilt synthetic datasets in stage I followed by training in a combination of stages II and III. Both stages II and III use the
L2 learners’ corpora or our rebuilt 34K synthetic datasets described in Sec. 4.1. To evaluate the performance of the trained models, we let each model correct grammatical errors in the sentences of the CoNLL2014 and BEA2019 test datasets. Note that we set the confidence bias and the minimum probability threshold to zeros for inference after stages I and II as on the website. We evaluated the performance of the models for the CoNLL2014 and BEA2019 test datasets using M2scorer9 and by submitting the corrected sentences to the server
9https://github.com/nusnlp/m2scorer/
referred to by the BEA2019 shared task website10, respectively.
Table 4 shows comparisons of GEC performance with metrics, precision (P), recall (R) and F0.5 scores for the test datasets, indicating train datasets each model used in stages I, II and III. The results for the PIE-9M synthetic dataset are summarized as follows. The baselines are the underlined results of the model trained on the conventional datasets, that is, Original+BEA2019 through stages I, II and III, resulting in F0.5 = 62.9 and F0.5 = 70.5 for the CoNLL2014 and BEA2019 test datasets, respectively. While the pre-trained Original performs F0.5 = 51.2 and F0.5 = 51.1, the pre-trained Proposed performs F0.5 = 61.2 and F0.5 = 66.7, respectively. For the partial pipeline training of stages I and III, the Original+BEA2019 performs F0.5 = 62.4 and F0.5 = 70.3, and the Proposed+BEA2019 performs F0.5 = 62.8 and F0.5 = 70.1, respectively. Proposed+PIE-34K, which was pre-trained and fune-tuned only on the rebuilt PIE-9M and PIE-34K synthetic datasets, performs F0.5 = 62.9 and F0.5 = 71.5, respectively. Proposed+C4-34K was pre-trained and finetuned only on the synthetic datasets as well, however, the training employed two different synthetic datasets, PIE and C4. For the full pipeline training of stages I, II and III, the Original+BEA2019 performs F0.5 = 62.9 and F0.5 = 70.5, and the Proposed performs F0.5 = 63.6 and F0.5 = 70.6, respectively. The results regarding the C4-200M synthetic dataset are also shown in the same manner in the figure.

heading: 5 Discussion and related work
text: This paper addresses the quality of synthetic parallel data due to the insufficiency of human-annotated L2 learners’ corpora and the effectiveness of training only on synthetic data. Note that the quality does not address grammatical correctness, but the validity of source-target sentence pairs for training and how well the data fits the characteristics of L2 learners’ mistakes. The overall results indicate that our method is more effective for the PIE-9M dataset than the C4-200M dataset, and it implies that the C4-200M dataset is of better quality.
Here, we discuss the experiments on the PIE9M dataset, which more likely needs the technique. The stage-I training by the proposed method outperforms the conventional training by 10.0 and
10https://www.cl.cam.ac.uk/research/nl/bea2019st/
15.6 with regard to F0.5 for the CoNLL2014 and BEA2019 test datasets, respectively. It results in only 1.7 and 3.8 less than the baselines, which were trained through the full pipeline, stages I, II, and III.
Furthermore, the stage-II training reduces the performance of the pre-trained models on the proposed method’s synthetic dataset. This suggests that the proposed method’s synthetic datasets could be of higher quality than the overall L2 learners’ corpora while each synthetic dataset itself could be inferior to the L2 learners’ corpora.
Unfortunately, the baseline of the training replaced with the rebuilt synthetic dataset does not improve its performance. Our synthetic datasets can be employed all through the pipeline of training, that is, training without L2 learners’ corpora. The results show that GEC model training without L2 learners’ corpora is as practical as conventional training with both L2 learners’ corpora and synthetic datasets in terms of accuracy. Note that the version of the model employed to rebuild synthetic data in the experiments achieves the scores of 64.0 and 71.8 on F0.5 for the CoNLL 2014 and BEA 2019 test datasets, respectively.
To summarize the achievements, the proposed method :
1. outperforms pre-training on the original synthetic datasets.
2. provides notably good training performance without human-annotated L2 learners’ corpora.
Trained GEC models can be used not only for predicting correct sentences but also for generating better synthetic data, and systems incorporating the proposed method are not limited to the synthetic data and model used in this paper.
Addressing training data for GEC models, Grundkiewicz and Junczys-Dowmunt (2014) introduce the WikEd Error Corpus generated from Wikipedia revision histories, corpus content and format. The corpus consists of more than 12 million sentences with a total of 14 million edits of various types. Kiyono et al. (2019), Grundkiewicz et al. (2019) and Choe et al. (2019) employ synthetically generated pseudo data for pre-training of GEC systems prior to fine-tuning on human-annotated corpora for the BEA2019 shared task(Bryant et al., 2019).
Mita et al. (2020) focus on human annotators’ errors in official datasets when they rewrite incorrect sentences to remove grammatical mistakes and denoise the target sentences of the official datasets using some trained GEC models with a perplexity criterion. Rothe et al. (2021) also apply the similar technique to the LANG-8 corpus, which is a large corpus of texts written by L2 learners with userannotated corrections, and correct human errors by the GEC models.
Our proposed method is effective not only for correcting human annotators’ errors, but also for adjusting source-target disparity to match the domain. Stahlberg and Kumar (2021) build a large synthetic pre-training dataset with error tag frequency distributions matching Seq2Edits (Stahlberg and Kumar, 2020). Parnow et al. (2021) trained a generator to generate increasingly realistic errors (in the form of token-based edit labels) and a discrimina-
tor to differentiate between artificially-generated edits and human-annotated edits. Stahlberg and Kumar (2021) propose tagged corruption models using both the Seq2Edits and a finite state transducer to match the observed error type distribution of the BEA2019 dev dataset, and generate synthetic data for pre-training GEC models. Yasunaga et al. (2021) apply BIFI algorithm (Yasunaga and Liang, 2021) and LM-Critic to synthetic data to generate better datasets for GEC. LM-Critic chooses the most likely grammatical sentence from multiple sentence candidates based on the sentence occurrence probabilities generated by a language model.

heading: 6 Conclusion
text: In this paper, we have addressed the effectiveness of synthetic parallel data and have proposed a method for rebuilding a corpus of synthetic parallel data using target sentences predicted by a GEC
model. While the original target sentences in synthetic parallel data are guaranteed to be error-free, the target sentences predicted by a GEC model contain grammatical errors because the GEC model has been developed through research and is not perfect in its performance. However, pre-training on our proposed synthetic data outperforms that on the original synthetic data, and our pre-trained GEC model showed performance only slightly lower than the conventional fine-tuned GEC model. In addition, our proposed method can provide notably good training performance without humanannotated L2 learners’ corpora.
The proposed method’s target sentences by an imperfect GEC model work better than the original error-free target sentences although the former may contain grammatical errors. The reason why this paradoxical result happens needs to be determined. In future work, we plan to investigate further reconfiguration and modification of synthetic parallel data, and fine-tune training using such data to improve the performance of GEC. Investigation of the source-target relationships on training data mentioned above should also be carried out to clarify the effectiveness of the proposed method.

heading: Acknowledgements
text: We gratefully thank Martin Chodorow at CUNY Hunter College for his valuable suggestions and feedback. Furthermore, we would like to thank the reviewers for their insightful comments. This work was supported by JSPS KAKENHI (Grant Numbers JP18K00904 and JP21K00806).

heading: Future Work
text: In future work, we plan to investigate further reconfiguration and modification of synthetic parallel data, and fine-tune training using such data to improve the performance of GEC. Investigation of the source-target relationships on training data mentioned above should also be carried out to clarify the effectiveness of the proposed method