Improving Grammatical Error Correction via Contextual Data Augmentation (2024)

Abstract: Nowadays, data augmentation through synthetic data has been widely used in the field of Grammatical Error Correction (GEC) to alleviate the problem of data scarcity. However, these synthetic data are mainly used in the pre-training phase rather than the data-limited fine-tuning phase due to inconsistent error distribution and noisy labels. In this paper, we propose a synthetic data construction method based on contextual augmentation, which can ensure an efficient augmentation of the original data with a more consistent error distribution. Specifically, we combine rule-based substitution with model-based generation, using the generative model to generate a richer context for the extracted error patterns. Besides, we also propose a relabeling-based data cleaning method to mitigate the effects of noisy labels in synthetic data. Experiments on CoNLL14 and BEA19-Test show that our proposed augmentation method consistently and substantially outperforms strong baselines and achieves the state-of-the-art level with only a few synthetic data.

heading: 1 Introduction
text: Grammatical Error Correction (GEC) aims to detect and correct grammatical errors in a text (Wang et al., 2020; Bryant et al., 2022). It is a challenging task with a wide range of application scenarios, including search engines, writing assistants (Omelianchuk et al., 2020), and Automatic Speech Recognition (ASR) systems. Due to the low frequency of grammatical errors in real corpus, obtaining and annotating a certain number of high-quality GEC datasets is usually difficult and costly. Therefore, the currently available high-quality annotated GEC data is very limited (Ye et al., 2023a), making synthetic data an important research direction for the data-starved task.
*Corresponding author.
Nowadays, using synthetic data or data augmentation to improve the performance of GEC models has become a mainstream approach (Madnani et al., 2012; Grundkiewicz and Junczys-Dowmunt, 2014; Grundkiewicz et al., 2019). Common construction methods can be categorized into rule-based substitution (Awasthi et al., 2019; Choe et al., 2019) and model-based generation methods (Xie et al., 2018; Lichtarge et al., 2019; Zhou et al., 2019; Fang et al., 2023a). However, the synthetic data constructed by the above methods are mainly used in the pre-training phase to initialize a better GEC model. The data augmentation methods used for the data-limited fine-tuning phase are of great research value.
There are two main reasons why previous synthetic data cannot apply to joint training in the fine-tuning phase. (1) Inconsistent Error Distribution. The high randomness of synthetic data makes it difficult to perfectly match the distribution of a certain high-quality data, leading joint training
1 10898
to performance degradation. Rule-based substitution methods are limited by the distribution and word frequency of the unlabeled corpus. Although model-based generation methods can generate different types of grammatical errors (Stahlberg and Kumar, 2021), they still do not have stable controllability for specific errors with a small amount of synthetic data. As shown in Figure 1, the distribution of our proposed augmentation method is most consistent with the original dataset. (2) Noisy Label. Synthetic data is not human-labeled and cannot avoid introducing some mislabeling (inappropriate substitution or ungrammatical generation). For example, "I think you are right" may be incorrectly annotated as "I think that you are right" in synthetic data. As a text generation task with token-level metrics, the GEC task is very sensitive to this type of noise. Directly joint training of synthetic and real data brings serious performance degradation (Zhang et al., 2019). Recently, Ye et al. (2023a) propose the MixEdit framework for grammatical error augmentation of the fine-tuning stage through pattern replacement. But it is still suffering from the two problems mentioned above.
In this paper, we propose a high-quality synthetic data construction method for the fine-tuning phase based on contextual augmentation. It can be viewed as a combination of a rule-based substitution approach and a model-based generation approach, where the model is utilized to generate a rich context for the extracted error patterns. An example of the augmentation data is shown in Table 1. Specifically, we first extract the error patterns (containing correct and incorrect token pairs) present
in the real corpus through a GEC tool (ERRANT) and construct a corresponding error pattern pool. After that, we sample error patterns from the pool based on the true frequency of the original dataset to regenerate contexts for them. We regard this process as a hard constraint generation task, allowing the model to generate contextual sentences containing the correct pattern, and then obtaining the wrong sentence by rule substitution. We attempt both GPT2 (Radford et al., 2019) supervised generation and LLaMA2-7b-chat (Touvron et al., 2023) few-shot generation for our experiments. Finally, we use the baseline GEC model to relabel the synthetic data for joint training to mitigate the noise in the synthetic data.
The main contributions of this paper can be summarized as follows:
• We propose a synthetic data construction method based on contextual augmentation, which can stably generate a rich context for specific grammatical errors.
• To mitigate the effect of noisy labels, we introduce the re-labeling method into the synthetic data which improves the performance of the GEC model in joint training.
• Experiments show that our approach effectively enhances the robustness and performance of the GEC model by augmenting the high-quality annotated data in the fine-tuning phase.
We will release our code and model on github1. 1https://github.com/wyxstriker/CDA4GEC
2

heading: 2 Method
text: The main flow of our proposed synthetic data construction method based on context augmentation is illustrated in Figure 2. First, we generate the synthetic data with contextual augmentation according to Section 2.1’s method. After denoising by relabeling (Section 2.2), we use the synthetic data to augment the original data in the joint training (Section 2.3).

heading: 2.1 Pattern-based Context Generation
text: Both rule-based substitution and model-based generation methods generate synthetic data that require large amounts of data (in the millions) to guarantee a wide range of errors (Kiyono et al., 2019). However, in the supervised fine-tuning phase, the amount of data is very limited (W&I+L only includes about 30k), and millions of synthetic data for joint training is unrealistic. A more stable augmentation approach is needed to ensure that the original high-quality errors are adequately trained.
The main motivation for our proposed context augmentation is to leverage the modeling capability of language models to generate rich contexts
for specific high-quality grammatical errors. Compared with other synthesis methods, we ensure that the error distribution is consistent with the original dataset through rule-based error patterns and the diversity of sample contexts through model-based generation. The synthesis method can be divided into three steps: building the error pattern pool (Section 2.1.1), synthesizing the contextual corpus (Section 2.1.2), and substituting 2.1.3 to get the parallel corpus.

heading: 2.1.1 Error Patterns Extraction
text: We follow Choe et al.’s (2019) setting and use the parsing tool ERRANT2 to extract the editing operations present in the parallel corpus as error patterns according to the rules. We also extracted error patterns of different lengths for our experiments to ensure that the synthetic data contains more realistic errors. Excessively long patterns will make it difficult to match them in unlabeled text with the original rule-based substitution method, but the contextual augmentation-based generation method can solve this problem well.
We finally extract the patterns for each human2https://github.com/chrisjbryant/errant
3
labeled GEC dataset, and merge the corresponding patterns into an error pattern pool for the construction of synthetic data. The statistics of extracted patterns can be found in Appendix A.

heading: 2.1.2 Contextual Corpus Generation
text: With the error pattern pool and the frequency of the corresponding errors, we can simply obtain a set of pattern datasets with the same distribution as the annotated corpus by sampling. The goal of the generator model can be viewed as a hard-constrained text generation task (Welleck et al., 2019), generating a context that fully contains the target pattern. Considering that existing pre-trained models are trained on grammatically correct corpora, we only generate the corresponding contextual corpus based on the correct patterns and subsequently construct the parallel corpus by rule-based substitution.
In particular, the input to the model will be a combination of several randomly sampled patterns, which can be formulated as:
Patterninput = Pattern1 [M ] Pattern2 (1)
where Patterni represents the correct pattern that was sampled and [M ] represents the context placeholder that needs to be generated. It should be noted that the number of patterns for each sample will be randomly selected between 1 and 2, and
Equation 1 represents the case of 2 patterns only. The output of the model is a piece of text containing the corresponding input pattern.
In this paper, we experiment with two models as context generators, GPT2 and LLaMA2, representing the two settings of supervised fine-tuning and few-shot generation, respectively.
Finetuning for GPT2 We choose GPT2 as the backbone network to represent the performance of the fine-tuned generative model in the contextual augmentation task. The model generates the target corpus directly from the provided pattern, which can be formulated as:
S = Patterninput <sep> Sentencetarget (2)
where Patterninput is the combination of the patterns mentioned in Equation 1 and Sentencetarget is the target corpus containing all the patterns. <sep> is the special token dividing the input S into two parts.
For the training phase, we use the autoregressive way consistent with the pre-training:
L = j∑
k=i
−log(P (tk|t0t1...tk−1; θ)) (3)
where θ is the set of parameters of the language model, i and j represent the start and the end index of Sentencetarget, and ti represents the i-th token in the model input S like Equation 2.
As for the training data, in order to ensure that the style of the generated text is consistent with the training data, we directly adopt the correct sentences in the non-native speaker GEC dataset CLang8 (Rothe et al., 2021) dataset as the target sentences to construct the training set. Specifically, we randomly replace multiple consecutive text segments in the corpus with [M ] label and train the model to generate the corresponding context based on the remaining text segments (error patterns during inference).
Few-shot generating for LLaMA2 Recently, LLMs (Brown et al., 2020; Wei et al., 2021; Touvron et al., 2023) have presented powerful incontext learning capabilities to accomplish complex NLP tasks based on a few example samples. Therefore we also try to use the LLM to generate a more appropriate context for the error patterns. We directly use the prompt with the 5-shot setting and ask the LLM to generate the conditional corpus. Details of the prompts and input format can
4
be found in Appendix C. In addition, the GEC corpus is usually large, and considering the cost issue, we choose the open-source LLM LLaMA2-7b-chat (Touvron et al., 2023) for our experiments.

heading: 2.1.3 Pattern Substitution
text: Given the contextual corpus and error pattern pairs, we can obtain the corresponding GEC parallel corpus by simple substitution (Choe et al., 2019). To ensure the diversity of the corpus, we choose the sampling decoding strategy during generation, and we directly ignore the patterns that can’t be matched exactly. Besides, we only substitute the error patterns for 50% of the synthetic data to ensure a consistent error rate with the annotated datasets during the joint training process.

heading: 2.2 Synthetic Data Denoising
text: Compared to human-annotated data, synthetic data is more accessible but inevitably noisy. Previous work (Zhang et al., 2019) has proven that direct joint training of synthetic and real data affects the metrics of the final model. Improper substitutions (grammatically correct both before and after substitution) are the main cause of noisy synthetic data. Yasunaga et al. (2021); Cao et al. (2023) propose some sentence-level filtering methods based on scores such as PPL, but the filtering granularity and accuracy are not sufficient in the joint training setting. We need an efficient way of filtering at the token level.
Inspired by Rothe et al.; Ye et al.’s (2021; 2023b) distillation method, which mitigates the effects of noisy data by relabeling the corpus with a powerful GEC model, we also want to denoise the corpus through relabeling. Specifically, we view the synthetic data as an unlabeled grammatical error-filled corpus and relabel it using a strong baseline model. Since the synthetic data is obtained by augmentation using the original dataset, relabeling using the original model effectively removes the noise while correcting most of the grammatical errors.

heading: 2.3 Joint Training Process
text: To obtain a strong baseline model, we follow Bout et al.’s (2023) approach of using three stages for training. Our proposed data augmentation method will also be applied to the stages of fine-tuning.
As shown in Figure 3, we divide the available dataset into three stages for training. We use C4200M (Stahlberg and Kumar, 2021) dataset for the pre-training phase (stage I), which generate
grammatical errors with type distribution consistent with BEA-Dev (Bryant et al., 2019) based on the seq2edit model. For Stage II, we used the complete available annotated dataset (see Table 2 for details) to fine-tune the model, including Lang-8, NUCLE, FCE, and W&I+L. As the highest-quality annotated GEC dataset, we individually fine-tune the stage III on W&I+L.
Due to data distribution and quality, previously synthetic data are mainly utilized in the pre-training phase (stage I). In contrast, our proposed context augmentation approach is mainly used to adapt a small amount of high-quality fine-tuned data (stage II&III). We generate different amounts of synthetic data (StageII-syn and StageIII-syn in Table 2) for different stages using the corresponding error pattern pool. After that, we directly train the synthesized data jointly with the real data of fine-tuning stages, as shown in Figure 3.

heading: 3 Experiment

heading: 3.1 Setting
text: Datasets In Table 2, we summarize the statistical information for the all relevant datasets. The C4200M (Stahlberg and Kumar, 2021) dataset used for stage I is synthetic data based on Seq2Edit (Stahlberg and Kumar, 2020) model generation. For the other stages, we use the following common GEC datasets: Lang-8 Corpus of Learner English (Lang-8) (Mizumoto et al., 2011; Tajiri et al., 2012) collects from non-native speaker online learning websites Lang-83; National University
3https://lang-8.com/
5
of Singapore Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013) consists of essays written by undergraduate students on a variety of topics and annotated by professional English teachers; First Certificate in English (FCE) (Yannakoudakis et al., 2011) primarily contains answers to the upper-intermediate level exams written by English language learners; Write & Improve + LOCNESS Corpus (W&I+L) (Bryant et al., 2019) includes two parts of data. The Write & Improve dataset consists of chunks of text (articles, letters, etc.) submitted to the W&I system written by English learner; In contrast, LOCNESS consists of essays written by native English-speaking students and is used for evaluation purposes only.
In addition to the existing publicly available datasets, we constructed synthetic data for different stages (corresponding to StageX-Syn in the Table 2) by contextual augmentation using the proposed method. As for the amount of synthetic data, we heuristically choose 2M pairs for Stage II, 200,000 pairs for Stage III. We perform ablation experiments on the amount of augmented data in
subsequent analyses.
Evaluation We use BEA19-Dev (Bryant et al., 2019) as a validation set to evaluate the performance of the GEC model. In the main experiments, we report results of Conll14-Test (Ng et al., 2014) using the official M2 scorer (Dahlmeier and Ng, 2012), and results of BEA19-Test (Bryant et al., 2019) using ERRANT (Bryant et al., 2017) on the online platform4.
Training Details As described in Section 2.1.2, we use GPT2-base and LLaMA2-7b-chat as context generators for our experiments. For the implementation of the baseline GEC model, we refer to previous setups (Zhang et al., 2022) and use Seq2Seq-based BART-large (Lewis et al., 2019) for the experiments, which is trained using the Fairseq5 framework. More details on hyperparameters can be found in Appendix B.
4https://codalab.lisn.upsaclay.fr/competitions/4057 5https://github.com/facebookresearch/fairseq
6

heading: 3.2 Baseline Approaches
text: We select several recent state-of-the-art methods as baselines for comparison. GECToR (Omelianchuk et al., 2020) is an efficient auto-encoder grammatical error correction model, which corrects errors by predicting edit tags. Rothe et al. (2021) verify the performance of T5 (Raffel et al., 2020) models of various scales (from small to xxl) on the GEC task. SynGEC (Zhang et al., 2022) incorporates the syntactic information of the text into the model using GCN. TemplateGEC (Li et al., 2023) fuses the seq2edit and seq2seq models to provide a new twostage framework for error detection and correction. Ye et al. (2023a) propose a data augmentation approach MixEdit that strategically and dynamically augments realistic data, without requiring extra monolingual corpora. Bout et al. (2023) propose a multi-task pre-training method and optimization strategy, which greatly improved the performance of the GEC model.
In this paper, we use the three-stage training model initialized by the C4200M datasets as strong baselines. With Contextual Data Augmentation (CDA), we integrate contextually augmented synthetic data for training in the fine-tuning phase, as shown in Figure 3.

heading: 3.3 Main Experimental Results
text: The experimental results of our proposed augmentation method on CoNLL14 and BEA19 are shown in Table 3. We obtain a strong baseline model by training in three stages according to the Bout et al.’s (2023) setting. The results show that contextual data augmentation can effectively improve the robustness and generalization of the original model, and bring significant improvements on both CoNLL14 and BEA19-Test datasets. Our 400M BART model achieves the state-of-the-art through contextual data augmentation, and is comparable to the 11B T5-XXL. In addition to this, we find that the impact of the augmented data’s noisy labels can be well mitigated by simple relabeling. It should be noted that the proposed method improves the modeling precision with a slight loss in recall, which is encouraged in GEC tasks since ignoring an error is not as bad as proposing a wrong correction (Ng et al., 2014).
In addition to this, we have analysed the impact of the proposed data augmentation approach on the model at different stages. As shown in Table 4, the enhancement of model effectiveness by contextual
data augmentation is more pronounced in the second stage where the quality of annotated data is relatively low.

heading: 4 Analysis

heading: 4.1 Impact of Different Generators
text: In this article, we have experiment with two generator settings, GPT2 fine-tuning, and LLaMA2 ICL, to generate synthetic data. The GPT2 fine-tuning model is relatively small, which has a faster generation efficiency and follows the task requirements better after training. On the contrary, LLaMA2 has a larger number of parameters and generates more diverse and fluent texts. But the model generates more slowly and follows the instructions more weakly.
To verify the effect of the different generators on the quality of the generated text, we use them to generate 200k synthetic data on the high-quality text of stage III respectively for joint training. The experiment results are shown in Table 4. We find similar conclusions to Xu et al. (2023), that whether the synthetic data generated by the finetuned model or the LLM in-context learning has little effect on the final model performance. So we mainly use GPT2 fine-tuning as the generator for experiments for efficiency considerations.

heading: 4.2 Impact of Pattern Length
text: Unlike previous rule-based substitution approaches (Choe et al., 2019), our proposed method is not restricted to the distribution of the unlabeled corpus, so the length of the error pattern is no longer limited to the token level. To obtain the optimal pattern length, we experiment with contextual augmentation using 1-gram, 3-gram, and 5-gram patterns as shown in Table 1. The results are shown in Table 4, the model has the best performance with syn-
7
thetic data generated by the 3-gram pattern. We hypothesize that appropriately long patterns make the synthetic data distribution closer to the source data, indirectly improving the quality of the relabeling.

heading: 4.3 Impact of Data Mixing Ratio
text: In joint training, the impact of the ratio of synthetic to real data is significant. We conduct joint training experiments with different amounts of synthetic data, and the results are shown in Figure 5. Consistent with our analysis, the ratio of synthetic data to real data is a trade-off. As the amount of synthetic data increases, the precision of the system gradually increases. At the same time, the proportion of higher-quality real data has declined, possibly leading to some decline in error recall.

heading: 4.4 Analysis of Error Categories
text: To evaluate the specific advantages of the proposed data augmentation approach, we have conducted experiments on different error categories of error correction. As shown in Table 5, the CDA method provides a richer context for low-frequency errors, allowing the model to achieve great improvements in this type of error. In contrast, high-frequency errors themselves have a large amount of real training data, and the addition of synthetic data may have a slight impact on the original results such as orthographic (ORTH) and punctuation (PUNCT). This conclusion is consistent with Rothe et al. (2021), indicating that the distillation model does not handle these two types of errors well.

heading: 4.5 Quality Assessment of Synthetic Data
text: To compare the quality of our proposed context augmentation with other synthetic data, we adopt the same three-stage training and denoising for synthetic data of different construction methods. We choose PIE (Awasthi et al., 2019) and C4200M (Stahlberg and Kumar, 2021) as our baselines, which represent the rule-based substitution method and the model-based generation method, respectively. The results are shown in Table 6, where our proposed CDA method is able to better fit the distribution of the original dataset within the limited data, thus improving the model performance in joint training.

heading: 5 Related Work

heading: 5.1 Synthetic Data for GEC Task
text: Construction of Synthetic Data As a datastarved field, synthetic data has been proven effective in improving the GEC systems (Kiyono et al., 2019), which can be categorized into rule-based
8
substitution and model-based generation methods. Rule-based methods are mainly constructed through direct noise addition (Xu et al., 2019; Zhou et al., 2019; Kiyono et al., 2020), pattern substitution (Choe et al., 2019), and parsing tools (Grundkiewicz et al., 2019). Model-based generation methods mainly include back-translation (Xie et al., 2018; Stahlberg and Kumar, 2021) and round-trip translation (Zhou et al., 2019) based on Seq2Seq architecture. (Stahlberg and Kumar, 2021) propose a synthesis method based on the Seq2Edit (Stahlberg and Kumar, 2020) architecture capable of generating synthetic data by specifying grammatical error types.
Utilization of Synthetic Data Zhang et al. (2019) have explored the use of synthetic data through detailed experiments. The experiments prove that using synthetic data in the pre-training phase achieves optimal results. Some unsupervised grammatical error correction work (Yasunaga et al., 2021; Cao et al., 2023) have used the self-training framework to label and co-train unlabeled error corpus to obtain an improvement in effectiveness.

heading: 5.2 LLM for GEC Task
text: LLMs (Brown et al., 2020; Wei et al., 2021; Touvron et al., 2023) have made significant improvements in a wide range of natural language processing tasks. However, LLMs do not perform well on common benchmarks (Coyne et al., 2023) due to traditional evaluation metrics and over-corrections. Although Fang et al. (2023b) have demonstrated that some improvement is achieved by few-shot and chain-of-thought settings, there is still a big gap between LLMs and traditional fine-tuning models. In view of this, using LLM to construct synthetic data for the GEC task (Fan et al., 2023) can be considered as another feasible direction.

heading: 6 Conclusion
text: In this paper, we propose a synthetic data construction method based on contextual augmentation. It stably augments the context of the source data and ensures a consistent error distribution. Previous methods suffer from noisy labels of synthetic data. We significantly improve the performance of synthetic data in joint training through a re-labelingbased denoising method. We validate the effectiveness of our proposed method on several common datasets.
Limitations
Firstly, compared to other current synthetic data construction methods, generating synthetic data based on contextual augmentation takes more time and resources. For each sample, we need to complete the inference process on both sides of context generation and denoising. Secondly, the repredicted results are not completely correct and can only alleviate noise. There are still a small number of incorrect labels in the synthetic data. In addition, it should be noted that the context augmentation method we proposed can only provide richer context for errors existing in the annotated dataset, and cannot introduce new grammatical errors. We will focus on investigating how to better generate high-quality synthetic data that contains a wider variety of grammatical errors utilizing LLMs in our future work.
Ethics Statement
In this paper, we explore the application of contextual augmentation-based synthetic data on the GEC task. The source data for these methods come exclusively from publicly available project resources on legitimate websites and do not involve any sensitive information. In addition, all baselines and datasets used in our experiments are also publicly available, and we have acknowledged the corresponding authors by citing their work.

heading: Acknowledgements
text: We gratefully acknowledge the support of the National Natural Science Foundation of China (NSFC) via grant 62236004 and 62206078.

heading: A Error Pattern Information
text: We have extracted the error patterns from the existing annotated dataset using the ERRANT tool (Bryant et al., 2017), as described in Section 2.1.1. The number of patterns for each dataset is shown in Table 7. We maintain a separate error pattern pool for each dataset, from which we sample each time to generate synthetic data.

heading: B Hyper-parameters
text: We illustrate the hyper-parameters during training of the baseline model (see Table 8 for details) and the GPT2-based generative model (see Table 9 for details) here.
C Instruction Format for Synthetic Data Generation
When using LLaMA2-7b-chat for synthetic data generation, we use the 5-shot setting to generate the corresponding context for the sampled error patterns. We have given an example to illustrate the specific input format as shown in Table 10.
12
13

heading: Future Work
text: We will focus on investigating how to better generate high-quality synthetic data that contains a wider variety of grammatical errors utilizing LLMs in our future work