Towards Better Utilization of Multi-Reference Training Data for Chinese Grammatical Error Correction (2024)

Abstract: For the grammatical error correction (GEC) task, there usually exist multiple correction ways for an erroneous input sentence, leading to multiple references. Observing the high proportion of multi-reference instances in Chinese GEC training data, we target a systematic study on how to better utilize multireference training data. We propose two new approaches and a simple two-stage training strategy. We compare them against previously proposed approaches, on two Chinese training datasets, i.e., Lang-8 for second language learner texts and FCGEC-Train for native speaker texts, and three test datasets. The experiments and analyses demonstrate the effectiveness of our proposed approaches and reveal interesting insights. Our code is available at https://github.com/ymliucs/MrGEC.

heading: 1 Introduction
text: Given a potentially erroneous sentence, the goal of the grammatical error correction (GEC) task is to generate an error-free sentence of the same meaning (Wang et al., 2021b; Bryant et al., 2023). Table 1 gives an example. There are two references, corresponding to two ways of correcting the sentence.
Similar to the machine translation (MT) task, GEC is a typical task for which multiple reference phenomena are ubiquitous, due to the great inherent flexibility of natural languages. For the sake of reliable evaluation, the test set is usually of multireference in both GEC (Ng et al., 2014; Bryant et al., 2019; Zhang et al., 2022a; Xu et al., 2022; Zhang et al., 2023) and MT research.
In contrast, in terms of training data, GEC differs from MT in that a large proportion of GEC training data is multi-reference. In the most widely used training data for second-language learner (L2) texts, i.e., Chinese Lang-8, about 48% sentences
∗Corresponding Author.
(274K among 566K) have multiple references.1 As a recently constructed dataset for native speaker (NS) texts, FCGEC-Train contains about 23% (8K among 36.2K) multi-reference instances.
Despite the ubiquitousness of multi-reference training instances, there lacks a systematic investigation of the question, i.e., how to better utilize multi-reference data for training GEC models. The most common way is to treat a multi-reference instance as multiple independent single-reference instances. Ye et al. (2022) propose to statically determine a single reference from multiple ones before training, and discard other references during training. They compare several ways for reference selection.
This work aims to conduct a comprehensive investigation of this issue. We compare several approaches, including two newly proposed ones, i.e., 1) averaging the loss of multiple references at the same training step, and 2) dynamically determining and using the minimum loss among multiple references. We also propose a two-stage training strategy. We conduct experiments on two Chinese training datasets, i.e., Lang-8 for L2 texts and FCGECTrain for NS texts, and three test datasets. The results and analyses reveal interesting insights.
1English Lang-8 data contains a tiny proportion of multireference instances, which is about 7% (67K among 943K) after merging duplicate input sentences (Mizumoto et al., 2012; Tajiri et al., 2012). Besides Lang-8, all other manually constructed training datasets for English are single-reference (Bryant et al., 2023). Partially due to the lack of multireference training data, our experiments avoid English but focus on Chinese.
3044

heading: 2 The Basic Seq2Seq GEC Model
text: Given an input sentence x = x1 · · ·xn, a GEC model aims to output another sentence y = y1 · · · ym that has the same meaning but is free of grammatical errors. In this work, we adopt the sequence-to-sequence (Seq2Seq) model, since it is consistently superior in performance and more flexible in handling complex errors, compared with the other mainstream model, i.e., sequence-to-edit (Seq2Edit) (Zhang et al., 2023; Bout et al., 2023).
The Seq2Seq GEC model adopts the standard encoder-decoder framework of Transformer (Vaswani et al., 2017). Following (Zhang et al., 2023), we employ the pre-trained BART (Lewis et al., 2020) as the model backbone.
Inference. At the τ -th timestamp, based on the encoded information of x and partially generated tokens y<τ , the model generates the probability distribution in an auto-aggressive manner for the next position.
pτ (t) = Transformer(x,y<τ ). (1)
where t refers to a token in the output vocabulary.
Training. Given a single-reference training dataset, D = {(x(j),y(j))}Nj=1, the model applies local cross-entropy loss to maximize pτ (yτ ) in the teacher forcing manner.

heading: 3 Two Baseline Approaches
text: Multi-reference training data. As discussed in Section 1, for GEC it is ubiquitous that an erroneous sentence may have multiple ways for correction, leading to multiple references, especially when the sentence contains more complex errors (vs. spelling errors). We denoted a multireference data as M = {(x(j),Y (j))}Nj=1, where Y (j) = {y(j)1 , . . . ,y (j) kj
} represents a set of reference sentences for x(j).
Undoubtedly, existing GEC models cannot directly utilize M as training data. Previous works propose two approaches to handle this issue.
The concatenation (CAT) approach. The CAT approach splits a multi-reference training instance into several independent singlereference instances, each corresponding to an input/reference pair. Formally, (x(j),Y (j)) becomes {(x(j),y(j)1 ), . . . , (x(j),y (j) kj
)}. During training, instances for the same x(j) may appear in different batches due to shuffling.
The minimum Levenshtein distance (MLD) approach. Intuitively, the CAT approach may make the training unstable, as the model swings among multiple references.
To mitigate the uncertainty, (Ye et al., 2022) proposes to select a single reference from multiple ones before training. After comparing several strategies, they find that it is best to select the reference that is closest to the input according to Levenshtein distance (Levenshtein, 1965), as follows.
Closeness(x,y) = −LD(x,y)
n+m . (2)
where LD(·) is the Levenshtein distance,2 and n = |x|, m = |y| is used for eliminating the sentence length factor.

heading: 4 Our Approaches
text: In this section, we propose two approaches for better utilization of multi-reference training data. The basic idea is to treat a multi-reference instance as a whole, and evaluate each reference after calculating losses of all references. Moreover, we propose a two-stage training strategy to combine the power of two approaches.

heading: 4.1 Average Training Loss (AvgL)
text: We use the average loss of all references as follows. Multiple references function as a single reference.
Loss(x(j),Y (j)) = 1
kj
∑
r
Loss(x(j),y(j)r ).
(3) Compared with CAT, AvgL allows the model to consider all references at the same time and to update parameters in a compromised fashion, which, intuitively, can alleviate the training instability issue to some extent.

heading: 4.2 Minimum Training Loss (MinL)
text: We use the minimum loss of all references as follows.
Loss(x(j),Y (j)) = min r Loss(x(j),y(j)r ). (4)
There are at least two implications for a reference to have the minimum loss. First, the model favors it among all references and will probably output it during inference. Second, minimum loss means the least parameter update. Therefore, it
2The costs of the three edit operations are 1/1/1 (delete/insert/substitute), respectively.
is reasonable to further encourage the model to converge toward the single reference, instead of swinging among several references.
MLD can be understood as a static selection strategy, in the sense one reference is selected and used all the time during training. In contrast, MinL dynamically selects one reference according to present losses, and different references may be selected at different training steps.

heading: 4.3 Two-stage Training (AvgL + MinL)
text: Our preliminary experiments show that under MinL, the model would often stick to a certain reference once the reference has the minimum loss when the instance is met for the first time. This is contrary to our expected dynamic selection process. We expect that all references can contribute to model training, and one reference wins gradually and at later stages.
Therefore, we propose a two-stage training strategy: 1) at the first S training steps, the model adopts AvgL; 2) afterward, the model adopts MinL.

heading: 5 Experiments
text: We conduct experiments on two types of datasets: L2 and NS.
L2 Data. For the L2 experiment, we utilize Lang8 (Zhao et al., 2018) as the training dataset,3 and MuCGEC (Zhang et al., 2022a) as both the validation and test dataset.
Following previous practice and based on our observation, we preprocess the L2 datasets to improve the performance of the baseline model. Details are given in Appendix A.1. Table 2 shows the statistics for the L2 datasets.
NS Data. For the NS experiment, we utilize FCGEC (Xu et al., 2022) as the training, validation, and test dataset. Meanwhile, NaSGEC-Exam (Zhang et al., 2023) serves as an additional test set in the NS experiment.
3The HSK(Cui and Zhang, 2011; Zhang and Cui, 2013) is excluded since most of the instances in it are single-reference.
We find that the NS datasets have a severe data leakage problem. There are many same or very similar (input) sentences that are contained in both the training and test datasets, especially when we use NaSGEC-Exam as the test set, since FCGEC and NaSGEC-Exam are from the same data source.
This data leakage problem introduces uncertain factors and may affect the reliability of the experiment results. Therefore, we propose a preprocessing procedure to deal with the issue. Appendix A.2 gives the details. We strongly suggest that future research can follow our practice or even improve our procedure.
Besides FCGEC and NaSGEC-Exam, our preprocessing procedure also considers NaCGEC (Ma et al., 2022), which has the same data source, to facilitate future experiments, though we have not conducted experiments on NaCGEC in this work.
Table 3 gives the statistics of the NS datasets, both before and after handling the data leakage problem.
Evaluation metrics. In all the experiments, we employ P/R/F0.5 scores calculated by ChERRANT (Zhang et al., 2022a) for evaluation.

heading: 5.1 Implementation Details
text: Our code is based on the PyTorch (Paszke et al., 2019) and Transformers (Wolf et al., 2020) frameworks. We use fnlp/bart-large-chinese and HillZhang/pseudo_native_bart_CGEC for L2 and NS experiments, respectively. For the settings of hyperparameters, please refer to Appendix B. We complete all the experiments on two Tesla V100-SXM2-32GB GPUs.

heading: 5.2 Main Results and Analyses
text: The main result is shown in Table 4. Benefiting from our preprocessing strategy, the CAT approach
achieves performance on par with NaSGEC-BART without using the HSK dataset in the L2 experiment. Besides, the CAT approach significantly outperforms NaSGEC-BART in the NS experiment, responding to the vast differences between GEC datasets sourced from L2 and NS.
Compared to the CAT approach, the MLD approach improves the precision, but the recall decreases accordingly, resulting in a drop of F0.5 score. This shows that only learning the reference closest to the input and ignoring valuable GEC knowledge in other references causes the model to become excessively cautious, which will lead to a decrease in model performance.
In comparison with the CAT approach, our AvgL + MinL approach achieves an increase of 0.69, 2.35, 1.43, and 0.82 in the F0.5 score on MuCGEC-Test, FCGEC-Dev, FCGEC-Test, and NaSGEC-Exam datasets, respectively. It’s noteworthy that although AvgL + MinL achieves the best performance as expected in the NS experiment, AvgL obtains the top performance (+1.16 compared to CAT) in the L2 experiment. This may be because Lang-8 contains many annotations with incomplete corrections. Although MinL is capable of considering multiple references, it gradually focuses on one single reference during the training process. Therefore, despite an improvement in performance over MLD, it re-
mains prone to the effects of noise. In contrast, AvgL can combine the advantages of each reference for training. Unlike Lang-8, FCGEC-Train comes from high-quality manual annotation, and the MinL model can pick out the reference with the best quality for training. We also conduct experiments on the original NS datasets. The performance of the models is much higher than those after handling the data leakage problem, which is intuitively reasonable. More importantly, the overall trends are consistent regarding the effectiveness of our proposed approach. Appendix C gives the detailed results.

heading: 5.3 Overlap and Quality Analyses
text: We extract multi-reference instances from Lang-8 and FCGEC-Train, and use MLD and MinL to select a reference per instance. The results are shown in Table 5. The references selected by the two approaches show notable differences, especially on FCGEC-Train which has fewer average references.
We conduct manual annotation on 600 instances to compare the quality of references selected by the two approaches, and the details are presented in Appendix D. The results are shown in the Table 6. In Lang-8, we find that MLD tends to select higher-quality references more frequently, whereas
FCGEC shows the opposite trend. we suggest that this discrepancy might stem from MinL’s propensity to favor higher-fluency references. As a result, the chosen references are prone to overcorrection, this phenomenon is also observed in the case study in Appendix E. Concurrently, many references only partially correct the errors in the input and are marked as “Poor”, making MLD more susceptible to noise. In contrast, MinL can dynamically learn from other references, thereby being less affected by such disturbances. In FCGEC, where references are derived from manual annotations, the quality of references selected by both approaches is generally high. Under these circumstances, MLD lacks the abundance of useful knowledge, leading to a lower recall compared to MinL. Moreover, in instances of selection inconsistency, MinL predominantly chooses references of higher quality, while MLD prefers references with minimal modifications, resulting in insufficiently thorough corrections.

heading: 6 Related Works
text: As the most closely related work with ours, Zheng et al. (2018) investigate multi-reference training for the MT task. They compare three methods, all of which are similar to the CAT method in this work. Unlike the circumstances for GEC, the proportion of multi-reference instances is extremely low in training data for MT. Therefore, they first train the model on large-scale single-reference data, and then continue training on small-scale multireference data that is actually composed of several other evaluation datasets. Another interesting contribution of their work is to generate more references given a multi-reference instance.
Lin et al. (2022) target the diverse MT task, which requires that a single model outputs multiple diverse but correct translations. They generate pseudo multi-reference training data using an offthe-shelf neural MT model, and during training adjust the loss function according to the quality of each reference and the diversity among references.
Similar to the MT task, the GEC task also involves multi-reference instances, and the GEC community has paid attention to this issue. Wang et al. (2021a) annotate YACLC, a multi-reference GEC dataset, which contains 14.6 references per input sentence on average. For a more accurate evaluation, CLEME (Ye et al., 2023) combines all references of the same input and evaluates the model performance at the chunk level.
Among these works, the question of how to better train models with multiple references has not been fully explored. The only previous work that we are aware of is Ye et al. (2022), which proposes selecting a reference from multiple references for each input sentence based on handcrafted rules during the preprocessing phase. However, this approach may not fully utilize the information contained in multiple references. In contrast, our MinL approach dynamically selects references in real time during the training phase.

heading: 7 Conclusion
text: This paper focuses on optimizing the utilization of multi-reference GEC training data. We introduce two new methods and a simple two-stage training strategy. The experiments on the L2 and NS datasets reveal that our method significantly improves the performance of the model training on multi-reference data, and outperforms previous proposed methods in the quality of selected references.

heading: 8 Limitations
text: Although the proportion of multi-reference training data is much higher for GEC than for MT, the average reference number for each instance is quite low. Most multi-reference instances have only two references to be considered during training. With more references to consider, different approaches may display greater potential in improving GEC performance. To this end, we may consider increasing the number of references for multi-reference training instances, similar to the work of Zheng et al. (2018), which we think is possible, especially for sentences having complex grammatical errors.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful feedback and constructive suggestions. We thank Yu Zhang for developing and maintaining Supar, on which we built our code.4 Our gratitude extends to Houquan Zhou for his vigilant discovery and invaluable assistance in the issue of NS data leakage, and the help in perfecting our paper.
This work was supported by the National Natural Science Foundation of China (Grant No. 62176173 and 62336006), Alibaba Group through the Alibaba Innovative Research Program, and a Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.
4https://github.com/yzhangcs/parser

heading: A Data Preprocessing
text: We merge and deduplicate the references of the same input for all L2 and NS datasets.
A.1 Preprocessing L2 Data
We remove references unchanged from the input in Lang-8 and then clean it for denoising. First, we delete samples whose reference number is inconsistent with the manual annotation. Then, we remove references that differ by more than 1.5 times in length from the input sentence following (Zhou et al., 2023). Besides, we use regular expressions to reduce punctuation errors in references. For example, we retain only one punctuation when continuous Chinese punctuations (commas, periods, etc.) appear. Please refer to our code for all cleaning techniques. Additionally, we strictly removed all input sentences that coincide with MuCGEC from Lang-8 following Zhang et al. (2022a).
A.2 Handling the Leakage Issue of NS Data
When analyzing the model performance on the NS GEC benchmarks, several peculiar cases lead us to suspect the existence of data leakage. After carefully examining the input sentences in the NaSGEC benchmarks and the training set of the FCGEC, we find many identical input sentences in both datasets. In addition to these identical input sentences, we also find numerous slightly different input sentences that contain the same erroneous portions and require the same corrections, as shown in Table 7. These cognate sentences cause data leakage issues in GEC.
After analyzing a substantial number of samples, we take 60 as the threshold of the BLEU score (after removing punctuation) to determine whether two sentences are cognate. Our statistics show that 60.8% of the sentences in NaSGEC-Exam and 50.5% of the sentences in NaCGEC-All (a concatenation of NaCGEC-Dev and NaCGEC-Test) are cognate sentences with FCGEC-Train-Filtered.
To address the data leakage issue, we propose two general strategies, which can be used alone or in combination. The first strategy is moving the cognate sentences in the test datasets to the training datasets, and the second strategy is deleting cognate sentences from the training datasets.
In this work, we use these two strategies in combination to solve the data leakage issue. Specifically, we first move the cognate sentences in NaSGEC-Exam and NaCGECAll to FCGEC-Train-Filtered, obtaining FCGECTrain-V2, NaSGEC-Exam-V2, and NaCGEC-AllV2. Next, we delete the cognate sentences in FCGEC-Dev, FCGEC-Test, NaSGEC-Exam-V2, and NaCGEC-All-V2 from FCGEC-Train-V2, obtaining the final version, FCGEC-Train-V3.
Our code for handling the data leakage problem is available in our GitHub repository, along with a detailed log of preprocessing FCGEC, NaSGEC, and NaCGEC.
We do not report the results of NaCGEC-All in this paper because its results are quite similar to those of NaSGEC-Exam in our preliminary experiments.

heading: B Hyperparameters Settings
text: We configure the warm-up steps to be 4,000 and 200 for L2 and NS experiments, respectively. For two-stage model training, the training steps S in the first stage are designated as 10,000 for L2 exper-
Dataset Sample BLEU Score
FCGEC-Train-Filtered Input 与作家不同的是，摄影家们把自己对山川、草木、城市、 乡野的感受没有倾注于笔下，而是直接聚焦于镜头。 Unlike writers, photographers pour their feelings about mountains, plants, cities and countryside do not into their writings, but focus directly on the lens.
60.4
Ref
与作家不同的是，摄影家们没有把自己对山川、草木、城 市、乡野的感受没有倾注于笔下，而是直接聚焦于镜头。 Unlike writers, photographers do not pour their feelings about mountains, plants, cities and countryside into their writings, but focus directly on the lens.
NaSGEC-Exam Input 与作家不同的是，摄影家们把自己对大自然中山川、草木、河流的独 特感受没有倾注于笔下，而是直接聚焦于镜头，用画面与读者交流。 Unlike writers, photographers pour their unique feelings about mountains, plants and rivers in nature do not into their writings, but focus directly on the lens and communicate with readers through images.
Ref
与作家不同的是，摄影家们没有把自己对大自然中山川、草木、河流的 独特感受没有倾注于笔下，而是直接聚焦于镜头，用画面与读者交流。 Unlike writers, photographers do not pour their unique feelings about mountains, plants and rivers in nature into their writings, but focus directly on the lens and communicate with readers through images.
FCGEC-Train-Filtered Input 从这件平凡的小事中，说明了一个深刻的道理。 In this ordinary little thing illustrates a profound truth.
61.7 Ref 从这件平凡的小事中，说明了一个深刻的道理。 This ordinary little thing illustrates a profound truth.
NaCGEC-All Input 从这件平凡的小事中，却说明了一个重大问题。 In this ordinary little thing illustrates a major problem.
iments and 460 for NS experiments. The learning rate is set to 3× 10−5, and the dropout rate is set to 1× 10−2. We set the gradient clip to 1.0 and our model has a max input length 1024. In addition, we use the batch size 16384, and the update frequency is set to 5. The number of total epochs is set to 60, and we use the early stopping strategy whose patience is 3 and 5 for L2 and NS experiments, respectively. We use the AdamW (Loshchilov and Hutter, 2019) with default parameters as the optimizer. In the inference stage, we use batch size 1024. The beam search strategy is conducted with a beam size of 12.

heading: C Original NS Data Experiments
text: In addition to utilizing the leakage-processed NS data, we also conduct experiments with the original NS datasets. The resultant findings, depicted in Table 8, mirror the trends observed in Section 5.2.

heading: D Manual Quality Annotation
text: When the training of the “AvgL+MinL” model is completed, we randomly sampled a total of 600 instances from Lang-8 and FCGEC-Train (300 for each) that receive different references according to the MLD and MinL metrics. Then we manually evaluate the quality of the references, in order to understand which metric can select better references.
Before the annotation process, we ensure that the annotators cannot identify which method selects each reference, and we shuffle the order of the references for all instances to maintain impartiality. Then we assigned them to two graduate students (Yumeng and Haochen) who specialized in GEC for independent annotation. The two annotators subsequently engage in discussions to resolve disagreements and reach a consensus to get the final annotation result.
At first, we have annotated 200 instances. As suggested by an anonymous reviewer, and in order to make our analysis findings more reliable, we increased the number of annotation instances from 200 to 600. Table 9 shows the annotation results for the initial 200 instances, which are consistent with the results for Table 6.

heading: E Case Study
text: The case study to compare the selection of MLD and MinL is shown in Table 10. In Lang-8, the references selected by both methods fail to correct all the errors in the input. Concurrently, MinL prefers to select more fluent references than the MLD, allowing the model to correct more errors but may lead to over-correction. This could be why MinL has a higher recall rate but a lower precision rate than MLD in Table 4. In contrast, the references for FCGEC come from manual annotations. As
shown in Table 9, most instances have at least one qualified reference, and MinL is more capable of finding the optimal reference than MLD.

heading: Future Work
text: We strongly suggest that future research can follow our practice or even improve our procedure. Besides FCGEC and NaSGEC-Exam, our preprocessing procedure also considers NaCGEC (Ma et al. , 2022), which has the same data source, to facilitate future experiments, though we have not conducted experiments on NaCGEC in this work. Table 3 gives the statistics of the NS datasets, both before and after handling the data leakage problem. Evaluation metrics. In all the experiments, we employ P/R/F0. 5 scores calculated by ChERRANT (Zhang et al. , 2022a) for evaluation