Are Pre-trained Language Models Useful for Model Ensemble in Chinese Grammatical Error Correction?
Year: 2023

Abstract:
Model ensemble has been in widespread use for Grammatical Error Correction (GEC), boosting model performance. We hypothesize that model ensemble based on the perplexity (PPL) computed by pre-trained language models (PLMs) should benefit the GEC system. To this end, we explore several ensemble strategies based on strong PLMs with four sophisticated single models. However, the performance does not improve but even gets worse after the PLM-based ensemble. This surprising result sets us doing a detailed analysis on the data and coming up with some insights on GEC. The human references of correct sentences is far from sufficient in the test data, and the gap between a correct sentence and an idiomatic one is worth our attention. Moreover, the PLM-based ensemble strategies provide an effective way to extend and improve GEC benchmark data. Our source code is available at https://github.com/JamyDon/PLMbased-CGEC-Model-Ensemble.

heading: 1 Introduction
text: Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text (Bryant et al., 2022). Nowadays, there are two mainstream GEC approaches. The first is treating GEC as a low-resource machine translation task (Yuan and Briscoe, 2016), where sequence-tosequence models like BART (Lewis et al., 2020) are used. This approach simply inputs the incorrect text to the encoder and gets the corrected result from the decoder. The second is treating GEC as a sequence tagging task, where the incorrect text is still taken as the input, but the output is edit tags (keep, delete, add, replace, etc.) for each token. After applying all the edits to the input text, the corrected result is then generated. The model used in this approach is also known as sequence-to-edit
∗ Corresponding author.
models and GECToR (Omelianchuk et al., 2020) is a typical one.
However, most researches on GEC focus on English while Chinese GEC (CGEC) has just started up. The Chinese language is different from English in many ways and its GEC is thus much harder. Instead of word inflection in many Western languages, the Chinese grammar is expressed by function words and word order, making CGEC more difficult and complex for that we can’t take word form as a handle. In addition, unlike English, we have very few datasets for training and testing CGEC, which sets us exploring training-free methods like model ensemble to further improve the performance of CGEC systems.
Because of the nature of GEC that corrections can be represented as several independent edits, model ensemble has been a popular way to improve GEC systems. In CGEC, Li et al. (2018), Liang et al. (2020) and Zhang et al. (2022) ensemble their models by majority voting on edits and achieve considerable improvement. Besides, Xie et al. (2016) adopt language models to improve neural language correction, following whom Junczys-Dowmunt et al. (2018) ensemble their GEC models using a language model probability. Today, transformer-based (Vaswani et al., 2017) Pre-trained Language Models (PLMs) have been in predominant use in NLP. However, we find few works on model ensemble using PLMs in CGEC.
In this work, we hypothesize that choosing the best ensemble output with the help of perplexity (PPL) computed by PLMs should boost the final performance of CGEC. We experiment on ensemble of four CGEC models, including two sequenceto-sequence ones and two sequence-to-edit ones. We try four ensemble strategies: traditional voting, sentence-level ensemble, edit-level ensemble, and edit-combination ensemble, the last three exploiting the power of PLMs.
To our surprise, the results of model ensemble
893
with PLMs do not exceed those of traditional voting and are even worse than most of the single models. To find out why a low PPL cannot lead to a better GEC performance, we carry out a detailed analysis on the ensemble results and get some insights on GEC:
1) In the test data, human references are insufficient, while PLM-based ensemble strategies produce valuable candidates, after being human checked, which may be considered as necessary complement to human references.
2) When facing an erroneous sentence, a human expert corrects it with the minimal effort, while PLM-based ensemble strategies generate more natural and idiomatic text, which is of great help for oversea language learners.
3) With the powerful ability, PLM-based models try to generate fluent sentences but sometimes ignore the original meaning of the source sentence, resulting in over-correction that should be addressed in future work.

heading: 2 Basic Models

heading: 2.1 Single CGEC Models
text: We implement four single models as baselines, with two seq2seq models and two seq2edit ones. All the models use the Lang-8 1 dataset for training.
Sequence to Sequence Models. The two seq2seq models are both based on BART-base-Chinese (Shao et al., 2021), and are implemented using fairseq 2(Ott et al., 2019). Besides Lang-8, the HSK data 3 is also used for training. One seq2seq model adopts the "dropout-src" strategy, where each token in input sentences is replaced with "[PAD]" with a probability of 10%. The other one is pre-trained on the synthetic data constrcted on THUCNews 4 (Sun et al., 2016) before the normal training.
Sequence to Edit Models. We apply GECToRChinese 5 (Zhang et al., 2022) as our seq2edit models, with the pre-trained Structbert-large-Chinese 6 (Wang et al., 2019) as backbone. Our two seq2edit models only differ in random seeds.

heading: 2.2 Pre-trained Language Models
text: We adopt three PLMs to carry out model ensemble.
1http://tcci.ccf.org.cn/conference/2018/taskdata.php 2https://github.com/facebookresearch/fairseq 3https://github.com/shibing624/pycorrector 4http://thuctc.thunlp.org 5https://github.com/HillZhang1999/MuCGEC 6https://huggingface.co/bayartsogt/structbert-large
BERT-base-Chinese 7. It is pre-trained on two tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). In MLM, each token has a chance of 15% to be replaced with a "[MASK]" (80%), a random word (10%), or itself (10%). Please refer to Devlin et al. (2019) for details.
MacBERT-base-Chinese 8. It is similar to BERT, but employs whole word masking, N-gram masking and similar word replacing in MLM. Besides, Sentence-Order Prediction (SOP) is exploited instead of NSP. Please refer to Cui et al. (2020) for details.
GPT2-Chinese 9. It is an unofficial Chinese version of GPT-2 (Radford et al., 2019). It employs generative pre-training, by predicting the next word in a sentence with only previous words provided.

heading: 3 Ensemble Strategy
text: With the source sentence and the outputs of four single models as the input, we present four ensemble strategies. The diagram of our PLM-based ensamble strategies is shown in Figure 1.

heading: 3.1 Traditional Voting
text: Different models vote for the final results. For each sentence, we consider edit operations suggested by no less than T models as the correct one. In our work, we experiment on T from 2 to 4. We implement the original code provided by Zhang et al. (2022) to carry out this voting strategy.

heading: 3.2 Sentence-level Ensemble
text: Using different PLMs, we compute the perplexities (PPLs) of the source sentence and the outputs of four single models. Specifically, given a sentence S = (w1, w2, ..., wn) and the probability of the word wi computed by a PLM denoted as pi, then PPL = ( ∏n i=1 1 pi )1/n. The sentence with the lowest PPL is chosen to be the final output.

heading: 3.3 Edit-level Ensemble
text: Given a source sentence S, all the edits suggested by single models constitute a candidate set A, and the number of edit spans is denoted as m. An edit span means the start-end pair of an edit’s position in the sentence. The set of all the edits (from different single models) on the i-th edit span (including
7https://huggingface.co/bert-base-chinese 8https://huggingface.co/hfl/chinese-macbert-base 9https://github.com/Morizeyao/GPT2-Chinese
"noop") is denoted as Ai. Thus, we can divide A =⋃m i=1Ai, where Ai = {eij | j = 1, 2, ..., |Ai|}, and eij means the j-th edit on the i-th edit span. For each edit span (Ai in A), we generate |Ai| new sentences, each corresponding to a single edit in Ai. Then we consult PLMs about PPLs of these new sentences and accept the edit corresponding to the sentence with the lowest PPL, which we mark as eibest. In other words, e i best is the best edit (decided by PLMs) in Ai, or on span i. With each span’s best edit, the final edit set Efinal combines these best edits, described as:
Efinal = {eibest | i ∈ {1, 2, ...,m}}, (1)
The final hypothesis sentence is then produced on the basis of Efinal.

heading: 3.4 Edit-combination Ensemble
text: One source sentence may contain more than one errors. For each sentence, this strategy applies all edit combinations to the source sentence and generates many new sentences.
To be specific, given a source sentence S, the edit candidates A are still divided as A = ⋃m i=1Ai, and then we get all possible edit-combinations by:
U = {{e1j1 , e2j2 , ..., emjm} | ji ∈ {1, 2, ..., |Ai|}}. (2)
Thus we generate ( ∏m
i=1 |Ai|) new sentences, each corresponding to an edit-combination in U . The
sentence with the lowest PPL will be accepted as the final output.
Taking the computational complexity into consideration, we only apply this strategy on sentences whose number of edit-combinations is no more than 300. Such simple sentences make up 95.15% of MuCGEC-test and 98.90% of NLPCC-test. We do nothing to the left not-so-simple sentences.

heading: 4 Experiments

heading: 4.1 Dataset and Evaluation Metrics
text: We carry out experiments on MuCGEC test data (Zhang et al., 2022) and NLPCC test data (Zhao et al., 2018). MuCGEC contains 7063 sentences and each have at most three references, but is not available at present. NLPCC contains 2000 sentences, each with one or two references, and about 1.1 references on average. We carry out analysis on NLPCC test data.
On MuCGEC, we submit the results of our systems to the public evaluation website 10. On NLPCC, we implement the tools provided by Zhang et al. (2022) to compute the P (Precision), R (Recall), and F0.5 of the output on char-level. Also, we report word-level results on NLPCC-test for reference with previous works.
10https://tianchi.aliyun.com/dataset/131328

heading: 4.2 Results
text: Table 1 shows the experimental results. The traditional voting strategy achieves the best performance, with a 44.09 F0.5 score on char level that is significantly higher than the best single model. With the threshold T increasing, the precision rises while the recall drops. When T = 3, F0.5 score reaches the peak, in line with the finding of Tarnavskyi et al. (2022).
However, the PLM-based ensemble strategies get much worse performance than the simple voting strategy, and are even lower than most of single models. In terms of precision and recall, traditional voting achieves higher precision but lower recall than single models while PLM-based strategies are on the contrary. Among three ensemble strategies, the sentence-level one performs best.
Among different PLMs, GPT2-Chinese achieves the best results in all three ensemble strategies. This may be because BERT-based models are naturally good at mask prediction rather than computing PPLs for whole sentences. Later, we base GPT2-Chinese to make further analysis.

heading: 5 Analysis and Discussion
text: We design three ensemble strategies to choose the sequence with the lowest PPL as the final output, but why does F0.5 score drop? In our work, all single models are made up of their own PLMs, which means ensembling them exploiting another PLM is just like using PLMs to judge PLMs, so the performance may benefit little. This is in line with the work of Junczys-Dowmunt et al. (2018), where pre-trained single models gain little and even have worse performance after PLM-based ensemble while other simple single models benefit a lot. Besides this, are there any other reasons?

heading: 5.1 Statistical Results
text: In order to find out the cause of the poor performance of PLM-based ensemble strategies, on NLPCC test data, we randomly select 200 samples from the results of all the three strategies along with the best single model (seq2seq-1) for comparison, and ask two graduate students to analyze the output sentences with a double-blind manner. After that, a third expert arbitrates for the inconsistency. Instructions for human annotators are shown
in Appendix A. According to human judgement, four types are summarized. Exact (E): the output is fluent and correct, in line with the reference. Good (G): the output is fluent and correct but different with the reference, which indicates that the references are not sufficient enough. Over-corrected (O): the output is fluent but doesn’t meet the original meaning of the source sentence. Wrong (W): the output has other problems that we don’t care in this work.
The result of human annotation is reported in Table 2, and some examples of G and O are shown in Table 3.

heading: 5.2 Discussion
text: The insufficiency of GEC references. In the outputs of PLM-based ensemble strategies, about 1/4 ("G") are automatically judged to be wrong according to the golden references, but indeed correct after human check. Actually, if we assume class G is also correct, the number of sentences corrected by PLM-based ensemble strategies (except edit-
level ensemble) exceeds that by seq2seq-1, the best single model.
This indicates that GEC references are not sufficient enough, even though datasets like NLPCC provide multi-references. Since artificially generating a correct sentence is much harder than judging a machine-generated sequence correct or not, continuously adding human checked results of PLMensemble systems to the references may be a good solution to improve the quality and diversity of the GEC test data.
The goal of GEC. This is a significant issue. Is it enough to just get a sentence rid of errors? Taking coding into example, can we say a piece of code "good" when all the "errors" are clear but pages of "warnings" are flashing? In "Good" samples, we compare the human references and automatically generated sentences, and find many of references are only correct but not so idiomatic. On the other hand, many output sentences of PLM-based ensemble strategies are more natural and like native speakers. If a GEC system is aimed at helping overseas students with their language learning, for example, then idiomaticity should be taken into consideration.
The over-correction of PLM-based models. About 1/10 of sentences generated in PLM-based ensemble ("O") are over-corrected, i.e., the model corrects a correct token and thus produces a wrong sentence. PLMs always choose the most fluent sentence with the lowest PPL, sometimes ignoring the original meaning of the source sentence. The over-correction of PLM-based generative models should be addressed in future work.

heading: 6 Conclusion
text: This paper introduces novel ensemble strategies for the GEC task by leveraging the power of pretrained language models (PLMs). We compare different strategies of model ensemble in CGEC. Surprisingly, PLM-based ensemble strategies do not benefit the system. This suggests that PPL and F0.5 have diverging goals. According to our analysis, the insufficiency of references in GEC remains a major problem, which should be continuously improved in future work.

heading: Acknowledgement
text: This work is supported by the National Hi-Tech RD Program of China (No.2020AAA0106600),
the National Natural Science Foundation of China (62076008) and the Key Project of Natural Science Foundation of China (61936012).
Limitations
First, we don’t use any single models without PLMs in their structures to carry out comparative experiments, even though few advanced models nowadays can get rid of PLMs. Second, because of the wrapping of fairseq, we don’t have access to all the output probabilities of the single models and thus cannot apply the strategy of using the weighted sum of single models and PLMs used in Junczys-Dowmunt et al. (2018). Third, while BERT-based PLMs are good at mask prediction, we haven’t found a strategy to make use of that capacity without being embarrassed by conditional probability. Fourth, we carry out our experiments only on Chinese.
Ethics Statement
About Scientific Artifacts. Since we focus on CGEC, all the code and tools are for the Chinese language and all data is in Chinese. All the scientific artifacts are used for GEC only. The artifacts provided by Zhang et al. (2022) are publicly available based on the Apache-2.0 license, on which we base our own codes and models.
About Computational Budget. We run all the experiments of model ensemble on an Intel® Xeon® Gold 5218 CPU. Processing times are shown in table 4.
About Reproducibility. All the experiments of model ensemble is completely reproducible when the PLMs are frozen (i.e., no matter how many times we run the experiments, the results are just the same).
About Human Annotators. Each of the annotators is paid $20 per hour, above the legal minimum wage. The instructions are shown in Appendix A.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1
3 B1. Did you cite the creators of artifacts you used? Section 4.1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Ethics Statement
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Statement
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Ethics Statement
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Ethics Statement
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix B
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics Statement
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: According to our analysis, the insufficiency of references in GEC remains a major problem, which should be continuously improved in future work