SlowGuess/NAACL_2022_OCR huggingface dataset:
Yue Zhang $^{1\dagger}$ , Zhenghua Li $^{1*}$ , Zuyi Bao $^{2}$ , Jiacheng Li $^{1}$ , Bo Zhang $^{2}$ , Chen Li $^{2}$ , Fei Huang $^{2}$ , Min Zhang $^{1}$
$^{1}$ Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China;  $^{2}$ DAMO Academy, Alibaba Group, China  
 $^{1}\{yzhang21, jcli20\} @stu.suda.edu.cn, ^{1}\{zhli13, minzhang\} @suda.edu.cn$ $^{2}\{zuyi.bzy, klayzhang.zb, puji.1c, f.huang\} @alibaba-inc.com$
This paper presents MuCGEC, a multi-reference multi-source evaluation dataset for Chinese Grammatical Error Correction (CGEC), consisting of 7,063 sentences collected from three Chinese-as-a-Second-Language (CSL) learner sources. Each sentence is corrected by three annotators, and their corrections are carefully reviewed by a senior annotator, resulting in 2.3 references per sentence. We conduct experiments with two mainstream CGEC models, i.e., the sequence-to-sequence model and the sequence-to-edit model, both enhanced with large pretrained language models, achieving competitive benchmark performance on previous and our datasets. We also discuss CGEC evaluation methodologies, including the effect of multiple references and using a char-based metric. Our annotation guidelines, data, and code are available at https://github.com/HillZhang1999/MuCgEC.
Given a potentially noisy input sentence, grammatical error correction (GEC) aims to detect and correct all errors and produce a clean sentence. Recently, GEC has increasingly gained attention for its vital value in various downstream scenarios (Grundkiewicz et al., 2020; Wang et al., 2021).
To support GEC research, high-quality manually labeled evaluation data is indispensable. For English GEC (EGEC), such datasets are abundant (Yannakoudakis et al., 2011; Dahlmeier et al., 2013; Ng et al., 2014; Naples et al., 2017; Bryant et al., 2019; Naples et al., 2019; Flachs et al., 2020). However, Chinese GEC (CGEC) evaluation datasets are relatively scarce. The two publicly available CGEC evaluation datasets are NLPCC18 and CGED, contributed by the NLPCC-2018 (Zhao
Source	我不知道他何时返回回来。
I don’t know when he will return back.
Ref. 1	我不知道他何时返回回来。
I don’t know when he will return.
Ref. 2	我不知道他何时返回回来。
I don’t know when he will be back.
Table 1: A CGEC example with two references.
et al., 2018) and the series of CGED shared tasks (Rao et al., 2018, 2020), respectively.
Most EGEC evaluation datasets provide multiple references for each input sentence, such as CoNLL14-test (Ng et al., 2014) and BEA19-test (Bryant et al., 2019). In contrast, sentences in existing CGEC evaluation datasets usually have only one reference (i.e.,  $87\%$  of the sentences in NLPCC18 and all in CGED). This is probably due to the different annotation workflows adopted.
As suggested by Bryant and Ng (2015), enforcing multi-reference annotation is crucial for both GEC model evaluation and GEC data annotation, because there usually exist more than one acceptable reference with similar meanings for an incorrect sentence, as illustrated by the example in Table 1. On the one hand, if a GEC model outputs a correct reference, which is yet different from the one given in the evaluation data, then the model performance will be unfairly underestimated. To mitigate this issue, a straightforward solution is increasing the number of references (Sakaguchi et al., 2016; Choshen and Abend, 2018). On the other hand, imposing a single-reference constraint makes data annotation problematic. If annotators submit different equally acceptable corrections, which is very common, it will be difficult for the senior annotator to decide which one is the best.
Besides the lack of multiple references, existing CGEC datasets collect sentences from a single text source, which may be insufficient for robust model evaluation (Mita et al., 2019). In addition, we be
† This work was partially done during the first author's internship at Alibaba DAMO Academy.
Human Language Technologies, pages 3118 - 3130
lieve that it is beneficial for improving data quality to compile comprehensive annotation guidelines.
To fill these gaps, this paper presents a multi-reference multi-source evaluation dataset for CGEC, named MuCGEC. After investigating previous works on constructing GEC datasets, we compile comprehensive annotation guidelines. Based on a browser-based online annotation tool, each sentence is assigned to three annotators for independent correction, and one senior annotator for final review. An annotator may submit multiple references, and the senior annotator may also supplement new references besides rejecting incorrect submissions. In this way, we aim to produce as many references as possible. In summary, this work makes the following contributions.
(1) Our newly constructed MuCGEC consists of 7,063 sentences from three representative sources of Chinese-as-a-Second-Language (CSL) learner texts. Each sentence obtains 2.3 references on average. We conduct detailed analyses on our new dataset to gain more insights.
(2) We conduct benchmark experiments using two mainstream and competitive CGEC models, i.e., the sequence-to-edit (Seq2Edit) and sequence-to-sequence (Seq2Seq) models, both enhanced with pretrained language models (PLMs). We also experiment with an extremely effective ensemble strategy. Moreover, we investigate the effect of multiple references on model evaluation, and propose to use a char-based evaluation metric, which we believe is simpler and more suitable than previous word-based ones for CGEC.
2.1 Multi-Source Data Selection
This work focuses on CSL learner texts. In order to investigate diverse types of Chinese grammatical errors, we select data from the following three sources.
(1) We re-announce the NLPCC18 test set (Zhao et al., 2018), which contains 2,000 sentences from the Peking University (PKU) Chinese Learner Corpus.
(2) We select and re-announce sentences from CGED-2018 and CGED-2020 test datasets (Rao et al., 2018, 2020), which come from
the writing section of the HSK exam (Hanyu Shuiping Kaoshi, translated as the Chinese level exam), an official Chinese proficiency test. After removing sentences marked as correct from the total 5,006 ones, we obtain 3,137 potentially erroneous sentences for re-annotation.
(3) Lang8 $^{1}$  is a language learning platform, where native speakers voluntarily correct texts written by second-language learners. The NLPCC-2018 shared task organizers collect about 717K Chinese sentences with their corrections from Lang8 and encourage participants to use them as the training data. We randomly select 2,000 sentences with 30 to 60 characters for re-annotation.
In the end, we obtain 7,137 sentences. For simplicity, we discard all original corrections and directly perform re-annotation from scratch following our new annotation guidelines and workflow.
2.2 Annotation Paradigm: Direct Rewriting
There are mainly two types of annotation paradigms for constructing GEC data, i.e., error-coded and direct rewriting. The error-coded paradigm requires annotators to explicitly mark the erroneous span in the original sentence, then choose its error type, and finally make corrections. Ng et al. (2013, 2014) adopt the error-coded paradigm for constructing data for the CoNLL-2013/2014 EGEC shared tasks. For CGEC, the original NLPCC18 and CGED datasets both follow the error-coded paradigm as well.
As discussed by Sakaguchi et al. (2016), the error-coded paradigm poses two challenges. First, it is extremely difficult for different annotators to agree upon the boundaries of the erroneous spans and their error types, especially when there are many categories to consider (Bryant et al., 2017). This inevitably leads to an increase in annotation effort and a decrease in annotation quality. Second, under such a complex annotation paradigm, annotators would pay less attention to the fluency of the resulting reference, sometimes even leading to unnatural expressions.
Instead, the direct rewriting paradigm asks annotators to directly rewrite the input sentence and produce a corresponding grammatically correct one, without changing the original meaning. In order to
https://lang-8.com/
Major Types	Minor Types
Punctuation	Missing; Redundancy; Misuse
Spelling	Phonetic confusion; Glyph confusion; Character disorder
Word	Missing; Redundancy; Misuse
Syntax	Word order; Mixing syntax patterns
Pragmatics	Logical inconsistency; Ambiguity; Commonsense mistake
evaluate model performance, edits can be extracted automatically from parallel sentences by additional tools (Bryant et al., 2017).
This annotation paradigm has proven to be efficient and cost-effective (Sakaguchi et al., 2016), and has been adopted by several recent GEC data construction works (Napoles et al., 2017, 2019; Syvokon and Nahorna, 2021; Náplava et al., 2022).
In this work, we adopt the direct rewriting paradigm. Besides above-mentioned advantages, we believe this paradigm can help improve the diversity of references since annotators can correct errors more freely.
2.3 Annotation Guidelines
After an extensive survey of previous work on GEC data construction, we compiled 30 pages of comprehensive guidelines for CGEC annotation. During the course of the annotation process, we gradually improved our guidelines according to feedback from annotators.
To facilitate learning, our guidelines adopt a two-tier hierarchical error taxonomy, including 5 major error types and 14 minor types, as shown in Table 2. Our guidelines describe in detail how to handle each minor error type and provide typical examples. We will release our guidelines along with the dataset, which we hope can benefit future research.
For dealing with word-missing errors, we found that it was unreasonable to simply insert certain words when the missing words are context-dependent, which means the missing words are related to context beyond the given sentence. Table 3 shows a sentence in which a verb is missing. However, under sentence-level GEC annotation, annotators are unable to decide the specific missing verb. According to our observation, previous CGEC datasets directly insert specific words like
Table 2: The 5 major and 14 minor error types adopted by our guidelines for organizing the content.
Source	我的爸爸经常我。
My dad usually me.
Previous	我的爸爸经常骂我。
My dad usually scolds me.
Ours	我的爸爸经常[MC]我
My dad usually [MC] me.
Table 3: An example for handling context-dependent missing components. The inserted tokens are underlined. "Previous" means annotation in previous datasets, and "Ours" refers to our annotation.
"scolds" under such circumstances, which we think is inaccurate and may cause trouble for GEC model evaluation, because there are many other acceptable candidates. To handle this problem, we instead insert a special tag named context-dependent missing components (MC). We find about  $1\%$  of sentences in MuCGEC contain "[MC]" tags. Current GEC models cannot handle "[MC]", since "[MC]" is not included in existing training data and vocabulary. We leave this issue as future work.
2.4 Annotation Workflow and Tool
In order to encourage more diverse and high-quality references, we assign each sentence to three random annotators for independent annotation. Their submissions are then aggregated and sent to a random senior annotator (reviewer) for review. An annotator may submit multiple references for one sentence if he/she thinks they are all correct according to the guidelines. The job of the senior annotator includes: 1) modifying incorrect references into correct ones (sometimes just rejecting them); 2) adding new correct references according to the guidelines. After review, the accepted references are defined as final golden references.
For the sake of self-improvement, we employ a self-study mechanism that allows annotators to learn from their mistakes if they submit an incorrect reference. Concretely, if an annotator submits a reference that is not included in the final golden references, he/she has to modify his/her submission into a correct one. Moreover, the annotator can also make complaints if he/she insists that his/her submission is correct. We find that the self-study and making-complaints mechanisms can trigger very helpful discussions.
To improve annotation efficiency, we have developed a browser-based online annotation tool to support the above workflow and mechanisms. Due
Dataset	#sent	#err. sent (perc.)	chars/sent	edits/ref	refs/sent
Original NLPCC18	2000	1983 (99.2%)	29.7	2.0	1.1
MuCGEC (NLPCC18)	1996 (4)	1904 (95.4%)	29.7	2.5	2.5
MuCGEC (CGED)	3125 (12)	2988 (95.6%)	44.8	4.0	2.3
MuCGEC (Lang8)	1942 (58)	1652 (85.1%)	37.5	2.8	2.1
MuCGEC	7063 (74)	6544 (92.7%)	38.5	3.2	2.3
Table 4: Data statistics, including the number of sentences, the number (proportion) of erroneous sentences, the average number of characters per sentence, the average number of edits per reference, and the average number of references per sentence. Some sentences in our source data were discarded since annotators could not understand their meaning and thus were unable to correct them. Numbers in parentheses in the "#sent" row refer to such sentences.
to the space limitation, we show the visual interfaces for annotation and review in Appendix B.
2.5 Annotation Process
We employed 21 undergraduate students who are native speakers of Chinese and familiar with Chinese grammar as part-time annotators. Annotators received intensive training before real annotation. In the beginning, two authors of this paper, who were also in charge of compiling the guidelines, served as senior annotators for review. After one month, when the annotators were familiar with the job, we selected 5 outstanding annotators as senior annotators to join the review.
All participants were asked to annotate for at least 1 hour every day. The whole annotation process lasted for about 3 months.
2.6 Ethical Issues
All annotators and reviewers were paid for their work. The salary was determined by both submission numbers and annotation quality. The average salary of annotators and reviewers is 24 and 35 RMB per hour respectively.
All the data of the three sources are publicly available. Meanwhile, we have obtained permission from organizers of the NLPCC-2018 and CGED shared tasks to release our newly annotated references in a proper way. We are deeply grateful to them for their kind support.
3 Analysis of Our Annotated Data
Overall statistics of MuCGEC are shown in Table 4. We also include the original NLPCC18 dataset (Zhao et al., 2018) for comparison $^{2}$ .
First, from the proportion of erroneous sentences, we can see that most of the sentences are considered to contain grammatical errors in the original annotation, but a considerable part of them are not corrected in our annotation. We attribute this to our strict control of the over-correction phenomenon.
Second, regarding sentence lengths, NLPCC18 has the shortest sentences, whereas CGED sentences are much longer. This may be because candidates on the HSK examination, an official Chinese proficiency test, tend to use long sentences to show their ability in Chinese.
Third, each sentence in the re-annotated NLPCC18 receives 2.5 references on average, which is more than twice that in the original NLPCC18 data. Overall, each sentence obtains 2.3 references. We believe the multi-reference characteristic makes our dataset more reliable for evaluation, which is further discussed in Section 6.3.
Finally, we compare the number of char-based edits per reference in different datasets. We describe how to derive such edits in detail in Section 6.2. We can see that the number of edit is tightly correlated with sentence length. The difference in the average sentence length and number of edits indicates that the three data sources may have a systematic discrepancy in quality and difficulty, which we believe is helpful for evaluating the generalization ability of models. Moreover, compared with NLPCC18-orig, we annotate  $25\%$  more edits (2.0 vs. 2.5) in each reference. We believe the major reason is that the original NLPCC18 data are annotated under the minimal edit distance principle (Nagata and Sakaguchi, 2016), which requires annotators to select a reference with fewer edits when correcting errors.
Here we do not compare with the original CGED and Lang8 datasets since: 1) the CGED-orig mainly focuses on error detection annotation and does not provide corrections for
word-order errors; 2) the Lang8-origin is automatically collected from the internet, and its correction is quite noisy.
Figure 1: The proportion of sentences with different number of references in MuCGEC.
Sentence distribution with respect to numbers of references is shown in Figure 1. Here, we only consider erroneous sentences. Identical references from different annotators are counted as one reference. Overall, most sentences have 2 references, accounting for  $39.4\%$ ;  $29.1\%$  of sentences have 3 references;  $21.8\%$  of sentences have only 1 reference, most of which are short and easy to correct.
We believe that the average number of references could be further increased if more annotators were assigned to each sentence. It is also worth noticing that annotators tend to submit a single reference, despite the fact that our annotation tool allows annotators to submit multiple ones. We suspect the reason may be that it is more economical for annotators to do so. One the one hand, it may be easy to come up with the most suitable correction, whereas thinking of alternatives is more time-consuming. On the other hand, we did not give enough consideration to this issue when designing the salary computation formula. In the future, we plan to optimize (or simplify) our annotation workflow so that each annotator is required to give only one reference which he/she thinks is the best, and assign each sentence to more annotators if we need more references.
Human annotation performance. In order to assess the annotation ability of our annotators and human performance for CGEC task, we calculate char-based  $\mathrm{F}_{0.5}$  scores by evaluating all annotation submissions against the final golden references. We describe how to compute char-based metrics in detail in Section 6.2. Each reference submitted by an annotator is considered as a sample. Overall, the average  $\mathrm{F}_{0.5}$  is 72.12, which we believe could be higher if we discarded data that were annotated
Figure 2: The human performance of the 15 annotators who annotated the most sentences.
at the early stage of our project when annotators were less experienced and less familiar with our guidelines.
Figure 2 shows  $\mathrm{F}_{0.5}$  scores of 15 annotators who annotated the most sentences, in descending order of the number of annotated sentences. We can see that human performance varies across different annotators. The best annotator achieves an  $82.34\mathrm{F}_{0.5}$  score, while the annotator who completes the most tasks only gets a score of 68.32. This indicates that we should pay more attention to annotation quality when calculating salaries and prevent annotators from focusing too much on annotation speed.
Common mistakes made by annotators. We randomly select 300 invalid references rejected by reviewers and try to understand what mistakes are more frequently made by annotators. We manually classify all selected references into three mistake categories. The most frequent mistakes are caused by incomplete correction and account for  $56.7\%$  of the invalid references. We found that it was sometimes difficult for annotators to correct all errors without omissions, possibly due to the complexity or flexibility of Chinese grammar. The second frequent type of mistakes is erroneous correction, which means that the correction of old errors incurs new errors, with a proportion of  $32.3\%$ . Besides,  $11.0\%$  of submissions are rejected due to meaning change, which means that the correction changes the intended meaning of the original sentence.
To understand how well cutting-edge GEC models perform on our data, we adopt two mainstream GEC approaches, i.e., Seq2Edit and Seq2Seq. Both models are enhanced with PLMs. We also at-
tempt to combine them after observing their complementary power in dealing with different error types. This section briefly describes these benchmark models. Due to the space limitation, please kindly refer to Appendix C for more model details.
The Seq2Edit model treats GEC as a sequence labeling task and performs error corrections via a sequence of token-level edits, including insertion, deletion, and substitution (Malmi et al., 2019). A token corresponds to a word or a subword in English, and to a character in Chinese. With minor modifications to accommodate Chinese, we adopt GECToR (Omelianchuk et al., 2020), which achieves the SOTA performance on EGEC datasets.
Following recent Seq2Edit work like Awasthi et al. (2019) and Omelianchuk et al. (2020), we enhance GECToR by using PLMs as its encoder. After comparing several popular PLMs, we choose StructBERT (Wang et al., 2019) $^3$  due to its superior performance after fine-tuning (see Table 5).
The Seq2Seq model straightforwardly treats GEC as a monolingual translation task (Yuan and Briscoe, 2016). Recent works propose to enhance Transformer-based (Vaswani et al., 2017) Seq2Seq EGEC models with PLMs like T5 (Rothe et al., 2021) or BART (Katsumata and Komachi, 2020). Unlike BERT (Devlin et al., 2019), T5 and BART are specifically designed for text generation. Therefore, it is straightforward to continue training them on GEC data. We follow these works and utilize the recently proposed Chinese BART from Shao et al. (2021) to initialize our Seq2Seq model.
The ensemble model. Several previous works have proven the effectiveness of model ensemble for CGEC (Liang et al., 2020; Hinson et al., 2020). In this work, we clearly observe the complementary power of the above two models in fixing different error types (see Table 7), and thus attempt to combine them.
We adopt a simple edit-wise voting mechanism. The edits are at the char-based span level, and correspond to four error types. Please refer to Section 6.2 for detailed explanation of our char-based evaluation metric. More specifically, we aggregate edits from the results of each model, and only preserve edits that appear more than  $N / 2$  times, where  $N$  is the number of models. In other words, an edit is kept in the final result only if it is produced by a majority of models. $^4$
We experiment with two ensemble settings: 1) one Seq2Edit and one Seq2Seq, denoted by “ $1 \times$  Seq2Edit +  $1 \times$  Seq2Seq”, and 2) three Seq2Edit and three Seq2Seq, denoted by “ $3 \times$  Seq2Edit +  $3 \times$  Seq2Seq”. The three Seq2Edit models are obtained using different random seeds for initialization, and the same goes for the Seq2Seq.
Other settings. To obtain the single-model performance of both kinds of models, we run them three times separately with different random seeds for initialization and calculate average metrics. For “ $1 \times \text{Seq2Edit} + 1 \times \text{Seq2Seq}$ ”, we random select a pair of single models. For “ $3 \times \text{Seq2Edit} + 3 \times \text{Seq2Seq}$ ”, we aggregate the results of all six single models.
5 Experiments on NLPCC18-Orig Data
In order to show that our benchmark models are competitive among existing CGEC models, we conduct experiments on the original NLPCC18 test set, on which most previous CGEC systems are tested.
Training data. For the sake of easy replicability, we limit our training data strictly to public resources, i.e., the Lang8 $^5$  data (Zhao et al., 2018) and the HSK $^6$  data (Zhang, 2009). We filter duplicate sentences that appear in our dataset, and discard correct sentences. The final Lang8 and HSK data contains 1,092,285 and 95,320 sentence pairs, respectively. The HSK data is cleaner and of higher quality than Lang8, but is smaller. Following the re-weighting procedure of Junczys-Dowmunt et al. (2018), we duplicate the HSK data five times, and merge them with Lang8 data.
Comparison with previous works. Table 5 shows the results. For a fair comparison, we follow the official setting of the shared task, including the word-based MaxMatch scorer (Dahlmeier and Ng,