SlowGuess/NAACL_2022_OCR huggingface dataset:
to the adaption of two pieces of modification, which consistently improve performance in our preliminary experiments. First, for word-order errors, we set the preserving threshold to  $N / 2 - 1$  considering the Seq2Seq model is much superior in handling word-order errors than the Seq2Edit model. Imagine the scenario when all Seq2Seq models agree on correcting a word-order error, whereas all Seq2Edit models disagree. Using  $N / 2 - 1$  means that a word-order edit is kept in the final result even when it is produced by exactly half of models. Second, we use a set of simple heuristic rules to recognize spelling errors, a sub-type of substitution errors, and also use  $N / 2 - 1$  as the preserving threshold for them. This is also reasonable since both GEC models can obtain high precision scores on such errors.
5http://tcci.ccf.org.cn/conference/2018/ taskdata.php
<sup>6</sup>http://hsk.bclc.edu.cn
$^{3}$ https://github.com/alibaba/AliceMind/tree/main/StructBERT
4In fact, our voting strategy is a little more complex due
P	R	F0.5
Trained on Lang8
YouDao (Fu et al., 2018)◇	35.24	18.64	29.91
AliGM (Zhou et al., 2018)◇	41.00	13.75	29.36
BLCU (Ren et al., 2018)◇	47.63	12.56	30.57
HRG (Hinson et al., 2020)◇	36.79	27.82	34.56
MaskGEC (Zhao and Wang, 2020)◇	44.36	22.18	36.97
Our Seq2Edit	39.83	23.01	34.75
Our Seq2Seq	37.67	29.88	35.80
1×Seq2Edit+1×Seq2Seq◇	58.15	18.35	40.55
3×Seq2Edit+3×Seq2Seq◇	55.58	19.78	40.81
Trained on Lang8+HSK
TEA (Wang et al., 2020)◇	39.43	22.80	34.41
WCDA (Tang et al., 2021)◇	47.41	23.72	39.51
Our Seq2Edit (BERT)	39.61	28.53	36.76
Our Seq2Edit (RoBERTa)	39.74	30.44	37.54
Our Seq2Edit (MacBERT)	40.46	30.73	38.05
Our Seq2Edit (StructBERT)	42.88	30.19	39.55
Our Seq2Seq	41.44	32.89	39.39
1×Seq2Edit+1×Seq2Seq◇	60.72	22.48	45.31
3×Seq2Edit+3×Seq2Seq◇	59.38	24.18	45.99
Table 5: Performance comparison on the original NLPCC18 dataset (Zhao et al., 2018) using the official word-based evaluation script. The first group lists models that use only Lang8 for training, whereas the second group shows those using both Lang8 and HSK data. Models marked by  $\diamond$  use model ensemble, and those marked by  $\heartsuit$  use data augmentation.
2012) for calculating the P/R/F values. We segment model outputs by adopting the PKUNLP word segmentation (WS) tool provided by the shared task organizers (Zhao et al., 2018).
When only using Lang8 for training, our single Seq2Seq model is already quite competitive. Its performance is only lower than MaskGEC (Zhao and Wang, 2020) by 1 point in  $\mathbf{F}_{0.5}$ . Please notice that MaskGEC extra uses data augmentation. For now, our benchmark models do not use any synthetic data for simplicity, but we believe data augmentation could further boost the performance of our models.
Adding the HSK training data improves performance of all our models by about 4 points. Our two benchmark models already achieve SOTA performance under the single-model setting.
The model ensemble technique leads to obvious performance gains (more than 5 points) over single models. However, the gains from increasing the number of component models seem rather small. We try to explain this issue in Section 6.3.
For Seq2Edit, we additionally present results with other PLMs besides StructBERT, including BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and MacBERT (Cui et al., 2020) from the
Hugging Face<sup>7</sup> website. We use the "large" variants of all PLMs.
6 Experiments on MuCGEC
6.1 Data Splits
For hyperparameter tuning or model selection, previous works on other CGEC datasets often randomly sample some sentence pairs from training data as the dev set (Wang et al., 2020; Zhao and Wang, 2020; Hinson et al., 2020), which is inconvenient for reproducing or comparing.
In this work, we propose to provide a fixed dev set for our newly annotated dataset, by randomly selecting 1,125 sentences from the CGED source, denoted as CGED-dev. The remaining 5,938 sentences are used as the test set, in which each data source has a roughly equal amount of sentences, i.e., 1,996 sentences for NLPCC18-test, 2,000 for CGED-test, and 1,942 for Lang8-test.
Problems with word-based metrics. As discussed in Section 5, previous CGEC datasets are annotated upon word sequences and adopt word-based metrics for evaluation. Before annotation and evaluation, a sentence needs to be segmented into words using a Chinese word segmentation (CWS) model. We believe this will introduce unnecessary uncertainty in CGEC evaluation procedure. First, CWS models inevitably produce word segmentation errors (Fu et al., 2020). Second, there are multiple heterogeneous CWS standards. Finally, we found that a correct edit may be judged as wrong due to the word boundary mismatch.
Char-based span-level evaluation metrics are adopted in this work instead. First, given an input sentence and a correction, we obtain an optimal sequence of char-level edits with the minimal edit distance. We consider three types of char-level edits, corresponding to three error types:
- Deleting a char for a redundant error;
- Inserting a char for a missing error;
- Substituting a char with another one for a substitution error;
Then, we convert all char-level edits into span-level by merging consecutive edits of the same type, following previous practice in EGEC and CGEC (Felice et al., 2016; Hinson et al., 2020)
<sup>7</sup>https://huggingface.co/
NLPCC18-test	CGED-test	Lang8-test	All-test
P	R	F0.5	P	R	F0.5	P	R	F0.5	P	R	F0.5
Seq2Edit	50.09	32.09	45.04	42.87	27.69	38.64	39.65	21.62	33.98	44.11	27.18	39.22
Seq2Seq	47.99	35.12	44.71	46.04	26.97	40.34	36.10	25.01	33.16	43.81	28.56	39.58
1×Seq2Edit+1×Seq2Seq	74.13	24.11	52.39	68.59	20.35	46.53	62.25	14.23	37.17	68.92	19.68	45.94
3×Seq2Edit+3×Seq2Seq	72.82	26.38	53.81	67.95	21.58	47.52	60.65	16.39	39.38	67.76	21.42	47.29
Human	75.77	66.15	73.63	74.14	64.84	72.00	72.31	62.26	70.05	73.47	63.75	71.25
Table 6: Performance of models and our annotators on MuCGEC, using the char-based metric. For calculating the human performance, each submitted result is considered as a sample if an annotator submits multiple references.
Seq2Edit	Seq2Seq	Ensemble	Human
Missing (29.2%)	41.09	40.93	42.25	69.72
Redundant (16.1%)	43.11	37.65	54.18	72.78
Substitution (48.9%)	35.99	39.98	47.37	71.69
Word-order (5.8%)	28.28	40.33	42.44	72.58
Table 7:  $F_{0.5}$  scores for each error type on Alltest. The bold numbers in parentheses show the proportion of each error type. "Ensemble" refers to  $3 \times Seq2Edit + 3 \times Seq2Seq$ .
The above two steps are applied to both the system output sequence and golden reference, transforming them into sets of span-level edits. Finally, we can calculate the P/R/F value by comparing them. If there are multiple golden references, we will choose the one with the highest F-score.
Span-level word-order errors. When calculating overall metrics, we only consider above three types of errors. When analyzing, we distinguish the fourth error type — word-order. A span-level word-order error is usually composed of a redundant and a missing error, where the deleted span is the same as the inserted one. We use simple heuristic rules to identify such errors (Hinson et al., 2020).
Please kindly notice that we release our evaluation script as well.
Main results. Table 6 shows the char-based performance of the benchmark models and our annotators on MuCGEC. All models are trained on Lang8+HSK, as described in Section 4.
The overall trend of performance change is basically consistent with those on the original NLPCC18 dataset in Table 5. First, the Seq2Seq and Seq2Edit models perform quite closely on  $\mathrm{F}_{0.5}$ , but clearly exhibit divergent strength in precision and recall, giving a strong motivation for combining them. Second, the model ensemble approach improves performance by a very large margin.
One interesting observation is that on MuCGEC,
Figure 3: Effect of the number of references on  $F_{0.5}$ .
$3 \times \text{Seq2Edit} + 3 \times \text{Seq2Seq}$  substantially outperforms  $1 \times \text{Seq2Edit} + 1 \times \text{Seq2Seq}$  on All-test and all three subsets. In contrast, the improvement is only modest on the original NLPCC18 test data. We suspect this may indicate that a multi-reference dataset can more accurately evaluate model performance. However, it may require further human investigation for more insights.
Finally, there is still a huge performance gap between models and humans, indicating that CGEC research still has a long way to go.
Performance on four error types. Table 7 shows more fine-grained evaluation results on four error types.
It is clear that the Seq2Edit model is better at handling redundant errors, whereas the Seq2Seq model is superior in dealing with substitution and word-order errors. For missing errors, the two perform similarly well.
These phenomena are quite interesting and can be understood after considering the underlying model architectures. On the one hand, to correct redundant errors, the Seq2Edit model only needs to perform a fixed deletion operation, which is a much more implicit choice for the Seq2Seq model, since its goal is to rewrite the whole sentence. On the other hand, the Seq2Seq is suitable to substitute or reorder words due to its natural capability of uti
lizing language model information, especially with the enhancement of BART (Lewis et al., 2020).
Again, the model ensemble approach substantially improves performance on all error types. The ensemble model is closest to the human on redundant errors, probably because they are the easiest to correct. The largest gap occurs in word-order errors, which require global structure knowledge to correct and are extremely challenging.
Influence of the number of references. To understand the impact of the number of references on performance evaluation, we deliberately reduce the available number of reference in our dataset. For example, when the maximum number of references is limited to 2, we only kept the first two references in the dataset if a sentence has more than 2 golden references. The results are shown in Figure 3.
When the maximum number of references increases, the performance of both models and humans increases continuously, especially for humans. As only a few sentences have more than 3 references, the improvement is quite small when the maximum number of references increases from 3 to All. This trend suggests that compared with single-reference datasets, a multi-reference dataset reduces the risk of underestimating performance, and thus is more reliable for model evaluation.
EGEC resources. There has been a lot of work on EGC data construction. As the two earliest EGC datasets, FCE (Yannakoudakis et al., 2011) and NUCLE (Dahlmeier et al., 2013) adopt the error-coded annotation paradigm. In contrast, JFLEG (Napoles et al., 2017) collects sentences from TOFEL exams and adopts the direct rewriting paradigm. W&I (Bryant et al., 2019) also chooses the direct rewriting paradigm, and extra annotates a score indicating the language proficiency level of the writer for each input sentence. All four datasets are composed of essays from non-native English speakers and provide multiple references.
Recently, researchers have started to annotate small-scale EGEC data for texts written by native English speakers, including AESW (Daudaravicius et al., 2016), LOCNESS (Bryant et al., 2019), GMEG (Napoles et al., 2019) and CWEB (Flachs et al., 2020). In the future, we plan to extend this work to texts written by native Chinese speakers.
CGEC resources. Compared with EGEC, progress in CGEC data construction largely lags be
hind. As discussed in Section 1, NLPCC18 (Zhao et al., 2018) and CGED (Rao et al., 2018, 2020) are the only two evaluation datasets for CGEC research. Besides them, there are also a few resources for training CGEC models, e.g., the Lang8 corpus (Zhao et al., 2018) and the HSK corpus (Zhang, 2009).
Concurrently with this work, Wang et al. (2022) present a multi-reference CGEC dataset, named as yet another Chinese learner corpus (YACLC), containing 32,124 sentences from Lang8. Each sentence is annotated by 10 annotators.
Recent progress in CGEC. In the NLPCC-2018 shared task (Zhao et al., 2018), many systems adopt Seq2Seq models, based on RNN/CNN. Recent work mainly utilizes Transformer (Wang et al., 2020; Zhao and Wang, 2020; Tang et al., 2021). Hinson et al. (2020) first employ a Seq2Edit model for CGEC, and achieve comparable performance with the Seq2Seq counterparts. Some systems in the CGED-2020 shared task (Rao et al., 2020) directly employ the open-source Seq2Edit model, i.e., GECToR (Liang et al., 2020). Most Seq2Edit models use PLMs like BERT (Devlin et al., 2019) to initialize their encoders. Besides the above two mainstream models, Li and Shi (2021) for the first time apply a non-autoregressive neural machine translation model to CGEC.
Besides modeling optimization, techniques like data augmentation (Zhao and Wang, 2020; Tang et al., 2021) and model ensemble (Hinson et al., 2020) have proven to be very useful for CGEC.
This paper presents MuCGEC, a newly annotated evaluation dataset for CGEC, consisting of 7,063 sentences written by CSL learners. Compared with existing CGEC datasets, ours can support more reliable evaluation due to three important features: 1) providing multiple references; 2) covering three text sources; 3) adopting strict quality control (i.e., annotation guidelines and workflow). After describing the data construction process, we perform detailed analyses of our data. Then, we adopt two mainstream and competitive CGEC models, i.e., Seq2Seq and Seq2Edit, and carry out benchmark experiments. We also propose to adopt char-based evaluation metrics to replace previously used word-based ones.
We want to thank the anonymous reviewers and Sebastian Schuster for their great help. We are also grateful to Zeyang Liu for building the annotation system, and Lei Zhang, Fukang Yan, Jiayu Shen, Houquan Zhou, Yu Zhang for helping us improve this paper, and all annotators for their great effort in data annotation. This work was partially supported by National Natural Science Foundation of China (Grant No.62176173 and No.61876116) and by Alibaba Group through Alibaba Innovative Research Program. This work was also partially supported by Projected Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Parallel iterative edit models for local sequence transduction. In Proceedings of EMNLP-IJCNLP, pages 4260-4270.
Christopher Bryant, Mariano Felice, Øistein E Aandersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of BEA@ACL, pages 52-75.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of ACL, pages 793-805.
Christopher Bryant and Hwee Tou Ng. 2015. How far are we from fully automatic high quality grammatical error correction? In Proceedings of ACL, pages 697-707.
Leshem Choshen and Omri Abend. 2018. Inherent biases in reference-based evaluation for grammatical error correction. In Proceedings of ACL, pages 632-642.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Proceedings of EMNLP: findings, pages 657-668.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of NAACL-HLT, pages 568-572.
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner English: The nus corpus of learner English. In Proceedings of BEA@NAACL-HLT, pages 22-31.
Vidas Daudaravicius, Rafael E Banchs, Elena Volodina, and Courtney Naples. 2016. A report on the automatic evaluation of scientific writing shared task. In Proceedings of BEA@NAACL-HLT, pages 53-62.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In Proceedings of COLING, pages 825-835.
Simon Flachs, Ophélie Lacroix, Helen Yannakoudakis, Marek Rei, and Anders Søgaard. 2020. Grammatical error correction in low error density domains: a new benchmark and analyses. In Proceedings of EMNLP, pages 8467-8478.
Jinlan Fu, Pengfei Liu, Qi Zhang, and Xuan-Jing Huang. 2020. Is Chinese word segmentation a solved task? Rethinking neural Chinese word segmentation. In Proceedings of EMNLP, pages 5676-5686.
Kai Fu, Jin Huang, and Yitao Duan. 2018. Youdao's winning solution to the NLPCC-2018 task 2 challenge: a neural machine translation approach to Chinese grammatical error correction. In CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC), pages 341-350.
Roman Grundkiewicz, Christopher Bryant, and Mariano Felice. 2020. A crash course in automatic grammatical error correction. In Proceedings of COLING: Tutorial Abstracts, pages 33-38.
Charles Hinson, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Heterogeneous recycle generation for Chinese grammatical error correction. In Proceedings of COLING, pages 2191-2201.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of NAACL-HLT, pages 595-606.
Satoru Katsumata and Mamoru Komachi. 2020. Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. In Proceedings of AACL, pages 827-832.
Diederik P Kingma and Jimmy Ba. 2014. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL, pages 7871-7880.
Piji Li and Shuming Shi. 2021. Tail-to-tail non-autoregressive sequence prediction for Chinese grammatical error correction. In Proceedings of ACL, pages 4973-4984.
Deng Liang, Chen Zheng, Lei Guo, Xin Cui, Xiuzhang Xiong, Hengqiao Rong, and Jinpeng Dong. 2020. BERT enhanced neural machine translation and sequence tagging model for Chinese grammatical error diagnosis. In Proceedings of NLPTEA@AACL, pages 57-66.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag, realize: high-precision text editing. In Proceedings of EMNLP-IJCNLP, pages 5054-5065.
Masato Mita, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, and Kentaro Inui. 2019. Cross-corpora evaluation and analysis of grammatical error correction models—is single-corpus evaluation enough? In Proceedings of NAACL-HLT (Short), pages 1309-1314.
Ryo Nagata and Keisuke Sakaguchi. 2016. Phrase structure annotation and parsing for learner English. In Proceedings of ACL, pages 1837-1847.
Courtney Naples, Maria Nădejde, and Joel Tetreault. 2019. Enabling robust grammatical error correction in new domains: data sets, metrics, and analyses. TACL, 7:551-566.
Courtney Naples, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: a fluency corpus and benchmark for grammatical error correction. In Proceedings of EACL, pages 229-234.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of CoNLL: Shared Task, pages 1-14.
Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In Proceedings of CoNLL: Shared Task, pages 1-12.
Jakub Náplava, Milan Straka, Jana Straková, and Alexandr Rosen. 2022. Czech grammar error correction with a large and diverse corpus. TACL, 10:452-467.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. Gector-grammatical error correction: tag, not rewrite. In Proceedings of BEA@ACL, pages 163-170.
Gaoqi Rao, Qi Gong, Baolin Zhang, and Endong Xun. 2018. Overview of NLPTEA-2018 share task Chinese grammatical error diagnosis. In Proceedings of NLPTEA@ACL, pages 42-51.
Gaoqi Rao, Erhong Yang, and Baolin Zhang. 2020. Overview of NLPTEA-2020 shared task for Chinese grammatical error diagnosis. In Proceedings of NLPTEA@AACL, pages 25-35.
Hongkai Ren, Liner Yang, and Endong Xun. 2018. A sequence to sequence learning for Chinese grammatical error correction. In CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC), pages 401-410.
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of ACL-IJCNLP, pages 702-707.
Keisuke Sakaguchi, Courtney Naples, Matt Post, and Joel Tetreault. 2016. Reassessing the goals of grammatical error correction: fluency instead of grammaticality. TACL, 4:169-182.
Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation. arXiv preprint arXiv:2109.05729.
Oleksiy Syvokon and Olena Nahorna. 2021. UAGEC: grammatical error correction and fluency corpus for the Ukrainian language. arXiv preprint arXiv:2103.16997.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of ICCV, pages 2818-2826.
Zecheng Tang, Yixin Ji, Yibo Zhao, and Junhui Li. 2021. Chinese grammatical error correction enhanced by data augmentation from word and character levels. In Proceedings of the 20th Chinese National Conference on Computational Linguistics (CCL) (in Chinese), pages 813-824.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 5998-6008.
Chencheng Wang, Liner Yang, Yingying Wang, Yongping Du, and Erhong Yang. 2020. Chinese grammatical error correction method based on transformer enhanced architecture. Journal of Chinese Information Processing (in Chinese), 34(6):106-114.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. 2019. Struct-BERT: incorporating language structures into pretraining for deep language understanding. In Proceedings of ICLR.
Yingying Wang, Cunliang Kong, Liner Yang, Yijun Wang, Xiaorong Lu, Renfen Hu, Shan He, Zheng-hao Liu, Yun Chen, Erhong Yang, and Maosong Sun. 2022. YACLC: a Chinese learner corpus with multidimensional annotation. arXiv preprint arXiv:2104.05507.
Yu Wang, Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. 2021. A comprehensive survey of grammatical error correction. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1-51.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of ACL, pages 180-189.
Zheng Yuan and Ted Briscoe. 2016. Grammatical error correction using neural machine translation. In Proceedings of NAACL-HLT, pages 380-386.
Baolin Zhang. 2009. Features and functions of the HSK dynamic composition corpus (in Chinese). International Chinese Language Education, 4:71-79.
Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018. Overview of the NLPCC 2018 shared task: grammatical error correction. In CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC), pages 439-445.
Zewei Zhao and Houfeng Wang. 2020. MaskGEC: improving neural grammatical error correction via dynamic masking. In Proceedings of AAAI, pages 1226-1233.