SlowGuess/NAACL_2022_OCR huggingface dataset:
Junpei Zhou, Chen Li, Hengyou Liu, Zuyi Bao, Guangwei Xu, and Linlin Li. 2018. Chinese grammatical error correction using statistical and neural models. In CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC), pages 117-128.
A More Discussion on the Char-based Span-level Metric
We find that ERRANT_ZH (Hinson et al., 2020), a useful evaluation tool for CGEC, only merges edits for redundant errors and missing errors, and does not merge edits for substitution errors. In contrast, as discussed in Section 6.2, we also merge the consecutive edits for substitution errors for the sake of simplicity. We hope future research can adopt our simplified version unless there is a strong reason. Meanwhile, we should keep thinking about which evaluation metrics are more suitable for CGEC task.
B Interface
Figure 4 shows our design of annotation interface in our annotation tool, where annotators correct assigned sentences. Given an annotation task, this interface presents a potentially wrong sentence and
a text input box. The original sentence is copied into the input box below, so that the annotator can directly modify it. To support multiple corrections, we also provide a button to add additional input boxes. Two special buttons are provided to deal with special cases. The error free button means that the sentence is correct; the not annotatable button means that the annotator can not understand the sentence.
Figure 4: The screenshot of the annotation interface.
Figure 5 shows the review interface, where senior annotators judge whether the submitted corrections are correct. All corrections of a sentence from annotators are shown on the screen, and reviewers click a check box to mark each of them as correct or incorrect. The input box below allows reviewers to supplement extra valid corrections.
Figure 5: The screenshot of the review interface.
Table 8 and Table 9 shows the detailed hyperparameters for training our two benchmark models. Due to the GPU memory limitation, we truncated sentences longer than 100 characters when training the Seq2Seq model. In other words, extra characters in the input sentences and the references are discarded.
A useful trick. We find that some sentences in MuCGEC are actually composed of multiple sentences. Therefore, we split one input sentence into multiple ones based on punctuation marks such as
periods, exclamation marks, and so on. Then, we perform corrections on the smaller sentences and concatenate the results. We find this trick can consistently improve performance in our preliminary experiments. For now, we decide not to break the sentences when releasing in order to be consistent with the sources where the data comes from.
Configurations	Values
Model architecture	BART (Lewis et al., 2020)
Pretrained model	Chinese-BART-Large (Shao et al., 2021)
Number of max epochs	10
Devices	8 Nvidia V100 GPU (32GB)
Batch size per GPU	32
Optimizer	Adam (Kingma and Ba, 2014)
(β1 = 0.9, β2 = 0.999, ε = 1 × 10-8)
Learning rate	3 × 10-6
Learning rate scheduler	Polynomial
Gradient accumulation steps	4
Dropout	0.1
Gradient clipping	1.0
Label smoothed cross entropy
Loss function	label-smoothing=0.1)
(Szegedy et al., 2016)
Total training time	About 20 hours
Stopping criteria	Loss value on the dev set
Patience	3
Table 8: Hyperparameter values of our Seq2Seq model.
Configurations	Values
Model architecture	GECToR (Omelianchuk et al., 2020)
Pretrained model	Chinese-Struct-Bert-Large (Wang et al., 2019)
Number of max epochs	20
Number of cold epochs	2
Devices	1 Nvidia V100 GPU (32GB)
Optimizer	Adam (Kingma and Ba, 2014)
(β1=0.9, β2=0.999, ε=1×10-8)
Cold learning rate	1×10-3
Learning rate	1×10-5
Batch size	128
Loss function	Cross entropy
Total training time	About 10 hours
Stopping criteria	Label prediction accuracy on the dev set
Patience	3
Table 9: Hyperparameter values of our Seq2Edit model. "Cold" means that freeze the parameters of BERT.