RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation
Authors: Yue Zhang
Leyang Cui
Enbo Zhao
Wei Bi
Shuming Shi
Venue: EMNLP 2023 Main (2023)
Keywords: Grammatical Error Correction
Robustness

Abstract: Grammatical Error Correction (GEC) systems play a vital role in assisting people with their daily writing tasks. However, users may sometimes come across a GEC system that initially performs well but fails to correct errors when the inputs are slightly modified. To ensure an ideal user experience, a reliable GEC system should have the ability to provide consistent and accurate suggestions when encountering irrelevant context perturbations, which we refer to as context robustness. In this paper, we introduce RobustGEC, a benchmark designed to evaluate the context robustness of GEC systems. RobustGEC comprises 5,000 GEC cases, each with one original error-correct sentence pair and five variants carefully devised by human annotators. Utilizing RobustGEC, we reveal that state-of-the-art GEC systems still lack sufficient robustness against context perturbations. Moreover, we propose a simple yet effective method for remitting this issue.

RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation
Yue Zhang Leyang Cuist Enbo Zhao Wei Bi
Shuming Shi
Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, Suzhou, China Tencent AI Lab, Shenzhen, China
yzhang2l@stu. suda. edu.
wttp:///wooaoooicccciiiiiialeeaa shmmi wttpgssshhhhtteeeeeeeeeeee @etenent..coom.. com Qttppss//gttabb.ccm//llllll /
Abstract
like play basketball.
Grammatical Error Correction (GEC) systems play vital role in assisting people with their daily writing tasks. However, users may some- times come across a GEC system that initially performs well but fails to correct errors when the inputs are slightly modified. To ensure an ideal user experience, a reliable GEC system should have the ability to provide consistent and accurate suggestions when encountering irrelevant context perturbations, which refer to as context robustness. In this paper, we in- troduce RobustGEC, a henchmark designed to evaluate the context robustness of GEC systems. RobustGEC comprises 5,000 GEC cases, each with one original error-correct sentence pair and five variants carefully devised by human annotators. Utilizing RobustGEC, we reveal that state-of-the-art GEC systems still lack suf- ficient robustness against context perturbations. addition, we propose a simple yet effective method for remitting this issue.
like playing basketball.
GEC System
He loves play basketball.
He loves play basketball.
GDC System
like
play
hockey.
like playing ice hockey.
GEC System
Figure 1: real example of how minor changes in user input misleads T5-based GEC system wrong pre- dictions. We use Red and Green to highlight errors and corrections, and underline to mark user modifications.
1 Introduction
Grammatical Error Correction (GEC) is the task of automatically fixing various textual errors in given sentence. This task has many real-life appli- cations, such as writing assistance and language teaching, thus receiving considerable interest from both academia and industry (Grundkiewiez et al., 2020; Wang et al., 2021; Bryant et al., 2022; Zhang et al., 2023a).
like play basketball", the GEC system effectively corrects the error (play = playing), because "like" should be followed by gerund. However, when the user introduces minor modifications unrelated to the error correction, the GEC system starts mak- ing mistakes. including under-correction (play = play) and over-correction (hockey + ice hockey). This observation may lead confusion and frus- tration, as users would expect the GEC system have a more stable and reliable performance.
Nowadays, outting-edger GEC systems perform decently in academic benchmarks (Ng al., 2013, 2014; Napoles al., 2017: Bryant al., 2019). However, in practical applications, the GEC sys- tem may offer different enror correction suggestions when users perform irrelevant modifications. Fig- ure 1 presents a real example of this phenomenon, where user interacts with GEC system based on T5 (Rothe et al., 2021). Given the original input "I
A key desirable property of strong GEC systems is context robustness, which refers to the ability to maintain consistent and accurate results with- out being disrupted by irrelevant context perturba- tions. To test this, we contribute the RobustGEC benchmark based on traditional GEC datasets (53). We choose 5,000 coror-correct sentence pairs from three sources for annotation, i.e., CoNLL-14 (Ng
Work was done during the internship Tencent Lab. Conresponding author.

---

al., 2014), BEA-19 (Bryant et al., 2019), and our newly released TEM-8. The first two are col- lected from essays and corrected by native English- speaking teachers. The last is derived from the ungrammatical sentence modification question in TEM-8, a high-level exam in China for evaluating English major students' English ability. where er- rors designed by experts. We ask annotators proficient in English to carefully craft five variants for each original sample by perturbing the context unrelated to the correction. The annotators are al- lowed to freely replace, insert, or delete content without introducing new errors or altering the orig- inal errors. In this way, RobustGLEC can examine whether GEC systems consistently and accurately rectify errors disregarding natural and realistic ir- relevant context perturbations.
For example, after post-training with CPR, the GECToR model performs consistent corrections for 15.1% 1% more GEC cases in RobustGEC.
We hope this research could spur future inves- tigations into the robustness of GEC systems. To facilitate further studies, we have made all data and code publicly accessible at https: /ttps://github .com/ hi 1 1zzang1 999 /RobustEEC. In the context of the LLM era, we believe that there is still much more to explore for GEC, extending beyond the mere pursuit superior leaderboard scores.
Related Work
GEC Benchmarks. To better study GEC, re- searchers have dedicated significant effort to build solid evaluation benchmarks (Ng et al., 2014; Napoles al., 2017: Bryant al., 2019; Napoles et al., 2019; Flachs et al., 2020; Zhang et al., 2022a, 2023d). These benchmarks typically evaluate GEC performance using reference- based metrics like edit-level Fo.5 (Dahlmeier and Ng, 2012; Bryant et al., 2017). Despite their contributions, these benchmarks are limited in their ability to assess the robustness of GEC systems to context perturba- tions, as each error them is associated with only one specific context. Therefore, we decide to build RobustGEC to fill this gap.
Utilizing RobustGEE,, evaluate the context robustness of five state-of-the-art (SOTA) GEC systems (64), i.e., seg2seq-based BART (Kat- sumata and Komachi, 2020), seq2seq- Syn- GEC with linguistic knowledge (Zhang et al., 2022b), seq2edit-basee GECToR (Omelianchuk et al., 2020). LLM-based LLaMA with GEC fine- tuning (Touvron et al., 2023), and LLM-based Chat- GPT with zero-shot prompting (OpenAI, 2023). Despite their notable performance conventional GEC benchmarks, all evaluated systems show dra- matic performance fluctuations when facing con- text perturbations, revealing their lack of context ro- bustness. Taking BART as an example, only 43.5% of GEC instances exhibit consistent corrections during perturbation. One potential explanation is that current GEC systems do not truly comprehend grammar, but rather depend some spurious cor- relations (Tu et al., 2020; Yang et al., 2022) to make corrections. In order gain more in-depth insights, we also conduct detailed analyses to in- vestigate how context perturbations affect GEC systems, in terms of perturbing action, distance, word frequency, etc.
GEC Approaches. For building advanced GEC systems, there are three mainstream paradigms to- day. The first is sequence-to-sequenee to-suunnceee (seq2seq), which treats GEC as a monolingual machine trans- lation task and employs uncoder-decodee models such as Transformer (Vaswani et al., 2017) to gen- erate corrections (Junczys- Doowmuyt et al., 2018; Stahlberg and Kumar, 2021; Rothe al., 2021; Zhang al., 2022b). The second is sequence-to- edit (seq2edit), which predicts sequence of edits and applies them to source tokens to perform cor- rections (Awasthi et al., 2019; Omelianchuk et al., 2020; Stahlberg and Kumar, 2020). The last is based on the burgeoning large language models (LLMs), such as ChatGPT and LLaMA (Touvron et al., 2023), which can achieve promising GEC performance via prompting (Coyne and Sakaguchi, 2023; Fang et al., 2023) or fine-tuning (Zhang al., 2023b). In this work, we select five representative systems that cover three paradigms for evaluat- ing context robustness on RobustGEC.
To improve the context robustness, we further propose a Context Perturbation Robust (CPR) post- training approach ($5). The proposed CPR method optimizes the GEC system to output the same pre- diction distribution at the non-perturb positions by minimizing the bi-directional Kullback-Leibler (KL) divergence, which is lightweight and easy to train. Experiments demonstrate that the CPR method can improve context robustness by a large margin and maintain the original GEC performance.
Robustness Other NLP fields. Robustness NLP has long been an active research topic (Wang

---

Source
CoNLL-14
BEA-19
TEM-8
ID
Case
Wrong Reason
Number of Sentences
1,312
2,503
1,185
have [lot=>lots] of friend. will give talk yesterday.
Correctness Faithfulness
Number of Wrong Sentences
1,176
1,687
1,184
Average Length (Word) Average Number of Errors
22.98
20.97
30.65
2,60
3.12
1.47
Table 2: Examples of bad cases generated by automatic rules. Blue highlights the automatic perturbation and Red marks the grammatical error.
Table 1: Statistics our three data sources.
al., 2022). Using adversarial text generation (Ebrahimi et al., 2018; Ren al., 2019; Li al., 2020), existing work automatically builds synthetic robustmess benchmarks for various tasks, such as question answering (Jia and Liang, 2017). name entity recognition (Lin et al., 2021), and sentiment analysis (Xing et al., 2020). Meanwhile, there are improving NLP models' robustness from both continuous (Chen et al., 2020; Zhu al., 2020) and discrete (Kaushik al., 2019; Liu et al., 2021) spaces, which may offer useful insights for building robust GEC systems.
We collect it from the "LANGUAGE USAGE" sec- tion in TEM-8 the highest-level test for senior students majoring in English Language and Litera- ture in China. TEM-8 contains multifarious chal- lenging grammatical errors designed by linguistic experts, which aim examining whether students meet the English language proficiency specified in the "National College English Teaching Syllabus" for English Majors. We gather about 1,000 GEC samples from this source for annotation.
also lot of studies
The data statistics of each source are presented in Table 1. The total number of original GEC samples for annotation is 5,000. We see that there exist some discrepancies between the three data sources regarding the average sentence length and the aver- age number of errors, which may help RobustGEC support a more comprehensive evaluation.
Robustness in GEC. Compared with other NLP areas, research on robustness in GEC is relatively scarce. Wang and Zheng (2020) study the robust- ness of GEC models under the scenario of varying numbers of errors. Raina et al. (2022) instead fo- cus on security and reveal that GEC systems can be easily fooled by appending an ad adversarial phrase. Unlike them, our work explores the robustness of GEC systems when nuanced modifications irrel- evant errors are introduced by users, which is most like the recent work from Feng et al. (2023). Addressing this issue is crucial, as the robustness against irrelevant variations could significantly af- fect users' trust in the GEC system's 's reliability.
3.2 Annotation Procedure
As mentioned in 82. one common practice many NLP fields, e.g., text classification (Zhang et al., 2019) and named entity recognition (Lin et al., 2021), is automatically constructing synthetic benchmarks to probe model robustness. Compared with these tasks, GEC is more sensitive to addi- tional grammatical errors introduced by synthetic context perturbations. We present two illustrative cases in Table 2 to explain why this method is not well-suited for GEC. We perturb Case#l with auto- matic synonym substitution (Zhou et al., 2021) and find this introduces new eтror (a lots of). This occurs because "lot" and "lots" are considered syn- onymous, but "lots" is inappropriate in this context. We perturb Case#2 by randomly deleting word. However, this changes the original error since the deleted "yesterday" is the evidence of the error (will give=>
3 RobustGEC Benchmark
3.1 Data Collection
To ensure a systematic and comprchensive evalu- ation for context robustness, we collect realistic GEC samples from three sources for annotation, i.e., CoNLL--14, BEA-19 and TEM-8.
CONI.L-14 (Ng et al., 2014) and BEA-19 (Bryant et al., 2017) are two widely-used GEC evaluation datasets. Sentences in CoNLL-14 are collected from student essays written by English- as- as-Secondary-Languaae learners, while BEA-19 comprises essays written by both learners and na- tives. Sentences in them are corrected by native teachers. We utilize the whole CoNLL-14 test set and part of the BEA-19 19 dev set for annotation.
From these examples, it becomes evident that synthetic context perturbation often introduces new grammatical errors or alters existing ones, leading to unreliable evaluation of robustness in GEC. To make our benchmark convincing. we ultimately
TEM-8 is a newly released dataset this work.
Ihttp://tem, fltonl ine.cn/

---

Source
Now, all he-hau have turn radio
Peplace
Source
Now,
radio
Target
Target
Now, he has to do is jurn on the radio or TV
Source
Now, all has to do is turn the radio TV and play musicc
Prweett
Parturo
Annotator
Target
Now, all he has to do is turn on the radio or TV and play music
We Instoad of the intransitive turn" to take ann object.
Explanation
Source
Now, all has to do is lurn the radio or7VV,
Target
or-PV,
Figure 2: An illustrative example the annotation procedure RobustGEC.
opt for manual annotation in the creation of Robust- GEC. We engaged 10 annotators, all of whom hold either bachelor's 's master's ' degrees in English and have successfully passed the TEM-8. The demo- graphic features (e.g., gender, age, and homeplace) are uniformly distributed among our annotators.
Annotation Analysis. After annotation, there is average of 1.29 perturbing edits per sample. This indicates that the perturbations in RobustGEC are quite unobtrusive. Nevertheless, in the later $4.2, we demonstrate that all evaluated GEC sys- tems, despite performing well on traditional bench- marks, still remain susceptible to such subtle pertur- bations. In addition, we observe that annotators ex- hibit preference for substitutions over insertions and deletions, and they tend to perturb nouns and verbs more frequently than other part-of-speech types, as nouns and verbs typically serve as the core components sentences.
Our annotation workflow is illustrated in Fig- ure 2. During the annotation process, we em- ploy the ERRANT toolkit (Bryant et al., 2017)² align the source and target sentences of the orig- inal GEC sample, extracting errors and their cor- responding corrections. We then highlight these errors using prominent colors and assign the sam- ple to annotators for perturbation. The annotator is allowed to perturb the non-error context by replac- ing/deleting/iserring content freely. The perturba- tion should be subtle, ensuring that it neither alters the intended meaning of the sentence involves excessive content changes. More importantly, the perturbation must adhere two principles:
Quality Assurance. To ensure the quality of Ro- bustGEC, we have adopted strict quality control protocol. In the beginning, we assign few tasks to the annotators for trial annotation. Only partic- ipants who have reached an accuracy of 90% can join the next stage. During annotation, each sub- mission will be assigned to an experienced reviewer for double-checking, We also organize discussions regularly to address questions raised by annotators. To further validate the annotation quality, we also conduct a human evaluation. Concretely, we ask two experienced judges to evaluate the qual- ity of each annotation sample from Correctness and Faithfulness. We randomly select 300 annota- tion samples for inspection. We calculate the inter- annotator agreement (IAA) ratio of two judges and then resolve their disagreement after discussion. As shown Table 3, most samples are acceptable considering correctness (97.3%) and faithfulness (99.3%) with high IAA ratios, demonstrating the satisfactory quality of RobustGEC.
Correctness:
the perturbation must NOT in- troduce any errors.
Faithfulness: the perturbation must NOT change the original errors.
The underlying motivation for these two princi- ples is to emulate situation where users make mi- enror-irreeevat modifications while expecting the GEC system maintain accurate, consistent corrections. We instruct annotators carefully re- view their annotations guarantee adherence to these principles. Each sample undergoes annota- tion with one perturbation by five distinct anno- tators. Finally, we obtain six variations of a GEC case by combining the original sample with the five perturbed versions.
3.3 Evaluation Metrics
An ideal GEC system should have both strong GEC ability and context robustness, so we conduct eval-

---

Accept
Reject (%)
IAA (%)
ator is the number of perturbed samples that are corrected consistently with the original samples. For instance, consider a GEC case in Robust- GEC with one original sample and five perturbed samples, where four perturbed samples share the same corrections as the original sample. In such a case, CRS is O while P-CRS amounts to 1/5 0.8.
Correctness Faithfulness
97.3
2.7
95.6
99.3
0.7
96.3
Table 3: Results of data quality inspection.
uations from these two aspects.
3.3.1 Metries for GEC Ability
Evaluating Robustness of GEC Systems
GEC ability refers to the performance of GEC sys- tems in correcting errors. Following Bryant et al. (2019), calculate precision (P), recall (R), and Fo.5 value by extracting and comparing the hypoth- esis and reference edits with the ERRANT toolkit (Bryant et al., 2017) to measure this ability.
In this section, we employ the RobustGEC bench- mark to conduct comprehensive test and analysis on the context robustness of existing GEC systems.
4,1 Selected GEC Systems
For a comprehensive evaluation, choose five representative GEC systems that cover three main- stream paradigms as introduced in $2, including the seq2seq method-Bal (Katsumata and Ko- machi, 2020), the seq2seq method with syntactic Showledge-Syneee (Zhang et al., 2022b), the seq2edit GathoddGECTool (Omelianchuk et al., 2020). the LLM-based method with GEC fine- Muning-LLLave (Touvron et al., 2023; Zhang et al., 2023b), and the LLM-based method in the zero-shot setting ChatGPT (OpenAI, 2023; Wu et al., 2023; Fang et al., 2023).
Unlike previous work that calculates the similar- ity between hypothesis output and one multiple golden references, thanks to RobusIGEC compris- ing one original sample with five corresponding perturbed variants, we can evaluate the GEC ability from multiple perspectives to gain more insights.
We report original GEC performance to eval- uate GEC performance on the original samples, facilitating comparison with existing work evalu- ated CoNLL-14 and BEA-19. Perturbations may result in either improvement GEC performance. Therefore, we measure upper- bound GEC performance and lower-bound GEC performance by selecting the variant that produces the highest and lowest Fo.5 values for each GEC case, respectively. The difference between upper and nower-bound performance (A Fo.5) can also be considered indicator of context robustness.
decline in
For all systems except ChatGPT, we fine-tunc them on CLang8 (Rothe et al., 2021), which is a commonly-usee GEC training set with about 2.4M sentence pairs. For implementation, we adopt the authors' official code. We present more details such as hyper-parameters in Appendix A.1.
For the prompt-based method using ChatGPT, we query gpt- 5-turbo (on Apr. 15, 2023) with the following prompt: "I want you to act as a grammar checker to correct explicit mistakes in the following sentence. Please directly return the cor- recied sentence without explanation." " This prompt is chosen according to our preliminary experimen- tal results. When calling OpenAl's API, we adopt greedy search to climinate randomness.
3.3.2 Metrics for Context Robustness
We further devise two new metrics measure the context robustness of GEC systems, namely Con- text Robustness Score (CRS) and Pair-wise Con- text Robustness Score (P-CRS).
CRS measures the GEC system's stability across all variants of GEC case, indicating its ability to maintain strictly consistent corrections. CRS is for- mally defined as #Casec #Casep* where the denominator is the total number of GEC cases, and the numer- ator is the number of GEC cases with consistent corrections for all variants.
4.2 Main Evaluation Results
We present the main evaluation results for all se- lected systems on RobustGEC in Table 4.
P- P-CRS is more lenient and evaluates the stability between each prigina sample pair, assess- ing whether the GEC system can retain consistency before and after each perturbation. It can be calcu- #P=samplec. where the denominator the lated as #P-sampley, eep' total number of perturbed samples, and the numer-
Overall Performance. We observe that all evalu- ated systems, despite performing well on conven- tional GEC benchmarks, lack sufficient robustness against context perturbations. This is evident from their substantial performance fluctuations, as in- dicated by the high (>20) AFo. values. On the

---

Original R
Upper-Bound
Lower-Bound
Model
Fo.s CRS
A Fo.s
P-CRS
P
Fo.
P
R
Fo.5
P
R
RobusIGEC- Cotaal
BART
55.06
37.92
50.49
65.38
42.30
58.95
35.78
31,44
34.82
24.13
43.5
85,4
SynGEC
56.21
38.81
51.59
66.30
43.17
59.89
36.14
31.87
35.20
24.69
43.2
85.3
GECToR
54.99
34,96
49.34
65.26
39.11
57.56
35.54
28.49
33.87
23.69
46.8
86.9
LLaMA
51.83
37.83
48.26
62.79
42.94
57.47
31.99
29.90
31.55
25.92
38.9
83.3
ChatGPT
36.94
49.50
38.91
47.29
56.07
48.82
23.03
40.49
25.20
23.62
20.5
75.2
CoNLI-11-Subbn Subset
BART
47.32
29.66
42.29
57.63
33.15
50.21
31.89
25.23
30.29
19.98
43.4
86.8
SynGEC
48.04
30.27
43.00
59.08
33.85
51.42
31.89
25.25
30.30
21.12
41.4
86.5
GECToR
43.83
30.55
40.33
52.77
33.98
47.51
31.39
26.06
30.16
17.35
45.4
87.6
LLaMA
43.72
30.38
40.19
55.29
34.74
49.44
27.62
24.16
26.85
22.59
35.1
82.9
ChatGPT
31.62
38.32
32.77
42.55
45.22
43.06
19.51
30.45
21.02
22.04
17.0
74.3
BEA-19-Subset
BART
62.95
43.68
57.84
72.78
48.24
66.06
40.14
36.60
39.38
26.68
83.4
SynGEC
63.87
44.52
58.76
72.90
49.00
66.42
40.30
36.97
39.59
26.83
41.8
83.3
GECToR
63.26
39.86
56.61
73.28
44.05
64.69
40.49
32.72
38.65
26.04
85.1
LLaMA
61.44
42.90
56.55
71.78
47.95
65.29
38.13
34.46
37.33
27.96
40.4
82.3
ChatGPT
40.50
53.94
42.62
50.90
60.11
52.51
25.75
45.35
28.18
24.33
75.2
TEM-8-Subee
BART
47.19
36.79
44.67
58.34
41.40
53.93
30.29
28.62
29.94
23.99
48.6
88.2
SynGEC
49.41
38.28
46.69
60.36
42.68
55.74
31.60
29.93
31.25
24.49
48.7
88.1
GECTOR
47.96
32.53
43.81
59.56
36.96
53.07
29.76
25.61
28.82
24.25
50.5
89.9
LLaMA
42.63
37.13
41.40
53.48
42.75
50.92
24.89
27.62
25.39
25.53
40.4
85.8
ChatGPT
35.98
56.71
38.81
45.79
62.40
48.36
21.46
44.78
23.95
24.41
22.9
76.2
Table 4: Main evaluation results on the RobustGEC benchmark. The P/R/Fo.s metrics are all calculated using ERRANT. A Fo.s is the absolute difference between the upper and lower bound of the system's Fo.s score.
entire RobustGEC benchmark, all systems achieve a CRS below 50, suggesting that they would pro- vide inconsistent corrections for more than half of our cases. Regarding the P-CRS metric, the highest value is only 86.9. These observations confirm that there is still considerable room for existing GEC systems improve their context robustness.
based methods, including fine-tuned LLaMA and zero-shot ChatGPT, exhibit inferior context robust- ness compared with conventional methods. Al- though LLMs possess more knowledge than small- size models like BART, they also suffer from hal- lucinations and other issues al., 2023; Zhang et al., 2023c), which may result in instability. Given that more and more individuals rely on LLMs as their personal grammar checkers today?, improving the context robustness of L.LM- based GEC systems is an important and mcaningful future challenge for GEC study.
severely (Bang
Performance Different Systems. As re- flected by the higher CRS and P-CRS scores, GEC- ToR consistently exhibits better context robustness compared other systems. One possible explana- tion for this could be that GECToR has an encoder- only sequence labeling architecture and predicts target edits independently, whereas the seq2seq models will be affected by the context perturba- tions twice at both the encoder and decoder sides. Incorporating linguistic knowledge (SynGEC) does not further enhance the context robustness of BART. We speculate that this may be because the syntactic parser in SynGEC for providing linguistic informa- tion will also be affected by context perturbations. Another noticeable phenomenon is that the LLM-
Performance on Different Subsets. We also list in detail the evaluation results of all systems on different subsets. The systems collectively achieve their best context robustness on TEM-8. As men- tioned before, TEM-8 sourced from an official exam, so errors in this subset are challenging and have deterministic correction methods. We find GEC systems tend to exhibit more stable perfor-
www.writeerrooo wtrtoaaaycottttttoooooooooooomo coaatttg pt-va- tra-grammarlly

---

P-415
SYNGEC
DART
SHGGEC
CECTOR
LLN
CUrGFT
Blow B/6d gilggs
(a) Influence of perturbing actions.
(h) Influence of perturhing positions.
(c) Influence of word frequency.
Figure 3: Fine-grained analysis of how context perturbations affect GEC systems, reflected by the P-CRS score.
mance for such errors compared to the expression- related errors in the other two subsets. In addition, on BEA-19, although most systems achieve their best GEC performance, their context robustness is actually the worst. This suggests that the GEC performance and context robustness may not nec- essarily be positively correlated.
Effect of perturbing word frequency. We con- duct an experiment to study the effect of the word frequency of the perturbations. For simplification. we only consider the substitution perturbation (e.g.. replace A with B) and count the number of occur- rences of the target word (i.e., R) in the training data. We categorize perturbations into three levels: low (less than 10 occurrences), medium (10 to 50 occurrences), and high (more than 50 occurrences). As shown Figure 3c, supervised systems (ex- cept the zero-shot ChatGPT) exhibit significantly low context robustness when the target word rarely appears in the training data. This phenomenon dis- closes that existing GEC systems may highly rely on spurious correlation patterns learned from train- ing data rather than real grammar knowledge to fix errors. As result, they perform inconsistently when generalizing to unseen contexts.
4.3 Fine-grained Analysis
To gain deeper insights, we categorize context per- turbations from different perspectives and employ the P-CRS score to probe whether they alter the original predictions of GEC systems.
Influence of perturbing action. We begin by ex- ploring the impact of the perturbing action type. The effects of the perturbation action type shown in Figure 3a. In comparison to substitution actions, insertion and deletion actions yield signif- icantly lower P-CRS scores, implying that GEC systems are more likely to produce inconsistent corrections when users insert or delete irrelevant content. The insertion and deletion actions often alter the sentence structure. Such changes in sen- tence structure may easily mislead GEC systems into unstable predictions.
5 Improving Robustness of GEC Systems
After witnessing the unsatisfactory context robust- ness of existing GEC systems, we further explore the approach to improving it. In this section, we try to propose simple yet effective method enhance the context robustness of GEC systems.
5.1 Context Perturbation Robust Training
Influence of perturbing position. We proceed analyze the impact of the position of the pertur- bation. Specifically, we only consider the perturbed sample with only one error and one perturbation. We then calculate the absolute word distance be- tween the perturbation and the target error. Figure 3b shows the results. We observe that GEC systems are prone to perform inconsistent corrections when the perturbation is closer the error. Words that are close together often belong the same sen- tence constituent and are closely related in terms of grammar and semantics, which could explain why perturbing them has a greater impact.
We propose Context Perturbation Robust (CPR) training method. The basic idea constrain the GEC system to output the same results disregard- ing the perturbations in contexts. To this end, our method compares an original sample (ar y) with perturbed sample (æ', 3/),),), and forces the GEC model to output the same prediction distribution at the non-perturb positions by minimizing the bi- directional Kullback-Leibler (KL. divergence. For instance, we feed the original sample (x, y) = (x 2²x3 3 . y 2 and the perturbed sam- ple (x', = (x x 2nx xx yo'yys " " into the

---

Model
O-P
O-R
O-Fo.s
U-P
U-R
U-Fo.5
L-P
L.R
L-Fo..
CRS
P-CRS f
GECToR (Baseline)
54.5
33.4
48,4
65.8
37.7
57.3
35,7
27.8
33.8
46,6
87.5
GECToR CPR (ours)
60.9
27.1
48.6
70.4
34.1
58.0
38.8
22.1
33.7
61.7 (+15.1)
90.3 (+2.8)
w/o KL loss
$7.5
29.8
32.3
37.4
22.6
33.4
(+4.3)
87.9 (+0.4)
Real- -Synthetic
60.7
26.4
48.2
70.5
31.9
56.8
38.0
22.3
33.3
55.9 (+9.3)
88.8 (+1.3)
GECToR + Inf. Tweak
61.1
27.9
49.4
71.2
33.9
58.4
38.1
23.9
34.1
51.3 (+4.7)
89.0 (+1.5)
Table 5: Main results of our Context Perturbation Robust (CPR) training method for improving the robustness of the P/R/Fo.5 values, respectively.
GECToR model. O/U/L-//R/Fo. denote the
GEC model, where æ" x and y" "y are the per- turbed tokens the source/target sides. Subse- quently, we can get two prediction distributions of two samples, denoted as P(y) = p²p² and Q(y((z") = 94 Finally, our method calcu- lates the bi-directional KL-divergence loss at the non-perturb positions 2 and 3 as follows:
5.3 Results and Analysis
Our method improves context robustness by large margins. We present the primary results Table 5. As observed, post-training GECToR with our CPR method on real original sam- ples significantly enhances its context robustness. The CRS and P-CRS scores improve by +15.1 and +2.8 points, respectively. Notably, the post-training does not affect the GEC ability of GECToR, the changes O/U/L-Fo..3 values are minimal. These results demonstrate the effectiveness of our pro- posed CPR method as a lightweight sdug-and-play technique for improving context robustness. We also perform a case study in Appendix C.
CKI. = (xi(p2119)) + (xI(p"1l??)
(DKL(p + Dr DKu(o?11p?)
(1)
1 2 (DKI(D'lla)) DKL (KL
The original learning objective is the negative log-likelihood loss function, formally defined as:
Post-training without explicit constraints in- sufficient. We also report the results of removing the Kl--divergence loss (w/o KL loss), which en- tails post- pest-training on original=pperurrbel samples without an explicit constraint. After removing the KL-divergence loss, the boosts in CRS and P-CRS metrics become considerably smaller than before. This finding highlights the importance of explicitly enforcing the GEC system to produce consistent results using the KL-divergence loss. Furthermore, we examine the impact the weight factor a of the KL-divergence loss Appendix D.
ONLL- = logP(() - 10/1(( (2)
The final training object of our CPR method to minimize C, which can be calculated as:
C = CNLL + a LKL
where a is the coefficient weight to control LKL-
Synthetic perturbation data is also useful but not effective real data. In previous experi- ments, we utilized numan-annotated perturbed sam- ples from RobustGEC. We can also generate per- turbed samples using heuristic rules, which are eas- ier to obtain. For comparison, we generate the same amount of synthetic samples by randomly substi- uuing/inserring/teleing contents. More details can be found in Appendix B. As shown in Table 5 (Real-=Synthetic the improvement of context robustness diminishes after using synthetic data (+15.1-++9.3 for CRS, +2.8 for P-CRS), vealing a non-negligible gap between real and syn- thetic perturbations. Nonetheless, the improvement remains significant, indicating that our method has
5.2 Experimental Setup
For calculating the KL-divergence loss, concate- nate the original samples and the corresponding per- turbed samples to form training batch. We align them using the edit distance algorithm to achieve the perturb positions. We utilize real perturbed sam- ples in our method for better results. Consequently, we divide RobustGEC into train/dev/test splits with 3,000/500/2,500 GEC cases, respectively. We test method on top of GECTOR. As mentioned in 54.2, GECToR has better context robustness than other systems, so further enhancing it is more chal- lenging. We set the weight a LKL as 1.0. More details can found in Appendix A.2.

---

the potential to be extended scenarios where auman-annotated perturbation data unavailable.
Limitations
Benchmark Limitation. The scope of the data source of RobustGEC may be somewhat limited. We only consider English GEC, while there have been many studies in other languages like Chinese. Additionally, the three data sources we selected pertain only to formal writing, whereas informal writing also has a great demand for GEC. Given the aforementioned flaws, we plan to continuously update and improve RobustGEC in the future.
The effectiveness of our method does not come from minimal modification. As observed, our method improves the GEC model's precision but reduces recall. also possible that our boost in context robustness could stem from a more con- servative correction trend because there exists a shortcut-ifa a GEC system does not correct at all, it will achieve full robustness scores as its results are very "consistent". To discuss this, compare our method with directly tweaking the inference of GECToR, which involves adding bias to the probability of the KEEP tag to avoid changing the source token. From the results (GECToR + Inf. Tweak), we observe that tweaking the inference leads GECToR to achieve a similar precision/recall trade-off as us. However, the improvement in con- text robustness is less pronounced under this setting. This indicates that the gains in context robustness we achieved are not solely due making the GEC system correct more conservatively, but also come from teaching it to tolerate irrelevant perturbations.
Method Limitation. Regarding our approach enhancing context robustness, the primary limita- tion is that we only test it on a single type of GEC system, specifically, GECToR. In reality, our CPR method is model-agnostic, and we intend to eval- uate its effectiveness on a broader range of model structures in the future. Furthermore, our method is just simple and preliminary attempt. We hope fu- ture research will explore more effective techniques for improving robustness by using RobustGEC.
Ethics Statement
Data License. When building RobGEC, col- lect data from three sources, namely CoNLL-14, BEA-19, and TEM-8. The first two are publicly available GEC datasets, and the application of them in our study is consistent with their intended use and license. As for the TEM-8 data, we have consulted with professional legal advisors before collecting them. The advisors have confirmed that gathering exam questions from official institu- tional organizations on external educational web- sites does not infringe upon copyright laws, pro- vided that no additional analytical content from these external sites is used. Besides, we do not use the outputs from test-takers, only gold answers are used. So there is no privacy problem for test- takers. We commit that all data will be used only for research purposes.
6
Conclusion
This paper investigated the context robusiness of GEC systems. This is the ability to provide con- sistent correction suggestions when modifications unrelated to errors are introduced into the inputs. To quantitatively analyze this, we developed di- agnostic benchmark named RobusIGEC. Using Ro- bustGEC, we disclosed that popular GEC systems have sigmificant room for improvement in context robustness. Furthermore, we proposed simple yet effective method for enhancing context robustness. We hope our work can offer insights for future re- search aimed at developing robust GEC systems.
Acknowledgement
Annotation Payment. All annotators involved in our annotation were paid properly according to their annotated task numbers and quality. The average salary is about 60 RMB per hour, which is much higher than the legal standard in our country.
We would like to thank the anonymous reviewers and the meta reviewer for their valuable comments. We want to thank Prof. Zhenghua Li for his in- sightful suggestions. We also want to thank Dr. Xing Wang for helping us conduct the annotation experiment as mentioned in Appendix E. This work was partially supported by the Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.
References
Abhiject Awasthi, Sunita Sarawagi. Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Par- allel iterative edit models for local sequence trans-

---

duction. In Proceedings of EMNLP-IJCNLP" pages 4270.
Allennlp: A deep semantic natural language process- ing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A mul- titask, multilingual, multimodal evaluation of Chat- GPT on reasoning, hallucination, and interactivity. arXiv preprint urXiv:2302.04022 04003.
Roman Grundkiewicz, Christopher Bryant, and Mariano Felice. 2020. A crash course in automatic grammat- ical error correction. In Proceedings of COLING: Tutorial Abstracts, pages 33-38.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.0666
Christopher Bryant, Mariano Felice, Øistein E Ander- sen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of BEA AACI, pages 52-75.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of EMNLP, pages 2021-2031.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceed- ings of ACL, pages 793-805.
Marcin Junczys-oommun Roman Grundkiewicz. Shubha Guha, and Kenneth Heafield 2018. Ap- proaching neural grammatical error correction as a low-resource machine translation task. In Proceed- ings of NAACL-HLT, pages 595-606.
Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2022. Grammatical error correction: A survey of the state of the art arXiv preprini arXiv:2211 2itiv:221100055 1.05666..
Satoru Katsumata and Mamoru Komachi. 2020. Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. In Pro- ceedings of AACL, pages 827-832.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mixtext: Linguistically -informed interpolation of hidden space for semi-supervised text classification. In Procedd ings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 2147-2157.
Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2019. Learning the difference that makes a difference with counterfactually-agueee augmente data. Proceedings of ICLR.
Steven Coyne and Keisuke Sakaguchi. 2023. An anal- ysis of GPT-3's performance in grammatical error correction. ArXiv, /tbs/2303.143422
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: demoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of ACL, pages 7871- 7880.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of NAACL-HLT, pages 568-572.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018 Hotflip: White-box adversarial examples for text classification. In Proceedings of ACL (Short), pages 31-36.
I Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversarial Attack Against BERT Using BERT. In Proceedings of EMNLP, pages 6193-6202.
Tao Fang. Shu Yang. Kaixin Lan, Derek F Wong, Jin- peng Hu, Lidia Chao, and Yue Zhang. 2023. ChatGPT a highly fluent grammatical error correc- tion system? comprehensive evaluation. arXiv prepriat 204442230..01746.
Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. Rockner: A simple method to create adversarial examples for evaluating the robust- ness of named entity recognition models. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9728-3733
Kaiqi Feng, Zecheng Tang, Juntao Li, and Min Zhang. 2023. Beyond hard samples: Robust and effec- tive grammatical error correction with cycle self- augmenting. In Proceedings of NLPCC, pages 678- 689.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan. Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. 2021. Just train twice: Improving group robustness without training group information. In Proceedings of ICML, pages 6781- 6792.
Simon Flachs, Ophélie Lacroix, Helen Yannakoudakis. Marek Rei, and Anders Søgaard. 2020. Grammatical eтror correction in low error density domains: a new benchmark and analyses. In Proceedings of EMNLP, pages 1467-8478.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint 1907 11692.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018.

---

Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoaki Okazaki. 2023. Exploring effectiveness of gpt-3 in grammatical enror correction: A study on per- formance and controllability in prompt-basce meth- ods. arXiv preprinl vrXiv:2305./8/50
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1 2397.
Courtney Napoles, Maria Nädejde, and Joel Tetreault. 2019. Enabling robust grammatical error correction in new domains: data sets, metrics, and analyses. TACL, 7:551-566
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spuri- ous correlations using pre-trained language models. TACL, 8:621-633.
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: a fluency corpus and bench- mark for grammatical error correction. In Proceed- ings of EACL, pages 229-234.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 5998-6008.
Hwee Tou Ng. Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction, In Proceedings of CoNLL: Shared Task, pages 1-14.
Lihao Wang and Xiaoqing Zheng. 2020. Improving grammatical error correction models with purpose- built adversarial examples. Proceedings of EMNLP, pages 2858-2869.
Hwee Tou Ng. Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL- 2013 shared task grammatical error correction. In Proceedings of CoNLL: Shared Task, pages 1-12.
Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. Measure and improve robustness in nlp models: A survey. In Proceedings of NAACL-HLT, pages 4569- 4586.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chemoddu,, and Oleksandr Skurzhanskyi. 2020.
Yu Wang. Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. 2021. A comprehensive survey of grammatical error correction. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1-51.
GECToR-grammatien error correction: tag, not rewrite. In Proceedings of BEA@ACL, pages 163- 170.
OpenAl. 2023. ChatGPT. https://openai... /blog/chatggt..
Haoran Wu, Wenxuan Wang. Yuxuan Wan, Wenxi- ang Jiao, and Michael Lyu. 2023. ChatGPT or grammarly? evaluating ChatGPT grammati- cal error correction benchmark. arXiv preprint arXiv:2303. 1.33003.344. 13648.
Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of HLTDem) pages 48 53.
Xiaoyu Xing, Zhijing Jin, Di Jin, Bingning Wang, Qi Zhang, and Xuan-Jing Huang. 2020. Tasty burg- ers, soggy fries: Probing aspect robustness in aspect- based sentiment analysis. In Proceddiggy of EMNLP, pages 3594-3605.
Vyas Raina, Yiting Lu, and Mark Gales. 2022. Gram- matical error correction systems for automated assess- ment: Are they susceptible to universal adversarial attacks? In Proceedings of AACL, pages 158-171.
Linyi Yang, Lifan Yuan, Leyang Cui, Wenyang Gao, and Yue Zhang. 2022. Factmix: Using few labeled in-domain examples to generalize to cross-domaie named entity recognition. In Proceedings of COL- ING, pages
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial exam- ples through probability weighted word saliency. In Proceedings of ACL. pages 1085-1097.
Yu Zhang, Yue Zhang, and Leyang Cui Guohong Fu. 2023a. Non-autoreeressive text editing with copy- aware latent alignments. In Proceedings of EMNLP.
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of ACL-IJCNLP, pages 702-707.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of NAACL-IILT, pages 1298-1308.
Felix Stahlberg and Shankar Kumar. 2020. Seq2edits: Sequence transduction using span-level edit opera- tions. Proceedings of EMNLP, pages 5147-5159.
Yue Zhang, Leyang Cui, Deng Cai, Xinting Huang. Tao Fang, and Wei Bi. 2023b. Multi-task instruc- tion tuning of llama for specific scenarios: A pre- liminary study on writing assistance, arXiv preprint arXiv:2305.1322
Felix Stahlberg and Shankar Kumar. 2021. Synthetic data generation for grammatical error correction with tagged corruption models. Proceedings of BEA @ EACL, pages?3 37-47.

---

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang. Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023c. Siren's song in the ai ocean: survey on hallucination in large language models. arXiv preprini urXiv:2309.0120)
2021). This model has about 406M parameters. The training hyperparameters used are on par with Katsumata and Komachi (2020). For decod- ing, we use the beam search algorithm with a beam size of 12.
Yue Zhang, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022a. MuCGEC: a multi-reference multi-source evaluation dataset for Chinese grammatical error cor- rection. In Proceedings of NAACL-HLT, pages 3118- 3130.
SynGEC. SynGEC (Zhang et al., 2022b) is a GEC system built on the top of BART. addition- ally incorporates tailored dependency syntax infor- mation into BART by using a GFC-oriented parser, and demonstrates such linguistic knowledge can further lead to substantial performance improve- ments. We reproduce the SynGEC system using their official code?. We also train this model CLang8 using the default hyper-parameters as in their paper (Zhang et al., 2022b).
Yue Zhang, Bo Zhang, Haochen Jiang, Zhenghua Li, Chen Li, Fei Huang, and Min Zhang. 2023d. NaS- GEC: multi-domain Chinese grammatical error cor- rection dataset from native speaker texts. In Proceed- ings of ACL (Findings), pages 9935-9951.
Yue Zhang, Bo Zhang, Zhenghua Li, Zuyi Bao, Chen Li, and Min Zhang. 2022b. SynGEC: Syntax-enhanced grammatical error correction with tailored gec- oriented parser. In Proceedings of EMNLP, pages 2518-2531.
GECTOR. GECToR (Omelianchuk et al., 2020) is sequence-to-edie (Seq2Edit) GEC system that corrects ungrammatical sentences via predicting edits such as keeping and deleting. Unlike previous Seq2Seq methods, features only a BERT-based encoder followed by a softmax layer for predic- tion, which makes its prediction speed very fast, We train GECToR with their official implementa- tion We choose the Roberta-Large model (Liu et al., 2019) as the encoder of GECToR, which has about 354M parameters. Before training, we convert the error-correct sentence pairs in CLang8 to labels using the tool provided by the official GitHub repository. The hyper-paramette parameters for train- ing and predicting directly refer to the original pa- per (Omelianchuk et al., 2020).
Yi Zhou, Xiaoqing Zheng, Cho-Jui Hsieh, Kai-Wei Chang, and Xuanjing Huan. 2021. Defense against synonym substitution-bu basdd adversarial attacks dirichlet neighborhood ensemble. In Proceedings of ACT, pages 5482-5492.
Chen Zhu, Cheng Yu, Gan 7he, Sun Siqi, Goldstein Thomas, and Liu Jingjing. 2020. Freelb: Enhanced adversarial training for language understanding. Pro- ceedings of ICLR.
A More Implementation Details
A.1 Implementation of GEC Systems
The implementation details of the five GEC sys- tems evaluated on RobustGEC are listed below. All experiments are conducted using 8 Nvidia Tesla- V100 32 GB GPUs. Most experiments could be complemented within several hours.
LLaMA. LLaMA (Touvron et al., 2023) an open-sourced large language model (LLM) that exhibits remarkable performance across various tasks. It is a decoder-only model with a vast num- ber of parameters from 7B to 65B. Zhang et al. (2023b) found that LLaMA could achieve decent GEC ability after fine-tuning with GEC training data. Following Zhang al. (2023b), we fine-tune LLaMA-7B (Touvron al., 2023) on the CLang8 training data. Considering the time and resource costs, we conduct parameter-efficient fine-tuning by utilizing the low-rank adaptation technique (Hu et al., 2021). The taken from Zhang et al. (2023b).
BART. BART (Lewis al., 2020) is pre-trained encoder-decoder model which is widely used for sequence-to-sequencee (Seq2Seq) modeling. The pre-training task of BART re-constructing the corrupted texts. Katsumata and Komachi (2020) has proven that BART can achieve competitive per- formance on conventional GEC benchmarks. We
employ the Fairseq' toolkit (Ott et al., 2019) to fine-tune BaRT- model (Lewis al., 2020) on the CLang86 GEC training data (Rothe al.,
are directly
irseq httoss//aaagiinffeeec////eebee/
large
/https://g/ thub.commgoog http://t......oooooooooeeeeeeor. tasets/clang8
synGse
hhttte://githhu..com/grrs/

---

ChatGPT. Recently, many works have shown that ChatGPT can achieve promising GEC perfor- mance by using only zero/few-shot prompts (Wu et al., 2023; Fang et al., 2023; Loem et al., 2023). this work, we also take the prompt-based GEC approach with ChatGPT into consideration. The prompt we used is described in 64.2. We set the temperature parameter as 0.0, which means we perform the greedy decoding. The reasons are two- fold. First, this will make the results fixed and reproducible. Second, we find this will lead to better GEC performance.
GECToR Baseline
0-S
Such people never bump up other people.
0-T
Such people never bump into other people.
O-H
Such people never bump into other people.
Such people never bump up other people because they, are very, careful,
PS
x
Such poople never bump into other people
P-T
because they very eeryful
Such people never bump other people
P-H
because they are very careful,
GECToR R after CPR post-training
O-S
Such people never bump up other people.
O-T
Such people never bump into other people.
O-H
Such people never bump into other people.
Such people never bump up other people
P-
because they are very, careful.
Such people never bump into other people
P-T
A.2 Implementation of CPR Training
hecause they are very careful,
Such people never bump into other people
During the CPR post-training, we combine the original sample with each of the corresponding perturbed samples to form contrastive pair. We implement GECToR and our CPR method with the ALLENNLP toolkit (Gardner et al., 2018) We train GECToR on CLang8 until convergence as our baseline and post-train it using our CPR method on the train split of RobustGEE. During training. we set the learning rate as le-6, the batch size as 128, the dropout rate as 0.3, and the weight of CK.. as 1.0. The experiments are conducted on 8 Nvidia Tesla-V100 32 GB GPUs, and the results in Table are averaged over runs with different random seeds. The training cost is relatively inex- pensive and only takes approximately 20 minutes, thereby our method is lightweight and easy to train.
because they very careful.
Table 6: A real case shows that post-training GEC- ToR with our proposed CPR method makes it output consistent and accurate correction. O/P-S/T/H denotes Original/Perturbed- Soaedd-Souueeaaaaee Taarrr///Trre//yytohhessss We use Red and Green to highlight the grammatical error and the correct modification, respectively. We also use the underwave line to mark the perturbation.
65
55
50
45
40
0.1
0.5
3
B Synthetic Perturbation Details
a
As described in & 5.3, we generate synthetic per- turbed samples using predefined rules for compari- son with real human-annotated perturbed samples. We take 3,000 original GEC samples from the train split of RobustGEEC as the seed corpus and create five synthetic perturbed samples for each, ensuring a fair comparison. Each sample is perturbed only once, with substitution, insertion, or deletion ap- plied randomly with equal probability. For substi- tution, we randomly select word in the non-error context, mask it, and then use masked language model, such as Roberta, to fill in the masked slot. For insertion, we pick random token from the vocabulary and insert sition. For deletion, we randomly remove a word from the non-error context.
Figure 4: Original Fo.s and CRS curves with different
a.
C
Case Study
We present a case study in Table 6. As can be seen, the GECToR baseline successfully corrects the grammatical error (up = into) in the original sample, but fails to correct this error when an insertion perturbation is introduced. However, af- ter post-training GECToR with our method, can consistently and accurately correct errors in both original and perturbed samples, thereby demon- strating the effectiveness of our method.
random non-error po-
D Analysis of the Weight of KL Loss
"https::/chat..qeent "https: http:////gthaahhh.aaaaaaaaaaaa /alenn/nl
We conduct an experiment to investigate the impact of the weight factor a of the KL-divergence loss

---

LKL in Eq. 3. The results illustrated in Figure 4. As the value of increases, we observe that the context robustness of the GEC system improves, while the GEC ability deteriorates. is also evident that setting a to 1.0 achieves the best trade-off. Consequently, we set a to 1.0 in our experiments.
E Comparison between Native and Non-native Annotators
It is important to highlight that the annotators in- volved in constructing RobustGEC are not native English speakers. To examine the potential impact of the annotators' native language on the data di- versity, we launched experiment that engaged native English speakers to perform annotations. In our preliminary annotation, which includes 100 cases, we compared the perturbations annotated by both native and non non-native speakers. Interest- ingly, we found no significant linguistic differences, such as in edit actions and POS-tags, between the two groups of annotators. The CRS and P-CRS for BART are 41.7/83.5 on data annotated by non- native speakers and 40.8/82.8 on data annotated by native speakers, indicating only a minor difference. These findings suggest that, despite our annotators being non-native English speakers, their English proficiency is sufficiently high, enabling them to produce annotations comparable to those of native speakers.


Reviews:
[{"review_id": "rEChpnG0P9", "reviewer": "Reviewer_t6Xg", "summary": "", "questions": "My only slight concern is that the similarity in background of the annotators could result in a bias wrt the types of contextual perturbations the annotators generate. I think the authors would do well to add a few sentences about the diversity in the perturbations to show that the perturbations are not unnaturally uniform, as this could lead to learning a new set of spurious correlations rather than actually improving learning of grammatical patterns. I'm not saying this is the case, but it is something the authors need to think about.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Dear Reviewer t6Xg\nWe sincerely thank you for your great effort spent on our paper. We are also very glad to see your recognition of our work. Here are our replies to your concerns.\n\n> Q: The similarity in background of the annotators could result in a bias wrt the types of contextual perturbations the annotators generate.\n\nA: Yes, we agree. The potential bias arising from the annotator background is worth further exploration. We will add the following discussions in the revision of our paper.\n\n(1) The **demographic features** (e.g., gender, age, education) are uniformly distributed among our annotators. \n\n(2) We also analyze the **diversity** of the annotated perturbations from the linguistic aspects.\n\n|   Action   | Substitution | Insertion | Deletion |\n|:----------:|:------------:|:---------:|:--------:|\n| **Proportion** |     50.34    |   23.71   |   25.95  |\n\nIn terms of the *type of actions*, we can see that despite most perturbations being substitutions, the insertions and deletions still account for a consideration portion.\n\n|   Pos-Tag   | Adjective | Adverb | Noun | Preposition | Verb | Others |\n|:----------:|:------------:|:---------:|:--------:|:------------:|:---------:|:--------:|\n| **Proportion** |   19.43    |   8.90   |   34.18  |  4.22   |  27.05  |  6.22 |\n\nAs shown above, the *perturbing part-of-speech (pos) tag* is also naturally distributed. \n\nThese observations could demonstrate that the perturbations in RoubstGEC are naturally uniform. \n\n(3) To enhance the diversity of our annotators' backgrounds, we additionally employ **native English speakers** to conduct annotations following your suggestions. \n\nBased on our preliminary annotation (100 cases), we compare the perturbations annotated by native and non-native speakers, and find that there is no significant linguistic differences (e.g., in edit action and pos-tag) across two group annotators. \n\nThe CRS and P-CRS for BART are 41.7/83.5 on non-native-annotated data and 40.8/82.8 on native-annotated data. The difference is also marginal. \n\nThese suggest that despite our annotators being non-native English speakers, their proficiency in English is high, enabling them to perform annotations similar to native speakers.\n\nHope our responses can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well thought out and well written paper, and I think it is a significant contribution to the GEC community. The proposed dataset, especially, stands to benefit English GEC research. The authors raise a good point about the implications of lack of robustness to context perturbations; this may well be an indication that models are learning spurious correlations, and forcing the model to learn to output similar corrections in varied contexts may ameliorate the issue. Also, the authors experiments with synthetic data are useful as they make the authors' approach more widely accessible. Overall, I think this is a good paper and should be accepted.\n\nAlso, this annotation process was clearly a significant undertaking, and this type of data resource needs to be publicized.", "reasons_to_reject": "I don't have any specific reasons to reject. I think this paper is well thought out and argued. And the authors were quite thorough in describing their annotation procedure (which is challenging to evaluate given the fact that the annotators are allowed to make varied contextual changes, meaning there is no way to directly compare agreement between annotators). I think that they have done a good job of explaining and defending their decisions in the annotation process. ", "questions_to_authors": "My only slight concern is that the similarity in background of the annotators could result in a bias wrt the types of contextual perturbations the annotators generate. I think the authors would do well to add a few sentences about the diversity in the perturbations to show that the perturbations are not unnaturally uniform, as this could lead to learning a new set of spurious correlations rather than actually improving learning of grammatical patterns. I'm not saying this is the case, but it is something the authors need to think about.", "paper_topic_and_main_contributions": "This paper proposes a new dataset for improving model robustness to non-error variation in written English GEC. Specifically, the authors discuss and demonstrate the fact that several leading GEC models show marked variation in proposed corrections to the same error when the context is varied in ways that should not affect the correction. As the authors point out, this is a major concern as reliability is key to engendering confidence in GEC system users; a system that gives you different corrections when sentence context is slightly changed is not trustworthy to users. In order to demonstrate this effect, the authors annotate a new dataset containing contextual perturbations to show how GEC systems often output different, and sometimes incorrect, corrections when non-error context is varied. They draw the dataset from two existing GEC benchmarks (BEA-19 and CoNLL-14) as well as a new data source, the TEM-8 test for Chinese students of English. The start with a set of 5000 GEC sentence pairs and ask 10 annotators who are proficient (though non-native) English speakers to generate 5 perturbations for each sentence. They first use this data to test 5 different well-known GEC systems, and demonstrate that these systems all, to varying degrees, have difficulty in generating consistent corrections when faced with non-error perturbations. The authors argue that this may be evidence of these models not having actually learned the grammatical patterns of English, but rather spurious correlations that allow them to perform reasonably on the various benchmarks. The authors also develop a method of fine-tuning the non-LLM models to encourage consistency in generated corrections by penalizing larger KL divergence values between sentences which should result in the same correction. They show that this approach can substantially increase model robustness to perturbation. The authors also show that this type of fine-tuning with synthetic perturbations can also increase model robustness, though to a lesser degree. However, this means that even for situations in which annotating new data isn't feasible, the authors approach could help stabilize model output. The authors say they will release their code and dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "mkpHJKZ78c", "reviewer": "Reviewer_3EQP", "summary": "", "questions": "- A. For the coefficient weight $\\alpha$ in Eq. 3, the authors chose not to include a balancing weight (1-$\\alpha$) to the NLL loss. What are the reasons behind this choice?\n\n- B. Line 423: “The systems collectively achieve their best context robustness on TEM-8.” I noticed that the average number of errors of TEM-8 is much lower (1.47) than other data sources (2.60, 3.12). Is it possible that a lower number of errors in the TEM-8 dataset contribute to the high robustness phenomenon of GEC systems trained on it?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Dear Reviewer 3EQP\nWe are very grateful for your in-depth and insightful comments, which will greatly help us improve our paper. For your concerns, we provide the following responses.\n> Q: Copyright about TEM-8.\n\nA: We have consulted with professional legal advisors before collecting this data. Our advisors have confirmed that gathering exam questions from official Chinese institutional organizations on external educational websites does not infringe upon copyright laws, provided that no additional analytical content from these external sites is used. Besides, we do not use the outputs from test-takers, only gold answers are used. So there is no privacy problem for test-takers. We will make these clear in the next version. Thanks for this kind reminder.\n\n> Q: The paper suggests the potential use of LLMs for GEC systems without concrete evidence of tech giants developing such systems. \n\nA: Actually, some tech companies, such as Grammarly[1] and Google[2], have built writing assistant systems based on LLMs. Such systems can handle GEC and can be used by individuals. Moreover, individuals can query ChatGPT for GEC, which is affordable and effective, as shown in [3]. We will add such information in the revision.\n\n*Reference:*\n\n[1] Raheja, Vipul, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. \"CoEdIT: Text Editing by Task-Specific Instruction Tuning.\" arXiv preprint arXiv:2305.09857 (2023).\n\n[2] Shu, Lei, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng. \"RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting.\" arXiv preprint arXiv:2305.15685 (2023).\n\n[3] Fang, Tao, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. \"Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation.\" arXiv preprint arXiv:2304.01746 (2023).\n\n> Q: The training data of ChatGPT is unknown for now, how can we ensure the divided types of word occurrences are reasonable for ChatGPT?\n\nA: Yes. The analysis on perturbing word frequency is somewhat unreasonable for ChatGPT, as its training data is unknown. The experimental results in Figure 3(c) show that ChatGPT exhibits stable robustness for different types of word occurrences. We have mentioned this phenomenon on Line 473. We will make it clear in the revision.\n\n> Q: For the coefficient weight $\\alpha$ in Eq. 3, the authors chose not to include a balancing weight (1-$\\alpha$) to the NLL loss. What are the reasons behind this choice?\n\nA: Please kindly note that these two weighting methods are essentially equivalent. For $L = \\alpha L1 + (1-\\alpha) L2$, we can convert it to $L/(1-\\alpha) = (\\alpha/(1-\\alpha)) L1 + L2$, and set $\\beta=\\alpha/(1-\\alpha)$ and $L'=L/(1-\\alpha)$. Then we can get $L' = \\beta L1 + L2$. Hence, the selection of either method is valid. In our design, we opted for the simplicity of adding just one weight for the KL loss.\n\n> Q: Is it possible that a lower number of errors in the TEM-8 dataset contribute to the high robustness phenomenon of GEC systems trained on it?\n\nA: Yes. We also agree this could be an important factor, and will discuss this in the revision.\n\nWe will meticulously revise our paper according to other helpful suggestions about typos and unclear statements. We do hope our responses can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Novelty**: This paper is novel compared to the related works in the field of GEC robustness. It is the first to explore context perturbations and introduces a novel benchmark along with an assessment protocol. \n- **Implications**: The investigation into context perturbations is vital because these perturbations can be introduced by users, indicating the potential for improving the reliability of GEC systems in real-world applications.\n- **Resources**: Section 3 outlines a thorough creation procedure for the new datasets, including strict quality control. The collected perturbation data and TEM-8 benchmarks are of high quality and hold valuable utility for examining robustness problems within the GEC community. \n- **Methodology**: In addition to providing a robustness benchmark, this paper presents a practical solution for addressing context perturbations. Remarkably, the proposed method remains effective even for the most robust baseline (GECToR).", "reasons_to_reject": "- **Ethics**: The paper demonstrates a thorough consideration of ethics while creating the new benchmark. However, there is a notable concern regarding copyright. TEM-8 is a language test organized by an official Chinese institution, but the authors claim that it was collected from \"educational websites,\" without providing clarity on whether these websites are officially authorized sources. Besides, it is also unclear how the authors deal with the privacy problems of test takers of TEM-8.\n\n- **Analysis**: Some arguments in this paper may not be clear. I list them below:\n  - 1) Line 417: “Given that numerous individuals rely on LLMs as personal grammar checkers today…”. Running LLMs can indeed be expensive for individuals, and many might prefer using commercial products due to their availability and cost-effectiveness. However, the paper suggests the potential use of LLMs for GEC systems without concrete evidence of tech giants developing such systems. If the authors wish to maintain this point, they need to provide further evidence or references to support their claim.\n\n  - 2) Line 470: “...count the number of occurrences of the target word (i.e., B) in the training data…”. According to Appendix, the reproducible GEC systems are trained/fine-tuned on the CLang8 data. But the training data of ChatGPT is unknown for now, how can we ensure the divided types of word occurrences are reasonable for ChatGPT?", "questions_to_authors": "- A. For the coefficient weight $\\alpha$ in Eq. 3, the authors chose not to include a balancing weight (1-$\\alpha$) to the NLL loss. What are the reasons behind this choice?\n\n- B. Line 423: “The systems collectively achieve their best context robustness on TEM-8.” I noticed that the average number of errors of TEM-8 is much lower (1.47) than other data sources (2.60, 3.12). Is it possible that a lower number of errors in the TEM-8 dataset contribute to the high robustness phenomenon of GEC systems trained on it?", "paper_topic_and_main_contributions": "This paper aims to examine and improve the robustness problem of grammatical error correction (GEC) systems, with a particular focus on natural context perturbations. The main contribution of this study is the collection of five perturbed variants of GEC data instances through human labor, followed by the evaluation of systems' robustness in terms of correction consistency and performance drop. In addition to annotating the conventional benchmarks (CoNLL-14, BEA-19), the authors have also created a new benchmark, TEM-8, and annotated perturbed instances. The robustness evaluation covers prevalent GEC systems (seq2seq, seq2edit, LLMs) across the three benchmarks mentioned earlier. The experimental results reveal that context perturbations can mislead the correction behaviors of current GEC systems. Furthermore, the authors propose a preliminary KL-divergence-based method to alleviate the robustness problems.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Caption: “Table 3: Results of data quality inspection” -> “Table 3: Results of data quality inspection.”\n- Line 196: “1k GEC samples” -> “1,000 GEC samples”\n- Line 494: “Kullback-Leibler (KL) divergence.” -> “KL divergence.” (the abbreviation already appeared in the previous section)", "Justification_for_Ethical_Concerns": "- I am not sure whether the sources of TEM-8 data are authorized since the authors did not talk about them. As such, it remains uncertain if the data collection process adheres to copyright regulations and permissions.\n\n- The paper does not provide information on how the authors handled the privacy issues associated with the test takers of TEM-8. \n\nTo address these ethical concerns, the authors should provide transparent and detailed information about the sources of TEM-8 data, and privacy consents to ensure compliance with copyright and privacy laws."}}, {"review_id": "SNIq9KyAWS", "reviewer": "Reviewer_hW9g", "summary": "", "questions": "In section 3.1 (or link to an appendix), can you go into more detail on how you sampled data from TEM-8?  Was this random or were you sampling stratified on anything such as demographics, learner-level, etc.?\n\nWhen describing your inter-annotator agreement in section 3.2 under \"Quality Assurance\", can you specify which metric you used to measure inter-annotator agreement?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Dear Reviewer hW9g\n\nThank you for your hard work. Below are our responses to your concerns.\n\n>Q: This paper would find a very receptive audience at BEA, but unfortunately BEA 2023 already happened and the authors may not have been able to travel to Toronto.\n\nA:  We appreciate the reviewer's recognition that our paper would find a receptive audience at the BEA2023. Unfortunately, the deadline for BEA2023 was May 2, at which point we had not yet completed this paper. We believe that our work would appeal to NLP researchers with a focus on robustness and grammatical error correction, which is also suitable for EMNLP 2023. \n\n>Q: Can you go into more detail on how you sampled data from TEM-8?\n\nA: We sampled data from the TEM-8 dataset randomly. We will add this information in the revision.\n\n>Q: Can you specify which metric you used to measure inter-annotator agreement?\n\nA: The inter-annotator agreement in Sec.3.2 is calculated as the proportion of samples that two annotators reach a consensus. We will make it clear in the revision.\n\nWe will conduct proofreading to ensure the typo and grammar issues are fixed.  Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work is thorough and the paper is well-organized.  It is pretty well-written although I have some corrections (below).  It is a great fit for the NLP Applications track.", "reasons_to_reject": "This paper would find a very receptive audience at BEA, but unfortunately BEA 2023 already happened and the authors may not have been able to travel to Toronto.", "questions_to_authors": "In section 3.1 (or link to an appendix), can you go into more detail on how you sampled data from TEM-8?  Was this random or were you sampling stratified on anything such as demographics, learner-level, etc.?\n\nWhen describing your inter-annotator agreement in section 3.2 under \"Quality Assurance\", can you specify which metric you used to measure inter-annotator agreement?", "paper_topic_and_main_contributions": "This paper describes an ensemble data set and benchmark measuring the robustness of Grammatical Error Correction (GEC) systems against changes to the context around a grammatical error.  The authors use two pre-published GEC data sets and create a third using responses to a L2 English exam administered in China along with human annotation.  For all three data sets, humans create additional examples from those containing grammatical errors by altering the context around the error slightly.  The authors report results on this data set on five pre-published GEC systems and propose an improvement to the top-performing system.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "On line 014, \"cases\" should be \"examples\".\n\nOn line 026, \"application scenarios, such as writing aid\" should be \"applications, such as writing assistance\".\n\nOn line 056, replace \"i.e., CoNLL-14\" with \"annotation: CoNLL-14\".  \"i.e.\" is intended for examples, not for listing out the actual things.  This also applies on line 075.\n\nOn line 059, insert \"English-speaking\" after \"native\".\n\nOn line 174, \"systematical\" should just be \"systematic\".\n\nOn line 208, \", named entity recognition\" should be \" and named entity recognition\".\n\nOn line 210, \"Different from them\" should be \"Compared to these tasks\".\n\nOn line 310, insert \"the\" before \"original\".\n\nOn line 312, \"in\" should be \"that uses\".\n\nOn line 368, \"like\" should be \"including\".\n\nOn line 400, \"and predict\" should be \"that predicts\".\n\nOn line 411, \"much\" is unnecessary and slightly awkward (sorry).\n\nOn line 412, \"than\" should be \"compared to\".\n\nOn lines 431-432, remove \"on BEA-19,\" from where it is and insert it after \"performance\".\n\nOn line 432-433, insert \"with this data set\" after \"robustness\".\n\nOn line 453-454, \"the perturbed sample\" should be \"perturbed samples\".\n\nOn line 486-487, replace \"We propose a\" with \"With our\", insert a comma after \"method\", and make \"The\" lower-case to have this be one sentence.\n\nOn line 518, \"utilize\" should be \"use\".\n\nOn line 574, \"Because there\" should be \"There\".\n\nOn like 607, \"like\" should be \"including\".  (Using \"like\" in this way is too informal for this style of writing.)\n\nOn line 624, \"RobGEC\" should be \"RobustGEC\"."}}]

Author rebuttal:

Meta-review: This paper addresses the issue of robustness in GEC – they study how the contextual changes that are not relevant to the error being corrected affect model performance, and show that state-of-the-art models exhibit variation in proposed corrections when the context is varied in ways that should not affect the correction. The paper presents a new dataset, which is a sample of 2 English GEC benchmarks, combined with data from the TEM-8 dataset. They annotated the new dataset with contextual perturbations. The paper also proposes a method of fine-tuning GEC models to avoid variation and to encourage consistency.

The paper presents solid work, and addresses an interesting topic that should be relevant to GEC researchers and those working on model robustness in NLP in general. 

 Ethical concerns identified during the regular  reviewing and pointed out in the ethical reviews should be addressed.
Decision: Accept-Main