MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction
Authors: Jingheng Ye
Yinghui Li
Yangning Li
Hai-Tao Zheng
Venue: EMNLP 2023 Findings (2023)
Keywords: natural language processing
grammatical error correction
data augmentation
TL;DR: Revisiting data augmentation and beyond for grammatical error correction: two measures and a noval data augmentation method.

Abstract: Data Augmentation through generating pseudo data has been proven effective in mitigating the challenge of data scarcity in the field of Grammatical Error Correction (GEC). Various augmentation strategies have been widely explored, most of which are motivated by two heuristics, i.e., increasing the distribution similarity and diversity of pseudo data. However, the underlying mechanism responsible for the effectiveness of these strategies remains poorly understood. In this paper, we aim to clarify how data augmentation improves GEC models.
To this end, we introduce two interpretable and computationally efficient measures: Affinity and Diversity. Our findings indicate that an excellent GEC data augmentation strategy characterized by high Affinity and appropriate Diversity can better improve the performance of GEC models. Based on this observation, we propose MixEdit, a data augmentation approach that strategically and dynamically augments realistic data, without requiring extra monolingual corpora. To verify the correctness of our findings and the effectiveness of the proposed MixEdit, we conduct experiments on mainstream English and Chinese GEC datasets. The results show that MixEdit substantially improves GEC models and is complementary to traditional data augmentation methods. All the source codes of MixEdit are released at https://github.com/THUKElab/MixEdit.

MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction
Jingheng Ye', Yinghui Li Yangning Li Hai-Tao Zheng
TTinggu Shenzhen International Graduate School, Tsinghua University "Pnng Cheng Laboratory
(yejh22, Semangg tsinghua. edu. cn
Abstract
Numerous studies tend to improve the performance of Seq2Seq GEC models by increasing the amount of training data (Yc et al., 2022; Li et al., 2022c.b; Zhang et al., 2023).
Data Augmentation through generating pseudo data has been proven effective in mitigating the challenge data scarcity in the field of Grammatical Error Correction (GEC). Various augmentation strategies have been widely ex- plored, most of which motivated by two heuristics, ie.. increasing the distribution sim- ilarity and diversity of pseudo data. However, the underlying mechanism responsible for the effectiveness of these strategies remains poorly understood. In this paper, we aim to clarify how data augmentation improves GEC mod- els. To this end, we introduce two interpretable and computationally efficient measures: Affin- ity and Diversity. Our findings indicate that an excellent GEC data augmentation strategy characterized by high Affinity and appropriate Diversity can better improve the performance GEC models. Based on this observation, we propose MixEdit, data augmentation ap- proach that strategically and dynamically aug- ments realistic data, without requiring extra monolingual corpora. To verify the correct- ness of our findings and the effectiveness of the proposed MixEdit, we conduct experiments on mainstream English and Chinese GEC datasets. The results show that MixEdit substantially im- proves GEC models and is complementary to traditional data augmentation methods"
However, high-quality parallel data for GEC is not as widely available. Despite the great suc- cess of Seq2Seq models, they are prone to over- fitting and making predictions based on spurious patterns (Tu et al., 2020), owing to the vast gap between the number of model parameters and the limited high-quality data. This data sparsity is- sue has motivated research into data augmentation in the field of GEC, particularly in the context of resource-heavy Seq2Seq approaches (Rothe et al., 2021; Stahlberg and Kumar, 2021). Thanks for the ease of constructing pseudo grammatical errors, recent studies focus on generating synthetic paral- lel data from clean monolingual corpora. Various augmentation methods have been widely explored, whose major motivation comes from improving 1) distribution similarity between pseudo and realis- tic data, including error patterns (Choe et al., 2019) and back trccksrranlatiin (Sennrich et al., 2016a; Xic et al., 2018; Stahlberg and Kumar, 2021); and 2) diversity of pseudo data (Koyama et al., 2021b), such as noise injection (Grundkiewicz et al., 2019; Xu et al., 2019) and round-trip translation (Zhou et al., 2020). However, prior works have primar- ily focused on showing the effectiveness of their proposed augmentation methods, without consid- ering sample efficiency. Training GEC models with excessive samples (e.g. over 100M (Lichtarge et al., 2019; Stahlberg and Kumar, 2021)) for poor- scalable improvement is expensive and often un- feasible for most researchers. Additionally, exist- ing studies suffer from lack of consistent experi- mental settings, making it intractable to systemati- cally and fairly compare various data augmentation methods. this complex landscape, claims re- garding distribution similarity and diversity remain unverified heuristics. An uncomplicated approach
1 Introduction
Grammatical Error Correction (GEC) is a task that involves making locally substitutions in text to correct all grammatical errors in a text (Ye et al., 2023b; Li et al., 2023; Ye et al., 2023a; Ma et al., 2023). GEC is often considered monolingual machine translation (MT) task, which is typically tackled using sequence-to-sequemne (Seq2Seq) ar- chitecture (Bryant al., 2022: Dong al., 2023).
Coorreponning authors: Hai-Tao Zheng and Yinghui All the source codes of Mix are released at tptps:// gi thub.hccc com/ wtt./ooo.//e/eee THUKElab//MM wttEElaab/iixiiiiii t.dit.

---

Target
This will, already , cause problems as there is very limited space for us This will, already , caused problems as there are very limited spaces (mask) will I (mask) ulready (mask) will cause problems the (mask) (mask) (mask) space for (mask)
Source
Direct Noise (DN)
Pattern Noise (PN)
This will. not already cause problem as there is very limiting space for us
Backtranslation (BT)
This if not already cause the problems as there are very few space for us.
Soundtraantaaton (RT)
If that had not been done, it would have caused problems. because our space is very limited.
Target
We realize that burning of fuels produces large amount of greenhouse gases.
Source
relize that burning of fuels produce the large amount of greenliouse gases
Direct Noise (DN)
(mask) (mask) burning of looking (mask) a power (mask) (mask) greenhouse gases (mask)
Puttern Noise (PN)
we realized that burnings fuels produces a large amount of greenhouse guses.
Backtranslation (BT)
We realize that burn of fuel produce a greenhouse gases.
Round-translation (KT)
We recognize that large umounts of greenbouse gases are generated by combustion fuels.
Table 1: Examples of pseudo source sentences generated by Direct Noise (DN), Pattern Noise (PN), Backtranslation (BT) and Round-translatoon (RT), respectively.
to evaluate the effectiveness of an augmentation heuristic is to conduct experiments on all possible augmented datasets. However, it is computationally expensive and even impractical when confronted with numerous augmentation heuristic options.
tion among diverse augmented samples, instead of learning spurious patterns during training. We fur- ther apply Jensen-Shannon divergence consistency regularization to match the predictions between error patterns, and dynamically select candidate grammatical errors during fine-tuning.
In this paper, to determine the extent to which distributional similarity and diversity of data aug- mentation can improve GEC models, we quantify both heuristics. We use Affinity measure to evalu- ate the distribution similarity between pseudo and realistic grammatical errors, which is defined the inverse of Kullback-
We conduct experiments on two English GEC evaluation datasets: CoNLL-14 (Ng et al., 2014) and BEA-19 (Bryant et al., 2019). and two Chi- nese GEC evaluation datasets: NLPCC-18 (Zhao et al., 2018) and MuCGEC (Zhang et al., 2022a). Despite its simplicity, MixEdit consistently leads to significant performance gains compared to tradi- tional methods. By combining MixEdit with tradi- tional methods, we achieve state-of-the-art results on BEA-2019, NLPCC-2018, and MuCGEC.
(KL) divergence be- tween them. On the other hand, Diversity measure is used to assess the uncertainty of grammatical errors, defined as the entropy. Next, we revisit four mainstream GEC data augmentation methods using our proposed measures. Our findings illustrate that Affinity varies across these augmentation methods and is positively correlated with the performance of GEC models to some extent. By altering the corruption rate of an augmentation method, we ob- serve that the varying Diversity serves as a trade-off between Precision and Recall.
2 Background
To avoid confusion, we use the uppercase sym- bol X to represent a sentence, and the bold low- ercase symbol to indicate text segment. In this paper, we focus the discussion construct- ing grammatical errors for GEC data augmenta- tion. Generally, Seq2Seq-based GEC models pa- rameterized by learn the translation probability P(Y X: 0), where X denotes an ungrammatical source sentence and Y represents a grammatical target sentence. Given a parallel training dataset D, the standard training objective is to minimize the empirical risk:
To overcome the challenge of limiting aug- mentation strategy with high Affinity and appro- priate Diversity, we propose MixEdit, a data aug- mentation approach that strategically and dynami- cally augments realistic data. Unlike traditional approaches that rely on pseudo data generated from extra monolingual corpora, MixEdit regu- larizes over-parameteriize GEC models using a limited amount of realistic data during fine-tuning. MixEdit achieves this by replacing grammatical errors in the source sentence with other probable and label-preserving grammatical errors, avoiding undesired noise. These augmented samples differ only in the forn of grammatical errors, encourag- ing models to fully utilize the intrinsic informa-
(()) = Exx.))--lCco((Xx.00
(1)
where LCE is the cross entropy loss, D could be realistic dataset D, in a standard supervised learn- ing setting, or pseudo dataset DD typical GEC data augmentation settings, where the source sen- tences are usually generated from monolingual cor-

---

pora (Kiyono et al., 2020).
tion of Equation (1): 1) jointly optimizing GEC models by concatenating the realistic dataset D and the pseudo datset Pp. and 2) pre-training mod- els on the pseudo dataset before fine-tuning on real- istic datasets. In a study conducted by Kiyono et al. (2020), these two training settings were compared for two data augmentation methods (DN and BT), and it was found that pre-training was superior to joint optimization when large enough pseudo data was available. Therefore, avoid any adverse ef- fects resulting from noisy augmented samples, we adopt the pre-traine pre-training sctting in our work.
Recent works have concentrated on improving the performance GEC models by integrating var- ious data augmentation techniques, which usually fall under the categories listed in Table 1.
Direct Noise (DN). DN injects noise into gram- matically correct sentences in a rule-based man- ner (Kiyono et al., 2020). The noise can take the form of 1) masking, 2) deletion, and 3) insertion based on pre-defined probabilities. DN is applica- ble to all languages since its rules are language- independent. However, the generated errors are often not genuine and may even distort the original semantics of the sentences.
Method
3.1 Affinity and Diversity
Pattern Noise (PN). PN, on the other hand, in- volves injecting grammatical errors that are al- ready present in the realistic GEC dataset into sen- tences (Choe et al., 2019). Specifically, this process entails first identifying error patterns in the GEC dataset using automated error annotation tool such ERRANT (Bryant et al., 2017), followed by applying noising function that randomly sub- stitutes text segments with grammatical errors.
Affinity. In-distribution corruption (Choe et al., 2019) has motivated the design of GEC data aug- mentation policies (Grundkiewiez et al., 2019; Stahlberg and Kumar, 2021; Yasunaga et al., 2021), based on the idea that pseudo data with less distri- bution shift should improve performance on spe- cific evaluation datasets. Inspired by this focus, propose Affinity, which is used to qualify how augmentation shifts data with respect to the error pattern. We define Affinity of data augmentation method as the inverse of Kullback Leibler (KL) di- vergence between pseudo and realistic grammatical errors, which can be computed as follow:
Backtranslation (BT). With the help of Seq2Seq models, BT can generate more genuine grammat- ical errors by learning the distribution of human- written grammatical errors (Xie al., 2018). The noisy model trained with the inverse of GEC parallel dataset, where ungrammatical sentence are treated as the target and grammatical ones as the source. Xie et al. (2018) proposed several variants of BT, and showed that the variant BT (Noisy) achieved the best performance. Therefore, we focus on this variant in this work. When de- coding the ungrammatical sentences, BT (Noisy) adds rraadoom to the score of each hypothesis in the beam for each time step, where r drawn uniformly from the interval [o, 1]. and Brandom is syper-parameter that controls the degree of noise.
1 Affinity(D) (D,tty(D,,))
(L((p( y) Pt(x, y))
2
1 + (T(PP
Pa(x, y));
(2)
where D, and D, refer to the pscudo and realis- tic datasets, respectively. The pair of text correc- tions, denoted by æ and y, can be extracted using an automated error annotation toolkit such as ER- RANT (Bryant et al., 2017). Pp and P, denote the probabilities of pseudo and realistic grammatical errors, and KL represents KL divergence used to calculate the distance between two distributions. To prevent KL approaching infinity, limit the support set of the calculation. The first term of the above equation computed as follow:
(RT). RT an alternative method to generate pseudo data, which based on the assumption that NMT systems may produce translation errors, resulting in noisy outputs via the bridge languages (Lichtarge et al., 2019; Zhou et al., 2020). The diverse outputs, however, may change the structure of the sentence due to the heterogencity of different languages.
KL(P, 3) P, v))
Training Settings. There are two primary train- ing settings for incorporating D, into the optimiza-
Σ p,( y) log () Pa y)
(3)
(m.y)-Dr

---

P(P/))
Source
Nothing
absolutle
right
X
Reference
Nothing
absolutely
right
and
wrong.
Lox (X.))
very
2116:
Cross-entropy
(X,
absolutelly
05% abusoluteriy extreamely
12% atsclute
and
09% that
08%
Loss
Consistency Loss
(x
Pseudo
Source
Nothing
very
right
an
wrong
X
Error Selection
Grammatical Emvr
P(N)))
Figure 1: Overview of our approach MixEdit. MixEdit 1) first extracts the patterns from GEC realistic datasets and builds Error Pattern Pool, 2) replaces grammatical errors with alternative candidates from the Error Pattern Pool, and then computes the cross-entropy loss CeR and the consistency loss CKI..
With this definition, Affinity of a small value suggests the pscudo grammatical errors are more likely to be distributed within the realistic dataset.
from a GEC dataset D. The following corruption probability can be derived from the error patterns using Bayes' rule:
Diversity. Koyama al. (2021b) demonstrated the importance of diversity in pseudo grammatical errors for improving performance. Based on the observation, we introduce another perspective to evaluate GEC data augmentation policies, which we dub Diversity, defined as the entropy of pseudo grammatical errors:
P( y) = P(x, y) P(y)
(5)
The corruption probability describes the correc- tion feature of GEC dataset. For each parallel sam- ple (X, Y), the correction can be denoted by an edit sequence E (e1, E2, ... emm, where each edit e, consists of a source segment æ and a target segment As shown in Figure 1, MixEdit dynam- ically replace the original source segment æ with other candidates x' from a pool of error patterns during training. Since all the candidates share the same target segment y, this transformation is typi- cally label preservne,,,,,g preventing undesired noise. These preee-prs perturbations provides var- ious literal forms of the same grammatical errors. enabling correct predictions based on more com- plex contexts instead of spurious patterns.
Diversity
Σ P(x.y)log P(x, y), (2.y)
(4)
where P(.) can take on the values of (-( or P.(.).
Like Affinity, this definition of Diversity has the ad- vantage of capturing task- specifcc and interpretable elements, i.e., the error patterns of GEC datasets, Furthermore, these measures are off-the-shelf and require little computational cost.
should be kindly noted that the performance GEC models not purely a function of training data, as training dynamics and implicit biases in the model can also impact final performance. Both of these measures are introduced to provide new perspective characterizing and understanding GEC data augmentation.
Inspired by Shen al. (2020); Chen et al. (2021), incorporate consistency regularization in model to encourage stable and similar predictions across realistic and augmented samples. The train- ing objective written as:"
L = ==LoE(X,)) + aLcE(X', +X
3.2 MixEdit
(6)
Mixlidit aims to strategically and dynamically con- struct pseudo data with high Affinity and appro- priate Diversity, thus achieving the best of both worlds. Specifically, we first extract error patterns
"Shen al. (2020) considered multiple augmented samples for realistic sample, but we found a single augmented sample works just as well. Additionally, increasing the number of augmented samples significantly escalates the training cost.

---

where the augmented sample X' is generated dy- namically during training, with weights and used balance the contribution of learning from the original data and the augmented data. LCE denotes the cross-entropy loss, and LKL is KI-- divergence as follow:
Dataset
#Sentences
Usage
Gigaword
8,000,000
Pre-training
CLang8
2,372,119
Fine-tuning
FCE
34,490
Fine-tuning
NUCLE
57,151
Fine-tuning II
W&I+LOCNESS
34,308
Fine-tuning I&&III
BFA-2019-Dey
4,384
Validation
BFA-2201-Test
4,477
Testing
CoNLL-2014-Test
1,312
Testing
CKI(X, KL P(Y X') Pang] (7)
Table 2: Statistics English GEC datasets. Gigaword is only available for DN, PN, BT and RT.
where Pang represents the average prediction prob- ability across realistic and augmented samples.
PN is the most similar traditional GEC data aug- mentation method to MixEdit. However, there are several distinctions between these two methods: 1) MixEdit provides different views for the same grammatical errors, while PN randomly constructs pseudo errors. In our preliminary experiments we found matters to determine the positions for constructing grammatical errors. 2) PN is typically used to generate new pseudo parallel data from monolingual corpora (Grundkiewicz and Junczys- Dowmunt, 2019; White and Rozovskaya, 2020). Conversely, MixEdit is proposed to augment high- quality realistic data and can be combined with other traditional data augmentation methods intro- duced in Section 2 due their orthogonality.
M scorer (Dahlmeier and Ng, 2012) and the BEA- 19 test set (Bryant et al., 2019) evaluated by ER- RANT. The results are averaged over three runs with different random seeds, and the BEA-19 dev serves as the validation set. We provide statis- tics of all the involved datasets in Table 2.
GEC backbone model. We adopt Transformer- based BART-Large (Lewis et al., 2020) as our back- bone model, which has been shown as strong baseline for GEC (Katsumata and Komachi, 2020; Zhang et al., 2022b). We acquire subwords by titepair-enncodin (BPE) (Sennrich et al., 2016b) algorithm. We apply the Dropout-Sre mecha- nism (Junczys-Dowmunt et al., 2018) to source- side word embeddings following Zhang et al. (2022b). All experiments are conducted using the Fairseq (Ott et al., 2019) public toolkit. Most of the hyperparameter settings are identical to Zhang et al. (2022b), which are provided in Appendix A.1.
4 Experiments on English GEC
4.1 Experimental Settings
Datasets and evaluation. We decompose the training of the baseline model into three stages following Zhang al. (2022b). We train the model on 1) the CLang8 dataset (Rothe et al., 2021), 2) the FCE dataset (Yannakoudakis et al., 2011), the NUCLE dataset (Dahlmeier et al., 2013) and the W&I+LOCNESS train-set (Bryant et al., 2019). 3) We finally fine-tune the model on high-quality W&I-LOCNESS. As for the traditional data aug- mentation methods using extra corpora (i.e., DN, PN, BT and RT), we construct pseudo dataset using the seed corpus Gigaword* 4, which has been proven to be the best among three seed corpora by Kiyono et al. (2020). In total, we generate 8M pseudo data with the same target sentences for four data augmentation methods, which is used for the pre-training of GEC models.
Data Augmentation. We examine and analyze four mainstream GEC data augmentation methods discussed in Section 2, as well as our proposed MixEdit in Section 3.2. We introduce extra pre- training stage that utilizes the pseudo datasets gen- erated by each data augmentation method. Further details regarding the experimental settings for aug- mentation methods can be found in Appendix A.2.
4.2 Results of GEC Data Augmentation
Table 3 showcases the results of each data aug- mentation method. Based on the baseline model, PN achieves the highest Fo.5 score on CoNLL-14, while BT obtains the highest Fo.5 score BEA- 19. In contrast, DN produces lower Fo.s score than the baseline in both test sets, which contradicts the findings of Kiyono et al. (2019) that DN can benefit the GEC performance of scratch transformer mod- els. We attribute the phenomenon to the fact that
For evaluation, reports the results on the CoNLL-14 test (Ng al., 2014) evaluated by
We leave more detailed analysis to Section 4.3. <https: bhttps://cat //tat//acc..../////// 1dc. upenn, wdwe addu/LDC201100

---

Extra
Transformer
CoNLL-14Hte
BEA-19-test
System
Data Size
Layer, Hidden, FFN
Fo.5
Fo.s
Kiyono et al. (2019)*
70M
12+12.0024,409
67.9
44,1
61.3
59.4
64.2
w/o PLM
Lichtarge et al. (2020)
340M
12+12.1024.400
69.4
43.9
62.1
67.6
62.5
66.5
Stahlherg and Kumar (2021)
540M
12+12,1024,4086
72.8
49.5
66.6
72.1
64.4
70.4
Kaneko al. (2020)*
70M
12+12.1024,4088
69.2
45.6
62.6
67.1
60.1
65.6
Katsumata and Komachi (2020)
12+12.1024.4008
69.3
45.0
62.6
57.1
65.6
Omelianchuk et al. (2020)
9M
12+0,768,3072
77.5
40.1
65.3
79.2
53.9
72.4
Rothe et al. (2021)
2.4M
12+12, 11-12,1024,40000 4096
66.1
72.1
Sun et al. (2021)*
300M
12+2,1024,400
71.0
52.8
66.4
72,9
BART Baseline" (Zhang et al., 2022b)
2.4M
12+12, 12+12,1024,4000 1024,4096
73.6
48.6
66.7
74.0
64.9
72.0
SynGEC (Zhang ul., 20226)
2.4M
12+12, 1024, 1024,4096 4096
74.7
49.0
67.6
75.1
65.5
72.9
Our BART Baseline"
2.4M
12412.102444000
73.8
47.6
66.5
74.4
63.7
72.0
w/ PLM
+DN*
10.4M
12+12.1024,44
70.3
50.3
65.1
72.6
64.6
70.9
10.4M
12+12.1024,4008
73.3
51.1
67.4
75.0
65.1
72.8
+BT
10.4M
12+12.1024.400
72.7
51.3
67.1
65.9
73.0
+R
10.4M
12+12,1024, 1024,4096
73.0
48.7
72.5
2.4M
12+12,1024, 1029,4096
75.6
67.31
76.4
62.7
73.21
+D
10.4M
12+12,1024.400
72.6
48.3
66.01
74.4
63.0
71.81
+P
10.4M
12+12, 4096
75.1
48.1
67.57
75.3
64.7
72.91
+BT49
10.4M
12+12, 1112,,022,40000
74.4
48.4
67.21
76.4
63.7
73.41
+
10.4M
12+12,1024.4096
75.2
47.2
67.21
75.9
63.3
72.91
Table 3: The results of baselines and GEC data augmentation methods for single-model Layer. Hidden and FFN denote depth, hidden size and feed-forward network size Transformer. "Our BART Baseline" is re-implemented from the open-source "BART Baseline", both of which under the same experimental setting, making them fairly comparable. means the performance of "w/ MixEdit" is better than that of its "w/o MixEdit" counterpart. Besides the public numan-annotated training data, private and/or pscudo data sources also widely used in GEC systems, including: 'BT pscudo data from Gigaword (70M sentences). *Wikipedia revision histories (170M), RT pseudo data from Wikipedia (170M), BT pseudo data from Colossal Clean Crawled Corpus (200M). DD psuedo data from one-billion-worf (9M), * BT and DN pseudo data (300M), cleaned version of Lang8 (2.4M), our psuedo data from Gigaword (8M).
the pre-training task of BART similar DN, and it is unnecessary and possibly harmful to pre-train BART again with task-independent noise. Finally, RT performs slightly better on BEA-19 but slightly worse on CoNLL-14.
lens of quantified Affinity and Diversity measures. seeking to gain insight into the mechanisms under- lying GEC data augmentation.
To ensure a fair comparison, we avoid introduc- ing extra monolingual corpora that may influence result due to nuisance variables such as text do- main. Instead, we apply data augmentation to GEC realistic datasets comprising FCE, NUCLE and W W&I+LCCNSSS (collectively referred to BEA- train). Specifically, we retain the target sentences from BEA-train, and generate pseudo source sen- tences by enforcing data augmentation strategies, resulting in set of pscudo datasets with identical targets but different sources. It is worth noting that remove the dynamic training and the regulariza- tion of MixEdit to facilitate fair comparison with other data augmentation methods. We respectively train GEC models on these pseudo datasets and report the results in Table
The bottom group lists the results of MixEdit and combination with each traditional GEC data aug- mentation method. Without extra monolingual cor- pora, our MixEdit achieves 67.3/73.2 Fo.s scores, which are on par with or marginally superior to tra- ditional methods using the same backbone model. Notably, MixEdit also complements these tradi- tional methods. Each augmentation method based MixEdit produces higher Fo.5 scores than its counterpart, with PN and BT w/ MixEdit achiev- ing the highest Fo.5 scores on CoNLL-14 (67.5) and BEA-19 (73.4), respectively.
4.3 Analysis
The mechanism through which these corruptions work remains unclear, although the effectiveness of GEC data augmentation is well-established. this section, we investigate the relationship between data augmentation and performance through the
Affinity is positively correlated with the perfor- mance. At a high level, Affinity of MixEditt PN and BT, which achieve the highest Fo.s scores, is much higher than that of DN and RT. We compute the Pearson's correlation coefficient for the data

---

Method
Affinity!
Diversity
P
Fas
CoNLL-14-ted
BEA-19-es
Baseline
8.78
56.03
37.60
51.03
P/R/Fas
P/R/Fo
DN
0.41
10.70
20.49
12.94
18.35
MixEdit
76.81/45 00667.30
76.37/6.71173.11
RT
0.75
10.54
20.59
38.41
22.70
w/o Error Pattern
46/66.35
75.22/6308972.44
PN
w/o Consistency Loss
74.73447.80067.16
76.06/63.9973.. 12
Round=1
7.46
35.46
16.35
28.74
Dynamic Generation
75.41466.4466.11
75.73/62.35/22.6
Round=2
7,56
39.43
23.71
34.82
Pattern Noise
74. 74.7049.73667.66
74.78/64.7772.55
Round=4
7.70
39.41
36,90
Round=8
1.77
7.92
42.14
37.51
41.13
Table 5; Ablation results of MixEdit.
Round=16
1.59
8.24
33.83
39.24
34.79
BT
Prandom =0
0.65
6,96
22.20
7.36
15.82
P
R
Fo.5
Branlam=3
1.47
7.67
44.11
32.46
41.16
0.5
55.96
41.35
52.27
Brandom=6
1.57
8.22
49.29
42,99
47.89
0.8
55.65
41.35
52.05
Prandom
44.69
1.0
57.96
39.00
$2.83
Brandom =12
1.2
56.75
40.93
$2.68
MixEdit (Ours)
2.33
8.52
57.72
33.24
50.31
2.0
55.88
41.30
$2.20
Table 4: Affinity and Diversity of data augmentation methods. The baseline model is trained using realistic BEA-train dataset.
Table 6: Results of various on BEA-train.
8
P
R
Fo.5
0.5
57.56
38.51
52.38
augmentation methods involved: DN, PN, BT, RT and MixEdit. For PN and BT, we choose the hyper- parameter configurations that yield the highest F0.5 scores. The Pearson's correlation coefficient be- tween Fo.5 and Affinity is 0.9485. The results in- dicate a strong correlation between Affinity and Fo.5 on BEA-train. It should be noted that, despite having lower Affinity, BT achieves a higher Fo.5 score than PN. We attribute this to the advantage of BT in learning the distribution of grammatical errors using Seq2Seq models instead of adding er- rors crudely. MixEdit also skillfully avoids the drawbacks of PN by strategically applying label- preserving perturbations, resulting in an approxi- mate Fo.s score with the baseline.
1.0
57.96
39.00
$2.83
2.0
56.69
40.59
52.52
Table 7: Results of various 3 BEA-train.
ground truth. MixEdit only replaces the original grammatical errors with augmented ones, maintain- ing the same error density in the augmented dataset. As a result, the diversity of the augmented dataset closely resembles that of the original dataset.
To further explore the relationship between Affinity/Diversity and performance across different datasets, we also conduct additional experiments on English CLang8 and Chinese HSK. The relation- ship between Affinity/Divrrity and performance is similar to the results on BEA-train, which are pro- vided in Appendix B. Additionally, we provide further analysis on the complementary effective- ness of pseudo data in Appendix B.2.
Diversity responsible for the trade-off between Precision and Recall. We qualitatively investi- gate the effect of Diversity on GEC performance within a fixed method. The trade-off of Diversity is apparent when adjusting certain parameerrsss responsible for Diversity (Round for PN and 3 for BT). As Diversity increases, Precision and Fo.5 ini- tially increase and then decrease, reaching their peak at an appropriate setting. Meanwhile, Re- call continues to increase. For example, PN and BT reach their peak Fo.s score at Round=8 and Brandom =6, respectively, falling at an intermediate value of Diversity.
4.4 Ablation Study
Decomposition of MixEdit. We explore the ef- fectiveness of each component of our proposed MixEdit by conducting ablation studies shown in Table 5. Specifically, for the "w/o Error Pattern" variant, we randomly mask tokens in the sentence instead of sampling grammatical errors from the Error Pattern Pool. For "w/o Consistency Loss", we remove the consistency loss. For "w/o Dynamic Generation", we always generate fixed pscudo data for fine-tuning GEC models. Additionally, we tempt incorporate PN into generating dynamic pseudo data in the fine-tuning stage. The results demonstrate that task-specific information of error pattern is important to constructing high-quality
Therefore, argue that an excellent data aug- mentation technique should have high Affinity and appropriate Diversity, which motivated our pro- posed MixEdit. The high Affinity of MixEdit stems from the fact that the distribution pseudo gram- matical errors it generates is the same as that of the

---

Extra
NLPCC-18-
MuCGEC-tex
System
Data Size
P
R
Fo.5
Fa.5
Zhang al. (2022a)
42.88
30.19
39.55
43.81
28.56
39.58
SynGEC (Zhang al., 2022b)
49.96
45.32
29.10
46.51
Our Baseline
49.81
31.57
44.65
54.24
29.67
46.53
+DN
8M
49.57
31.80
44.59
54.93
29.61
46.91
+PN
8M
50.15
35.27
46.25
55.83
30.15
47.71
+BT
8M
47.64
36.43
44.88
54.82
30.27
47.17
+RT
8M
$1.06
30.74
$5.09
$6.96
27.41
46.86
MixEdit
32.93
45.03T
55.25
29.30
46.94
+DN
50.46
30.55
44.64T
56.48
28.12
47.00"
+PN
8M
52.26
33.37
46.94"
$6.99
29.73
48.16"
+BT
8M
48.99
35.45
45.52"
54.72
31.76
47.817
+RT
51.19
34.94
45.68"
55.04
30.53
47.43"
Table 8: Single-model results on Chinese datasets. All models are initialized with pre-trained Chinese BART weights. means the performance of "w/ MixEdit" is better than its "w/o MixEdit" counterpart.
pseudo data. Dynamic Generating is another crit- ical factor for the success of MixEdit, since can improve the diversity of pseudo data without the loss of Affinity. Surprisingly, adding pattern noise improves Fo.5 CoNLL-14-test, with per- Given that CoNLL-11 centered around essays written by language learners, our speculation is that it includes a greater number of stereotypical grammatical er- rors. As result, incorporating pseudo data with a higher corruption ratio can improve performance on CoNLL-14. We investigate how varying PN cor- ruption ratios affect the performance of the GEC model in Appendix B.3.
Dataset
#Sentences
Usage
News2016zh
8,000,000
Pre-training
Lang8
1,220.906
Fine-tuning
HSK
156,870
Finc-tuning
MuCGEC-der
1,125
Validation
MuCGEC-test
5,938
Testing
formance reduction
NLPCC-18-tstt
2,000
Testing
Table 9: Statistics of Chinese GEC datasets.
evaluated by CHERRANT. Table 9 provides statis- tics for the aforementioned datasets.
GEC backbone model. We employ Chinese BART (Shao et al., 2021) as our backbone model 6 We retained the original vocabulary because the updated version of Chinese BART has already in- corporated a larger vocabulary, and therefore no modifications were necessary.
Sensitivity to hyperparameters. We investigate the sensitivity to the choice of the hyperparameters a and 3 introduced in Equation 6. We explore the optimal values of them on BEA-train. The results of various and B are reported in Table and Table respectively, where the optimal hyperpa- rameter setting is 1.0 and 3 1.0.
5.2 Results
Table 8 presents our results. With the exception of PN, all data augmentation methods improve the Fu.5 score, demonstrating that low-affinity data aug- mentation methods can negatively impact the per- formance of pre- trained models like BART. More- over, incorporating MixEdit with traditional meth- ods can further increase the Fo.s scores on both evaluation datasets. These findings suggest that MixEdit is general technique that can be effec- tively employed in various languages and com- bined with other data augmentation methods.
5 Experiments on Chinese GEC
5.1 Experimental Settings
Datasets and evaluation. We adopt the seed cor- pus news2016zh* to generate 8M pseudo data for each traditional data augmentation method, simi- lar our English experiments. Following (Zhang et al., 2022a), we fine-tune GEC models on the Chi- nese Lang8 (Zhao et al., 2018) and HSK (Zhang, 2009) datasets. We reports the results on NLPCC- 2018-rest (Zhao et al., 2018) evaluated by 2 scorer, and MuCGCE-test (Zhang et al., 2022a)
Related Works
Data Augmentation. Data augmentation encom- passes methods of increasing training data diver-
//gn::///ttttbb ctthuuu.co//bbbb/eee com/bricc////ooooo httpbbiggttart/p.pcieeeese corpus

---

sity without requiring additional data collection. It has become staple component in many down- stream NLP tasks. Most researches apply range of perturbation techniques to increase training scale, with the aim of reducing overfitring and improv- ing the generalization of models (Feng et al., 2021; Shorten and Khoshgoftaar, 2019). Furthermore, many researchers develop various data augmenta- tion strategies based on unverified heuristics. De- spite the success of data augmentation, there ap- pears to be lack of research on why it works. To explore the effectiveness of these augmentation heuristics, Kashefi and Hwa (2020) proposed to quantify an augmentation heuristic for text clas- sification based on the idea that good heuristic should generate "hard to distinguish" samples for different classes. Gontijo-Lopes al. (2020) intro- duced interpretable and easy-to-compute measures qualify an augmentation heuristic in the field of Computer Vision (Liu et al., 2022).
7 Conclusion
This paper introduces two interpretable and com- putationally efficient measures, Affinity and Diver- sity, to investigate how data augmentation improves GEC performance. Our findings demonstrate that an excellent GEC data augmentation strategy char- acterized by high Affinity and appropriate Diversity can better improve the performance of GEC mod- els. Inspired by this, we propose MixEdit, which does not require extra monolingual corpora. Exper- iments on mainstream datasets in two languages show that MixEdit is effective and complementary to traditional data augmentation methods.
Limitations
One shall cautiously consider that our proposed measures, Affinity and Diversity, are a tool for gain- ing new perspective on understanding GEC data augmentation. Though positive correlations are ob- served between Affinity and performance, should not be relied upon as precise predictor for compar- ing data augmentation methods. Additionally, it is worth noting that our proposed MixEdit approach is only applicable to realistic datasets, where it can generate aabel-preserving grammatical errors. De- spite these limitations, we believe that our findings provide a solid foundation for further scientific in- vestigation into GEC data augmentation.
Pseudo Data Generation. Increasing the scale of training data has been instrumental in improving GEC systems. However, the lack of high-quality publicly available data remains a challenge in low- resource settings (Ma et al., 2022; Li et al., 2022a). To mitigate this issue, recent studies focus on gen- erating pseudo data from clean monolingual cor- pora. A common approach is to artificially per- turb a grammatically correct sentence through ran- dom word character-level insertion, substitution, or deletion (Kiyono et al., 2020; Xu al., 2019; Zhou et al., 2020), using spell checkers (Grund- kiewicz et al., 2019) or eтror patterns extracted from realistic datasets (Choe et al., 2019). How- ever, these rule-based methods struggle to emulate maden-made errors, which can lead to low sam- ple efficiency problem and performance degrada- tion (Yuan and Felice, 2013). Recent studies utilize models to generate genuine pseudo data, such as backtranslation (Xie et al., 2018) and round-trip translation (Zhou et al., 2020). On the other hand, Koyama et al. (2021a) shown that the performance of GEC models improves when pseudo datasets generated by various backtranslation models are combined. Stahlberg and Kumar (2021) proposed generate pseudo data based on given error type tag using the Seq2Edit model (Stahlberg and Ku- mar, 2020). Different from all these studies, we focus on generating high-quality pscudo data with- out extra corpora.
Ethics Statement
In this paper, we revisit the effectiveness of tradi- tional data augmentation methods for GEC, includ- ing direct noise (DN), pattern noise (PN), back- translation (BT) and round-trip translation (RT). The source data for these methods is obtained ex- clusively from publicly available project resources on legitimate websites, without any involvement of sensitive information. Additionally, all the base- lines and datasets used our experiments are also publicly accessible, and we have acknowledged the corresponding authors by citing their work.
Acknowledgements
This research is supported by National Natural Sci- ence Foundation of China (Grant No.62276154), Research Center for Computer Network (Shen- zhen) Ministry of Education, the Natural Science Foundation of Guangdong Province (Grant No. 2023A1515012914) Basic Research Fund of Shen- zhen City (Grant No. JCYJ202 0324 1324120012033

---

and JSCG200108021544020000 the Major Key Project of PCL for Experiments and Applications (PCL2021 A06), and Overseas Cooperation Re- search Fund of Tsinghua Shenzhen International Graduate School (HW2021008).
Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. 2023. A survey of natural language generation. ACM Comput. Surv.,
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chan- dar, Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A survey of data augmentation approaches for NLP. Findings of the Association for Computational Linguistics; ACL-IJCNLP 2021, pages 968-988, Online. Association for Computa- tional Linguistics.
References
Christopher Bryant, Mariano Felice, Øistein E. Ander- sen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-75, Florence, Italy, Association for Computational Lin- guistics.
Raphael Gontio-Lopes, Sylvia J Smullin, Ekin D Cubuk, and Ethan Dyer. 2020. Affinity and diver- sity: Quantifying mechanisms of data augmentation. arXiv preprint 200%02200...973.
Roman Grundkiewicz and Marcin Junczys- -owzyutt 2019. Minimally-augmeeneed grammatical error cor- rection. In Proceedings of the Sth Workshop on Noisy User-gemerated Text (W-NUT 2019), pages 357-363, Hong Kong, China, Association for Computational Linguistics.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793 805, Vancouver, Canada. Association for Computational Linguistics.
Roman Grundkiewicz, Marcin
and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In Proceedings of the Fourleenth Workshop Innovative Use of NLP for Building Educational Applications, pages 252-263, Florence, Italy. Association for Computational Linguistics.
Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2022. Grammatical error correction: A survey of the state of the art. arXiv preprinl arXiv:2211.0511
Jiaao Chen, Dinghan Shen, Weizhu Chen, and Diyi Yang. 2021. HiddenCut: Simple data augmentation for natural language understanding with better gener- alizability. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Pupers), pages 4380-4390, Online. Association for Computa- tional Linguisties.
Marcin Junczys-Dowmunt, Roman Grundkiewicz. Shubha Guha, and Kenneth Heafield. 2018. Ap- proaching neural grammatical error correction as a low-resource machine translation task. In Proceed- ings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Techmologies, Vol- ume (Long Papers), pages 595-60556900 5066 New Orleans, Louisiana. Association for Computational Linguis- tics.
Yo Joong Choe, Jiyeon Ham, Kyubyong Park, and Yeoil Yoon. 2019. A neural grammatical error correction system built on better pre-training and sequential transfer learning. In Proceedings of the Fourleenth Workshop on Innovative Use of NI.P for Building Educational Applications, pages 13--227, Florence. Italy. Association for Computational Linguistics.
Masahiro Kancko, Masato Mita Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248 4254,, On- line. Association for Computational Linguisties.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of the 2012 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 568-572 Montréal, Canada. Association for Compu- tational Linguistics.
Omid Kashefi and Rebecca Hwa. 2020. Quantifying the evaluation of heuristic methods for textual data augmentation. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 200-208, Online. Association for Computational Lin- guistics.
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner English: The NUS corpus of learner English. Pro- ceedings of the Fighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22-31, Atlanta, Georgia. Association for Computa- tional Linguistics.
Satoru Katsumata and Mamoru Komachi. 2020. Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. In Pro- ceedings of the 1st Conference of the Paca-Pacific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference

---

on Natural Language Processing, pages 827-832, Suzhou, China. Association for Computational Lin- guistics.
fine-tuning for chinese spell checking. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 238-249, Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014, Adam: A method for stochastic optimization. arXiv preprint 11X2::442..690.
Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zizhen Wang, Chao Li, Yunho Cao, and Hai-Tao Zheng. 2022c. The past mis- take is the future wisdom: Error-driven contrastive probability optimization for chinese spell checking. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3202 3213.221. Association for Computa- tional Linguisties.
Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizu- moto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error cor- rection. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 1236-1243 1242, Hong Kong, China. Association for Com- putational Linguistics.
Jared Lichtarge, Chris Alberti, and Shankar Kumar. 2020. Data weighted training strategies for grammat- ical error correction. Transactions of the Association for Computational Linguistics, 8:634-646.
Shun Kiyono, Jun Suzuki, Tomoya Mizumoto, and Ken- taro Inui. 2020. Massive exploration of pseudo data for grammatical error correction. IEEE/ACM trans- actions audio, speech, and language processing, 28:2134-2145.
Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019. Cor- pora generation for grammatical error correction. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3291-3301, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Aomi Koyama, Kengo Hotate, Masahiro Kaneko, and Mamoru Komachi. aa Comparison of grammat- ical error correction using back-translation models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Student Research Workshop, pages 126-135, Online. Association for Computa- tional Linguistics.
Ruiyang Liu, Yinghui Li, Linmi Tao, Dun Liang, and Hai-Tao Zheng. 2022. Are we ready for a new paradigm shift? A survey on visual deep MLP. Pat- lerns, 3(7):100520.
Shota Koyama, Hiroya Takamura, and Naoaki Okazaki. 202 Various errors improve neural grammatical error correction. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Com- putation, pages 251-261.
Shirong Ma, Yinghui Li, Haojing Huang, Shulin Huang. Yangning Li, Hai-Tao Zheng, and Ying Shen. 2023. Progressive multi-task learning framework for chi- nese text error correction. CoRR, Abs/2306.17441
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seqqencc pre-training for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Shirong Ma. Yinghui Li, Rongyi Sun, Qingyu Zhou, Shulin Huang, Ding Zhang, Li Yangning, Ruiyang Liu, Zhongli Li, Yunbo Cao. Haitao Zheng, and Ying Shen. 2022. Linguistic rules-based corpus gener- ation for native Chinese grammatical error correc- tion. In Findings of the Association for Computa- rional Linguistics: EMNLP 2022, pages 576-589, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yinghui Li, Haojing Huang. Shirong Ma. Yong Jiang, Yangning Li, Feng Zhou, Hai-Tao Zheng, and Qingyu Zhou. 2023. On the (in)effectiveness of large lan- guage models for chinese text correction. CoRR, abs/2307.09007
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto. Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natu- ral Language Learning: Shared Task, pages 1-14, Baltimore, Maryland, Association for Computational Linguistics.
Yinghui Li, Shulin Huang. Xinwei Zhang, Qingyu Zhou, Yangning Li, Ruiyang Liu, Yunbo Cao, Hai-Tao Zheng, and Ying Shen. 2022a. Automatic context pattern generation for entity set expansion. CoRR, abs/2207.08087 08087.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR grammatical emror correction: Tag, not rewrite. In Proceedings of the Fifleenth Workshop on Innovative Use of NLP for Building Educational
Yinghui Li, Shirong Ma, Qingyu Zhou, Zhongli Li, Yangning Li, Shulin Huang, Ruiyang Liu, Chao Li, Yunbo Cao, and Haitao Zheng. 2022b. Learning from the dictionary: Heterogeneous knowledge guided

---

Applications. pages 163-170, Seattle, WA, USA + Online. Association for Computational Linguistics.
Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021. Instantaneous grammatical error correction with shal- low aggressive decoding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 5937-5947, Online. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference the North American Chapter of the Associa- tionfoo for Compulational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-
MT - Building open translation services for the World. In Proceedings of the 22nd Annual Confer- enec of the European Association for Machine Trans- lation (EAMT), Lisbon, Portugal.
Sascha Rothe, Jonathan Mallinson, Fiic Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 702-707, Online. Association for Computational Linguisties.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spuri- ous correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621-633.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86 86-96, Berlin, Germany, Association for Computational Lin- guistics.
Max White and Alla Rozovskaya. 2020. A compara- tive study of synthetic data generation methods for grammatical error correction. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 198-208, Seattle, WA, USA Online. Association for Com- putational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguistics.
Ziang Xie. Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018. Noising and denois- ing natural language: Diverse backtranslation for grammar correction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 619-628, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Yunfan Shao, Zhichao Geng. Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. Cpt: A pre-trained unhalanced transformer for both chinese language understanding and generation. arXiv preprint 209.02210.
Shuyao Xu, Jiehao Zhang, Jin Chen, and Long Qin. 2019. Erroneous data generation for grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 149-158 158, Florence, Italy. Association for Computational Linguistics.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough- to-beat data augmentation approach for natural lan- guage understanding and generation. arXiv preprint arXiv::009.1 13818.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 189, Portland, Oregon, USA. Association for Computational Linguistics.
Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of big data, 6(1).1-48.
Felix Stahlberg and Shankar Kumar. 2020. Seq2Edits: Sequence transduction using sean-level edit opera- tions. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 5147-5159, Online. Association for Computational Linguistics.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. LM-critie: Language models for unsupervised grammatical error correction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7752-7763, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Felix Stahlberg and Shankar Kumar. 2021. Synthetic
data generation for grammatical error correction with tagged corruption models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Build- ing Educational Applications, pages 37-47, Online. Association for Computational Linguistics.
Jingheng Ye, Yinghui Li, Shirong Ma, Rui Xie, Wei Wu, and Hai-Tao Zheng. 2022. Focus is what you need for chinese grammatical error correction. CoRR, abs/2210.1 16992.

---

Jingheng Ye, Yinghui Li, and Haitao Zheng. 2023a. System report for CCL23-eval task 7: THU KELab (sz) - exploring data augmentation and denoising for Chinese grammatical error correction. In Proceed- ings of the 22nd Chinese National Conference on Computational Linguistics ( (Volume 3: Evaluations). pages 262-270, Harbin, China. Chinese Information Processing Society of China.
A
Implementation Details
A.1 Hyper-parameters
Configuration
Value
Pre-training
Backbone
BART-large (Lewis et al., 2020)
Devices
Tesla A A100 GPU (80GB3)
Epochs
60
Jingheng Ye. Yinghui Li. Qingyu Zhou, Yangning Li, Shirong Ma, Hai-Tao Zheng, and Ying Shen. 2023b. CLEME: debiasing multi-reference eval- uation for grammatical error correction. CoRR. fbs/2305.10819.
Batch size per GPU
4096 tokens
Adam (Kingma and Ba, 2014)
Optimizer
(91 = 0.9, 82 = 0.999, c = 1 10-3)
Learning rute Warmup updates
3 x 10-5
2000
Max source length Dropout Dropout-see
1024
Zheng Yuan and Mariano Felice. 2013. Constrained grammatical error correction using statistical ma- chine translation. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 52-61, Sofa, Bulgaria. Association for Computational Linguistics.
0.3 (Eaglish); 0.2 (Chinese) 0.2
Fine-tuning
Weights of
3 10 5.55, 1=1.0 f=1100 x 10-5 (English)
Learning rute
3 x 10 (Chinese)
Warmup updates
2000
Inference
Baolin Zhang. 2009. Features and functions of the hsk dynamic composition corpus. International Chinese Language Education, 4:71-79.
Beam size
12
Table 10: Hyper-parameter values used in our experi- ments.
Ding Zhang, Yinghui Li, Qingyu Zhou, Shirong Ma, Yangning Li, Yunbo Cao, and Hai- Tao Zheng. 2023. Contextual similarity is more valuable than charac- ter similarity: An empirical study for chinese spell checking. In ICASSP 2023 2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
We list the main hyper-parameters in Table 10. For the pre-training stage. we follow the same syper-parameters as described in (Zhang al., 2022b). We introduce MixEdit throughout the three-stage fine-tuning. To determine the optimal balancing weights for the training objective, we ex- periment with various values for a within (0.5, 1.0, 1.2, 2.0) and B within (0.5, 1.0, 2.0). Our exper- iments reveal that the configuration with a = 1.0 and B 1.0 achieves the highest Fo.s score on BEA-19 and CoNLL-14, which is used as the de- fault settings in all our experiments.
Yue Zhang, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022a. MuCGEC: a multi-reference multi-source evaluation dataset for Chinese grammatical error rection. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 3118-3130, Scattle, United States. Association for Computational Linguistics.
A.2 Details of GEC Data Augmentation
Yue Zhang, Bo Zhang, Zhenghua Li. Zuyi Bao, Chen Li. and Min Zhang. 2022b. SynGEC: syntax-enhanced grammatical error correction with a tailored GEC- oriented parser. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 2518-2531, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
We follow the default experimental settings of GEC data augmentation methods as proposed in their pa- pers or source code. our experiments, we set Pmask = 0.3 for DN, which has been proven to be the best by Kiyono et al. (2020). For PN, we generate pscudo data by following the default in- structions provided in the open-source project 7, We train the BT model, which is initialized with the weights of BART-large, on the CLang8 dataset, setting Brandom 6 to yields the best performance as evidenced Section 4.3. As for RT, we gen- crate pscudo data via Chinese as the bridge lan- guage, leveraging two off-the-shelf en-zh 8 and zh-
Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018 Overview of the nipce 2018 shared task: Grammatical error correction. In Natural Language Processing and Chinese Computing, pages 439-445, Cham. Springer International Publishing.
Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, and Ming Zhou. 2020. Improving grammati- cal error correction with machine translation pairs. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 318-328. Online. Association for Computational Linguistics.
hhttps://gthhut wwn/ www./aaaabrain/hell.... 'https: huttps//huuggingaace http://hggginaaaaa....ooo/sss/as hwwp/HelsiniiNNPPP..
mous-mt-en-zh

---

We have analyzed the effectiveness of GEC data augmentation methods from the lens of Affinity and Diversity measures in Section 4.3. However, one may argue that MixEdit gains unfair advan- translation models (Tiedemann and Thottingal,
Method
Data
Aft.'
Div.
P
R
Faus
2020).
Baseline
124K
8.78
$6.03
37.60
51.03
DN
249K
0.41
10.70
53.89
39.72
50.30
PN
B Extra Experiments
RT
249K
0.75
10.54
50.32
39.82
47.80
Round=1
249K
1.91
7.46
56.64
36.75
51.11
Round=2
249K
1.93
7.56
56.07
40.15
51.95
B.1
Affinity and Diversity of Extra Datasets
Round=4
249K
Round=8
249K
1.77
7.92
54,03
41.32
50.90
Round=16
249K
1.59
8.24
$4.43
40.88
51.04
Method
Affinity!
Diversity
P
Fas
BT
Baseline
8.81
63.04
44.69
58.25
Brandon-0
249K
56.46
DN
0.33
11.31
28.52
24.48
27.60
3
249K
1.47
7.67
54.94
42.87
52.01
RT
0.53
11,14
18.43
37.16
20.50
249K
1.57
8.22
$4.04
46.38
52.31
PN
Brandon
249K
1.53
8.59
53.63
45.76
51.84
Round=1
3.74
7.63
62.66
28.32
50.43
Panddm
249K
1.45
53.79
43.03
Round=2
3.55
7.79
60.24
34.12
52.24
MixFdit (Static)
249K
2.33
8.52
57.98
37.78
52.38
Round=4
3.17
8.05
56,70
38.58
51.83
MixEdit (Dynumic)
57.96
39.00
52.83
Round=8
49.70
48.29
Round=16
1.92
9.10
40.59
48.21
41.92
Table 13: Results on the combination of the realistic and pseudo BEA-train datasets for various data augmen- tation methods. MixEdit (Dynamic) generate pseudo data on-the-fly during fine-tuning.
BT
Brandom=
1.04
7.04
57.50
13.96
35.41
8randam =3
3.59
7.60
62.63
38.94
55.83
Arandom =
3:73
8.15
57.73
46.71
55.13
Prandum =9
3.08
8.55
53.07
51.19
52.68
Brandom=11
8.87
49.38
52.63
50.00
MixEdit (Ours)
19.29
8.85
64.50
38.32
56.75
worth noting that BT relies on an additional model to generate grammatical errors, which introduces efficiency concerns. The Pearson's correlation co- efficients between FO.5 and Affinity are 0.6239 CLang8. and 0.7717 on HSK. The results indicate strong moderate correlation between Affinity and FO.5 on different datasets, demonstrating the generalization of our proposed approach.
Table 11: Affinity and Diversity of data augmentation methods. All the models are trained using realistic or pseudo English CLang8 datasets.
Method
Affinity!
Diversity
P
Fas
Baseline
8.78
49.10
25.30
41.33
DN
0.39
10.10
28.84
26.56
28.35
RT
0.46
10.57
19.39
29.84
20.84
PN
B.2
Complementary Effeetiveness of Pseudo Data
Round=1
3.07
6.40
45.84
08.86 86
24.98
Round=2
2.73
6.59
43.54
13.28
29.91
Round=4
6.91
44.96
15.12
32.24
Round=8
32.76
Round=16
1.17
8.20
39.82
21.24
33.89
BT
=randam==0
1.30
5.44
27.07
03.34
11.18
Brandom
3.00
7.24
45.48
15.78
33.04
Arandom =6
2.73
8.10
41.72
26.32
37.35
Brandom =9
2.17
8.83
37.83
30.27
36.03
tages since the information about the positions of grammatical errors is only visible to MixEdit. To this end, we investigate the complementary effec- tiveness of pseudo data in this section. Specifically, wc construct a combination dataset by appending the realistic data of BEA-train to the pseudo data generated by data augmentation methods, where target sentence correspond to two source sentences.
Brandom
1.75
9.45
32.27
32.89
32.33
MixEdit (Ours)
8.52
47.81
18.31
36.15
Table 12: Affinity and Diversity of data augmentation methods. All the models are trained using realistic or pseudo Chinese HSK datasets.
We conduct extra experiments illustrated in Sec- tion 4.3 on English CLang8 and Chinese HSK datasets. The results are listed in Table 11 and Ta- ble 12, respectively. Surprisingly, backtranslation (BT) achieves the highest FO.5 on Chinese HSK. We speculate that Chinese grammatical errors are inherently more intricate, providing an advantage for BT as it can generate pseudo grammatical errors that are closer to authentic ones. Nonetheless, it is
We train GEC models using the combination dataset and report the results in Table 13. Similarly, DN and RT perform worse than the baseline since these methods inject considerable undesired noise, which makes the model prone to inaccurate correc- tions. PN arrives its peek of Fo.s score Round=2, rather than Round-8 in Table 4. BT arrives the peek at Brandom 6, and most selections of Brandom can improve GEC models. MixEdit (Static) performs the best among all static data augmentation meth-
htttps://hugginghee httpg//faaa...cesseeeeeeeeeeeeee wo/.eesinkiiNi.... opus zh-en

---

CoNL.L-14-test
BEA-19-test
Corruption Ratio
P/R/Fo.s
P/R/Foo.
0.00
76.81/45.00/67.
76.37/62.71/73..8
0.02
74.94/48.70/67.6
75.14/64.04/22
0.05
74.70/49.33/67.88
74.78/64.6772.51
72.42/50 68/66.70
73.70/65 25/71.25/1..88
0.15
73.26/48. 95/66.64
72.96/64.46711.
Table 14: Results of incorporating PN into MixEdit, The Corruption Ratio indicates the probability adding grammatical errors to token in the source sentence.
ods. Furthermore, MixEdit (Dynamic) achieves the highest Fo.s score, demonstrating the effectiveness dynamic pseudo data construction.
B.3 Incorporating PN into MixEdit
We also explore the effectiveness of incorporating PN into MixEdit in dynamic manner. In this approach, we applying PN after running MixEEdit, This means that randomly add grammatical er- to the pseudo source generated by MixEdit. We report the results of varying PN corruption ra- tios Table 14. Our findings suggest that when the correction ratio is low, PN can benefit GEC models on CoNLL-14-test. However, PN decreases the Fo.s scores on BEA-19-test regardless of different correction ratios. We attribute this to the differ- ent characteristics of the two evaluation datasets, where CoNLL-14 contains more typical grammati- cal errors written by language learners.


Reviews:
[{"review_id": "Nrh0HnHjf6", "reviewer": "Reviewer_ytRu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation of our work's contributions. We are honored to have the opportunity to discuss with you.\n\n---\n\n**Q1**: Could you elaborate the experiment results concerning comparable settings, especially with respect to the numerous experimental settings listed in Table 3?\n\n**A1**: Table 3 reports previous studies' results with different experimental settings, which make it hard to compare fairly with each other due to the selection of backbones, datasets and training procedure. However, the results of **SynGEC** [1] and our experiments are **comparable fairly**. We will clarify the difference of experimental settings and improve the discussion on the results in Table 3 in our final version.\n\n---\n\n**Q2**: Discussion regarding the sample efficiency should be added.\n\n**A2**: The comparison of various augmentation methods' sample efficiency is shown in Table 4 of our paper. With the same number of augmented samples, MixEdit can achieve the highest F0.5 score on BEA-train. To further confirm our finding, we have **conducted additional experiments on other datasets** (English CLang8 and Chinese HSK). \n\n**Table 1**: Affinity and Diversity of data augmentation methods on **English CLang8**. The experimental settings are identical to those mentioned in Section 4.3 of our paper, with the only difference being the choice of datasets.\n\n| Method        | Affinity | Diversity | P     | R     | F0.5  |\n| ------------- | -------- | --------- | ----- | ----- | ----- |\n| Baseline      | -        | 8.81      | 63.04 | 44.69 | 58.25 |\n| DN            | 0.33     | **11.31**     | 28.52 | 24.48 | 27.60 |\n| PN Round=1    | 3.74     | 7.63      | 62.66 | 28.32 | 50.43 |\n| PN Round=2    | 3.55     | 7.63      | 60.24 | 34.12 | 52.24 |\n| PN Round=4    | 3.17     | 8.05      | 56.70 | 38.58 | 51.83 |\n| PN Round=8    | 2.65     | 8.47      | 49.70 | 43.36 | 48.29 |\n| PN Round=16   | 1.92     | 9.10      | 40.59 | 48.21 | 41.92 |\n| BT $\\beta=0$  | 1.04     | 7.04      | 57.50 | 13.96 | 35.41 |\n| BT $\\beta=3$  | 3.59     | 7.60      | 62.63 | 38.94 | 55.83 |\n| BT $\\beta=6$  | 3.73     | 8.15      | 57.73 | 46.71 | 55.13 |\n| BT $\\beta=9$  | 3.08     | 8.55      | 53.07 | 51.19 | 52.68 |\n| BT $\\beta=12$ | 2.55     | 8.87      | 49.38 | **52.63** | 50.00 |\n| RT            | 0.53     | 11.14     | 18.43 | 37.16 | 20.50 |\n| MixEdit       | **19.29**    | 8.85      | **64.50** | 38.32 | **56.75** |\n\n**Table 2**: Affinity and Diversity of data augmentation methods on **Chinese HSK**.\n\n| Method        | Affinity | Diversity | P     | R     | F0.5  |\n| ------------- | -------- | --------- | ----- | ----- | ----- |\n| Baseline      | -        | 8.78      | 49.10 | 25.30 | 41.33 |\n| DN            | 0.39     | 10.10     | 28.84 | 26.56 | 28.35 |\n| PN Round=1    | 3.07     | 6.40      | 45.84 | 8.86  | 24.98 |\n| PN Round=2    | 2.73     | 6.59      | 43.54 | 13.28 | 19.91 |\n| PN Round=4    | 2.25     | 6.91      | 44.96 | 15.12 | 32.24 |\n| PN Round=8    | 1.43     | 7.43      | 42.83 | 16.89 | 32.76 |\n| PN Round=16   | 1.17     | 8.20      | 39.82 | 21.24 | 33.89 |\n| BT $\\beta=0$  | 1.30     | 5.44      | 27.07 | 3.34  | 11.18 |\n| BT $\\beta=3$  | 3.00     | 8.24      | 45.48 | 15.78 | 33.04 |\n| BT $\\beta=6$  | 2.73     | 8.10      | 41.72 | 26.32 | **37.35** |\n| BT $\\beta=9$  | 2.17     | 8.8.3     | 37.83 | 30.27 | 36.03 |\n| BT $\\beta=12$ | 1.75     | 9.45      | 32.27 | **32.59** | 32.33 |\n| RT            | 0.46     | **10.57**     | 19.39 | 29.84 | 20.84 |\n| MixEdit       | **3.95**     | 8.52      | **47.81** | 18.31 | 36.15 |\n\nThe results reveal that **MixEdit achieves the highest F0.5 score on CLang8 and comparable F0.5 on HSK**. Surprisingly, backtranslation (BT) achieves the highest F0.5 on Chinese HSK. We speculate that Chinese grammatical errors are inherently more intricate, providing an advantage for BT as it can generate pseudo grammatical errors that are closer to authentic ones. Nonetheless, it is worth noting that BT relies on an **additional model** to generate grammatical errors, which introduces **efficiency concerns**.\n\n---\n\n**Reference**\n\n[1] SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser [EMNLP 2022]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper quantifies the known insights into GEC's data augmentation by demonstrating its relationship with performance. \n\n- It also showed that state-of-the-art performance can be achieved by designing a learning objective function to control these quantities. \n\n- Through systematic experiments comparing multiple data augmentation techniques, as well as detailed experiments including ablation studies, the paper revealed that the affinity with real data and the use of an error pattern pool have significant effects on the GEC performance.\n", "reasons_to_reject": "While there is no major reason to reject the paper, the reader's understanding of the results may be enhanced if the interpretation of the results is discussed concerning substantially comparable settings, especially with respect to the numerous experimental settings listed in Table 3. \nAlthough the efficiency of sampling is mentioned in the introduction, it does not seem to be revisited later in the paper. If this is indeed the case, the claim needs to be revised.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents two efficiently computable and interpretable metrics that provide a better understanding of data augmentation in Grammar Error Correction (GEC). Previous studies have suggested that the similarity between real and pseudo data, as well as the diversity within pseudo data, play key roles in GEC augmentation. However, there's been no quantitative measure to evaluate the relationship between these perspectives and the performance of various data improvement methods in a coherent experimental setting. This paper offers a solution to this by proposing a method for measuring affinity using the inverse function of KL divergence and diversity in terms of entropy.\n\nThe findings using these measurements indicate that affinity has a direct correlation with performance. As for diversity, it presents a trade-off between precision and recall, according to the authors.\n\nBased on this analysis, the authors designed a loss function, named MixEdit, that appropriately mixes the losses generated from each of the real and augmented data using KL divergence loss and cross-entropy loss, achieving performance comparable to state-of-the-art models.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YNMEMyPhBj", "reviewer": "Reviewer_g98u", "summary": "", "questions": "- A. Regarding footnote$^2$ on Page 4, as it differs from previous research, could you provide additional evidence or data to support the claim that a single augmented sample performs as effectively as multiple samples? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive comments and insightful reviews. Please find below our point-by-point responses:\n\n---\n\n**Q1**: It is essential to prove a more concrete relationship between Affinity and performance.\n\n**A1**: We agree with your suggestion as it is technically sound. To further confirm the relationship between Affinity/Diversity and GEC performance, we have **conducted additional experiments on other datasets** (English CLang8 and Chinese HSK).\n\n**Table 1**: Affinity and Diversity of data augmentation methods on **English CLang8**. The experimental settings are identical to those mentioned in Section 4.3 of our paper, with the only difference being the choice of datasets.\n\n| Method        | Affinity | Diversity | P     | R     | F0.5  |\n| ------------- | -------- | --------- | ----- | ----- | ----- |\n| Baseline      | -        | 8.81      | 63.04 | 44.69 | 58.25 |\n| DN            | 0.33     | **11.31**     | 28.52 | 24.48 | 27.60 |\n| PN Round=1    | 3.74     | 7.63      | 62.66 | 28.32 | 50.43 |\n| PN Round=2    | 3.55     | 7.63      | 60.24 | 34.12 | 52.24 |\n| PN Round=4    | 3.17     | 8.05      | 56.70 | 38.58 | 51.83 |\n| PN Round=8    | 2.65     | 8.47      | 49.70 | 43.36 | 48.29 |\n| PN Round=16   | 1.92     | 9.10      | 40.59 | 48.21 | 41.92 |\n| BT $\\beta=0$  | 1.04     | 7.04      | 57.50 | 13.96 | 35.41 |\n| BT $\\beta=3$  | 3.59     | 7.60      | 62.63 | 38.94 | 55.83 |\n| BT $\\beta=6$  | 3.73     | 8.15      | 57.73 | 46.71 | 55.13 |\n| BT $\\beta=9$  | 3.08     | 8.55      | 53.07 | 51.19 | 52.68 |\n| BT $\\beta=12$ | 2.55     | 8.87      | 49.38 | **52.63** | 50.00 |\n| RT            | 0.53     | 11.14     | 18.43 | 37.16 | 20.50 |\n| MixEdit       | **19.29**    | 8.85      | **64.50** | 38.32 | **56.75** |\n\n**Table 2**: Affinity and Diversity of data augmentation methods on **Chinese HSK**.\n\n| Method        | Affinity | Diversity | P     | R     | F0.5  |\n| ------------- | -------- | --------- | ----- | ----- | ----- |\n| Baseline      | -        | 8.78      | 49.10 | 25.30 | 41.33 |\n| DN            | 0.39     | 10.10     | 28.84 | 26.56 | 28.35 |\n| PN Round=1    | 3.07     | 6.40      | 45.84 | 8.86  | 24.98 |\n| PN Round=2    | 2.73     | 6.59      | 43.54 | 13.28 | 19.91 |\n| PN Round=4    | 2.25     | 6.91      | 44.96 | 15.12 | 32.24 |\n| PN Round=8    | 1.43     | 7.43      | 42.83 | 16.89 | 32.76 |\n| PN Round=16   | 1.17     | 8.20      | 39.82 | 21.24 | 33.89 |\n| BT $\\beta=0$  | 1.30     | 5.44      | 27.07 | 3.34  | 11.18 |\n| BT $\\beta=3$  | 3.00     | 8.24      | 45.48 | 15.78 | 33.04 |\n| BT $\\beta=6$  | 2.73     | 8.10      | 41.72 | 26.32 | **37.35** |\n| BT $\\beta=9$  | 2.17     | 8.8.3     | 37.83 | 30.27 | 36.03 |\n| BT $\\beta=12$ | 1.75     | 9.45      | 32.27 | **32.59** | 32.33 |\n| RT            | 0.46     | **10.57**     | 19.39 | 29.84 | 20.84 |\n| MixEdit       | **3.95**     | 8.52      | **47.81** | 18.31 | 36.15 |\n\nWe compute the **Pearson's correlation coefficient** for the 5 GEC data augmentation methods involved: DN, PN, BT, RT and MixEdit. For PN and BT, we choose the hyper-parameter configurations that yield the highest F0.5 scores. The Pearson's correlation coefficients between F0.5 and Affinity are **0.9485 on BEA-train, 0.6239 on CLang8, and 0.7717 on HSK**. The results indicate a **strong or moderate correlation between Affinity and F0.5**.\n\n---\n\n**Q2**: The trade-off of Diversity seems to be hard to examine in a more scientific way.\n\n**A2**: As stated in Section 4.3 of our paper, We **qualitatively** analyze the effect of Diversity on GEC performance within a fixed method. The trade-off of Diversity is apparent when adjusting certain hyper-parameters responsible for Diversity (Round for PN and $\\beta$ for BT). As Diversity increases, Precision and F0.5 initially increase and then decrease, reaching their peak at an appropriate setting. Meanwhile, Recall continues to increase.\n\n---\n\n**Q3**: How does the relationship between Affinity/Diversity and performance show in other datasets?\n\n**A3**: To explore the relationship between Affinity/Diversity and performance across different datasets, we have conducted additional experiments on English CLang8 and Chinese HSK. The results and analyses are provided above. The relationship between Affinity/Diversity and performance is **similar to the results on BEA-train shown in our paper**.\n\n---\n\n**Q4**: Inconsistent logic of paper. Section 4.3 should be presented before Section 4.1 and 4.2.\n\n**A4**: The suggestion to rearrange the narrative order of Section 4 is valuable and we will make the necessary adjustments.\n\n---\n\n**Q5**: Could you provide additional evidence or data to support the claim that a single augmented sample performs as effectively as multiple samples?\n\n**A5**: In our preliminary experiments, we found using N=2 augmented samples can improve F0.5 scores by **less than 0.1** compared with N=1, and sometimes even have a negative impact. We speculate that too many augmented samples may dilute the valuable supervision information obtained from realistic sources. Additionally, increasing N would **significantly escalate the training cost**. For instance, with N=2, it would entail **three** feed forwards and back propagations for a single sample (two for augmented samples and one for the realistic sample). We will delve further into this issue in the final version of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Motivation**: I greatly appreciate the motivation behind this work, as it aims to understand the underlying mechanisms of data augmentation approaches in the field of GEC. The paper provides valuable insights by integrating the strengths of previous methods and presents an interesting finding: an effective augmentation method will generate data with high Affinity and appropriate Diversity.\n\n- **Novel Metrics**: The introduction of the proposed metrics (Affinity and Diversity) offers a statistical perspective to explain existing GEC data augmentation methods. The paper also justifies why these dimensions are crucial in GEC data augmentation.\n\n- **Experiments**: The experiments conducted are solid and effectively demonstrate the efficacy of MixEdit across different languages (English, Chinese). Although existing augmentation methods may not significantly benefit from MixEdit, it proves to be highly effective in on-the-fly augmentation scenarios.\n", "reasons_to_reject": "- **Soundness Problems**: The main argument throughout the paper suggests that a better augmentation method might produce data with high Affinity and appropriate Diversity, which serves as the motivation for introducing the MixEdit method (Line 261). However, I still have some concerns regarding the validity of this argument:\n\n  1. While Table 4 presents the observed trend and Section 4.3 discusses these findings, it is essential to prove a more concrete relationship between Affinity and performance. Utilizing statistical measures like Pearson's correlation coefficient could strengthen the argument. Moreover, the trade-off of Diversity seems to be hard to examine in a more scientific way.\n\n  2. In addition to the BEA-train dataset, it would be beneficial to explore other realistic datasets to analyze the relationship between Affinity/Diversity and augmentation performance.\n\n- **Inconsistent logic of paper**: The abstract and initial sections of the paper indicate that the analysis of data augmentation mechanisms using the proposed metrics will be presented first, followed by the introduction of a novel augmentation method based on the findings. However, the subsequent sections deviate from this logical order by presenting the method and experimental results first (Section 4.1, 4.2) before revealing observations related to the two metrics (Section 4.3). This inconsistency in the paper's structure creates confusion for readers.\n", "questions_to_authors": "- A. Regarding footnote$^2$ on Page 4, as it differs from previous research, could you provide additional evidence or data to support the claim that a single augmented sample performs as effectively as multiple samples? \n", "paper_topic_and_main_contributions": "This paper aims to determine the optimal conditions for when grammatical error correction (GEC) systems benefit the most from data augmentation methods and attempts to develop a more effective augmentation approach based on the identified characteristics of effective augmentations. The contributions of this paper are as follows:\n\n1) The paper introduces novel analysis perspectives for GEC data augmentation. Two interpretable metrics, Affinity, and Diversity, are proposed to uncover the underlying mechanisms of previous augmentation methods. Affinity measures the KL divergence, while Diversity quantifies the entropy of probabilistic distributions. \n2) Through empirical exploration, the paper identifies the key characteristics of an effective GEC data augmentation strategy. Based on these findings, the authors introduce a new augmentation approach called MixEdit. This approach optimizes the model using more appropriate pseudo data and objectives.\n\nThe experiments conducted on English and Chinese benchmarks demonstrate that MixEdit is an outstanding method, even without additional data. Furthermore, it can be seamlessly integrated with existing augmentation approaches.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 224: datasets respectively. -> datasets, respectively.\n- Line 241: axis -> perspective\n- Line 283: Inspired by (Shen et al., 2020; Chen et al., 2021), -> Inspired by Shen et al., (2020) and Chen et al. (2021),\n- Line 556: Following (Zhang et al., 2022a), -> Following Zhang et al. (2022a),\n"}}, {"review_id": "nvZhSv1FTP", "reviewer": "Reviewer_RBnA", "summary": "", "questions": "- (1) Table 8: I wonder if there exists a V100 card with 80GB memory. Is this a typo?\n- (2) Is MixEdit sensitive to the choice of hyperparameter alpha and beta? I wish to see the performance plot.\n- (3) Depending on the error pattern, the length of original sequence and edited sequence may not match. In such case, how do you compute P_avg?\n- (4) Why do the authors call their method MixEdit? To what extent does it \"Mix\" and \"Edit\"?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We attach great importance to your questions and hope to address your concerns.\n\n---\n\n**Q1**: There is no consistent trend for diversity metric. It seems elusive to determine an appropriate diversity.\n\n**A1**: We believe that the optimal diversity of augmented datasets varies across evaluation datasets. For **L2-speaker** evaluation datasets with a higher density of grammatical errors, augmented datasets with greater diversity may be preferred, while for **native-speaker** evaluation datasets with lower error density, the **opposite** may be true. Therefore, determining the ideal level of diversity without conducting experiments proves to be challenging. Additionally, we would like to emphasize that our proposed Affinity/Diversity metrics aim to **offer a fresh perspective** on characterizing and comprehending GEC data augmentation, as mentioned in line 256.\nTo support our claims regarding the relationship between Affinity/Diversity and GEC performance, we have **conducted additional experiments on English CLang8 and Chinese HSK**.\n\n**Table 1**: Affinity and Diversity of data augmentation methods on **English CLang8**. The experimental settings are identical to those mentioned in Section 4.3 of our paper, with the only difference being the choice of datasets.\n\n| Method        | Affinity | Diversity | P     | R     | F0.5  |\n| ------------- | -------- | --------- | ----- | ----- | ----- |\n| Baseline      | -        | 8.81      | 63.04 | 44.69 | 58.25 |\n| DN            | 0.33     | **11.31**     | 28.52 | 24.48 | 27.60 |\n| PN Round=1    | 3.74     | 7.63      | 62.66 | 28.32 | 50.43 |\n| PN Round=2    | 3.55     | 7.63      | 60.24 | 34.12 | 52.24 |\n| PN Round=4    | 3.17     | 8.05      | 56.70 | 38.58 | 51.83 |\n| PN Round=8    | 2.65     | 8.47      | 49.70 | 43.36 | 48.29 |\n| PN Round=16   | 1.92     | 9.10      | 40.59 | 48.21 | 41.92 |\n| BT $\\beta=0$  | 1.04     | 7.04      | 57.50 | 13.96 | 35.41 |\n| BT $\\beta=3$  | 3.59     | 7.60      | 62.63 | 38.94 | 55.83 |\n| BT $\\beta=6$  | 3.73     | 8.15      | 57.73 | 46.71 | 55.13 |\n| BT $\\beta=9$  | 3.08     | 8.55      | 53.07 | 51.19 | 52.68 |\n| BT $\\beta=12$ | 2.55     | 8.87      | 49.38 | **52.63** | 50.00 |\n| RT            | 0.53     | 11.14     | 18.43 | 37.16 | 20.50 |\n| MixEdit       | **19.29**    | 8.85      | **64.50** | 38.32 | **56.75** |\n\n**Table 2**: Affinity and Diversity of data augmentation methods on **Chinese HSK**.\n\n| Method        | Affinity | Diversity | P     | R     | F0.5  |\n| ------------- | -------- | --------- | ----- | ----- | ----- |\n| Baseline      | -        | 8.78      | 49.10 | 25.30 | 41.33 |\n| DN            | 0.39     | 10.10     | 28.84 | 26.56 | 28.35 |\n| PN Round=1    | 3.07     | 6.40      | 45.84 | 8.86  | 24.98 |\n| PN Round=2    | 2.73     | 6.59      | 43.54 | 13.28 | 19.91 |\n| PN Round=4    | 2.25     | 6.91      | 44.96 | 15.12 | 32.24 |\n| PN Round=8    | 1.43     | 7.43      | 42.83 | 16.89 | 32.76 |\n| PN Round=16   | 1.17     | 8.20      | 39.82 | 21.24 | 33.89 |\n| BT $\\beta=0$  | 1.30     | 5.44      | 27.07 | 3.34  | 11.18 |\n| BT $\\beta=3$  | 3.00     | 8.24      | 45.48 | 15.78 | 33.04 |\n| BT $\\beta=6$  | 2.73     | 8.10      | 41.72 | 26.32 | **37.35** |\n| BT $\\beta=9$  | 2.17     | 8.8.3     | 37.83 | 30.27 | 36.03 |\n| BT $\\beta=12$ | 1.75     | 9.45      | 32.27 | **32.59** | 32.33 |\n| RT            | 0.46     | **10.57**     | 19.39 | 29.84 | 20.84 |\n| MixEdit       | **3.95**     | 8.52      | **47.81** | 18.31 | 36.15 |\n\n\nThe relationship between Affinity/Diversity and performance is **similar to the results on BEA-train shown in our paper**.\n\n---\n\n**Q2**: Inconsistent Notation.\n\n**A2**: The symbols $\\rm{x}$ and $\\rm{y}$ in Section 2 represent source and target sentences, whereas $\\boldsymbol{x}$ and $\\boldsymbol{y}$ in Section 3.1 indicate an edit from $\\boldsymbol{x}$ to $\\boldsymbol{y}$. Additionally, the symbols $X, Y$ in Figure 1 should be **lowercase** to represent source and target sentences. The typo will be corrected in the final version.\n\n---\n\n**Q3**: How does MixEdit manage to achieve diversity and affinity?\n\n**A3**: The explanation for MixEdit's high Affinity and appropriate Diversity is provided in Section 3.2, specifically in Lines 260-276. The high Affinity of MixEdit stems from the fact that the **distribution** of pseudo grammatical errors it generates is the **same** as that of the ground truth. MixEdit only replaces the original grammatical errors with augmented ones, **maintaining the same error density** in the augmented dataset. As a result, the diversity of the augmented dataset **closely resembles** that of the original dataset.\n\n---\n\n**Q4**: Table 8: I wonder if there exists a V100 card with 80GB memory. Is this a typo?\n\n**A4**: It is a typo, and \"V100\" is actually \"A100\".\n\n---\n\n**Q5**: Is MixEdit sensitive to the choice of hyperparameter alpha and beta?\n\n**A5**: In Appendix A.1, we explore the optimal values of $\\alpha$ and $\\beta$, which are chosen from the set {0.5 0.8, 1.0, 1.2, 2.0} and {0.5, 1.0, 2.0}, respectively. We conduct additional experiments on BEA-train and report the results below.\n\n\n**Table 3**: Results of various $\\alpha$ on BEA-train.\n\n| $\\alpha$ | P     | R     | F0.5  |\n| -------- | ----- | ----- | ----- |\n| 0.5      | 55.96 | 41.35 | 52.27 |\n| 0.8      | 55.65 | 41.35 | 52.05 |\n| 1.0      | 57.96 | 39.00 | **52.83** |\n| 1.2      | 56.75 | 40.93 | 52.68 |\n| 2.0      | 55.88 | 41.30 | 52.20 |\n\n**Table 4**: Results of various $\\beta$ on BEA-train.\n\n| $\\beta$ | P     | R     | F0.5  |\n| ------- | ----- | ----- | ----- |\n| 0.5     | 57.56 | 38.51 | 52.38 |\n| 1.0     | 57.96 | 39.00 | **52.83** |\n| 2.0     | 56.69 | 40.59 | 52.52 |\n\nAs shown in Appendix A.1, the **optimal** hyperparameter setting is $\\alpha=1.0$ and $\\beta=1.0$.\n\n---\n\n**Q6**: How to compute P_avg?                                                                       \n\n**A6**: Actually, $P_{avg}$ is computed solely based on the target sentences and is **not** influenced by the source sentences. Therefore, the computation of $P_{avg}$ is straightforward.\n\n---\n\n**Q7**: Why does MixEdit get its name?\n\n**A7**: \"Mix\" is so named as the augmented grammatical errors are **sampled from the Error Pattern Pool**, as depicted in Figure 1 of our paper. The term \"Edit\" is used to describe the **unit** of our proposed data augmentation method. We have coined the name \"MixEdit\" for our proposed method to **reflect the augmentation process**.\n\n---\n\n**Q8**: Table 4 should be presented as a graph, to better illustrate the effectiveness of two proposed metrics.\n\n**A8**: We did consider presenting the results as a graph, but it was more feasible to showcase the **trade-off between Precision and Recall** in tabular form.\n\n---\n\n**Q9**: How to acquire subwords by bype-pair-encoding?\n\n**A9**: The subwords are obtained using a off-the-shelf tokenizer following previous study [1], ensuring the **comparability** of our experiments to theirs.\n\n---\n\n**Reference**\n\n[1] SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser [EMNLP 2022]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors conduct experiments on multiple languages (English and Chinese). These experiments strongly support the effectiveness of the proposed method (MixEdit).\n- The proposed method achieves the state-of-the-art performance on benchmark datasets.", "reasons_to_reject": "## Diversity Metric?\n\nMy biggest concern with this paper is that the usefulness of the proposed metrics are questionable. On one hand, the analysis on Table 4 presents that high affinity value seems to consistently improve the model performance. On the other hand, there is no consistent trend for diversity metric. For example, high diversity does not seem to always improve the performance. Given this result, the authors recommends that an \"appropriate\" diversity is important. However, I believe that such recommendation is unfriendly for those who wishes to utilize these metrics; one still needs to conduct an extensive hyperparameter search on diversity to maximize the performance.\n\n## Clarity Issue\n\nI had difficulty with understanding the details of this paper.\n\n(1): Notation is inconsistent. In Section 2, \"x\" and \"y\" represent ungrammatical source sentence and grammatical target sentence, respectively. However, in Section 3.1, \"x\" and \"y\" seem to represent different concepts. I assume that they represent a pair of corrections (L225). In addition, there is yet another notation in Figure 1. Here, \"X\" and \"Y\" are used for representing source and reference. As a result I was unable to understand how the authors compute both affinity and diversity.\n\n(2): How does MixEdit manage to achieve diversity and affinity? I was unable to find a discussion on this aspect.", "questions_to_authors": "- (1) Table 8: I wonder if there exists a V100 card with 80GB memory. Is this a typo?\n- (2) Is MixEdit sensitive to the choice of hyperparameter alpha and beta? I wish to see the performance plot.\n- (3) Depending on the error pattern, the length of original sequence and edited sequence may not match. In such case, how do you compute P_avg?\n- (4) Why do the authors call their method MixEdit? To what extent does it \"Mix\" and \"Edit\"?", "paper_topic_and_main_contributions": "This paper presents two metrics that characterizes pseudo data for grammatical error correction (GEC), namely, affinity and diversity.\nBase on these two metrics, this paper proposes a novel method for data augmentation (MixEdit).\nMixEdit applies data augmentation to real training data by generating label-preserving grammatical errors.\nThe contribution of this paper is that MixEdit achieves the state-of-the-art result on three benchmark datasets.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The authors might want to use a consistent notation for x and y throughout the paper\n- Table 4 should be presented as a graph, to better illustrate the effectiveness of two proposed metrics.\n- L344: \"We acquire subwords by byte-pair-encoding\": I am confused. Do the authors build the tokenizer by themselves, or do they use off-the-shelf tokenizer provided by BART model? I believe that the latter is more common practice."}}]

Author rebuttal:

Meta-review: The paper addresses the topic of data augmentation for Grammatical Error Correction (GEC). They introduce two measurements of data quality – Affinity (based on KL-divergence) and Diversity (measured in terms of entropy). They propose a MixEdit loss function that optimizes the model using more appropriate pseudo data based on the two metrics.

Experiments on English and Chinese benchmarks show that high Affinity seems to improve model performance, but there is no consistent trend for the Diversity metric performance-wise. They conclude that a data augmentation strategy should be characterized by high Affinity and appropriate Diversity.   

The paper could be substantially improved by strengthening  the argument regarding the utility of the proposed metrics. Specifically, it  is not clear  how to quantify “appropriate diversity” for a dataset. The paper should make a better effort to explore this issue and to make a more convincing argument regarding the utility of the metric. The individual reviews have specific suggestions about this. The paper would also be strengthened significantly if experiments on other languages were added.
Decision: Accept-Findings