System Combination via Quality Estimation for Grammatical Error Correction
Authors: Muhammad Reza Qorib
Hwee Tou Ng
Venue: EMNLP 2023 Main (2023)
Keywords: Grammatical Error Correction
Quality Estimation
System Combination
TL;DR: We propose a new state-of-the-art quality estimation (QE) model for grammatical error correction (GEC) and propose novel methods to utilize QE models to combine GEC models

Abstract: Quality estimation models have been developed to assess the corrections made by grammatical error correction (GEC) models when the reference or gold-standard corrections are not available. An ideal quality estimator can be utilized to combine the outputs of multiple GEC systems by choosing the best subset of edits from the union of all edits proposed by the GEC base systems. However, we found that existing GEC quality estimation models are not good enough in differentiating good corrections from bad ones, resulting in a low F0.5 score when used for system combination. In this paper, we propose GRECO, a new state-of-the-art quality estimation model that gives a better estimate of the quality of a corrected sentence, as indicated by having a higher correlation to the F0.5 score of a corrected sentence. It results in a combined GEC system with a higher F0.5 score. We also propose three methods for utilizing GEC quality estimation models for system combination with varying generality: model-agnostic, model-agnostic with voting bias, and model-dependent method. The combined GEC system outperforms the state of the art on the CoNLL-2014 test set and the BEA-2019 test set, achieving the highest F0.5 scores published to date.

Grammatical error correction (GEC) is the task of automatically detecting and correcting errors in text, including but not limited to grammatical er- rors, misspellings, orthographic errors, and seman- tic errors (Chollampatt et al., 2016; Chollampatt and Ng. 2018a; Qorib et al., 2022; Bryant et al., 2023). A GEC model is evaluated by calculating the F-score (van Rijsbergen, 1979) from compar- ing the edits proposed by the GEC model against gold (human-annotatee reference edits. GEC ed- are set of insertion, deletion, or substitution System Combination via Quality Estimation for Grammatical Error Correction
Muhammad Reza Qorib and Hwee Tou Ng
Department of Computer Science, National University of Singapore mrqoribēu. nos..uuusdduu nght@comp.. nus. edu.
Abstract
operations that are applied to the original (source) sentence to make free from errors. An edit is represented by three values: start index, end index, and correction string (Table 1). Since CoNLL-2014 (Ng et al., 2014), Fo.5 has become the standard metric for GEC. Grundkiewicz et al. (2015) and Chollampatt and Ng (2018c) reported that the Fo.; score correlates better with human judgment than other GEC metrics.
Quality estimation models have been developed to assess the corrections made by grammatical error correction (GEC) models when the refer- ence gold-standard corrections are avail- able. An ideal quality estimator can be utilized combine the outputs multiple GEC sys- tems by choosing the best subset of edits from the union of all edits proposed by the GEC base systems. However, we found that existing GEC quality estimation models not good enough differentiating good corrections from bad ones, resulting a low Fo.5 score when used for system combination. In this paper, we pro- pose GRECO', a new state-of-the-an quality estimation model that gives better estimate of the quality of a corrected sentence, as in- dicated by having higher correlation Fo.s score of corrected sentence. It results in a combined GFC system with a higher Fo.s score. We also propose three methods for utiliz- ing GEC quality estimation models for system combination with varying generality: model- agnostic, model-agnostic with voting bias, and model-dependent method. The combined GEC system outperforms state of the art on CoNLL-2014 test set and the BEA-2019 test set, achieving the highest Fo.s scores published date.
Qorib and Ng (2022) reported that GEC models have outperformed humans when measured by the Fo.5 metric, but still make occasional mistakes in simple cases. Thus, we need way evaluate the corrections proposed by a GEC model before accepting their corrections as a replacement for our original sentences. In real-world use cases where the gold reference not available, we can use a GEC quality estimation model to assess the quality of the correction made by GEC model.
GEC quality estimation model accepts a source sentence and its correction and produces a quality score. The quality score characterizes the accuracy and appropriateness of correction with respect to the source sentence. The higher the score, the more accurate and appropriate the correction is. A quality estimation model is typically used as a fil- tering method to accept or reject correction made by GEC model (Chollampatt and Ng. 2018b). It can also be used for choosing the best correction from the top-k outputs of a GEC system (Liu et al., 2021).
1 Introduction
In this paper, we propose to extend that use case further. Instead of choosing the best correction (hy- pothesis) from GEC models, we can use GEC quality estimation model to produce new and more accurate correction, based the edits that appear in the hypotheses. We generate all pos- sible hypotheses from the edit combinations and score them using a quality estimation model. The highest-scoring hypothesis is then deemed as the most appropriate correction of the source sentence.
Source code available at https::// //gi thub. cwt.:bsocc//u/np//////mmm greco.

---

Source
To sum it up I still consider having their own car is way more safe and convinient.
Correction Differences
To sum up I still consider having your own car way more safe and convenient. To sum (it) up (-) I still consider having (their your) own car (is) way more safe and (convinient convenient).
Edits
3,.3,% "), (4, 4, ;)), (8, 9, 'your), (11, 12, "), (16, 17, 'convenient')
Table I: Example GEC edits.
We discuss this in more detail in Section 3. The main contributions of this paper are:
each beam search output to learn the interaction between hypotheses, then summarizes and aggre- gates the information of the hypotheses' interaction using two custom attention mechanisms. VER- Net trains the model using the top-5 outputs of the Riken& Tohoku (Kiyono al., 2019) model on the FCE, NUCLE, and W W&I+LOCNESS (Bryant al., 2019; Granger, 1998) datasets.
We present novel methods for utilizing GEC quality estimation models for system combi- nation.
We reveal and highlight the low performance of existing GEC quality estimation models when used for system combination.
2.1.3 SOME
We present new state-of-the-arr GEC quality estimation model that has better correlation to the Fo.5 score and produces higher Fo.s scores when used for system combination.
SOME (Yoshimura et al., 2020) reference-less GEC metric that scores a GEC correction based on three scoring aspects: grammaticality. fluency, and meaning preservation. SOME consists of three BERT models, one for each scoring aspect. Differ- ent from the aforementioned GEC quality estima- tion models, SOME does not aim to estimate the Fo.5 score. Instead, it estimates the aspect scores directly. The authors created new dataset to train the BERT models by annotating outputs of various GEC systems on the CoNLL-2013 test set with the three scoring aspects. The authors argue that reference-less metrics are better than Fo.5 score be- cause it is difficult to cover all possible corrections in the gold reference.
We report new tatte-of-theaitt scores the CoNLL-2014 and BEA-2019 test sets.
2 Related Work
2.1
GEC Quality Estimation Models
In this section, we briefly discuss existing neural GEC quality estimation models, including a neural reference-less GEC metric.
2.1.1 NeuQE
NeuQE (Chollampatt and Ng, 2018b) is the first neural quality estimation model for GEC. NeuQE uses the
estimator framework (Kim et al., 2017) which trains word prediction task on the predictor network and trains the quality score the estimator network. The estimator is trained ing knowledge from the predictor. NeuQE has two types of model, one for Fo.s score estimation and the other for post-editing effort estimation. NeuQE is trained on the NUCLE (Dahlmeier et al., 2013) and FCE (Yannakoudakis et al., 2011) corpora.
2.2 GEC System Combination Methods
In this section, we briefly discuss state-of-the-art GEC system combination methods.
2.2.1 ESC
ESC (Qorib et al., 2022) a system combination method that takes the union of all edits from the base systems, scores each edit to decide whether the edit should be kept or discarded, and generates the final corrections using the selected edits. ESC uses logistic regression to score each edit based on the edit type and inclusion in the base systems, and filters the overlapping edit based on a threshold and a greedy selection method. ESC is trained on the BEA-2019 development set.
2.1.2 VERNet
VERNet (Liu et al., 2021) estimates the quality of GEC model from the top-k outputs of beam search decoding of the GEC model. VERNet uses BERT-like architecture get the representation of each token. then constructs a fully-connected graph between pairs of (source, hypothesis) for

---

Apply edit
Apply edit
Apply edit
QE model
cp
QE model
QE model
((5,8)
Qxx s + e2)
8
Q6s, (s (2)
$ c2
Q + e2)
1s 22) 2 e3
Source sentence
eg)
Q ① ez)
Qxx @ e!)
s @
Q ① ey)
(s ($@e)) ez)
Q (s 4 e1) + )2)
1s et)
Q(s, (s 1 ey) + eg)
+ 62
Figure Beam search with beam size (b) The blue arrow denotes generation of new hypothesis, and the orange circle denotes the hypotheses with the highest scores. At each step, new hypotheses are generated by applying edit e; to the top-b hypotheses of step i-
2.2.2 MEMT
behaves closely assumption will be good enough be useful for combining GEC systems.
MEMT (Heafield and Lavie, 2010) is system com- bination method that combines models' outputs by generating candidate hypotheses through token alignments and scoring each candidate according to its textual features, which include n-gram lan- guage model score, n-gram similarity to each base model's output, and sentence length. MEMT was originally designed for machine translation system combination, but Susanto et al. (2014) successfully adapted it for use in GEC.
3.1 Problem Formulation
For a source sentence {81, 82, ..., 81) with length / and a hypothesis = {h1, 101,hy h2, hm) with length m, a quality estimation model produces quality score ((s, h)) to assess how good is as a correction to s. When combining GEC systems, we have multiple hypotheses from different base GEC systems. From these hypotheses, we can extract all the edits. Let E denote the union of all edits.
2.2.3 EditScorer
A new hypothesis can be generated by applying an edit e; E E to the source sentence s. If it is a correct edit (eft), the quality score of the resulting hypothesis should be higher than when the edit is not applied or when wrong edit (e, ) is applied. Let h @ denote the operation of applying edit to sentence h. For any hypothesis h (including the case of h = s), an ideal quality estimation model should have the following property:
EditScorer (Sorokin, 2022) is model that scores each edit based on its textual features to generate a better correction. The model can be used to re- rank edits from a single model or combine edits from multiple models. It has a similar principle to ESC but uses the textual features of the edit and its surrounding context instead of the edit type. The textual fcature is acquired from RoBERTa-large' s (Liu et al., 2019) token representation of the can- didate sentence. The model is trained with more than 2.4M sentence pairs from cLang8 (Rothe et al., 2021) and the BEA-2019 training set.
Q(,,h >(s,,))) ((s,hec)) (I)
3.2 Beam Search
From an edit union of size E., we can get 2 pos- sible hypotheses. However, scoring all possible hy- potheses is too costly, so we use beam search with size b to generate the potential candidates in rea- sonable time. We apply each edit in E one by one to the hypotheses in the current beam to generate new candidates, with time complexity O(b x |E]) Initially, the beam contains the source sentence and all edits in are sorted from left to right, i.e.,
3 GEC System Combination via Quality Estimation
To be able to GEC quality estimation model for system combination, we assume an ideal quality estimation model that can discern good hypothe- ses from bad ones and produce appropriate quality scores. Even though a perfect quality estimation model does not exist yet, quality estimation that

---

edits with smaller start and end indices are pro- cessed earlier. each step, we generate new candi- dates by applying the current edit to all candidate sentences in the beam if it does not create a con- flict with previously added edits. We use the edit conflict definition of Qorib et al. (2022). Next, we compute the quality scores for the new candidates and add them to the beam. At the end of each step, we trim the beam by only keeping the top-b can- didates with the highest quality scores. After we finish processing all edits, the candidate with the highest quality score becomes the final correction. We illustrate this process in Figure I.
GAP LABELS +>
WORD LABELS
M1
W2
W3
W4
Mg
Pre-trained Language Model
CLS
+
f
+
hg
fz
Figure 2: Model architecture
should be word phrase inserted right after the current word and before the next word. gi is when
the gap is correct (i.e., there should be no words inserted) and 0 otherwise. The gold word labels and gap labels are computed by extracting the dif- ferences between the hypothesis and the gold refer- ence sentence using ERRANT (Bryant et al., 2017). The word label w,; and gap label gi computed from the projection of the embeddings learned by the pre-trained language model value in [0.1] using two-layered neural network with tanh acti- vation (d). We formally describe them in Equation (2) and (3), where LM denotes the pre-trained lan- guage model, a denotes the sigmoid function, Aw- aw b and b.. denote the weights and biases for the weight label projector, and Ag. ag- bg and by denote the weights and biases for the gap label projector, The size of A,, and Ag is dLM x dLM- while the size of aw- ag. bw. and by is d 1, with dr. being the language model dimension.
4 Quality Estimation Method
A correction produced by GEC model can be wrong in three aspects: keeping wrong words or phrases from the source sentence, changing the words or phrases into the wrong ones, or missing some words or phrases. In other words, a qual- ity estimation model needs to know which words are correct and which are wrong, well as deter- mine whether the gaps between words are correct wrong (in which case word or phrase needs to be inserted).
GEC quality estimation model should also produce the quality scores proportionately. A better correction of the same source sentence should get higher score than a worse one. That is, quality estimation model should be able to rank hypotheses by their quality scores correctly.
V LM(s; h)
In this section, we describe our approach to build a quality estimation model with the aforementioned qualities, which we call GRECO (Gammaticality- scorer for re-ranking corrections).
LM(s0, 81, .... 81, ho, h1, hm)
(vg vi,..., of, v/: v!...., v
w; (aT + bw:) + bw)
(2)
9: a oa (A, bby + by)
(3)
4.1 Architecture
The length of the word label vector is the same as the hypothesis (m)) while the length of the gap label vector g is m + 1. The gap vector length is one more than the hypothesis' length to account for the potentially missing words at the start of the hypothesis. If the pre-trained language model uses sub-word tokenization, tokens that are not the beginning of word are masked. The quality score Q (s, h) is calculated from the normalized product of the word label and gap label probabilities from all words in the hypothesis.
Our model uses BERT-like pre-trained language model its core architecture (Figure 2). We draw inspiration from quality estimation for machine translation models (Kim et al., 2019; Lee, 2020; Wang et al., 2020). The input to the model is the concatenation the source sentence and a hypoth- esis, with the source sentence and the hypothesis prefixed with [CLS] (so) and [SEP] (ho) pscudo- tokens respectively.
For every word in the hypothesis, the model learns to predict its word label w; and gap label gi. The word label denotes whether the current word is correct. w; is when the word is correct and 0 otherwise. The gap label denotes whether there
Q(s, h) 2m+1
t0;
9i
(4)

---

4.2 Loss Function
4.3.1 Voting Bias
The model is trained on two objectives: predict- ing the word label and gap label, and ranking the hypotheses correctly with the quality score, i.c., hypotheses with higher Fu.s scores should have higher quality scores than hypotheses with lower Fo.s scores. This translates into three loss func- tions: word label loss (Cw), gap label loss (Lg), and rank loss (L,). The first two losses are based on binary cross-entropy loss and the rank loss is based on RankNet (Burges et al., 2005; Burges, 2010) with a slight modification to amplify the power term with multiplier j..
Model voting common ensemble method that chooses a prediction label based on how many base systems predict that label. The rationale behind it is straightforward: the more systems propose a label, the more likely for it to be correct. In GEC, voting ensemble has also been used to combine edit labels from multiple GEC sequence-tagging models (Tarnavskyi ct al., 2022).
h = + En
(9)
Q'(s, - ((s V(E))
(10)
V(E,) = 1
count(e)
(11)
E
c
n wwi)
1
n
CEEA
C-
+ Σ 1=1
C0u) +7.6
(5) We incorporate voting bias into beam search by multiplying the quality score with voting score V (Eq. 10). The voting score is calculated by (6) the average number of base systems that propose an edit (count(e)) for all edits in the hypothesis (EE,), normalized by the number of base systems c. The effect of voting bias is governed by a hyper-
1
Lw
(yl"
log +
m
(1 Ur"
log(1 - wi))
m
Lg
f log g
m +
(7) parameter a, 0 < a < 1. If a = O, voting bias is not used.
1=0
(1 v/)) log(1 1 gi))
C.
Σ log
4.3.2 Edit Score
Qorib et al. (2022) reported that the best hypothe- sis is the one that maximizes the strengths of the base systems. bdsie-baeddoo GEC system combination methods approximate the strengths of GEC models through their performance on each edit type and learn the best combination from the edit type fea- ture. If we have edit scores that reflect the base GEC models' strength, we can incorporate them into the hypothesis scoring function.
We formalize the loss functions in Equation (5) to (8), where n is the number of training instances, y and y are the correct labels for the word label and gap label respectively, ve and Qu are the Fo.s score and quality score of hypothesis v respectively, and y is a hyper- .aramter.
4.3 System Combination Biases
One way to incorporate the edit scores is by mul- tiplying the hypothesis score with the edit scores. However, if we only multiply the scores of the edits that applied to the hypothesis, we will reward hypotheses with fewer edits, even if we normalize the edit score?. Instead, we want reward hypothe- ses that contain good edits and penalize hypotheses that miss good edits. Thus, we design the edit score to be the product of all edits in the edit union EE
The Q score from quality estimation models is model-agnostic it fully depends on the source sentence and the hypothesis sentence, independent of the system that proposes the hypothesis. With a perfect quality estimation model, it should be enough to get the best hypothesis. If we use an imperfect quality estimation model, some valuable information in the system combination task can be useful to get better hypothesis, such as how many systems propose edit and which systems pro- pose it. We incorporate the former through a voting bias and the latter through edit scores from an edit- based system combination method. In this section, we discuss how we replace the quality score Q in the beam search with a biased hypothesis score Q'.
V(BA) E.S Ey@ (12)
=
"For example, if edit e1 has an edit score of 0.95 and edit
e2 has an edit score of 0.9, the score of applying both edits is
lower than just applying es if we only multiply the scores of
edits that appear in the hypothesis, even though ea is also a good edit.

---

Single system evaluation
Multi-system evaluation
Model
p
P
R
Fo.5
P
R
Fo.5
NeuQE
-0.00
52.53
12.83
32.45
0.212
38.30
10.84
25.43
VERNet
0.199
72.13
35,93
60.04
0.354
65.06
22.41
47.12
SOME
0.002
53.02
51.06
52.62
0.392
47.30
34.62
44.07
GPT-2
0.088
54.09
50.98
53.43
0.116
46.67
33.32
43.21
GRECO
0.445
71.23
47.72
64.84
0.415
67.39
30.71
54.40
Table 2:
Quality estimation and re-ranking results on the CoNLL-2014 test set. denotes Spearman's rank correlation coefficient.
Pedtil (e, lEas) = (es(e) (ES(e)
i E En
with tokens not present in the source sentence are given higher weight z than other word and gap labels in the calculation of L and Lg- We choose the hyper-parameters based the model's perfor- mance on the BEA-2019 development set (Bryant et al., 2019) and the CoNLL-2013 test set (Ng al., 2013). We list the syper-parameters and explain our hyper-parameter search in Appendix B.
(13)
otherwise
(B,, R)
II oCK
Podit (e,, BA)
(14)
We formulate the hypothesis score with voting bias and edit score ES in Equation (12) - (14), where PEs denotes the probability of each edit. The effect of edit score is governed by the syper-paramettr 3, 0 < 3 < 1. If B O, the edit score is not used.
5.2 Evaluation
We evaluate our model on quality estimation, re- ranking, and system combination tasks. We com- pare our models to other GEC quality estimation models (NeuQE, VERNet, SOME) and language model baseline, GPT-2 Large (Radford al., 2019), which has been reported perform relatively well on unsupervised GEC task (Alikaniotis and Raheja, 2019). We use the RC variant of NeuQE and the ELECTRA variant of VERNet which produce the highest scores on the CoNLL-2014 test set (Ng et al., 2014). For the system combination task, we compare our model to state-of-the-an GEC system combination methods in Section 2.2.
5
Experiments
5.1 Model Training
We use DeBERTA-V3-Larg (He et al., 2023) as the pre-trained language model. We train the quality estimation model using the unique cor- rections of nine GEC systems, which are BART- GEC (Katsumata and Komachi, 2020), GECToR RoBERTa (Omelianchuk al., 2020), GECToR XLNet, GECTOR BERT, Kakao& Brain ensemble (Choe et al., 2019), Kakao& Brain Transformer- base, Fiken&Tohoke ensemble (Kiyono et al., 2019), T5-Large (Rothe et al., 2021), and UEDIN- MS ensemble (Grundkiewicz et al., 2019), on the W&I+LOCNESS, training set.
For the quality estimation and re-ranking tasks, we perform experiments in two scenarios: single system and multi-system evaluation. For single system evaluation, we follow Liu et al. (2021) on evaluating the models on the top-5 outputs of Riken& Tohoku (Kiyono et al., 2019) on the CoNLL-2014 test set. For multi-system evaluation, evaluate the models the outputs of 12 partic- ipating teams of the CoNLL-2014 shared task. For the quality estimation task, we follow Chol- lampatt and Ng (201 8b) in comparing the correla- tion coefficient of the quality score to the sentence- level Fo.s score. We use Spearman's rank corre- lation coefficient (Spearman, 1904), which is the primary metric of the WMT-2022 shared task quality estimation (Zerva et al., 2022).
The training data are grouped into small groups of size n, with corrections of the same source sen- tence grouped together as much as possible, and each group must contain least n corrections from 2
the same source sentence. This way, the rank loss (Eq. 8) computes more comparisons of hypotheses from the same source. Corrections with no edits are filtered out so that the model can focus more on predicting the labels on edit words. Perfect cor- rections are also filtered out to maintain the label distribution balance. W&I+LCCNESS has 34,308 sentences and the resulting training data, obtained after filtering the unique corrections of the nine GEC systems above, has 65,824 hypotheses. Dur- ing training, word labels and gap labels associated
For the re-ranking task, we pick one hypothe- sis with the highest quality score for each source

---

BEA-2019
BEA-2019 Test
CoNLL-2014 Test
Model
Dev (Fo.s)
P
R
Fo.5
P
R
Fo.5
1. T5-Large
56.21
74.30
66.75
72.66
69.66
51.50
65.07
2. GECToR XLNet
55.62
79.20
53.90
72.40
77.49
40.15
65.34
3. GECToR RoBERTa
54.18
77.20
55.10
71.50
73.91
41.66
64.00
4. Riken&
Riken&Tohoku Tohoku
53.95
74.7
56.7
70.2
73.26
44.17
64.74
5. UEDIN-MS
53,00
72.28
60.12
69.47
75.15
41.21
64.52
6. Kakao&Brain
53.27
75.19
51.91
69.00
NeuQE
29.30
68.48
20.19
46.32
66.48
15.87
40.59
VERNet
54.80
73.19
58.42
69.67
74.08
39.12
62.85
SOME
52.23
66.40
67.83
66.68
68.39
54.23
65.00
GPT-2
52.00
67.20
68.08
67.38
68.30
52.65
64.47
ESC
63.09
86.65
60.91
79.90
81.48
43.78
69.51
МЕMT
60.72
82.20
63.00
77.48
76.44
48.06
68.37
EditScorer
61.66
88.05
58.71
80.05
74.32
51.44
68.25
GRECO
60.74
80.03
66.22
76.83
76.39
50.35
69.23
GRECO, voting
62.22
82.86
65.10
78.58
79.36
48.69
70.48
GRECO, ESC
63.40
86.45
63.13
80.50
79.36
48.69
70.48
Table 3: System combination results. The first group of rows shows the base GEC systems that are combined. while the second and the third group of rows show the combination results of existing quality estimation models and system combination methods respectively. We do not show Kakao& Brain's 's score on CoNLL-2014 as is not used in the CoNLL-2014 combination. ESC and MEMT results are taken from (Qorib al., 2022). GRECO refers to our model with voting bias, and GRECO VeSing+ESC refers to our model with voting bias and edit scores from ESC.
sentence, and then compute the corpus-level Fo.5 score for the chosen hypotheses of all source sen- tences. We also add the source sentence as one of the hypotheses so that a model has the option to not make any corrections if the hypotheses are bad. We use the 2 Scorer (Dahlmeier and Ng, 2012) to compute the Fo.5 score.
nificantly outperforms all other quality estima- tion models on the correlation score and Fo.s score in both experimental settings (p < 0.001). Our re-ranking result has higher precision, recall, and Fo.5 score compared to the top-1 output of Riken& Tohoku, which has precision, recall, and Fo.5 score of 68.59, 44.87, and 62.03 respectively?.
For the system combination task, combine the base systems given in Table using ESC, MEMT, and EditScorer. We also combine the same base systems using beam search via quality esti- mation for the different quality estimation meth- ods NeuQE, VERNet, SOME, GPT-2, GRECO, GRECO voting. and GRECO, ESC. GRECO Fo.5 scores on the BEA-2019 test set and CoNLL- 2014 test set.
6.2 System Combination
We report the results of the system combination experiments in Table 3. Existing GEC quality timation models fail produce better corrections. More surprisingly, all of them produce combination scores that are lower than the fourth best base sys- tem on the BEA-2019 experiment and the second best base system on the CoNLL-2014 experiment. Our model without any additional biases (GRECO) successfully produces better corrections with 4.17 points and 3.89 points higher than the best basc system the BEA-2019 (T5-Large) and CoNLL- 2014 (GECToR XLNet) test sets respectively.
We report the
We use William's test (Williams, 1959) to mea- sure the statistical significance of the correlation coefficients. We use bootstrap resampling on 100 samples for the statistical significance of the Fo.s scores in the re-ranking and system combination tasks.
By adding the voting bias, our model outper- forms MEMT on both datasets and ESC and Ed-
6 Results
"The top-1 performance for Riken& periment is different from Table 3 because the data for this experiment was from reproduction by Liu et al. (2021), which does not include right-to-les not being publicly available according to them.
this ex-
6.1
Quality Estimation and Re-Ranking
We report the results of quality estimation and re- ranking evaluation Table 2. Our model sig-
hott-to-led left re-ranking due to that component

---

BEA-2019 Test
CoNLL-2014 Test
Model
Model
P
R
Fo.5
P
R
Fo.5
1. T5-XL
76.85
67.26
74.72
1. T5-XL
74.40
52.02
68.50
Laage
80.70
53.39
73.21
7. C4-200M
75.47
49.06
68.13
+ Model [2). [4],
[5]. [6]
from Table
3
+ +Model [2] - [5] from Table 3
ESC
86.64
61.54
80. 10
GRECO
76.68
51.54
69.86
GRECO
80.46
66.59
77.24
GRECO veting
79.60
49.86
71.12
GRECO voting
83.24
65.12
78.85
GRECO voting+ESC
86.66
63.72
80.84
Combination of the same base systems in Table but with Ta-Large replaced with TS-XL and the C4-200M model added to the base systems.
Table 5:
Table 4: Combination of the same base systems in Table
3, but with model [1] replaced by 15-XL and model [3] replaced by GECToR-Large RoBERTa.
T5-Large with T5-XL. From this experiment, our model achieves the highest CoNLL-2014 test score reported to date, 71.12 (Table 5). We also evaluate on the CoNLL-2014 test set with 10 annotations, using the same approach Bryant and Ng (2015)* and report the highest score date, 85.21.
itScorer the CoNLL-2014 test set. Our model when augmented with voting bias and edit scores from ESC outperforms ESC and EditScorer the BEA-2019 test set by 0.6 and 0.45 points respec- tively. Note that EditScorer is trained with 70 times more data than GRECO. Using edit scores from ESC on the CoNLL-2014 test set does not change the result since the optimal edit weight (3) is zero. In other experiments, we found that combining with ESC can also improve the Fo.; score the CoNLL-2014 test set (Appendix Table 11). Our final model has significantly higher scores than all other methods (p- < 0.005).
7.2 Fluency
We analyze the output fluency of our methods by measuring the perplexity of the generated correc- tions using GPT-2, since prior work (Kann et al., 2018) has found that perplexity correlates with hu- man fluency score. found that our method gen- erates more fluent corrections than all other meth- ods, and adding more biases to our model makes the model less fluent (Figure 3). Based on the gen- erated sentences, edit-based methods like ESC and EditScorer are too optimized toward picking the correct edits which can make the sentence unnatu- ral. Our model with three modes of generality of- fers a flexible trade-off between Fo.5 score and flu- ency. Our GRECO ESC achieves a higher Fo.5 score and better fluency at the same time compared to the previous best system combination methods ESC and EditScorer.
We also evaluate our model on the combination of stronger GEC models. We replace La-Large by T5-XL and GECToR RoBERTa by GECToR-Large RoBERTa (Tarnavskyi al., 2022) from the base systems and achieve the highest BEA-2019 test score, 80.84, reported to date. (Table 4).
7 Discussions
This section discusses important characteristics of our model. From this section onward, we some- times refer to our model GRECO voting as G,, and GRECO voSin G
37.0
38,0
35.8
85.2
7.1 Model-Agnostic
35.0
34.8
34.0
GRECO model-agnostic, which means we can add or change the base systems during inference. We run an experiment of adding the C4-200M GEC system (Stahlberg and Kumar, 2021) to the base systems of the CoNLL-2014 system combi- nation while using the model weights and hyper- parameters from Table This experiment setting is not possible with ESC and MEMT which re- quire the same set of base systems during training and testing. In this experiment, we also replace
33.0
28.8
28.0
Torget
GRECD
GRECOV GRECOWE
ESC
НENT
Edi3corer
Figure 3: Median perplexity of the generated corrections on the BEA-2019 development set (lower is better).
Evaluate the output on 10 sets of 9--annotation references and average the Fo.s scores from all 10 sets.

---

BEA
BEA
CoNLL
BEA
BEA
CoNLL
Model
b
-Dev
-Test
-2014
-Dev
-Test
-2014
GRECO vor
16
63.40
80.50
70.48
62.96
80.18
67.48
GRECO, vothngESSC
62.96
80.18
67.48
63.13
80.17
69.38
GRECO MASCOvorti voting
62.22
78.58
70.48
63.31
80.31
70.25
GRECO, voting
60.66
77.04
67.48
63.34
80.47
70.36
GRECO
16
60.74
76,83
69.23
12
63.34
80.49
70.34
GRECO
60.18
76.77
66.84
16
63.40
80.50
70.48
GRECO without rank loss
16
59.78
75.62
68.51
20
63.38
80.53
70.44
GRECO without rank loss
57.50
73.05
65.64
24
63.33
80.53
70.36
32
63.37
80.55
70.56
Table 6: Ablation study for each component of the model (left) and different beam sizes (right).
7.3 Ablation
We run an ablation study evaluate the contri- bution of each component of our method and the effect of beam size on model performance (Table 6). We found that both the training (rank loss) and inference (voting bias, edit scoring, beam search) techniques contribute to the final model perfor- mance. We also found that the performance does not change much with beam size >= 8. However, the difference between greedy (beam size and beam search (beam size 16) decoding quite set. Beam search has more effect on the CoNLL-2014 test set because its number of edits per sentence is more than the BEA-2019 development set.
voting
Figure Average Fo.s the CoNLL-2014 test set for varying number of base systems.
substantial, especially on the CoNLL-201
ferentiate good corrections from bad ones, which is shown by their ineffectiveness on the re-ranking and system combination tasks. Hence, there is need for new quality estimation model for GEC.
7.4 Number of Base Systems
We present a new state-of-the-ard quality estima- tion model, GRECO. Our model outperforms ex- isting models on quality estimation and re-ranking evaluation. Our re-ranking of the top-5 hypotheses of Tohoku beats the performance of the top- hypothesis. Our source code and model weights are publicly available, making our model directly usable as post-processing tool for re-ranking GEC systems' outputs.
We investigate the performance of our model when the number of base systems is reduced. We use the base systems in Table 3 and randomly sample combinations of base systems for each experiment (except when combining systems where there is only one combination). We evaluate the combina- tion on the CoNLL-2014 test set and calculate the average Fo.5 score for each number of base systems (Figure 4). We found that ESC's performance dete- riorates rapidly when the number of base systems is reduced, while GRECOwniig and EditScorer can maintain their performance.
Our model, combined with a voting bias and an edit-based system combination method, success- fully improves the Fo.5 scores on the GEC system combination task and produces the highest Fo.s scores on the CoNLL-2014 test set and BEA-2019 test set to date, which are 71.12 and 80.84 respec- tively.
8 Conclusion
In this paper, present novel methods uti- lize GEC quality estimation models for system combination with varying generality: model- agnostic, model-agnostic with voting bias, and model-dependent method. We report that existing GEC quality estimation models are not able to dif-
Limitations
Our model is trained on English GEC data with a single reference, and we only report experimental results on English GEC. Future work can apply our

---

model to other languages. We believe our work does not pose any risks to society.
Shamil Chollampatt and Hwee Tou Ng. 2018b. Neural quality estimation grammatical error correction. In Proceedings of EMNLP, pages 2528-2539.
Acknowledgements
Shamil Chollampatt and Hwee Tou Ng. 2018c. A re- assessment of reference-basee grammatical error cor- rection metrics. Proceedings of COLING, pages 2730 2741.
We thank the anonymous reviewers for their help- ful comments. This research is supported by research grant from TikTok (WBS No. A- 8000972-00-00), The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (httss//www.nsccc.g)) Cloud resources involved in this research work were also partially supported by NUS IT's Cloud Credits for Research Programme and Amazon Web Services.
Shamil Chollampatt, Kaveh Taghipour, and Hwee Tou Ng. 2016. Neural network translation models for grammatical error correction. Proceedings of IJ- CAI, pages 2768-2774.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of NAACI, pages 568-572.
Daniel Dahlmeier, Hwee Tou Ng. and Siew Mei Wu. 2013. Building large annotated corpus learner English: The NUS corpus of learner english. In Proceedings of BEA, pages 22-31.
References
Dimitris Alikaniotis Vipul Raheja. 2019. The unrea- sonable effectiveness of transformer language models in grammatical error correction. In Proceedings of BEA, pages 127-133.
Sylviane Granger. 1998. Developing automated writ- ing placement system for EST. learners. Learner English on Computer, pages 3-18.
Roman Grundkiewicz, Marcin Junezys-Dowmunt Dowmunt, and Edward Gillian. 2015. Human evaluation of gram- matical error correction systems. In Proceedings of EMNLP, pages 461-470.
Christopher Bryant, Mariano Felice, Øistein E. Ander- sen, and Ted Briscoe. 2019. The BEA-2019 shared task grammatical error correction. In Proceedings of BEA, pages 52-75.
Roman Grundkiewicz, Marein
and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre-training synthetic data. In Proceedings of BEA, pages 252-2633
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceed- ings of ACL, pages 793-805.
Christopher Bryant and Hwee Tou Ng. 2015. How far are we from fully automatic high quality grammatical error correction? In Proceedings of ACI., pages 697- 707.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre- training with gradient-disen disentangled embedding shar- ing. In Proceedings of ICLR.
Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical Error Correction: A Survey of the State of the Art. Computational Linguistics.
Kenneth Heafield and Alon Lavie. 2010. CMU multi- engine machine translation for WMT 2010. In Pro- ceedings of WMT, pages 301-306.
Katharina Kann, Sascha Rothe, and Katja Filippova. 2018. Sentence-level fluency evaluation: References help, but can be spared! In Proceedings of CoNLL, pages 313-323 323..
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of ICML, page 89-96.
Satoru Katsumata and Mamoru Komachi. 2020.
Chris Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Technical Report MSR- TR-2010-82, Microsoft Research.
Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. Pro- ceedings of AACI, pages 827-832.
Yo Joong Choe, Jiyeon Ham, Kyubyong Park, and Yeoil Yoon. 2019. A neural grammatical error correction system built better pre-training and sequential transfer learning. In Proceedings of BEA, pages 213- 227.
Hyun Kim, Hun Young Jung, Hongseok Kwon, Jong- Hyeok Lee, and Seung- Hoon Na. 2017. Predictor- estimator: Neural quality estimation based target word prediction for machine translation. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 17(1).
Shamil Chollampatt and Hwee Tou Ng. 2018a. A multi- layer convolutional encoder-decoder neural network for grammatical error correction. In Proceedings of AAAI, pages 5755-5762.
Hyun Kim, Joon-Ho Lim, Hyun-Ki Kim, and Seung- Hoon Na. 2019. QE BERT: Bilingual BERT using multi-task learning neural quality estimation. Proceedings of WMT, pages 85-89.

---

Shun Kiyono. Jun Suzuki, Masato Mita, Tomoya Mizu- moto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In Proceedings of EMNLP, pages 1236- 1242.
C. Spearman. 1904. The proof and measurement of sociation between two things. The American Journal of Psychology, 15(1):72-101.
Felix Stahlberg and Shankar Kumar. 2021. Synthetic data generation for grammatical error correction with tagged corruption models. In Proceedings of BEA, pages 37-47.
Dongjun Lee. 2020. Two-phase cross-lingual language model fine-tuning for machine translation quality es- timation. In Proceedingy of WMT, pages 1024-1028.
Raymond Hendy Susanto, Peter Phandi, and Hwee Tou Ng. 2014. System combination for grammatical error correction. In Proceedings of EMNLP, pages 951- 962.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/ 16692.
Maksym Tarnavskyi, Artem Chermodub, and Kostiantyn Omelianchuk. 2022. Ensembling and knowledge distilling of large sequence taggers for grammatical error correction. In Proceedings oCL, pages 3842- 3852.
Zhenghao Liu, Xiaoyuan Yi, Maosong Sun, Liner Yang, and Tat-Seng Chua. 2021. Neural quality estima- tion with multiple hypotheses for grammatical error correction. In Proceedings of NAACL, pages 5441- 5452.
Cornelis J. van Rijsbergen. 1979. Information Retrieval, 2nd edition. Butterworths.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In Proceedings of ICLR.
Minghan Wang, Hao Yang. Hengchao Shang, Daimeng
Wei, Jiaxin Guo, Lizhi Lei, Ying Qin, Shimin Tao, Shiliang Sun, Yimeng Chen, and Liangyou Li. 2020. HW-TSC's participation at WMT 2020 quality esti- mation shared task. In Proceedings of WMT, pages 1056-1004 1061.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe. Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of CoNLL, pages 1-14.
E.J. Williams. 1959. Regression analysis, volume 14. Wiley New York.
Hwee Tou Ng. Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL- 2013 shared task on grammatical error correction. In Proceedings of CoNLL, pages 1-12.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 201 I. new dataset and method for automatically grading esol texts. In Proceedings of the ACL 2011, pages 180-189.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020.
Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Ka- jiwara, and Mamoru Komachi. 2020. SOME: Reference-less sub-metrics optimized for manual evaluations of grammatical error correction. In Pro- ceedings of COLING, pages 6516 6-222.
GECToR grammatical error correction: Tag, not rewrite. In Proceedings of BEA, pages 163-170.
Muhammad Qorib, Seung-Hooe -Hon Na, and Hwee Tou Ng. 2022. Frustratingly easy system combination for grammatical error correction. In Proceedings of NAACL, pages 1964-1974.
Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjor, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Oräsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In Proceedings of WMT, pages 69 99.
Muhammad Reza Qorib and Hwee Tou Ng. 2022. Grammatical error correction: Are we there yet? In Proceedings of COLING, pages 2794-288 2800.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei. and Ilya Sutskever. 2019. Language models are unsupervised multitask learn- ers. httpss://ddmucfpksyww.do clw.://swwwwwwwcooooooooooooooae ront. net/ bet Detterr-langugge-mmmaass//ggggg models. wwr.uusspprriieeetulttiiaa....ae
A Oracle Performance
In this section, we analyze the performance of an or- acle system combination method on the BEA-2219 development set and CoNLL-2014 4 test set. Here, assume the oracle always picks the correct edits from the union of all edits from the base systems, making the precision 100%. From Table 7, we see that the oracle combination of T5-Large and GEC- ToR XLNet on the CoNLL-2014 test set already produces a very high score, 13.97 points higher
Sascha Rothe, Jonathan Mallinson, Fric Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of ACL, pages 702-707.
Alexey Sorokin. 2022. Improved grammatical emror rection by ranking elementary edits. Proceedings of EMLLP, pages 11416-11429

---

than our state-of-the-art Fo.s score, while the com- bination of systems is 16.56 points higher than our state-of-the-art Fo.5 score. This analysis shows that there is much room for further investigation on the GEC system combination task. Research 7
HP
Bound
Search Interval
Ir
Ir
<1
(1,2,3) 10 32, 64, 128, 256 4,
bs
bs
n
1 1 S bs
S 1 0 0 d <1 2 =2 0
0, 0.2, 0.3, 0.5, 1.0
on GEC system combination is important to better understand how and why GEC models have com- plementary strengths and why it is not trivial to combine them.
0.2, 0.25, 0.3
2
2.0
μ>0 0
1,4,5,7
0 S 05 0 3 <<1
0.1, 0.9 0.1, 0.2, .... 0.9
Base systems
BEA
CoNLL-
6
b 621
1.2, 8, 12, 16, 20, 24, 32
Dev
2014
T5-Large GECTOR-
82.94
84.45
XLNet CEECTOR-ROBERT
Table Hyper-parameters introduced by our method.
84.35
85.05
+Riken&Tohoku +UEDIN-MS
85.81
86.53
and a gap classifier with 1M parameters. In total, our model has 436,113,410 effective parameters. We detail the hyper-parameters of our final model in Table 9.
86.68
87.04
Table The oraele Fo.s scores with increasing number of base systems.
HP
Value
x 10-5
B Hyper-Parameters
scheduler
linear
Our method introduces seven hyper-parameters (HP), one for training data generation (n), four for training the model (?, μl, dropout rate d for the classifier layers, and weight for weighting the cross-entropy loss of word labels and gap labels associated with tokens not present in the source sen- tence), and two for inference (a and (). We also search the batch size (bs), learning rate (Ir), and beam size (b). See Table 8. Note that bs needs to be divisible by n. We search the myyer-purametten through manual tuning and select them based on the Fo.5 scores on the validation sets. For the in- ference syper-parameters, we evaluate the model times to find the optimal a on the validation sets, then another evaluations (with the a applied) to find the optimal B.
bs Optimizer
128 (32 x 4 gradient acc)
Adam (31 0.9, B2 = 0.999)
(Loshchilov and Hutter. 2019)
Max epochs
15
7
0.2
0.25
z
2.0
0.4
B
0.7 for BEA-2019 in Table 3
0.0 for CoNLL-2014 in Table
0.6 for BEA-2019 in Table 4
b
16
Table Syper-parameteer of our final model. Our model converged after epoch 4.
B.1 Training Data
As mentioned in Section 5.1, arrange the train- ing data in small groups of size n, in which hy- potheses from the same source are put together. In
C Running Time
We run all of our experiments on a single NVIDIA A100 40GB GPU. We report the training and test- ing time in Table 10.
our experiment, n = so the number rank loss computations within one group is (2) -() 6.
B.2 Final Hyper-Parameters
D Experimental Results
We use DeBERTa- V3-Large as the backbone of our model, which has about 434M parameters. Our model has a word classifier with IM parameters
We repeat our experiments five times with different random seeds for training the models. We report the Fo.5 scores in Table and report the mean and standard deviation in Table 12.
Chttps://githun haatts:://ggituu..commmm//r// http:///////ccccoo///eeeeeee

---

Step
Running Time
(HH:MM:SS)
Training
09:53:52
Inference
BEA-2019 Dev
00:06:08
BEA-2019 Test
00:05:22
CoNLL-2013 CoNLL-2014
04:01:46
00:01:51
Table 10: Running time of each process on single
NVIDIA A100 40GB GPU.
BEA-2019 Test
No
GRECO
GRECO, GRECO,+E
76.83
78.58
80.50
2
76.74
78.54
80.42
3
76.00
77.94
80.17
4
76.31
78.43
80.63
5
76.78
78.93
80.49
CoNLL-2014 Test
No
GRECO
GRECO, GRECO,+E
69.23
70.48
70.48
Model
BEA-Test
CoNLL-14
2
68.81
70.31
70.24
ESC
79.86=1.007
69.47:10.14
3
68.48
69.60
70.08
MEMT
76.66
68.14:1..0
4
68.38
69.77
69.77
GRECO EsCn
80.14
70.20:10.29
5
68.57
69.83
70.45
Table 12:
Average and standard deviation of Fo.3 + a) on the BEA-2019 test set from 5 experiments. The ESC and MEMT results are taken from (Qorib et al., 2022). Sorokin (2022) only reported a single data point, so we do not include EditScorer in this table.
Table 11: The details of our experimental results. The cell values are the Fo.n scores. Run the best score the development set it was chosen in experi- ments.
CoNL1-2014
E Resources
We list the data sources in Table 13. use the standard train/validationn/ess splits for all datasets. We list the GEC quality estimation model sources in Table 14, the GEC model sources in Table 15, and the scorer source code sources in Table 16.

---

Data
URLs
1. W&I+LOCNESS
https: //wbs. cl cam ac hwt...//ww.eeearrc///////22222//
2. BEA-2019 Dev
https://www. cam. wttp:.cc.ca..cc.../eeeeeeeetaaae
3. BEA-2019 Test
CoNLL-2013 Test
CoNLL-2014 Test
Participants of CoNLL-2014
Top-5 outputs Riken& Tohoku
Table 13: Data sources.
Model
URLs
1. NeuQE
2. VERNet
com/ wetp:tthbbb.omm//uumn/////eeee
3. SOME
4. GPT-2
Table 14: Quality estimation models.
Model
URLs
T5-XL
2. T5-Large
3. GECToR-Large
4. GECToR XLNet
5. GECToR RoBERTa
6. Tohoku
7. UEDIN-MS
8. Kakao&Brain
9. BART-GEC
Table 15: GBC systems' sources.
Scorer
URLs
1. ERRANT
thub. htwptt/t....//rrrispppr////rrrrr
2. M2Scorer
https://www wwwp.ccmm..oo nus. edu. edt...s//nllpcco1111ttt html
Table 16: Scorer source code sources.


Reviews:
[{"review_id": "BR1CmaKkJm", "reviewer": "Reviewer_9qEn", "summary": "", "questions": "A) in formula (1) in the Subsection 3.1 you say that the score of a hypothesis extended with incorrect edit should be lower than of the hypothesis itself. I think this is incorrect, consider the sentence \"The game I plays today was interesting before I began to lose\". The score of the edit plays->play is lower than the one of the plays->play, but probably higher than the score of the original hypothesis without edits.\nB) you introduce several nonstandard components to the model, it would be useful to understand their proper role via ablation. First, I mean beam search for edits application. For example, (Sorokin, 2022) applies edits in a greedy fashion beginning from the most probable. I would be glad to know what part of your improvement over him is due to better training setup and what part belongs to improved inference.\nC) the same ablation is required for ranking loss\nD) probably, the model of (Sorokin, 2022) performs poorly on spelling data because it was trained on GECTOR edits which rarely include spelling by design of GECTOR. Can you provide statistics analogous to Table 10 for some other type of error?\nE) probably, the model of (Sorokin, 2022) is rather not a method of model combination since it works with one model as well. May be, you should mention it.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and for appreciating our work.\n\nQuestions for the Authors:\n\nA. It seems there is a typo in the example given in your review? Both edits in the example are the same. In our problem formulation in Section 3.1, we follow the philosophy of GEC that not producing any correction is better than producing a wrong correction; a hypothesis with a wrong edit, e.g., (Q(s, s⊕(plays -> play)), will have a lower F0.5 score than keeping the source sentence as is (Q(s, s)), since the wrong edit reduces the precision without increasing the recall. We believe that the quality estimation score should have the same property as the official GEC score.\n\nB. and C. We have run an ablation study for the beam search. We found that the performance does not change much with beam size >= 8. However, the difference between greedy (beam 1) and beam 16 is quite substantial, especially on the CoNLL-2014 test set. Beam search has more effect on the CoNLL-2014 test set since the number of edits per sentence is more than the BEA-2019 dev set. The rank loss also has a considerable effect on the model performance.\n\n| Model                                       | Beam | BEA-Dev | BEA-Test | CoNLL-2014 test set |\n| ----------------------------------- | ------- | ----------- | ----------- | --------------- |\n| With rank loss + voting + ESC | 16       | 63.40 | 80.50 | 70.48 |\n| With rank loss + voting + ESC | 1         | 62.96 | 80.18 | 67.48 |\n| With rank loss + voting            | 16       | 62.22 | 78.58 | 70.48 |\n| With rank loss + voting            | 1         | 60.66 | 77.04 | 67.48 |\n| With rank loss                          | 16       | 60.74 | 76.83 | 69.23 |\n| With rank loss                          | 1         | 60.18 | 76.77 | 66.84 |\n| Without rank loss                     | 16       | 59.78 | 75.62 | 68.51 |\n| Without rank loss                     | 1         | 57.50 | 73.05 | 65.64 |\n\nWe believe both our training and inference setups contribute to the performance gain. We will add this analysis to our paper.\n\nD. We agree that it may be the cause. We also found that the recall of our model on the R:VERB:SVA (edits that fix the verb form to conform to subject-verb agreement) error type performs much better than ESC and EditScorer, and is comparable to MEMT. See the table below. Both our model and MEMT have a similar beam search method that takes into account the interaction of edits. Even though MEMT has a slightly higher recall than our HyRank+voting+ESC method, its precision (70.11) is much lower than our model (77.07), making the F0.5 score of our model on this error type (77.96) much higher than MEMT (72.27).\n\nHyRank | Hyrank+voting | Hyrank+voting+ESC | ESC  | MEMT | EditScorer\n----------- | ------------------- | --------------------------- | ------- | -------- | ---------------\n77.7       | 79.73               | 81.76                        | 78.38 | 82.43  | 66.22\n\nE. Yes, we will mention this point.\n\nTypos Grammar Style and Presentation Improvements:\n\nA. Thank you for your suggestion on Equation 8. We will follow your suggestion.\n\nB. We use the CoNLL-2014 test set to evaluate the re-ranking performance in Table 2, as mentioned in lines 452 – 457. We are sorry if this was not clear. We will also add this explanation to Table 2’s caption directly to improve clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* new SOTA on English GEC\n* a new method of model combination by edits rescoring\n* several other innovations, such as multicomponent loss for model training and beam-like inference are proposed\n* thorough analysis of different models in terms of their quality and fluency", "reasons_to_reject": "No reasons", "questions_to_authors": "A) in formula (1) in the Subsection 3.1 you say that the score of a hypothesis extended with incorrect edit should be lower than of the hypothesis itself. I think this is incorrect, consider the sentence \"The game I plays today was interesting before I began to lose\". The score of the edit plays->play is lower than the one of the plays->play, but probably higher than the score of the original hypothesis without edits.\nB) you introduce several nonstandard components to the model, it would be useful to understand their proper role via ablation. First, I mean beam search for edits application. For example, (Sorokin, 2022) applies edits in a greedy fashion beginning from the most probable. I would be glad to know what part of your improvement over him is due to better training setup and what part belongs to improved inference.\nC) the same ablation is required for ranking loss\nD) probably, the model of (Sorokin, 2022) performs poorly on spelling data because it was trained on GECTOR edits which rarely include spelling by design of GECTOR. Can you provide statistics analogous to Table 10 for some other type of error?\nE) probably, the model of (Sorokin, 2022) is rather not a method of model combination since it works with one model as well. May be, you should mention it.", "paper_topic_and_main_contributions": "The paper proposes a new method of system combination for grammatical error correction. Given the edits of several basic models, the proposed model is trained to answer, whether the proposed edit is correct. It allows to improve significantly the quality of the basic model, also surpassing the previous methods of model combinations and even the current SOTA on English GEC. Besides this main idea, the work applies several clever solutions to model training and inference. The paper also contains deep analysis of different methods' properties.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A) if I understood correctly the subscript $Q_w$ in equation 8, here *w* refers to an incorrect hypothesis. But two equations higher *w* stands for \"word\". I advice to change the subscript to $Q_u$ or some other letter not used above in a different sense.\nB) it is not easy to understand what dataset is used in Table 2, please mention it explicitly."}}, {"review_id": "XeXjw2QgLr", "reviewer": "Reviewer_nJd5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and for appreciating our work.\n\n1.a. In this paragraph, what we meant is that since a GEC model that has a high corpus-level F0.5 score may occasionally still make mistakes, we need a quality estimation model to evaluate the corrections proposed by a GEC model before trusting its corrections in a real-world setting where we don’t have the gold reference.\n\n1.b. and 1.c. Thank you for the suggestion. We will improve Figure 2 and the clarify gap labels at the beginning of Section 4.\n\n1.d. We rerank the output of Riken&Tohoku (Kiyono et al., 2019) on the CoNLL-2014 test set as explained in Section 5.2 line 454. We will make this clearer by explaining it in Table 2’s caption.\n\n2. The F0.5 scores from the M2-Scorer and ERRANT have been reported to highly correlate with human judgment [1,2], so we believe that reporting the F0.5 score is sufficient to demonstrate the models’ capability in combining GEC corrections.\n\n3. We report our training and inference time in Appendix Table 14. The time needed for combining GEC systems using our model with beam=16 is fast, about 0.07s per sentence, even though it is not as fast as statistical methods like MEMT and ESC.\n\nInference time (per sentence)\n* HyRank: 0.07 s\n* EditScorer: 0.06 s\n* ESC: 0.004 s\n* MEMT: 0.005 s\n\nReferences:\n\n[1] Peiyuan Gong, Xuebo Liu, Heyan Huang, and Min Zhang. 2022. Revisiting Grammatical Error Correction Evaluation and Beyond. EMNLP.\n\n[2] Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Edward Gillian. 2015. Human Evaluation of Grammatical Error Correction Systems. EMNLP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper highlights the low performance of existing GEC quality estimation models when used for system combination and introduces a new method for system combination.\n2. The paper achieves a new state-of-the-art F0.5 score on the BEA-2019 test set and CoNLL-2014 test set.\n3. The proposed method for GEC system combination is model-agnostic, making it easy to apply to GEC systems.", "reasons_to_reject": "1. Some details are not very clear:\n- In the second paragraph of the introduction, the author uses the word \"Thus,\" but the reasoning relationship is not evident.\n- Figure 2 could be improved for better aesthetics.\n- In the method section, the term \"gap labels\" should be clarified from the beginning, as it might cause confusion for readers.\n- In section 6.1, it is not clear on which test set the comparison of reranking results and the output of Riken&Tohoku is based.\n\n2. It will be more convincing to conduct a human evaluation experiment on the system combination results.\n\n3. Reporting the time taken for system combination will be beneficial. Is the proposed method faster than previous work?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new GEC quality estimation model and three methods for utilizing GEC quality estimation models for system combination with varying generality: model agnostic, model-agnostic with voting bias, and model-dependent methods. The results suggest that the proposed method improves the F0.5 scores on the GEC system combination task and achieves the highest F0.5 scores on the CoNLL-2014 test set and the BEA-2019 test set.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F5jrHswGAL", "reviewer": "Reviewer_8ah8", "summary": "", "questions": "A) Sec. 3.1 and 3.2:\nThe proposed method tested applicability of each edit one-by-one.  If there are simultaneous edits such as agreement, is there any concern that only one of them will be applied?  Was the beam size 16 enough?\n\nB) Lines 274-284:\nIt looks the unit size of the intermediate layer was not written anywhere.  What were the sizes of $A_{w}$, $A_{g}$, $a_{w}$, and\n$a_{g}$?\n\nC) Eq. 8:\nThe combination number of $y^{r}_{v} > y^{r}_{w}$ becomes huge if the\nnumber of element systems increases.  Were the systems sampled to\nreduce the computational cost?\n\nD) Line 411:\nIs this the same as 65,824 hypotheses?\n\nE) Lines 476-477:\nI could not find these values in Table 3.\nIf they are not shown in the table, please note it.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and for appreciating our work.\n\nQuestions for the Authors:\n\nA. Like any application of beam search, there is always a risk of missing long dependencies. However, if the model can identify the correct edit well, there will be no issue, even with a very small beam size. We ran an ablation study for the beam search and found that the performance does not change much with beam size >= 8. We found that beam size = 16 is sufficiently good, and our model has a higher F0.5 score on the subject-verb agreement error type compared to ESC, EditScorer, and MEMT.\n\nBeam      | BEA-Dev | BEA-Test | CoNLL-Test\n------------ | ------------- | -------------- | ---------\nBeam 1  | 62.96 | 80.18 | 67.48\nBeam 2  | 63.13 | 80.17 | 69.38\nBeam 4  | 63.31 | 80.31 | 70.25\nBeam 8  | 63.34 | 80.47 | 70.36\nBeam 12 | 63.34 | 80.49 | 70.34\nBeam 16 | 63.40 | 80.50 | 70.48\nBeam 20 | 63.38 | 80.53 | 70.44\nBeam 24 | 63.33 | 80.53 | 70.36\nBeam 32 | 63.37 | 80.55 | 70.56\n\nB. The size of $A_w$ and $A_g$ is model_dim x model_dim and the size of $a_w$ and $a_g$ is model_dim x 1. In our case, we use DeBERTaV3-Large which has a model_dim of 1024. We will clarify this in the paper.\n\nC. We group the hypotheses into smaller groups of size $n$ in Section 5.1 (second paragraph). In our experiment, we use $n=4$ (a hyper-parameter). Note that n is independent of the number of base (element) systems that we combine. This way, no matter how many systems we combine, the combination number of $y^{r}{v} > y^{r}{w}$ is capped to $n \\choose 2$. \n\nD. That is correct.\n\nE. The scores of Riken&Tohoku in Table 3 are from the official model outputs of the Riken&Tohoku paper (Kiyono et al., 2019 [1]), where the model is an ensemble with right-to-left reranking. The values in line 477 (and the data used for the single-system reranking evaluation in Table 2) are from a reproduction without right-to-left reranking by Liu et al., (2021) [2]. Liu et al. mentioned that the right-to-left reranking model is not publicly available and is trained with unpublished data.\n\nTypos Grammar Style and Presentation Improvements:\n\n1. We ran system combination experiments on the CoNLL-2014 test set with different numbers of base systems. We used the base systems in Table 3 and randomly sampled 5 combinations of base systems for each experiment (except when combining 5 systems where there was only one combination). We report the average F0.5 score for each number of base systems in the table below. We found that ESC’s performance deteriorates rapidly when the number of base systems is reduced, while HyRank+voting and EditScorer can maintain their performance.\n\n| # systems | ESC  | EditScorer  | Hyrank+voting |\n| ------------- | -------- | --------------- | ------------------- |\n| 5               | 69.51 | 68.25         | 70.48 |\n| 4               | 68.50 | 68.51         | 70.42 |\n| 3               | 67.57 | 68.82         | 70.14 |\n| 2               | 65.61 | 68.77         | 69.40 |\n\n2. When we add gold edits to the base systems, we found that on the BEA-2019 development set, 77% of the selected edits are from the gold edits, while on the CoNLL-2014 test set, 80.69% of the selected edits are the gold edits.\n\nReferences:\n\n[1] Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction. EMNLP.\n\n[2] Zhenghao Liu, Xiaoyuan Yi, Maosong Sun, Liner Yang, and Tat-Seng Chua. 2021. Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction. NAACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Although the proposed method is a combination method of multiple GEC systems, it is model-agnostic, and therefore scalability and usability must be high.\n* F_(0.5) score of the existing test set was the highest, and the experiments were carefully carried out.", "reasons_to_reject": "I could not find crucial problems except for the questions written in the next section.", "questions_to_authors": "A) Sec. 3.1 and 3.2:\nThe proposed method tested applicability of each edit one-by-one.  If there are simultaneous edits such as agreement, is there any concern that only one of them will be applied?  Was the beam size 16 enough?\n\nB) Lines 274-284:\nIt looks the unit size of the intermediate layer was not written anywhere.  What were the sizes of $A_{w}$, $A_{g}$, $a_{w}$, and\n$a_{g}$?\n\nC) Eq. 8:\nThe combination number of $y^{r}_{v} > y^{r}_{w}$ becomes huge if the\nnumber of element systems increases.  Were the systems sampled to\nreduce the computational cost?\n\nD) Line 411:\nIs this the same as 65,824 hypotheses?\n\nE) Lines 476-477:\nI could not find these values in Table 3.\nIf they are not shown in the table, please note it.", "paper_topic_and_main_contributions": "This paper presented a system combination method, which selects the best combination of edits output from multiple grammatical error correction (GEC) systems.\n\nBasically, the method computed quality scores when the edits are applied to the source sentence and searched for the best combination using beam search.  The quality score, which is based on word scores of substitution and deletion and gap scores of insertion, was computed by the pretrained encoder that learned the order of F_{0.5} scores.  The proposed system also used the voting score and the scores output from element GEC systems.\n\nIn their experiments, F_{0.5} score of the proposed method was better than those of each QE system. In comparison with other system combination methods, the proposed method achieved the highest scores on the BEA-2019 and CoNLL-2014 test sets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Because the proposed method is system combination, the result will be changed along with the combination of element systems. It is better to discuss the other combination of the element systems.\n\n* It is interesting whether only the gold edits will be selected or not if the proposed system is applied to the mixture of the gold edits and edits output from multiple GEC systems."}}]

Author rebuttal:

Meta-review: The paper proposes a novel quality estimation model for GEC that has a higher correlation with the F0.5 score used to evaluate GEC model quality. Then they propose 3 methods of system combination based on edit rescoring and beam search, that use the QE model, and show new SOTA results on two English benchmarks.

The paper is well-written, the experimental results are solid, with new SOTA on English benchmarks. The paper also offers a thorough analysis of model outputs. It would be interesting to see how this work applied for other languages.
Decision: Accept-Main