CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction
Authors: Jingheng Ye
Yinghui Li
Qingyu Zhou
Yangning Li
Shirong Ma
Hai-Tao Zheng
Ying Shen
Venue: EMNLP 2023 Main (2023)
Keywords: natural language processing
grammatical error correction
evaluation metric
TL;DR: A novel GEC metric designed to debias multi-reference evaluation for grammatical error correction, supporting two distinct evaluation hypotheses.

Abstract: Evaluating the performance of Grammatical Error Correction (GEC) systems is a challenging task due to its subjectivity.  Designing an evaluation metric that is as objective as possible is crucial to the development of GEC task. However, mainstream evaluation metrics, i.e., reference-based metrics, introduce bias into the multi-reference evaluation by extracting edits without considering the presence of multiple references. To overcome this issue, we propose Chunk-LE Multi-reference Evaluation (CLEME), designed to evaluate GEC systems in the multi-reference evaluation setting. CLEME builds chunk sequences with consistent boundaries for the source, the hypothesis and references, thus eliminating the bias caused by inconsistent edit boundaries. Furthermore, we observe the consistent boundary could also act as the boundary of grammatical errors, based on which the F$_{0.5}$ score is then computed following the correction independence assumption. We conduct experiments on six English reference sets based on the CoNLL-2014 shared task. Extensive experiments and detailed analyses demonstrate the correctness of our discovery and the effectiveness of CLEME. Further analysis reveals that CLEME is robust to evaluate GEC systems across reference sets with varying numbers of references and annotation styles. All the source codes of CLEME are released at https://github.com/THUKElab/CLEME.

Grammatical Error Correction (GEC) a task Debiasing Multi-reference Evaluation for Grammatical Error Correction
CLEME:
Jingheng Ye'; Yinghui Li" Qingyu Zhou', Yangning Li
Shirong Ma', Hai-Tao Zheng 133f Ying Shen*
T singhua Shenzhen International Graduate School, Tsinghua University OPPO Research Institute, 3Peng Cheng Laboratory
School of Intelligent Systems Engineering, Sun- Yat Sen University
(yejh22, liyinghu20 ( femailo wee.maai...ssssssuu.....e tsinghua. edu. cn
Abstract
EHRANT
Source Newadays
technologies were Impeoved compared for centary.
Ref.
(Nowadays
teclnologiees
have
[arpeoved a compared
Uhe century.
Evaluating the performance of Grammatical Er- ror Correction (GEC) systems is challenging task due to its subjectivity. Designing an evalua- tion metric that is as objective as possible eru- cial to the development of GEC task. However, mainstream evaluation metrics, i.c., reference- based metries, introduce bias into the multi- reference evaluation by extracting edits without considering the presence multiple references. To overcome this issue, propose Chunk- LEvel Multi celli-rrfeeence Evaluation (CLEME), designed to evaluate GEC systems in the multi- reference evaluation setting, CLEME builds chunk sequences with consistent boundaries for the source, the hypothesis and references, thus eliminating the bias caused by inconsistent edit boundaries. Furthermore, we observe the con- sistent boundary could also act as the boundary of grammatical errors, based on which the Fo.s score is then computed following the correction independence assumption. We conduct exper- iments on six English reference sets based on the CoNLL-2014 shared task. Extensive ex- periments and detailed analyses demonstrate the correctness of our discovery and the effec- tiveness of CLEME. Further analysis reveals that CLEME is robust to evaluate GEC systems across reference sets with varying numbers of references and annotation styles!
Ref.2
Nowsaaay
bechnology
bos
Imgroved let congared)
with the lest century.
Hyp.
Nowadays
Inchnolgies
have
Ingraved a compared with
the century.
CLEME
Source
Neweddays
(he technologie were
impeowed lot compared
the Last century:
Rof.
Nowodays
Iechaologies have
Imgroved comgared
The last century.
Ref.
Nowodays
technology has Improved compared
wit
The last contary.
Itpp
Newordays
In hnchhiggim have
insprosed let compardd
with (ee Lest centary..
Figure I: A comparison of edits automatically extracted by ERRANT and CLEME. orange block is edit.
et al., 2022; Li et al., 2023; Ye et al., b023a,b; Zhang et al., 2023). However, it is intractable to evaluate GEC systems due to the highly subjective nature of the task and the low inter-annotator agree- ment (IAA) (Choshen and Abend, 201 8). There- fore, most datasets improve compatibility by incor- porating multiple references to guarantee more realistic evaluation of the model performance.
There are two broad categories of GEC metrics: reference-based and reference-less. Reference- based metrics evaluate GEC systems by compar- ing their hypotheses and human-annotate ref- erences in terms of edits (Dahlmeier and Ng. 2012; Bryant et al., 2017) or n-grams (Napoles et al.. 2015). Reference-less metrics are pro- posed to evaluate GEC systems without references. However, Deutsch et al. (2022) demonstrate that reference-less metrics are inherently biased and limited in their ability to evaluate generated text. Therefore, we focus on reference-based metrics, which can evaluate in an interpretable manner, thus providing useful insights for model analysis.
1 Introduction
that involves making local substitutions to cor- rect grammatical errors in given ungrammatical text (Bryant et al., 2022; Ma et al., 2022; Ye et al., 2022; Ma et al., 2023). The practical value of GEC in daily life has led to increasing attention being paid to this task (Li et al., 2021, 2022a,b; Kaneko
Figure 1 illustrates how existing reference beeerence-based metrics, such as ERRANT, extract the edit and then compute the Fo.s score by comparing hypotheses and references. However, these metrics often fail to consider multiple references, which can result in bias during multi-reference evaluation. We argue that this bias arises because the current approach rewards equally good corrections unfairly. For in-
"indicates equal contribution.
Corresponding author: Hai-Tao
Zheng.
(E-mail:
@ sw..sinuuaa.du..cm
All the source codes of CI EME are released at tptps:/
gi thub. com/ wwt/TooOE/eeaaeaeeeeeeceeeeeeeee THUKElab/CLERE

---

stance, the ungrammatical phrase the technologies were equally well-corrected by both Ref. 1 and Ref. 2. However, if hypothesis aligns with Ref. I's corrections (i.e., [the + e] and [were- + havel, TP=2), it will be rewarded less than the correc- tions of Ref. 2 (i.e., [the + e], [techmologies + technology] and [were - has], TP=3).
(3) Extensive experiments and human evaluation are conducted confirm the effectiveness and robustness our approach.
2 Preliminary Study
2.1 Consistent Boundaries
We determine consistent chunk-level boundaries by chunk partition process to debias the multi- reference evaluation, as depicted in Figure 2. We first extract the edit sets of the hypothesis and ref- erences, and then merge the overlapping edits into chunk. It's worth noting that the source, hypoth- esis and references are all segmented into chunk sequences with the same number of chunks, regard- less of the number of their tokens. This process is straightforward since we can locate and exam- ine all possible corrections of an erroneous chunk. For example, the chunk by the can be corrected in two ways, i.e., with in Ref. and through in Ref. 2. The resulting chunks fall into three categories: I) unchanged chunks, which contain the same text segments as the source sentence, 2) corrected chunks, which consist of non-empty text segments different from the source sentence, and 3) dummy chunks are empty chunks.
In this paper, we propose Chunk-LEvel Multi- reference Evaluation (CLEME), which enables unbiased Fo.5 scores for GEC multi-reference eval- uation. Inspired by (Gotou et al., 2020), CLEME transforms the source, the hypothesis and all the references into chunk sequences with consistent boundaries, thereby eliminating the bias in GEC multi-reference evaluation.
Existing metrics assume that corrections of gram- matical errors are dependent. That is, whenever there is more than one reference for a source, the metries try each reference in turn, and then the highest score is taken as the final score. However, we observe that grammatical errors corrections in terms of chunks can be considered approximately independent. For example, the ungrammatical phrases the technologies were and for shown in Figure can be corrected independently, i.e., the correction of the technologies were has no bearing on the correction of for. Based on this observation, we compute Fo.5 scores following the assumption that corrections of grammatical eтrors are indepen- dent. Specifically, we iterate through the chunks of hypothesis and consider chunk correct if it matches any of the corresponding chunks in the references. In this case, the hypothesis in Figure would be rewarded 2TP, rather than 1TP and IFP, which is the traditional case. To demonstrate the ef- fectiveness and robustness of CLEME, we conduct experiments on six English reference sets with vary- ing numbers of references and annotation styles, either calculating the Fo.5 sentence-level.
2.2 Boundaries of Grammatical Errors
Figure illustrates the merging of overlapping edits into either corrected or dummy chunks, which are then separated by unchanged chunks. This raises the question, are chunk boundaries the boundaries of grammatical errors?
Dataset. To answer the question, conduct ex- periments on BN-10GEC (Bryant and Ng, 2015). The dataset comprises 1,312 source sentences that are identical to the CoNLL-2014 test data (Ng et al., 2014). Each source sentence is associated with 10 references annotated by 10 native English speakers, including two official annotators of CoNLL-2014 the first author of the paper, and seven freelancers recruited via online recruitment website.
the corpus- or
In summary, our contributions are three folds:
(1) We propose CLEME, reference-based metric that evaluates GEC systems at the chunk-level, aiming to provide unbiased Fo.s scores for GEC multi-referencee evaluation.
Experiment Setup. For each source sentence, we sample 9 references and run the chunk parti- tion process described in Section 2. The result- ing chunk sequences are determined collectively by all 9 references. The edits of the remaining reference {er, {M} are then used to calculate the following three statisties: 1) The In-Corrected- Chunk (ICC) ratio indicates the proportion of edits included by corrected/dummy chunks of the other
(2) We observe that the corrections of grammat- ical errors in terms of chunks are approxi- mately independent. Therefore, we propose to compute Fo.5 scores based on the correction independence assumption.

---

references. An edit included by chunk if the interval of the edit falls within that of the chunk. 2) The In-Unchanged-Chee Chankk (IUC) ratio gives the proportion of edits included by unchanged chunks the other references. 3) The Cross-Chunk (CC) ratio computes the proportion of edits that extend beyond the original boundaries. These statistics are calculated as follows:
Item
Number (perc.)
Length
Sentences
1,312
23.0
References
13,120
22.9
Fdits
36,677
Unchanged Chunks
93,469 (77.63%)
2.5
Corrected/Dummy Chunks ICC
26,948 (22.37%)
2.4
33,251 (90.66%)
IUC
2,837 (7.74%)
CC
589 (1.61%)
Table I: Statistics of the BN-IOGEC dataset.
ICC 1 E fee)) M
(1)
Method
1 IUC
M Σ f2(e.).
(2)
3.1 Chunk Evaluation
As shown in Figure 2, each chunk consists of edit operation(s), start index, end index, and correct to- kens. Conventional reference- as MaxMatch (M²) and ERRANT compute Fo.s scores based on the correction dependence assump- tion. They evaluate the performance for each ref- erence separately and select the one that yields the best result for the source sentence. CLEME- dependent also computes Fo.5 scores in this way by treating corrected/dummy chunks as edits. On the other hand, CLEME-independendl is proposed to compute Fo.5 scores based on the correction independence assumption. A corrected/dummy chunk from the hypothesis is considered correct if it matches one of the corresponding chunks from the references. It is worth noting that CLEME is able to fully inherit pre-classified errors from ER- RANT, where each corrected/dummy chunk may consist of multiple error with different types.
CC ICC- IUC,
(3)
metrics such
where M is the number of edits from the remain- ing reference. If the edit e, is included in cor- rected/dummy chunk, the function fi (e:) returns 1, otherwise O. Likewise, if the edit e; included in an unchanged chunk, the function f2(4;) returns 1, otherwise O. We sample 9 different references for chunk partition in each run and repeatedly calculate the statistics using the remaining reference.
Results. As shown in Table 1, the number of corrected and dummy chunks are less than that of edits since overlapping edits are merged into chunk. total of 90.66% edits are included by the corrected/dummy chunks, which suggests the grammatical errors be corrected have been con- sidered by the other references. However, only 7.74% edits are included by corrected chunks, indi- cating that these edits may be over-corrected since the other references believe no grammatical errors needed correction. Interestingly, 1.61% % edits cross the chunk boundaries, suggesting that the chunk boundaries are stable enough to serve as the bound- aries of grammatical errors to some extent. Addi- tionally, human evaluation in Section 4.2 could be used as another argument to support this conclu- sion. Therefore, we have the following assumption.
3.2 Length Weighting
The average length of chunks is much longer than that of edits shown in Table 1, resulting in the un- fairness of chunk evaluation if a longer chunk is rewarded equally with a shorter one. Therefore, we add length weighting to the chunk evaluation. The intuition of length weighting is to compensate for long chunk matching. The weights of True Positives (TPs), False Positives (FPs), and False Negatives (FNs) are computed as follows:2
Correction independence assumption: grammatical envrr corrections are independent.
1 (ai exp((E - x) Cmin, Cras w =clip
(4)
That is, the correction of a grammatical error does not impact the correction of other grammatical errors. With this assumption, Fo.s scores can be calculated using an alternate method, which will be introduced in Section
w clip
1) exp(r r 8) Cmin
(5)
1+ (a2
We da not apply length weighting to TNs since it is unneces- sary for Fo.5 scores.

---

10
People build misitin- ship with others with social media.
Hef.1
People
bulld
relationship
with others
with
social media...
Ins, 2,
arelationship mp, 7
rep, wrthh
People bulld ielation-
Bel.2
stips witti others through social media.
People
build
rebiiionhips
with others
through
social zeeiia...
riu. 2, relatimnshiss
rip. 2,3, Intationships
rip, through
nep, dell 5 through
Peuple builé rebttoonhii others by the social
diliii
People
relationship
others
media...
Ins. 1, tend te
lns, 1, lend
rng, relationships
mu,, indafionshiss
People
others
People
boad
build
rebationships
with others
with
media.
rep,
Edit Extroction
Dummy
Conrected Chunk
Unchanged Chunk
Correction Dependence A ssumption
Correction Independence ssumption
ReG.1
indationnhip
Rel.2
relationships
V
through
Ref. 2
reattonships
through
reletiunships
Нур
Lend
P 0.33
R
2
0.50
P ==
0.67
1.00
p.
P-R
Fos = (1 0.5")
0.36
Fas 0.5*)
0.5 P
0.5
Figure 2: Overview of our approach CLEME. CLEME first 1) extracts edits of the hypothesis and the references, 2) merges the overlapping edits into chunks, then 3) computes the Fo.5 scores based on two different assumptions.
3.0
PN clip
I+ explt Crins Crax
03
(6)
2.5
01
where and 0: are scale factors for TPs, FPs and FNs respectively, is the length of the chunk, f is the average length of chunks, and the function clip(r, a, b) clips the value between and b. The curves of length weighting are depicted in Figure 3. Formally, given system corrected/dummy chunk set H'Hl and a gold corrected/dummy chunk set R we apply length weighting on each chunk to compute precision, recall and Fo.5 as follows:
Length Weight
2.0
02
1.0
0.5
0.0
2
3
1
5
Length Chunk
Figure 3: Curves of length weighting with different a
for 2. All curves pass through point (f, 1.0). curve with larger scale factor has greater slope.
Σ
w
TP
CC
R +
P=
(7)
WFF
DECHnCR
eECH CR
w TP
3.3 Corpus-level v.s. Sentence-level
CECHNCR
R
8 We compute Fo.s scores of GEC systems at both corpus-level and sentence-level following (Gong et al., 2022). Corpus-level metrics compute an Fo.s score over the entire dataset. Sentence-level metrics compute an Fo,5 score over each sentence of the dataset and evaluate GEC systems by us-
TP +
WFN
GEC nR
P.R
Fg (1
(9)
[5 P
where 3 0.5 is usually used, which weighs pre- cision twice as much as recall.

---

ing the average Fo.s score. CeeMeedependent and CLEME-independent are corpus-level met- rics, and their sentence-level variants are respec- tively SentCLEME-dependent and SentCLEME- independent. Both levels of the GEC metric are developed to provide more user- seer-friendly options. Sentence-level metrics should be used if consis- tent evaluation weight for each sample is desired. This ensures that the evaluation result of each sam- ple has the same influence on the final score. On the other hand, if harder samples containing more edits should have larger weight, then corpus-level metrics should be used instead.
ing corpus- and sentence-level variants 3:
GLEU and SentGLEU (Napoles et al., 2015) are n-gram based metries, which reward hy- pothesis n-grams that overlap with the refer- ence but not the source and penalize hypoth- esis n-grams that overlap with the source but not the reference.
M and SentM (Dahlmeier and Ng, 2012) dynamically extract the hypothesis edits with the maximum overlap of gold annotations.
ERRANAT and SentERRANT (Bryant al., 2017) extract edits by utilizing a linguistically- enhanced alignment algorithm.
4 Experiments
4.1 Correlations with Human Judgments
PT-M2 and SentPT-M2 (Gong et al., 2022) are recently proposed reference and PLM- based GEC metric, which score edits using the knowledge of pre-trained language model.
Dataset. To verify the effectiveness of CLEME, we measure correlations between reference-basee metries and human judgments on multiple English reference sets, including CoNLL-2014 (Grund- kiewicz al., 2015), BN-10GEC (Bryant and Ng, 2015) and SN-&GEC (Sakaguchi et al., 2016). All the reference sets are based on CoNL.L-2014 (Ng et al., 2014), consisting of 1,312 source sentences. SN-8GEC collected 8 references sets of annota- tions from both experts and non-experts, including 4 sets of minimal edits and 4 sets of fluency edits (2 by experts and 2 by non-experts). Reference sets statisties are reported in Appendix A.
Additionally, CLEME can evaluate GEC sys- tems by accuracy scores, which is usually not im- plemented by conventional reference- based metrics. Please refer to Appendix C for the introduction and analyses of evaluating GEC systems by accuracy.
Table 2 reports the correlations between reference-based metrics and human judgments. For the corpus-level metrics, GLEU achieves the high- est correlations BN- B0-IOGEC and NE-fluency ref- erence sets. However, GLEU suffers from negative correlations on NE-Minimal, which is caused by low-quality annotations of NE-Minimal, indicat- ing that GLEU may not be a robust metric, consis- tent with the findings of (Sakaguchi et al., 2016). ERRANT performs slightly better than M on most reference sets, while PT-M² is strong corpus-level metric, which achieves the highest or comparable correlations all reference sets at the cost of more than 10x running time than other reference-basee metrics. Our proposed CLEME-dependent and CLEME-independent make better use of consistent chunk boundaries, thus performing slightly better than M² and ERRANT on most reference sets. No- tably, CLEMEE-nndeeenden achieves comparable performance to CLEME-dependent showing the
Results.
The human judgments for the outputs of 13 GEC systems (including the unchanged source text) are presented by (Grundkiewicz et al., 2015), where eight native speaker were asked to rank the output of all the systems from best to worst. Two system ranking lists are generated using Expected Wins (EW) (Macháček and Bojar, 2013) and TrueSkill (TS) (Sakaguchi et al., 2014) respectively.
Experiment Settings. Following (Gong et al., 2022; Chollampatt and Ng. 2018), compute the Pearson 7 and Spearman correlation coefficient p between reference-bard -base metrics and human judg- ments based on corpus-level ranking. We tune the hyperparameters on CoNLL-2014 and keep the hy- perparameters on the other reference sets, in order to demonstrate the adaptability of our approach. The detailed hyperparameters of CLEME are re-
We not experiment with I-measure (Felice and Briscoe, 2015) due to negative correlation and high computing complexity (Grundkiewicz al., 2015). "The phenomenon exists on all sentence-level metries. We remove unchanged references from some reference sets to avoid it.
ported in Appendix B.
Evaluation Metrics. We compare our approach with the following reference-bate based metrics, includ-

---

CoNLL-2014
BN-I0GEC
E-Minimal
E-Fluency
NE-Minimal
NE-Fluency
Metric
EW
TS
FWW
TS
EW
TS
EW
TS
EW
T8
EW
TS
M
0.623
0.672
0.547
0.610
0.597
0.650
0.590
0.659
0.575
0.634
0.582
0.649
0.687
0.720
0.648
0.692
0.654
0.703
0.654
0.709
0.577
0.648
0.648
0.703
GLEU
7
0.701
0.750
0.678
0.761
0.533
0.513
0.693
0.771
-0.044
-0.113
0.674
0.767
0.467
0.555
0.754
0,806
0.577
0.511
0.710
0.757
0,005
0.055
0.819
0.642
0.688
0,586
0.644
0.578
0.631
0.594
0.663
0,585
0.637
6,597
0.659
ERRANT
0.659
0.698
0.637
0.698
0.742
0,786
0.720
0.775
0.747
0,797
0.753
0.797
0.69%
0.737
0.650
0.706
0.626
0.667
0.621
0.630
0.675
0.620
"0.682"
PT-M²
0.758
0,769
0.690
0.824
0.709
0,736
0.758
0.802
0.736
0,758
0.758
0.802
0.648
0.691
0.602
0.656
0.594
0.614
0.589
0.654
0.545
0.643
0.612
"06675"
CLEME-dspendent (Ours)
0.709
0.742
0.692
0.747
0.797
0.714
0.775
0.786
0,835
0,720
0.791
0.649
0.691
0.609
0.593
0.643
0.653
0.601
0.647
0.672
CLEME-indeeendent (Ours)
0.709
0.731
0.692
0.747
0.791
0.802
0.731
0.791
0.797
0.841
0,714
0.786
SentM
7
0.871
0.8664
0.805
0,836
0.655
0.732
0,729
0,785
0.699
0.731
0.758
0.593
0.806
0.845
0.731
0.764
0,7974
0,846
0.632
0.687
0.784
0.828
0.756
0.826
0.742
0.7734
0.785
0,846
0,723
0,762
0.718
0.848
SentGLEU
0.720
0.775
0.769
0.824
0.764
0.797
0.791
0,846
0,764
0.830
0,768
0.846
0.870
0.846
0.885
0.896
0,768
0.803
0.806
0.732
0,710
0.765
0,793
0.847
SemERRANT
0.742
0.747
0.786
0.830
0.775
0.819
0.813
0,764
0.780
0,841
0.830
0.857
0.949
0.938
0.602
0.682
0.831
0.855
0.689
0,763
0.770
0.8224
0.648
0.725
SenEPT-MF
0.907
0.874
0.626
0.670
0.808
0.819
0.797
0,841
0.813
0.857
0.742
0.786
0.856
0.842
0.915
0.913
0.506
0.838
0.849
0.886
0,742
0.795
0.876
0.927
SeutCLEMME-dependeet (Ours)
0.824
0.808
0.835
0.874
0,775
0.819
0.824
0.863
0,797
0,846
0,791
0.846
0.868
0.x37
0.855
0,876*
0.821
0.K56
0.841
0.77
0.762
0.314
0.852
00.96
SentCLEME-indepeddett (Ours)
0.725
0.758
0.659
0.714
0.775
0.8194
0.808
0.846
0.819
0.874
0,762
0.825
Table 2: Overview of correlations between mainstream GEC metrics and human judgments. We highlight the highest score in bold and the second- second-highest score with underlines. SN-8GEC consists of four reference sets, i... E-Minimal, E-FFueeccy, NE-Minimal and NE--Fluency. & We remove unchanged reference sentences for higher correlations due low-quality annotations. Otherwise, negative correlations are possible.
effectiveness of computing Fo.. scores based on the correction independence assumption.
tions. Additionally, our approach achieves higher correlations on (N)E-Fluency rather than (N)E- Minimal, while SentM and SentERAAT per- form worse on E-Fluency than E-Minimal. This is because CLEME evaluates GEC systems using longer chunks rather than scrappy edits, which could better reflect whether grammatical error is fluently corrected. Overall, our approach achieves higher or comparable correlations on sentence- level than existing reference- based methods.
The majority of the sentence-level metrics out- perform their corpus-level counterparts because they weigh samples equally, which is in line with the bias of human annotation. Despite the strong performance of PT-M², SentPT-M2 achieves lower correlations on BN- E-FFluency and NE- Fluency compared to other sentence-level metrics. It suggests that scoring edits using pre-trained lan- guage models may not generalize well to unseen reference sets for sentence-level metrics. Our ap- proach aligns better with human judgments than existing reference-basee metrics for most refer- ence sets. Specifically, SenCLEMEE-depeedeen per- forms best on forms on a par with the best metric on E-I'luency, indicating it is more suitable for fluent reference
4.2 Human Evaluation
Experiments have shown the effectiveness of eval- uating GEC systems based on the correction inde- pendence assumption. In this section, aim to demonstrate whether the correction independence assumption makes sense for humans. We define the correction independence of a pair of chunks as the irrelevance of the correction of one chunk to the correction of the other. A simple case is presented in Appendix E. To evaluate this assump- tion, we conduct human evaluation experiments on 1,000 sentences randomly sampled from BN- 10GEC (Bryant and Ng, 2015). Three annotators were asked to judge whether a pair of chunks is correction -ineeeednnndeeeeeent.
and NE-Fluency, and per-
sets. This phenomenon aligns with our intuition since fluent editing is more likely to follow the correction dependence assumption. In contrast, achieves higher correla- tions on E-Minimal and NE-Minimal, we would expect from minimal editing that is more likely to follow the correction independence assumption. These results suggests that reference sets may have a preference towards one of the correction assump-
Table 3 reports the ratio of correction indepen-

---

GLEU
We observe that the number of false negatives (FNs) identified by CLEME is significantly lower than that of ERRANT. This difference can be at- tributed to the distinet definitions used by each system. While ERRANT considers FNs as ed- ServCLIT ITU
FRRANT
Ovneleton
StHERRANT
dependent
SenCLEME dependent
CL CLEME-inkgenaar
SeneCl ittl-ntteeee
ScaCLEME dependett
independent
0.68
0.64
8 9
10
10
Number of References
Scale Factor
Figure 4: Effect of references number BN-10GEC.
Figure 5: Effect of scale factors BN-10GEC.
of Correction Independence
Text
FP FN
A
90.85%
Source
It has improved compared for the last contory. improved compared
Aa
93.55%
Hyp.
century.
ERRANT
Aa
91.14%
Ref. has improved compared to the century. improved compured with century.
Annotator
Cohen Ss::
CLEME
A vs. Ao
38
Rel. It bas improved cotupared to the century.
0
A. vs. Ag
43.10%
improved compared with century.
Aa v.S. Ag
39.34%
Table 4: Cases of ERRANT and CLEME, ERRANT gives FP=1 and FN=1 since the hypothesis does not match one edits references. CLEME gives only FP=1 since the hypothesis tries to correct the error.
Table 3: comparison correction independence annotations across three annotators.
dence and Cohen's-A s-n (Cohen, 1960) inter-annotator agreement (IAA) across the three annotators. Re- sults show that more than 90% pairs of chunks are prrrectonnindependeen for all the annotators, indicating that it reasonable to evaluate GEC systems based on the correction independence as- sumption. Moreover, considering the subjectivity of GEC task, the iAA statistics show that it is rel- atively easy to judge whether a pair of chunks is correction-indeeendeed compared with the previ- study (Bryant and Ng, 2015)
overestimate FN counts in grammatical error cor- rection (GEC) systems, which is evident from the examples presented in Table 4. On the other hand, CLEME's definition also includes true negatives (TNs), making possible calculate accuracy.
5.2 Ablation Study
We present ablation analyses of our approaches on BN- we have similar findings on other reference sets. We report Pearson correlations y using Expected Wins ranking. The trend is similar for Spearson correlations and TrueSkill ranking.
5 Analysis
5.1 False Negative
Number of References. Since CLEME is de- signed for multi-reference evaluation, it degrades to conventional reference-based metrics such M² and ERRANT when only one reference is available. Here we demonstrate how correlations change against an increasing number of available references. The results reported in Figure 4 indi- cate that the correlations of corpus-level metrics do not change significantly with the increasing num- ber of available references. However, except for SentGLEU, correlations of sentence-level metric are consistently higher than corpus-level metrics. and steadily increase with more references. There- fore, we recommend evaluating GEC systems us- ing sentence-level metrics rather than corpus-level
its in the reference that do not match those made the hypothesis, CLEME identifies FNs as cor- rected/dummy chunks in the reference that do not match the chunks in the hypothesis. We argue the definition of ERRANT is problematic, as it tends to
'Bryant and Ng (2015) attempted to compute IAA at the sen- tence level. Three raters were asked simply to decide whether 200 sentences were correct or not. The authors reported IAA of just 0.16, 0.4 and 0).23.

---

Chunk
Chunk
Chunk 3
Chunk 4
Chunk 5
Chunk 6
Source
On other hand, there are
ways can
help us control
cure the discase, can cure the disense , can
going.
Hyp.
On other hand, there are
ways that
help us to contrl
go.
Ref.
On other hand, there are
ways that can
help us to control
cure the diseuse, can
go.
Ref.
On other hand, are
things that can
help us to
cure the disense can
go.
Chunk
Chunk
Chunk
Chunk 4
Chunk 5
Source
On
one hand, we do not want this potential danger one hand, we do not want this potential donger one hand, we do not want this potential danger hand, not want this potential danger
causing figgttenning aflects in causing frightening affects in having frightening effects in to have frightening effects on
our lives
Hyp.
On
our lives
Ref. 1
On
our lives. our lives
Ref.2 2
On
the
Chunk
Chunk
Chunk 3
Chunk
Source
Especially for the Especially for the
young people young people
without marrige if he/she is without marriage if the lutter is marrige who are who are unmarried, if hefshe is
known to have some genetic risk. known to have some genetic risk.
Hyp.
Ref.
Espocially for unmarried
young people
known to have some genetic risk
Ref. 2
This is especially the case for
young people
known to have some genetic risk.
Table 5: Cases of chunk partition. These tables are automatically generated by CLEME. More cases from multiple datasets and language are provided in Appendix F.
metrics for the multi-reference evaluation setting.
Reference Metrics
Granularity
Score
Deterministic
(Tahlmmeer (Napoles
N-gnen
Wrightsd Procssio
Parameter Sensitivity Analysis. The scale fac- tors introduced in Section 3.2 dictate how much the weights of chunks change with their length. We report the corrections for various scale factors, as shown in Figure 5. The results demonstrate that CLEME is resilient to hyperparameter selection.
ERRANT (Bryau at 2017) CLEME (Curs)
Pluase level Bait
Fo
Chucklevel Bai
F.
Table 6: A comparison of mainstream reference- based GEC metrics. GLEU is indeterministic since it involves sampling operation.
experiments.
5.3 Case Study
Table 5 presents additional examples of CLEME. In the top group, chunk 2 and chunk 4 of the hy- pothesis respectively match those of Ref. and Ref. 2. In this case, CLEMEE-depeneen gives TP=1 and FP=1, while CLEME-independent gives TP=2. the second group, the hypothesis exactly corrects the ungrammatical word firghtenning in chunk 4. However, it cannot be rewarded since the entire chunk is not corrected. In the bottom group, two given references have made extensive modifi- cations, with an unchanged chunk young people. Evaluating hypotheses in terms of chunks is gener- ally more challenging than fragmented edits, but it provides a more comprehensive diagnosis.
Related Work
6.1 Reference-based Metrics
Reference-based metrics score GEC systems under the guidance of manually written references. M scorer (Dahlmeier and Ng. 2012) determines an optimal edit sequence between a source sentence and a system hypothesis that achieves the highest overlap with the gold-standard annotation. The per- formance of each system is then represented using the F0.5 score. However, optimality in terms of overlap does not guarantee optimality in GEC eval- uation. Bryant et al. (2017) showed that M scorer exploits its dynamic edit boundary prediction to artificially maximize true positives and minimize false positives, thus producing slightly inflated scores. Therefore, (Bryant et al., 2017) proposed ERRANT, which improves edit extraction using a linguistically alignment algorithm and merging rules, improving the alignment of tokens with similar linguistic properties. Despite its effec- tiveness, ERRANT language-dependen and bias still exists in multi-reference evaluation. Inspired by BLEU (Papineni et al., 2002) in NMT, Napoles
Even though there are larger grammatical er- rors spanning a significant portion of a sentence, CLEME would not necessarily collapse, i.e., pro- ducing one single correction chunk spanning the entire sentence. If collapse happens, the quality of the reference set should be checked first. This is because that collapse happens only if the input sen- tences of chunk partition are completely different, resulting in a trivial chunk partition result, which is an extreme case that has not been observed in our

---

al. (2015) proposed GLEU, an n-gram based met- ric for GEC evaluation. To remedy the shortcoming that Fo.5 is unable to differentiate a do-nothing sys- tem and a bad system unless TP >> 0, I-measure (Fe- lice and Briscoe, 2015) generates an exact (global optimal) alignment using three-way alignment al- gorithm and computes weighted accuracy to score GEC systems in terms of relative textual improve- ment. The comparison of reference-based GEC metrics is shown in Table 6.
ing the unchanged source) from the CoNLL-2014 shared task (Ng et al., 2014) were ranked based on human rankings collected by two ranking methods: Expected Wins (EW) and TrueSkill (TS) (Grund- kiewicz et al., 2015). Sakaguchi et al. (2016) col- lected 8 (2 x 2 x 2) annotations (minimal and flu- ency, expert and non-expert, with two corrections each), revealing that GEC metrics work differently across reference sets. Napoles et al. (2019) ex- plored how GEC metries operate in new domains (formal and informal writing by native English speakers). They constructed multi-referenn GEC test set called GMEG- Data and a new ensemble metric GMEG-Metric.
6.2 Reference-less Metries
To avoid the prerequisite of references for GEC evaluation, recent works focus on scoring GEC systems without
references. In- spired by quality estimation in neural machine translation, Napoles et al. (2016) propose three Grammaticality-Basse Metries (GBMs), which are calculated by benchmark GEC system a pre- trained ridge regression model. Asano et al. (2017) extend GBMs by introducing three assessment criteria for Grammaticality, Fluency and Mean- ing preservation (GFM). SOME (Yoshimura et al., 2020) further improves GFM by optimizing each Grammaticality, Fluency and Meaning preservation metric to more closely correlate with human judge- ments. Scribendi Score (Islam and Magnani, 2021) overcomes the limitations of SOME, requiring nei- ther benchmark GEC system nor fine-tuning. IM- PARA (Maeda et al., 2022) comprises a quality estimator (QE) and similarity estimator (SE) based on BERT (Devlin et al., 2019), which evaluate the quality of GEC output and semantic similarity of two sentences, respectively.
Conclusion
This paper proposes CLEME, reference- GEC metric that aim to provide unbiased Fo.s scores for multi-reference evaluation. We explore evaluating GEC systems based on either the correc- tion dependence assumption or the correction inde- pendence assumption. Several possible approaches can be suggested to further improve CLEME. For example, developing (1) a GEC metric that adap- tively combines dependent and independent as- sumptions, and (2) a weighting strategy by utilizing the knowledge of pre-trained model. In the future. would like to develop CLEME for all languages and admonstrate the effectiveness of CLEME on other languages. It is also worthwhile explore accuracy-based metrics.
Limitations
Although recent reference-less metrics may be highly consistent with human judgments, they al- ways suffer from the lack of interpretability and robustness (Bryant et al., 2022), which are crucial factors for GEC evaluation. Additionally, evalu- ating GEC systems by leveraging pre-trained or fine-tuned language models could pose potential risks. Efficiency of reference-less metrics that rely on language models is also critical if they are used for GEC benchmarks.
Although CLEME can be extended to other lan- guages, we have not tested its effectiveness any language other than English. Furthermore, all the reference sets used in our experiments are based on the CoNLL-2014 shared task, second- language dataset. To demonstrate the robustness of our approaches, further experiments on evaluation datasets with multiple text domains are required. We believe that introducing the correction indepen- dence assumption perspective into GEC datasets of other languages and domains could lead to more in-depth analysis and exploration.
6.3 Meta Evaluation Methods
It is intractable determine the best GEC metric. A reasonable GEC metric should take into account multiple factors, including correlation with human judgments, interpretability and efficiency. Inspired by WMT human evaluation campaigns (Callison- Burch et al., 2008), 13 system outputs (includ-
While recent PLM- PLM-based metries have shown superior correlations compared to reference-based metrics, including ours on some reference sets, our approach enables the evaluation of GEC systems in an interpretable manner, which is a significant advantage over reference-less metrics. We leave the

---

exploration of incorporating the PLM's knowledge into CLEME for future work.
pages 793-805. Vancouver, Canada. Association for Computational Linguistics.
Christopher Bryant and Hwee Tou Ng. 2015. How far are we from fully automatic high quality grammatical error correction? In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 697-707, Beijing, China. Association for Computational Linguistics.
Ethics Statement
In this paper, we verify the effectiveness of our pro- posed approach using CoNLL--2014, BN-10GEC, and SN-8GEC reference sets, all of which are from publicly available datasets resources on legiti- mate websites without sensitive data involved. All the baselines used in our experiments are also pub- licly available GEC metrics and we have cited the corresponding authors. We confirm that all datasets and baselines used in our experiments are consis- tent with their intended use.
Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2022. Grammatical error correction: survey of the state of the art. arXiv preprint or 2..5
Chris Cellison-Burcch Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation machine translation. Proceed- ings of the Third Workshop on Statistical Machine Translation, pages 70- 10-, Columbus, Ohio. Associ- ation for Computational Linguistics.
Additionally, we conduct human evaluation ex- periments show the rationality of correction in- dependence assumption. To do so, three postgradu- ate students specializing foreign linguistics and applied linguistics were employed as part-time an- notators, Each annotator could complete the entire annotation process within approximately 6 working hours. All annotators were paid for their work, with an average salary of approximately $5 per hour.
Martin Chodorow, Markus Dickinson, Ross Israel, and Joel Tetreault. 2012. Problems in evaluating gram- matical error detection systems. In Proceedings of COLING 2012, pages 61 1-628, Mumbai, India, The COLING 2012 Organizing Committee.
A Aeknowledgements
Shamil Chollampatt and Hwee Tou Ng. 2018. A re- assessment of reference-based grammatical error cor- rection metries. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, pages 2730-2741, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
This research is supported by National Natural Sci- ence Foundation of China (Grant No.62276154). Research Center for Computer Network (Shen- zhen) Ministry of Education, the Natural Science Foundation of Guangdong Province (Grant No. 2023A1515012914 Basic Research Fund of Shen- zhen City (Grant No. JCY120210322410000 10000222220000000 2003 and RCGG202108021440000 Project of PCL for Experiments and Applications (PCL2021 A06)), and Overseas Cooperation Re- search Fund of Tsinghua Shenzhen International Graduate School (HW2021008).
Leshem Choshen and Omri Abend. 2018. Automatic metric validation for grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Vooluee 1: Long Pupers), pages 1372-1382, Melbourne, Aus- tralia. Association for Computational Linguistics.
the Major Key
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement,
46.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of the 2012 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568-572 Montréal, Canada. Association for Compu- tational Linguistics.
References
Hiroki Asano, Tomoya Mizumoto, and Kentaro Inui. 2017. Reference-barre based metrics can be replaced with reference-less metrics in evaluating grammatical er- ror correction systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 343- 348, Taipei. Taiwan. Asian Federation of Natural Language Processing.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-fre evaluations of gener- ated text. arXiv preprint 2rXiv:2210.12566
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers),
Jacob Devlin, Ming-Wai Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zizhen Wang, Chao Li, Yunbo Cao, and Hai-Tao Zheng. 2022b. The past mis- take is the future wisdom: Error-driven contrastive probability optimization for Chinese spell checking. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 3202-3213, Dublin, Ireland. Association for Computational Linguistics.
Mariano Felice and Ted Briscoe. 2015. Towards stan- dard evaluation method for grammatical error de- tection and correction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 578-587, Denver, Col- orado, Association for Computational Linguisties.
Zuchao Li, Kevin Parnow, Masao Utiyama, Eiichiro Sumita, and Hai Zhao. 2021. MiSS: An assistant for multi-style simultaneous translation. In Proceedings of the 2021 Conference Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 1-10, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Peiyuan Gong, Xuebo Liu, Heyan Huang, and Min Zhang. 2022. Revisiting grammatical error cor- rection evaluation and beyond. arXiv preprint arXiv:221100133.
Shirong Ma. Yinghui Li, Haojing Huang, Shulin Huang, Yangning Li, Hai-Tao Zheng, and Ying Shen. 2023. Progressive multi-task learning framework for chi- nese text error correction. CoRR, abs/2306.1740 174447.
Takumi Gotou, Ryo Nagata, Masato Mita, and Kazuaki Hanawa. 2020. Taking the correction difficulty into account in grammatical error correction evaluation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2085-2095, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Shirong Ma, Yinghui Li, Rongyi Sun, Qingyu Zhou, Shulin Huang, Ding Zhang, Li Yangning, Ruiyang Liu, Zhongli Li, Yunbo Cao, Haitao Zheng, and Ying Shen. 2022. Linguistic rules-based corpus gener- ation for native Chinese grammatical error correc- tion. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 576-589, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Roman Grundkiewicz. Marcin Junezys- -owwmutt, and Edward Gillian. 2015. Human evaluation of gram- matical error correction systems. In Proceedings of the 2015 Conference on Empirical Methods in Natu- ral Language Processing, pages 461-
Matouš Machášek and Ondřej Bojar. 2013. Results of the WMT13 metries shared task. In Proceedings of the Eighth Workshop on Statistical Machine Transla- tion, pages 45-51, Sofia, Bulgaria. Association for Computational I Linguistics.
Md Asadul Islam and Enrico Magnani. 2021. Is this the end of the gold standard? straightforward reference- less grammatical error correction metric. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3009-3015, Online and Punta Cana. Dominican Republic. Asso- ciation for Computational Linguistics.
Koki Maeda, Masahiro Kaneko, and Naoaki Okazaki. 2022. IMPARA: Impact-based metric for GEC us- ing parallel data. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. pages 3578-8588, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
Masahiro Kaneko, Sho Takase, Ayana Niwa, and Naoaki Okazaki. 2022. Interpretabilityy for language learners using example-based grammatical error correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 7176-7188 Dublin, Ireland. Association for Computational Linguistics.
Courtney Napoles. Maria Nädejde, and Joel Tetreault. 2019. Enabling robust grammatical error correction in new domains: Data sets, metrics, and analyses. Transactions of the Association for Computational Linguistics, 7:551-566.
Yinghui Li, Haojing Huang, Shirong Ma, Yong Jiang, Yangning Li, Feng Zhou, Hai-Tao Zheng, and Qingyu Zhou. 2023. On the (in)effectiveness of large lan- guage models for chinese text correction. CoRR, fbs/2307.09007.
Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing ( (Volume 2: Short Papers), pages 588-593, Beijing, China. Asso- ciation for Computational Linguistics.
Yinghui Li. Shirong Ma. Qingyu Zhou, Zhongli Li, Li Yangning, Shulin Huang. Ruiyang Liu, Chao Li, Yunbo Cao, and Haitao Zheng, 2022a. Learning from the dictionary: Heterogeneous knowledge guided fine-tuning for Chinese spell checking. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 238-249, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016. There's 's no comparison: Reference- less evaluation metries in grammatical error correc- tion. In Proceedings of the 2016 Conference on

---

Empirical Methods in Natural Language Process- ing, pages 2109-2200 2109-2115, Austin, Texas. Association for Computational I Linguistics.
Ding Zhang, Yinghui Li, Qingyu Zhou, Shirong Ma. Yangning Li, Yunbo Cao, and Hai-Tao Zheng. 2023. Contextual similarity is more valuable than charac- ter similarity: An empirical study for chinese spell checking. In ICASSP 2023 2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and bench- mark for grammatical error correction. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229-234, Valencia, Spain. Association for Computational Linguistics.
Yue Zhang. Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang. Chen Li, Fei Huang, and Min Zhang. 2022. MuCGEC: a multi-referene multi- source evaluation dataset for Chinese grammatical error correction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 3118831.3 Seattle, United States. Association for Computational Linguistics.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference Compatational Natu- ral Language Learning: Shared Task, pages 1-14. Baltimore, Maryland. Association for Computational Linguisties.
A Statistics of Reference Sets
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
Table 7 presents the statistics of all reference sets in- volved in our experiments, including In-Corrected- Chunk (ICC) ratio, Sechangge-Chunlee (IUC) ratio and Cross-Chunk (CC) ratio. It is worth noting that all reference sets exhibit a low CC ratio with varying ICC and TUC ratios, indicating the ratio- nality and feasibility of evaluating GEC systems following the correction independence assumption.
Keisuke Sakaguchi. Courtney Napoles, Matt Post, and Joel Tetreault. 2016. Reassessing the goals of gram- matical error correction: Fluency instead of grammat- icality. Transactions of the Association for Computa- tional Linguistics. 4:169-182.
Keisuke Sakaguchi. Matt Post, and Benjamin Van Durme. 2014. Efficient elicitation of annota- tions for human evaluation of machine translation. In Proceedings of the Ninth Workshop on Statistical Ma- chine Translation, pages 1-11, Baltimore, Maryland, USA. Association for Computational Linguistics.
B Hyperparameters
The hyperparameters of our proposed CLEME con- sist of scale factors a and thresholds. We tune the hyperparameters on CoNLL-2014 14 and keep them on the other reference sets to demonstrate the adapt- ability of our method. The hyperparameters of CLEME are listed in Table 8.
Jingheng Ye, Yinghui Li. Yangning Li, and Hai-Tao Zheng. 2023a. Mixedit: Revisiting data augmenta- tion and beyond for grammatical error correction.
Jingheng Ye, Yinghui Li, Shirong Ma. Rui Xie, Wei Wu, and Hai-Tao Zheng. 2022. Focus is what you need for chinese grammatical error correction. CoRR, abs/2210.1 12692.
C Evaluate by Accuracy
Conventional reference metrics such Max- Match (M²) and ERRANT are unable calcu- late accuracy because they do not define True Negatives (TNs) 6. In order to implement the computation of accuracy, CLEME defines TNs as hypothesis unchanged chunks that match the chunks of references. Similar to Fo.., accu- racy can be computed based on correction de- pendence or independence assumptions in both corpus- and sentence-level, resulting in four new variants: 1) CLEMEE-dependenttaacc, 2) CLEME- independent-ace, 3) SenCCLEME-depeedeett acc, and 4) CentCLEMEE-nddeeeneent
Jingheng Ye, Yinghui Li, and Haitao Zheng. 2023b. System report for CCL23-eva task 7: THU KELab (sz) - exploring data augmentation and denoising for Chinese grammatical error correction. In Proceed- ings of the 22nd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations), pages 262-270, Harbin, China. Chinese Information Processing Society of China.
Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Ka- jiwara, and Mamoru Komachi. 2020. SOME: Reference-less sub-metries optimized for manual evaluations of grammatical error correction. In Proceedings of the 28th International Conference on Computational Linguisties, pages 6516-6522 Barcelona, Spain (Online). International Committee on Computational Linguistics.
"n exception is I-measure (Felice and Briscoe, 2015). which adopts an extended version of the Prtee-Annntatiion-S evaluation scheme (Chodorow et al., 2012).

---

Item
CoNLL-2004
BN-10GEC
E-Minimal
E-Fluency
NE-Minimal
NE-Fluency
Scats (Length)
1,312 (23.0)
1,312 (2.02 (23.0)
(23.0
1,312 (23.0)
1,312 (23.0)
(23.0)
Refs (Length)
2,624 (22.)
13,120
2,624 (21.2)
(22.8)
2.624 (23.0)
2,624 (22.2)
Edits (Length)
5,937 (1.0)
36,677 (1.0)
4,500 (1.0)
8,373 (1.1)
4,964 (0.9)
(1.2)
Unchanged Chunks (Length)
11,174 (4.8)
93,496 (2.5)
8,887 (6.3)
12.823 (3.8)
10,748 (5.1)
14,086 (2.9)
Comrecee/DDmmyy Chunks (Length)
4,994 (1.3)
26,948 (2.4)
3,963 (1.2)
6,305 (1.7)
4,221 (1.2)
6,892 (2.6)
ICC (Number)
51.05% (3,031)
90.66% (33,251)
62.47% (2.811)
52.69% (4,412)
43.84% (2,176)
43.32% (4.780)
(Number)
45.61% (2,708)
(2,837)
36.6% (1.647)
(3,589)
54.3% (2,698)
48.92% (5,397)
OC (Number)
3.34% (198)
1.61% (589)
0.93% (42)
4.8% (402)
1.81% (90)
7.76% (856)
Table 7: Statisties of CoNLL--2014 (Ng et al., 2014), BN-IOGEC (Bryant and Ng. 2015) and SN-8GEC (Sakaguchi et al., 2016) reference sets. We use ERRANT (Bryant et al., 2017) for edit extraction.
Configuration
CLEMM-ddepensennttaecc
CLEME-independenntt acc))
SennCLEEEE-ineeeeneeentt
Scale factor of TPs
10
10
Seale factor of FPs og
10
Scale factor of FNs
Scale factor of TNs 04
Threshold TPs
(0:75, 1.25)
(0.75, 1,25)
(1.00, 10.00)
(2.50, 10.00)
Threshold of HPS
(0.75, 1.25)
(0.75, 1,25)
(0.25, 10.00)
(0.25,1.00)
Tareshuld of FNs
(0.75.
(0.75, 1.25)
(1.00,
(1.00,
Threshold
(1.00, 1.00)
(1.00, 1.00)
(1.00,1.00)
(1.00,1.00)
Table 8: Hyperparameter values used in our experiments.
The results of human correlations are reported in Table 9, Accuracy-basee metrics perform very dif- ferently at the corpus- and sentence-level, which is similar to the findings (Napoles et al., 2016, 2019). Surprisingly, two accuracy- based corpus-level met- rics, i.e.. CLEME-dependent-acc and CLEME- independent-ace, result in negative correlations on all reference sets. However, their sentence- level variants, i.e., Sen:CLEMEEdeeeenden LenCLLEME--iddeeenddn--..., perform well and achieve the highest correlations on some reference sets. Regarding the disparity between the perfor- mance of accuracy-based metrics and Fo.s-based metrics at the sentence level, one notable differ- ence is their stability robustness on reference sets with varying numbers of references and an- notation styles. Fo.s-based metrics are more ro- bust different reference sets, where SenICLEME- (in)dependent achieves comparable correlations with the best metric on all reference sets. However, the performance of accuracy-based metrics lags far behind other metrics on some reference sets (BN- 10GEC, E-Minimal and NE-Fluency). A deeper investigation into this phenomenon is needed to un- derstand the instability of accuracy-baseds metrics. We leave the exploration and further analysis of accuracy-based metric for future work.
to ERRANT is attributed to the chunk partition process, where overlapping edits are merged into
chunks. It is worth noting that the FN counts of CLEME are significantly lower than those of ER- RANT because of their distinct definition. While ERRANT considers FNs the edits of references that are not identical to hypotheses, CLEME de- fines them as the corrected/dummy chunks of ref- erences that do not exactly match the chunks of hypotheses. We believe that the definition of ER- RANT could be problematic, as it has a tendency to overestimate the FN counts of GEC systems. This may result in an underestimated Recall rate in turn.
and
dependent v.s. -independent. Comparing the Precision and Recall of (Sem) is observed
to those of (Sent/CLEMEddependeel that the former has a slightly higher value. This is because (Sen)[LLMM-iindeeedee) has the po- tential to overestimate the performance of GEC systems, whereas (Sent)CLLEMEE-deeendee could result in underestimating the same. It is noteworthy that both metrics provide an upper bound and lower bound for GEC performance, respectively.
Corpus-level v.s. Sentence-level. The precision, recall, and Fo.s scores of sentence-level metrics are considerably higher than those of corpus-level vari- ants. There might be several factors contributing to this difference, but one possible explanation is that precision and recall values get affected by a limited number of challenging samples that contain numerous corrected/dummy chunks.
Detailed Analysis
Table 10 reports the detailed evaluation results of 13 systems on CoNLL-2014, The reason behind the lower TP and FP counts of CLEME as compared

---

CoN1.-20014
BN-10GEC
E-Minimal
E-Fluency
NE-Minimal
NE-Fluency
Metric
EW
T8
EW
TS
EW
T8
EW
T8
EW
TS
EW
TS
0.623
0.672
0.547
0.610
0.597
0.590
0.659
0.575
0.634
0.649
M'
0.687
0.720
0.648
0.692
0.654
0.703
0.654
0.709
0.577
0.648
0.648
0,203
0.701
0.750
0.678
0.761
0.535
0.515
0.693
0.771
-0.4
-0113
0.674
0.767
GLEU
0.467
0.555
0.754
0.806
0.577
0.511
0.710
0.757
-0.055
-0.055
0.725
0.819
0.842"
0.688
0.566
0.34
0.578
0.63
0.594
0.M
0.585
0.637
0:397
0.699
ERRANT
0.659
0.637
0.698
0.742
0.720
0.775
0.747
0.797
0.797
0.648
0.691
0.602
0,656
0.594
0.644
0.589
0.664
0.595
0.41
0612
0.623
-LEME-depeedeett (Ours)
0.709
0.742
0.692
0.742
0.797
0.714
0.275
0.786
0.36
0.291
0.649
0.691
0.609
0.659
0.595
0.643
0.587
0.653
0.601
0.647
0.611
0.672
Indeeendeeendentt (Ouns)
0.709
0.731
0.692
0.747
0.291
0.802
0.731
0.791
0.797
0.841
0.714
0.986
T6.26T
-0,342
0.286
-6.371
-0.522
-0,313
-0,216
"0330
-0,310
-0:15
-6440
00.313
CL CLEMM--depeddeet-acc (Ours)
-0.407
-0.478
00.45
-0.516
00.335
-0.423
-0,347
-0.437
00429
-0.516
-0.473
-0,555
CL EME--indeppeedeet--ncc (Ours)
-0.175
-0.262
-02266
-0384
-0.195
-0.283
-0.105
40.189
0.338
-0420
-0.338
-0.412
-0.176
40,264
-0.341
-0.418
-0.291
-0.379
-0.132
-0,231
40,499
-0.516
-0.451
-0.522
SentM
0.871
0.864
0.567
0.646
0,805
0,836
0.655
0.732
0.729
0,7854
0.621
0.699
0.731
0.758
0.593
0,648
0.808
0.44
0.731
0.764
0.797
0.846
0.632
0.687
0.784
0.628
0.756
0.826
0.742
0773
0,785
0,846
0.723
0,762
0.778
0.848
SentGLEU
0,720
0,775
0.769
0.824
0.764
0.797
0,791
0,846
0,764
0
0.768
0.846
D.300
0.846
0.885
6
0,806
0.79
0.710
0,16
0795
0.647
SentERRANT
0.742
0,747
0.786
0.830
0
0.819
0.813
0.764
0,780
0,841
0.830
0,857
SeatCLEME-dependent (Oun))
0.896
0.844
0.915
0,913
0.806
0.849
0.886
0.742
0.795
0.876
0.921
0.824
0.808
0.835
0,874
0.775
0.819
0.824
0.863
0.79%
0.846
0.291
0.846
SenCCLEME-indqeeeneent (Ours)
0.868
0.657
0.655*
0.876
0,22
0,556
0.81
0,817
0.785
6,31
0.352
0.596
0.758
0.639
0.714
0,775
0.819
0.808
0.846
0.819
0,87
0.525
D.R.E""
01337
0.850
077T5"
0.69
0.740
0.757
033F
"664
SentCIEEMEE-rrennnnttac (Ours)
0.813
0.841
0,682
0,740
0,530
0.8524
0.73]
0.786
0.853
0.8944
0.655
0,702
0.900*
0,921/*
0.6044
0.555
0.60
0.637
0.84
0.891
0.791
0,763
0.756
0.822
SentCIEMEAindepebbettacc (Ouss)
0.830*
0.848
0.363
0.303
0.588
0.544
0.857*
0.890
0.634
0.626
0.7474
0819
Table 9:
Overview of correlations between reference-based metrics and human judgments. We highlight the highest score in bold and the second-highest score with underlines. & We remove unchanged reference sentences for higher correlations due to low-quality annotations. Otherwise, negative correlations are possible.
Metric
AMU
CAMB
CUUI
IITB
INPUT
IPN
NTHU
PKU
POST
RAC
SJTU
UFC
UMC
TP
483
725
607
28
0
52
409
294
508
319
104
36
311
FP
795
1329
985
488
991
697
1152
794
261
14
774
FN
1934
1886
1946
2064
2070
2078
1976
2007
1985
2044
2036
2069
2020
ERRANT
37.79
35.30
38.13
30.11
100.0
9.63
29.21
29
30.60
28.66
28.49
72.00
28.66
R
19.98
27.77
23.78
1.34
0.00
2.44
17.15
12.78
20.38
13.50
4.86
1.71
13.34
FOG
32.08
33.48
34.02
5.68
0.00
6.06
25.61
23.46
27.81
23.40
14.44
7.81
23.31
314
482
379
195
203
213
w/o LW
382
588
477
22
39
330
246
412
254
85
322
216
FP
872
1392
1034
72
$29
975
776
1246
782
292
19
844
w/o
1303
964
788
EN
1182
987
1169
1564
1592
1445
1191
1259
1158
1278
1471
1583
1266
depenedependeet
wro LW
1345
1132
1333
1751
1782
1634
1366
1426
1333
1453
1657
1772
1439
6312
6347
6245
6314
6310
6377
P 26.45
25.74
26.81
19.29
100.0
5.85
21.42
20.06
21.07
20.60
19.02
56.40
20.14
R 20.97
32.84
24.48
1.09
0.00
2.22
18.23
13.39
22.31
13.71
4.45
1.52
14.40
Fas
25.14
26.90
26.31
4.45
0.00
4,41
20.69
18.24
21.31
18.72
11.50
6.85
18.65
F
63.05
41.07
57.60
99.22
106.0
67:45
55.11
63.62
53.26
66.665
8...00
98.70
'67.78
SentCLEMME-deeendee
R
48.87
59.94
51.21
32.37
31.33
07
.9.71
44.84
50.61
44.57
36.06
32.15
44.43
Fos
36.24
32.94
37.39
31.51
31.33
23.15
32.24
32.56
33.34
32.37
31,93
32.30
31.46
TP
318
488
392
17
0
33
272
196
339
204
69
24
214
wio
388
596
487
39
248
255
32
262
FF
864
1382
1016
$29
965
773
1236
781
292
19
843
win L.W
809
1295
948
67
488
897
707
1136
781
272
18
787
928
701
1362
1393
1246
937
12666
1372
1026
CL Clddeeeneeen
w/o LW
1029
778
984
1497
1530
1382
1045
1129
990
1135
1398
1506
1136
TN
6629
6701
6897
6567
6560
6664
6617
6611
6793
6628
6583
6846
(680
26.90
26.11
27.85
19.29
100.0
22.00
20.23
21.50
20.69
56.40
20:22
R
25.83
41.06
30.71
1.25
0.00
2.57
22.52
16.10
27.72
16.59
5.14
1.78
17.23
Foz.
26.61
28.16
28.38
4.97
0.00
4.66
22.10
19.24
22.51
19.71
12.35
7.77
19.54
65.36"
"45.39
60.92
99.22
160.0"
61.31
57.44
63.03"
56626
666.5
85.17"
58.70
63.36
SentCLEMER Inddeeeennnt
R
$7.20
15.15
60.76
36.87
35.29
40.56
58.08
52.20
59.83
52.14
41.31
36.59
51.94
Fos
42,00
39.63
43.76
35.49
35.29
25.90
38.13
37.43
39.38
37.04
36.25
36.48
36.59
Table 10: Detailed evaluation results across 13 GEC systems CoNI.I-2014,

---

Chunk
Chunk 2 Chunk
Chunk 4
Chunk
Chunk 6
Chunk 7
Source
not
their family then
who else
that are
willing to do that
Ref.
If
not
their family then
who else
will be
willing to do that
Ref.
not
their family then
who else
would
willing to do that
Ref. 3
not
from your family then
who else
is
willing to do that
Ref. 4
If
nol
their family then
who else
will be
willing to do that
Ref.
not
their family then
who else
will be
willing do that
Ref.
not
their family
who clse
would
willing to do that
Ref. 7
If
not
their family then
who clse
will be
willing to do that
Ref. 8
If
not
their family,
who clse
is
willing to do that
family do Ref. 10
Ref.
help then
who else
would
willing to do that
not
their family then
who else is
willing to do that
Table 11: A case of correction independence. We apply chunk partition to the source and all the references.
Chunk 1
Chunk 2
Chunk 3
Chunk 4
Source
For not
Ref.
Not for
with
Ref.
Do
car
Ref.
Car for
Ref.
Can not
use
the car
Chunk 1
Chunk 2
Chunk 3
Chunk
Chunk 5
Chunk 6
Source
One person if do n't
have good health
that means
many things
they could lost could be lost
Ref.
If person does n't
have good health
many things
Ref.
If one person does
have good health
that they could that means they could lose
many things
Ref.
person does n't
have good health
many things
Ref.
person does
have good health
that means
many things
they could lost
Chunk
Chunk
Chunk
Chunk
Chunk 5
Chunk
Source
4*
11 7 7
R.
Ref. 1
R
Ref. 2
$
R
4
x.
Ref. 3
4
Chunk
Chunk 2
Chunk
Chunk
Chunk
Chunk
Chunk
Chunk 8
Source
PTEC
18.0
x
Ref.
#1 #98. 22 12
10
Ref.2
.
EE 2
ZUE
J
BFul BFEl
Ref.
J
A
t
Table 12: More cases of chunk partition. These tables are automatically generated by CLEME. The first cases are from JELEG-dev, and the next two cases are from MuCGEC-dev.

---

E Correction Independence
We introduce the term correction independence to describe pair of chunks where the correction of each chunk is not related to the correction of the other, as illustrated in Table 11. Specifically, chunk 2 and chunk 4 are considered correction- dependent because the correction of chunk 2 family do from Ref.9 must be matched with the correction of chunk help then from Ref.9. However, chunk 6 is correction-indepee indeeeddnt with chunk (or 4) since the correction of chunk has no impact on the correction of chunk (or 4).
F More cases
We list more cases in Table 12, which involve JFLEG (Napoles et al., 2017) for English, and MuCGEC (Zhang et al., 2022) for Chinese.


Reviews:
[{"review_id": "dtOU9e3w9V", "reviewer": "Reviewer_XNd5", "summary": "", "questions": "Could you please clarify, not just in your response to the reviewers, but in the paper itself:\n\nA) Can you elaborate why the IAA you get is to be considered good \"given the subjectivity of the task\" in Section 4.2? Why is it subjective to tell whether two edits are dependent of each other or not? Is it not mostly about basic grammar?\n\nB) Can you elaborate on the impact of local vs. less local errors (mentioned above)? How would CLEME manage with more challenging error types?\n\nC) Could you explain more clearly what reference-based and reference-less evaluation mean, mentioned for instance in the introduction?\n\nD) You sort of make it sound as if you have come up with the idea of merging adjacent chunks, but in the ERRANT algorithm there are different merge strategies as well: rule-base, all-split, all-merge, all-equal. Could you describe more clearly your contribution in relation to theirs? (It is not that difficult to guess, but it would be good to elaborate.)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive comments and insightful reviews. Please find below our point-by-point responses:\n\n---\n\n**Q1**: Can you elaborate why the IAA you get is to be considered good \"given the subjectivity of the task\" in Section 4.2? Why is it subjective to tell whether two edits are dependent of each other or not? Is it not mostly about basic grammar?\n\n**A1**: The IAA of our paper is compared with previous studies. [1] attempted to compute IAA at the sentence level. Three raters were asked simply to decide whether 200 sentences were correct or not. The authors reported **IAA of just 0.16, 0.4 and 0.23**. Therefore, we claim in our paper that the IAA of correction independence is **relatively higher**. On the other hand, our annotators may be confused with some hard cases of correction independence, when two chunks are close. More discussion on IAA of GEC can refer to [2].\n\n---\n\n**Q2**: Can you elaborate on the impact of local vs. less local errors (mentioned above)? How would CLEME manage with more challenging error types?\n\n**A2**: If there are \"global\" errors spanning the whole sentence, CLEME, and even other metrics such as ERRANT, will indeed \"collapse\". However, it is an extreme case, and we have not observed similar situations in CoNLL-2014 [3], BEA-dev [4] (not shown in the paper), JFLEG [5] (not shown in the paper) and Chinese MuCGEC [6] (not shown in the paper). **If collapse happens, the quality of the reference set should be checked first**. It is also feasible to detect latent invalid references leading to collapse using CLEME in future work. Given that a source with multiple references, **CLEME can be improved to filter the references that lead to collapse and use the remaining references to evaluate**.\n\n---\n\n**Q3**: Could you explain more clearly what reference-based and reference-less evaluation mean, mentioned for instance in the introduction?\n\n**A3**: Both reference-based and reference-less metrics are distinguished based on **whether they rely on human-annotated reference sets**. The latter metrics usually evaluate GEC systems' output using PLMs. However, **it is also possible to combine reference-based metrics and PLMs, which is one of our future work**.\n\n---\n\n**Q4**: You sort of make it sound as if you have come up with the idea of merging adjacent chunks, but in the ERRANT algorithm there are different merge strategies as well: rule-base, all-split, all-merge, all-equal. Could you describe more clearly your contribution in relation to theirs? (It is not that difficult to guess, but it would be good to elaborate.)\n\n**A4**: This suggestion is highly valuable. All the merge strategies in ERRANT are only available in the single-reference setting. The chunk partition, proposed in our paper, can be treated as **a variant of merge strategies**, which aims to **debias evaluating in the multi-reference setting**. We will elaborate our contribution in relation to theirs in our final version.\n\n---\n\n**Q5**: Figures 1 and 2 (with Figure 1, in particular) is unreadable because of the minimal font used.\n\n**A5**: We acknowledge that the font used in Figures 1 and 2, particularly in Figure 1, makes them difficult to read. We will ensure that these Figures are redrawn using a more readable font in the final version.\n\n---\n\n**References**:\n\n[1] Annotating ESL errors: Challenges and rewards. NAACL Workshop 2010.\n\n[2] How Far are We from Fully Automatic High Quality Grammatical Error Correction? ACL 2015.\n\n[3] The CoNLL-2014 Shared Task on Grammatical Error Correction. ACL Shared Task 2014.\n\n[4] The BEA-2019 Shared Task on Grammatical Error Correction. ACL Workshop 2019.\n\n[5] JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction. EACL 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and can be followed rather easily. The experiments and analysis are thorough and multifaceted. The results are decent in comparison to a handful of benchmark models on a number of standard data sets.\n\nThe problem is relevant. It addresses real challenges observed for existing models. My guess is that multi-reference or multi-hypothesis systems will become more popular with the new opportunities offered by text generation based on large language models (LLMs). The community will need algorithms that take into account multiple parallel references (or hypotheses) and align corresponding spans (chunks) successfully, treating them as dependent or independent of each other, across references. \n", "reasons_to_reject": "Table 2 suggests that CLEME performs decently compared to the baselines, but not phenomenally well. The approach that the authors seem to advocate the most (SentCLEME-independent) works best on only 3 out of 12 data sets (however, being second best on 5 additional sets).\n\nThe independence claim does get strong support from the three annotators individually (> 90% of the cases match the assumption), but strangely enough the annotators seem to disagree which of the cases are such, because the Cohen's kappa values are not that high (Table 3). However, the authors suggest that the IAA statistics are good considering the subjectivity of the task. But I am not sure why that should be so.\n\nThe algorithm proposed by the authors probably works well only when the errors are very local (as mentioned in the introduction section). If there are larger errors involving incorrect word order, my guess is that the approach might \"collapse\", producing one single correction chunk spanning the entire sentence. (However, other metrics, such as ERRANT, do have the same limitations.) It would be interesting to the community to see algorithms and metrics that can cope with more challenging text with multiple errors.\n\nFigures 1 and 2 (with Figure 1, in particular) is unreadable because of the minimal font used. Since it is crucial to be able to read the text, in order to understand these figures and the algorithm, the font size should be bigger. I have been told that footnote size is the smallest acceptable font size for any text in a paper. Reading a paper in digital format does give the reader the opportunity to zoom in on the figure, but that is not good enough. The paper already occupies the maximum length allotted, so cramming in additional info by using very small text in figures, makes one wonder whether this is one way of extending the actual length of the paper beyond the given limits.", "questions_to_authors": "Could you please clarify, not just in your response to the reviewers, but in the paper itself:\n\nA) Can you elaborate why the IAA you get is to be considered good \"given the subjectivity of the task\" in Section 4.2? Why is it subjective to tell whether two edits are dependent of each other or not? Is it not mostly about basic grammar?\n\nB) Can you elaborate on the impact of local vs. less local errors (mentioned above)? How would CLEME manage with more challenging error types?\n\nC) Could you explain more clearly what reference-based and reference-less evaluation mean, mentioned for instance in the introduction?\n\nD) You sort of make it sound as if you have come up with the idea of merging adjacent chunks, but in the ERRANT algorithm there are different merge strategies as well: rule-base, all-split, all-merge, all-equal. Could you describe more clearly your contribution in relation to theirs? (It is not that difficult to guess, but it would be good to elaborate.)", "paper_topic_and_main_contributions": "The paper proposes CLEME, a new metric for the evaluation of grammatical error correction (GEC). The novelty is based on two major claims: 1) When there are multiple references, the alignment of the edits (corrections) should be consistent across all references, not just one at a time, as opposed to an established metric called ERRANT. 2) Edits are often independent of each other; therefore corrections can and should take multiple of the references into account at the same time (see Figures 1 and 2 for an illustration). To support the second claim, the authors perform human evaluation (Sec. 4.2).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Increase the font size in the figures please (as discussed above)."}, "extra_text": {}}, {"review_id": "3TbuDWrLLX", "reviewer": "Reviewer_WBcg", "summary": "", "questions": "Question A: The experimental setup about ♣ in the tables is unclear. In what case is the unchanged reference set excluded? \n\nQuestion B: What are the advantages of CLEME-independent compared to CLEME-dependent?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We attach great importance to your questions and hope to address your concerns.\n\n---\n\n**Q1**: Can CLEME be applied to other GEC datasets, like JFLEG?\n\n**A1**: Yes, it can. We have not considered experiments on other datasets like JFLEG [1] because of **unavailability of human evaluation results**. However, we are able to provide **qualitative results** on these datasets. Below are some evaluation examples from JELEG. **We will enhance our evaluation analysis on other datasets and cite relevant papers in our final version**.\n\n| Sentence | chunk-0 *   | chunk-1 | chunk-2  *  | chunk-3 |\n| -------- | ----------- | ------- | ---------- | --------- |\n| Source   | For not     | use     | car        | .         |\n| Ref. 0   | Not for     | use     | with a car | .         |\n| Ref. 1   | Do not      | use     | in the car | .         |\n| Ref. 2   | Car not for | use     |            | .         |\n| Ref. 3   | Can not     | use     | the car    | .         |\n\n\n| Sentence | chunk-0 *              | chunk-1          | chunk-2  *                   | chunk-3      | chunk-4 *      | chunk-5 |\n| -------- | ---------------------- | ---------------- | ---------------------------- | -------------- | --------------- | --------- |\n| Source   | One person if do n't   | have good health | that means                   | so many things | they could lost | .         |\n| Ref. 0   | If a person does n't   | have good health | ,                            | so many things | could be lost   | .         |\n| Ref. 1   | If one person does not | have good health | , that means they could lose | so many things |                 | .         |\n| Ref. 2   | If one person does n't | have good health | , that means they could lose | so many things |                 | .         |\n| Ref. 3   | If one person does n't | have good health | , that means                 | so many things | they could lost | .         |\n\n\nSimilar to Table 5 in our paper, CLEME can automatically generate visualization of chunk partition results for JFLEG. Since JELEG is built upon fluency correction, we guess CLEME-dependent is more applicable.\n\n---\n\n**Q2**: The experimental setup about ♣ in the tables is unclear. In what case is the unchanged reference set excluded?\n\n**A2**: We exclude the unchanged reference set if we observe **slight or even negative correlations** for some or most metrics. This intervention is only applied to sentence-level metrics since they are more sensitive to noisy reference sets. For example, the NE-Minimal reference set contains many under-corrections since it was annotated by non-experts following the minimal edit principle. We observe slight or negative correlations for all the metrics in the NE-Minimal reference set without intervention. Similarly, we observe slight correlations in the E-Minimal reference set due to the same reason. An exception is BN-10GEC, where we observe a significant drop for SentPT-M^2 and SentCLEME-independent. Further investigation on the **sensitivity of GEC metrics** is part of our future work.\n\n---\n\n**Q3**: What are the advantages of CLEME-independent compared to CLEME-dependent?\n\n**A3**: CLEME-independent is built upon the new evaluation assumption, i.e., **correction independence assumption**. It is a **relaxed variant of correction dependence assumption**, and is particularly applicable when grammatical error density is low. It become apparent that texts written by native speakers generally exhibit a lower grammatical error density than L2-speak datasets (CoNLL-2014). In such cases, correction independence assumption seems to be a promising means to **alleviate the scarcity of human-written references** because different grammatical errors within a sentence are more likely to be independent of each other.\n\n---\n\n**References**\n\n[1] JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction. EACL 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed metric is a simple algorithm and achieves performance comparable to existing methods.\n- The proposed metric design is based on preliminary research, and the experiments and analysis are comprehensive.\n- The manuscript is well written.\n", "reasons_to_reject": "The correction independence assumption is an interesting finding, but it is unclear whether it can be applied to GEC tasks in general. Three datasets are used in this paper, all based on CoNLL-2014 (line 285). CoNLL-2014 is the standard evaluation dataset in GEC, but experiments with other data sets, for example, JFLEG, should also be considered.", "questions_to_authors": "Question A: The experimental setup about ♣ in the tables is unclear. In what case is the unchanged reference set excluded? \n\nQuestion B: What are the advantages of CLEME-independent compared to CLEME-dependent?\n", "paper_topic_and_main_contributions": "This paper proposes a new reference-based evaluation metric for grammatical error correction (GEC), CLEME, which is characterized by the consideration of multiple reference sentences and the calculation of F0.5 at the chunk level. The preliminary study (Section 2) investigates chunk-level boundaries, and the proposed metric is designed based on the analysis. The effectiveness of the metric was evaluated in a comparison experiment with existing reference-based metrics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://aclanthology.org/E17-2037) (Napoles et al., EACL 2017)", "Typos_Grammar_Style_and_Presentation_Improvements": "- line 359: Suggests → suggests"}, "extra_text": {}}, {"review_id": "TM1qd5NNze", "reviewer": "Reviewer_oXwi", "summary": "", "questions": "+ Can you provide more detailed information about the human evaluation?\n+ Can you provide more details about the CLEME toolkit, such as the core pseudo code, the screenshot of the visualization results, etc.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation of our work's contributions. We are honored to have the opportunity to discuss with you.\n\n---\n\n**Q1**: Is CLEME somewhat too strict to count a true positive (TP), which could lead to an underestimation of GEC performance?\n\n**A1**: We acknowledge that CLEME may be too strict in recognizing a TP if a chunk is too long. To alleviate this problem, we have introduced the **length weighting** to encourage the matching of longer chunks.\n\n---\n\n**Q2**: Can more experiments on other languages be conducted?\n\n**A2**: Unfortunately, only CoNLL-2014 datasets are equipped with human ranking, which is crucial for quantitative experiments. However, we will present **qualitative results and analyses** on other multi-reference datasets, including BEA-19 [1], JFLEG [2] and MuCGEC [3] in our final version. **We will also cite relevant papers in our final version**.\n\n---\n\n**Q3**: Can you provide more detailed information about the human evaluation?\n\n**A3**: Yes, we will provide the details about the human evaluation in Section 4.\n\n---\n\n**Q4**: Can you provide more details about the CLEME toolkit, such as the core pseudo code, the screenshot of the visualization results, etc.\n\n**A4**: Absolutely, we will include more information about CLEME  in the final version. We also plan to **open-source a comprehensive GEC evaluation toolkit**, including CLEME, ERRANT, MaxMatch, GLEU, etc.\n\n---\n\n**References**\n\n[1] The BEA-2019 Shared Task on Grammatical Error Correction. ACL Workshop 2019.\n\n[2] JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction. EACL 2017.\n\n[3] MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction. NAACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The paper is well-written and easy to understand.\n+ This paper highlights an important yet under-explored aspect of GEC, specifically, reducing bias in multi-reference evaluation. The authors provide ample preliminary experiments to support their motivations, such as the chunk-based evaluation and the independent correction assumption.\n+ The final metric designed in this study demonstrates strong correlations with human judgments, outperforming most previous metrics, including both reference-based and reference-free approaches. \n\n", "reasons_to_reject": "+ In my opinion, the chunk-based design has a potentially unreasonable aspect. Using the example in Figure 1 as an illustration, if a GEC system correctly deletes *the*, it would receive a true positive (TP) when evaluated using ERRANT, while it would receive a false positive (FP) when evaluated using CLEME. This example suggests that CLEME might be somewhat too strict due to its chunk-based evaluation setting, which could lead to an underestimation of GEC systems' performance.\n+ I suggest the authors conduct more experiments on other languages such as Chinese, which already has some multi-reference datasets, e.g., YACLC and MuCGEC, to further confirm the superiority of their metric.", "questions_to_authors": "+ Can you provide more detailed information about the human evaluation?\n+ Can you provide more details about the CLEME toolkit, such as the core pseudo code, the screenshot of the visualization results, etc.", "paper_topic_and_main_contributions": "This paper introduces the Chunk-Level Multi-Reference Metric (CLEME) to address a significant issue in Grammatical Error Correction (GEC): multi-reference evaluation. By comparing all references and the source input, CLEME converts them into chunk sequences with consistent boundaries. The authors assert that CLEME can eliminate bias in multi-reference GEC evaluation, as the number of true positives (TP) remains equal for all references.\n\nAdditionally, the authors assume that all corrections in the references are independent, which is referred to as the correction independent assumption. This allows them to calculate metrics by comparing hypothesis corrections with all golden corrections across references.\n\nThe final experimental results demonstrate the effectiveness of the proposed CLEME metric in addressing the targeted issue.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "+ MuCGEC (NAACL'22), a Chinese multi-reference dataset along with a span-level metric proposed.", "Typos_Grammar_Style_and_Presentation_Improvements": "+ line 76: should use \\citet"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new metric for the evaluation of grammatical error correction (GEC) that takes advantage of multiple references and operates on chunk level.

The reviewers agree that this is a well-written paper and contains extensive experiments and analyses. The proposed technique is relatively simple but effective. However, the reviewers raised some questions about the proposed methods, in particular the minor improvements compared to existing metrics, the generalizability of the independence assumptions, and the behavior of the proposed method in cases of non-local errors.
Decision: Accept-Main