Venue: NAACL 2021

# Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction

Zhenghao Liu\({}^{1,2}\), Xiaoyuan Yi\({}^{1,2}\), Maosong Sun\({}^{1,3}\)+, Liner Yang\({}^{4}\), Tat-Seng Chua\({}^{5}\)

\({}^{1}\)Department of Computer Science and Technology, Tsinghua University, Beijing, China

Institute for Artificial Intelligence, Tsinghua University, Beijing, China

Beijing National Research Center for Information Science and Technology

\({}^{2}\)State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China

\({}^{3}\)Beijing Academy of Artificial Intelligence

\({}^{4}\)Beijing Language and Culture University, Beijing, China

\({}^{5}\)School of Computing, National University of Singapore, Singapore

Corresponding author: M. Sun (sms@tsinghua.edu.cn)

###### Abstract

Grammatical Error Correction (GEC) aims to correct writing errors and help language learners improve their writing skills. However, existing GEC models tend to produce spurious corrections or fail to detect lots of errors. The quality estimation model is necessary to ensure learners get accurate GEC results and avoid misleading from poorly corrected sentences. Well-trained GEC models can generate several high-quality hypotheses through decoding, such as beam search, which provide valuable GEC evidence and can be used to evaluate GEC quality. However, existing models neglect the possible GEC evidence from different hypotheses. This paper presents the Neural Verification Network (VERNet) for GEC quality estimation with multiple hypotheses. VERNet establishes interactions among hypotheses with a reasoning graph and conducts two kinds of attention mechanisms to propagate GEC evidence to verify the quality of generated hypotheses. Our experiments on four GEC datasets show that VERNet achieves state-of-the-art grammatical error detection performance, achieves the best quality estimation results, and significantly improves GEC performance by reranking hypotheses. All data and source codes are available at [https://github.com/thunlp/VERNet](https://github.com/thunlp/VERNet).

## 1 Introduction

Grammatical Error Correction (GEC) systems primarily aim to serve second-language learners for proofreading. These systems are expected to detect grammatical errors, provide precise corrections, and guide learners to improve their language ability. With the rapid increase of second-language learners, GEC has drawn growing attention from numerous researchers of the NLP community.

Existing GEC systems usually inherit the seq2seq architecture Sutskever et al. (2014) to correct grammatical errors or improve sentence fluency. These systems employ beam search decoding to generate correction hypotheses and rerank hypotheses with quality estimation models from \(K\)-best decoding Kiyono et al. (2019); Kaneko et al. (2020) or model ensemble Chollampatt and Ng (2018) to produce more appropriate and accurate grammatical error corrections. Such models thrive from edit distance and language models Chollampatt and Ng (2018); Chollampatt et al. (2019); Yannakoudakis et al. (2017); Kaneko et al. (2019, 2020). Chollampatt and Ng (2018) further consider the GEC accuracy in quality estimation by directly predicting the official evaluation metric, F\({}_{0.5}\) score.

The \(K\)-best hypotheses from beam search usually derive from model uncertainty Ott et al. (2018). These uncertainties of multi-hypotheses come from model confidence and potential ambiguity of lin

Figure 1: The Grammaticality of Generated Hypotheses. The hypotheses are generated by Kiyono et al. (2019) with beam search decoding. The hypothesis is compared to the source sentence with a BERT based language model and classified into Win (the hypothesis is better), Tie (the hypothesis and source are same) and Loss (the source is better). The ratios of different classes are plotted with different beam search ranks.



guistic variation (Fomicheva et al., 2020), which can be used to improve machine translation performance (Wang et al., 2019b). Fomicheva et al. (2020) further leverage multi-hypotheses to make convinced machine translation evaluation, which is more correlated with human judgments. Their work further demonstrates that multi-hypotheses from well-trained neural models have the ability to provide more hints to estimate generation quality.

For GEC, the hypotheses from the beam search decoding of well-trained GEC models can provide some valuable GEC evidence. We illustrate the reasons as follows.

* _Beam search can provide better GEC results_. The GEC performance of the top-ranked hypothesis and the best one has a large gap in beam search. For two existing GEC systems, Zhao et al. (2019) and Kiyono et al. (2019), the F\({}_{0.5}\) scores of these systems are 58.99 and 62.03 on the CoNLL2014 dataset. However, the F\({}_{0.5}\) scores of the best GEC results of these systems can achieve 73.56 and 76.82.
* _Beam search candidates are more grammatical_. As shown in Figure 1, the hypotheses from well-trained GEC models with beam search usually win the favor of language models, even for these hypotheses ranked to the rear. It illustrates these hypotheses are usually more grammatical than source sentences.
* _Beam search candidates can provide valuable GEC evidence_. As shown in Figure 2, the hypotheses of different beam ranks have almost the same Recall score, which demonstrates all hypotheses in beam search can provide some valuable GEC evidence.

Existing quality estimation models (Chollampatt and Ng, 2018) for GEC regard hypotheses independently and neglect the potential GEC evidence from different hypotheses. To fully use the valuable GEC evidence from GEC hypotheses, we propose the Neural Verification Network (VERNet) to estimate the GEC quality with modeled interactions from multi-hypotheses. Given a source sentence and \(K\) hypothesis sentences from the beam search decoding of the basic GEC model, VERNet establishes hypothesis interactions by regarding \(\langle\text{source}\), hypothesis\(\rangle\) pairs as nodes, and constructing a fully-connected reasoning graph to propagate GEC evidence among multi-hypotheses. Then VERNet proposes two kinds of attention mechanisms on the reasoning graph, _node interaction attention_ and _node selection attention_, to summarize and aggregate necessary GEC evidence from other hypotheses to estimate the quality of tokens.

Our experiments show that VERNet can pick up necessary GEC evidence from multi-hypotheses provided by GEC models and help verify the quality of GEC hypotheses. VERNet helps GEC models to generate more accurate GEC results and benefits most grammatical error types.

## 2 Related Work

The GEC task is designed for automatically proofreading. Large-scale annotated corpora (Mizumoto et al., 2011; Dahlmeier et al., 2013; Bryant et al., 2019) bring an opportunity for building fully data-driven GEC systems.

Existing neural models regard GEC as a natural language generation (NLG) task and usually use sequence-to-sequence architecture (Sutskever et al., 2014) to generate correction hypotheses with beam search decoding (Yuan and Briscoe, 2016; Chollampatt and Ng, 2018). Transformer-based architectures (Vaswani et al., 2017) show their effectiveness in NLG tasks and are also employed to achieve convinced correction results (Grundkiewicz et al., 2019; Kiyono et al., 2019). The copying mechanism is also introduced for GEC models (Zhao et al., 2019) to better align tokens from source sentence to hypothesis sentence. To further accelerate the generation process, some work also comes up with non-autoregressive GEC models and leverages a single encoder to parallelly detect and correct grammatical errors (Awasthi et al., 2019; Malmi et al., 2019; Omelianchuk et al., 2020).

Recent research focuses on two directions to im

Figure 2: The GEC Performance of Generated Hypotenses. The hypotheses generated by Kiyono et al. (2019) are evaluated on the CoNLL2014 dataset. The average scores of Precision and Recall are calculated according to the two annotations of CoNLL2014.

 

prove GEC systems. The first one treats GEC as a low-resource language generation problem and focuses on data augmentation for a grammar sensitive and language proficient GEC system (Junczys-Dowmunt et al., 2018; Kiyono et al., 2019). Various weak-supervision corpora have been leveraged, such as Wikipedia edit history Licharge et al. (2019), Github edit history Hagiwara and Mita (2020) and confusing word set Grundkiewicz et al. (2019). Besides, lots of work generates grammatical errors through generation models or round-trip translation Ge et al. (2018); Wang et al. (2019); Xie et al. (2018). Kiyono et al. (2019) further consider different data augmentation strategies to conduct better GEC pretraining.

Reranking GEC hypotheses from \(K\)-best decoding or GEC model ensemble Hoang et al. (2016); Chollampatt and Ng (2018) with quality estimation models provides another promising direction to achieve better GEC performance. Some methods evaluate if hypotheses satisfy linguistic and grammatical rules. For this purpose, they employ language models Chollampatt and Ng (2018a); Chollampatt et al. (2019) or grammatical error detection (GED) models to estimate hypothesis quality. GED models Rei (2017); Rei and Sogaard (2019) estimate the hypothesis quality on both sentence level Kaneko et al. (2019) and token level Yannakoudakis et al. (2017). Chollampatt and Ng (2018b) further estimate GEC quality by considering correction accuracy. They establish source-hypothesis interactions with the encoder-decoder architecture and learn to directly predict the official evaluation score F\({}_{0.5}\).

The pre-trained language model BERT Devlin et al. (2019) has proven its effectiveness in producing contextual token representations, achieving better quality estimation Kaneko et al. (2019); Chollampatt et al. (2019) and improving GEC performance by fuse BERT representations Kaneko et al. (2020). However, existing quality estimation models regard each hypothesis independently and neglect the interactions among multi-hypotheses, which can also benefit the quality estimation (Fomicheva et al., 2020).

## 3 Neural Verification Network

This section describes Neural Verification Network (VERNet) to estimate the GEC quality with multi-hypotheses, as shown in Figure 3.

Given a source sentence \(s\) and \(K\) corresponding hypotheses \(C=\{c^{1},\ldots,c^{k},\ldots,c^{K}\}\) generated by a GEC model, we first regard each source-hypothesis pair \(\langle s,c^{k}\rangle\) as a node and fully connect all nodes to establish multi-hypothesis interactions. Then VERNet leverages BERT to get the representation of each token in \(\langle s,c^{k}\rangle\) pairs (Sec. 3.1) and conducts two kinds of attention mechanisms to propagate and aggregate GEC evidence from other hypotheses to verify the token quality (Sec. 3.2). Finally, VERNet estimates hypothesis quality by aggregating token level quality estimation scores (Sec. 3.3). Our VERNet is trained end-to-end with supervisions from golden labels (Sec. 3.4).

### Initial Representations for Sentence Pairs

Pre-trained language models, _e.g._BERT Devlin et al. (2019), show their advantages of producing contextual token representations for various NLP tasks. Hence, given a source sentence \(s\) with \(m\) tokens and the \(k\)-th hypothesis \(c^{k}\) with \(n\) tokens, we use BERT to encode the source-hypothesis pair \(\langle s,c^{k}\rangle\)and get its representation \(H^{k}\):

\[H^{k}=\text{BERT}([\text{CLS}]~{}s~{}\text{[SEP]}~{}c^{k}~{}\text{[SEP]}). \tag{1}\]

The pair representation \(H^{k}\) consists of token-level representations, that is,

Figure 3: The Architecture of Neural Verification Network (VERNet). The estimated token (\(c^{k}_{p}\)) and potentially supporting evidence towards \(c^{k}_{p}\) are annotated.

 

\(H^{k}=\{H_{0}^{k},\ldots,H_{m+n+2}^{k}\}\). \(H_{0}^{k}\) denotes the representation of "[CLS]" token.

### Verify Token Quality with Multi-hypotheses

VERNet conducts two kinds of attention mechanisms, _node interaction attention_ and _node selection attention_, to verify the token quality with the verification representation \(V^{k}\) of \(k\)-th node, which learns the supporting evidence towards estimating token quality from multi-hypotheses.

The node interaction attention first summarizes useful GEC evidence from the \(l\)-th node for the fine-grained representation \(V^{l\to k}\) (Sec. 3.2.1). Then node selection attention further aggregates fine-grained representation \(V^{l\to k}\) with score \(\gamma^{l}\) according to each node's confidence (Sec. 3.2.2). Finally, we can calculate the verification representation \(V^{k}\) to verify the token's quality of each node.

#### 3.2.1 Fine-grained Node Representation with Node Interaction Attention

The node interaction attention \(\alpha^{l\to k}\) attentively reads tokens in the \(l\)-th node and picks up supporting evidence towards the \(k\)-th node to build fine-grained node representations \(V^{l\to k}\).

For the \(p\)-th token in the \(k\)-th node, \(w_{p}^{k}\) , we first calculate the node interaction attention weight \(\alpha_{q}^{l\to k}\) according to the relevance between \(w_{p}^{k}\) and the \(q\)-th token in the \(l\)-th node, \(w_{q}^{l}\):

\[\small\begin{split}&\alpha_{q}^{l\to k}=\text{softmax}_{q}(( H_{p}^{k})^{T}\cdot W\cdot H_{q}^{l}),\\ \end{split} \tag{2}\]

where \(W\) is a parameter. \(H_{p}^{k}\) and \(H_{q}^{l}\) are the representations of \(w_{p}^{k}\) and \(w_{q}^{l}\). Then all token representations of \(l\)-th node are aggregated:

\[\small V_{p}^{l\to k}=\sum_{q=1}^{m+n+2}(\alpha_{q}^{l\to k}\cdot H_{ q}^{l}). \tag{3}\]

Based on \(V_{p}^{l\to k}\), we further build the \(l\)-th node fine-grained representation towards the \(k\)-th node, \(V^{l\to k}=\{V_{1}^{l\to k},\ldots,V_{p}^{l\to k},\ldots,V_{ m+n+2}^{l\to k}\}\).

#### 3.2.2 Evidence Aggregation with Node Selection Attention

The node selection attention measures node importance and is used to aggregate supporting evidence from the fine-grained node representation \(V^{l\to k}\) of the \(l\)-th node. We leverage attention-over-attention mechanism Cui et al. (2017) to conduct source \(h^{ls}\) and hypotheses \(h^{lh}\) representations to calculate the \(l\)-th node selection attention score \(\gamma^{l}\). Then we get the node verification representation \(V_{p}^{k}\) with the node selection attention \(\gamma^{l}\).

To calculate the node selection attention \(\gamma^{l}\), we establish an interaction matrix \(M^{l}\) between the source and hypothesis sentences of the \(l\)-th node. Each element \(M_{ij}^{l}\) in \(M^{l}\) is calculated with the relevance between \(i\)-th source token and \(j\)-th hypothesis token (include "[SEP]" tokens):

\[\small M_{ij}^{l}=(H_{i}^{l})^{T}\cdot W\cdot H_{m+1+j}^{l}, \tag{4}\]

where \(W\) is a parameter. Then we calculate attention scores \(\beta_{i}^{ls}\) and \(\beta_{j}^{lh}\) along the source dimension and hypothesis dimension, respectively:

\[\small\beta_{i}^{ls}=\frac{1}{n+1}\sum_{j=1}^{n+1}\text{softmax}_{ i}(M_{ij}^{l}), \tag{5}\] \[\small\beta_{j}^{lh}=\frac{1}{m+1}\sum_{i=1}^{m+1}\text{ softmax}_{ j}(M_{ij}^{l}). \tag{6}\]

Then the representations of source sentence and hypothesis sentence are calculated:

\[\small h^{ls}=\sum_{i=1}^{m+1}\beta_{i}^{ls}\cdot H_{i}^{l},~{}~{}h^{lh}=\sum_ {j=1}^{n+1}\beta_{j}^{lh}\cdot H_{m+1+j}^{l}. \tag{7}\]

Finally, the node selection attention \(\gamma^{l}\) of \(l\)-th node is calculated for the evidence aggregation:

\[\small\gamma^{l}=\text{softmax}(\text{Linear}((h^{ls}\circ h^{lh});h^{ls};h^ {lh})), \tag{8}\]

where \(\circ\) is the element-wise multiplication operator and \(;\) is the concatenate operator.

The node selection attention \(\gamma^{l}\) aggregates evidence for the verification representation \(V_{p}^{k}\) of \(w_{p}^{k}\):

\[\small V_{p}^{k}=\sum_{l=1}^{K}(\gamma^{l}\cdot V_{p}^{l\to k}), \tag{9}\]

where \(V^{k}=\{V_{1}^{k},\ldots,V_{p}^{k},\ldots,V_{m+n+2}^{k}\}\) is the \(k\)-th node verification representation.

### Hypothesis Quality Estimation

For the \(p\)-th token \(w_{p}^{k}\) in the \(k\)-th node, the probability \(P(y|w_{p}^{k})\) of quality label \(y\) is calculated with the verification representation \(V_{p}^{k}\):

\[\small P(y|w_{p}^{k})=\text{softmax}_{y}(\text{Linear}((H_{p}^{k}\circ V_{ p}^{k});H_{p}^{k};V_{p}^{k})), \tag{10}\]

where \(\circ\) is the element-wise multiplication and \(;\) is the concatenate operator. We average all probability \(P(y=1|w_{p}^{k})\) of token level quality estimation as hypothesis quality estimation score \(f(s,c^{k})\) for the pair \(\langle s,c^{k}\rangle\):

\[\small f(s,c^{k})=\frac{1}{n+1}\sum_{p=m+2}^{m+n+2}P(y=1|w_{p}^{k}).\] (11) 



### End-to-end Training

We conduct joint training with token-level supervision. The source labels and hypothesis labels are used, which denote the grammatical quality of source sentences and GEC accuracy of hypotheses.

The cross entropy loss for the \(p\)-th token \(w_{p}^{k}\) in the \(k\)-th node is calculated:

\[L(w_{p}^{k})=\text{CrossEntropy}(y^{*},P(y|w_{p}^{k})), \tag{12}\]

using the ground truth token labels \(y^{*}\).

Then the training loss of VERNet is calculated:

\[L=\frac{1}{K}\frac{1}{m+n+2}\sum_{k=1}^{K}\sum_{p=1}^{m+n+2}L(w_{p}^{k}). \tag{13}\]

## 4 Experimental Methodology

This section describes the datasets, evaluation metrics, baselines, and implementation details.

**Datasets.** We use FCE Yannakoudakis et al. (2011), BEA19 Bryant et al. (2019) and NUCLE Dahlmeier et al. (2013) to construct training and development sets. Four testing scenarios, FCE, BEA19 (Restrict), CoNLL-2014 Ng et al. (2014) and JFLEG Napoles et al. (2017), are leveraged to evaluate model performance. Detailed data statistics are presented in Table 1. We do not incorporate additional training corpora for fair comparison.

**Basic GEC Model**. To generate correction hypotheses, we take one of the state-of-the-art autoregressive GEC systems Kiyono et al. (2019) as our _basic GEC model_ and keep the same setting. The beam size of our baseline model is set to 5 Kiyono et al. (2019), and all these beam search hypotheses are reserved in our experiments.

We generate quality estimation labels for tokens in both source sentences and hypothesis sentences with ERRANT Bryant et al. (2017); Felice et al. (2016), which indicate grammatical correctness and GEC accuracy, respectively. As shown in Table 2, ERRANT annotates edit operations delete, insert, and replace) towards the ground truth corrections. In terms of such annotations, each token is labeled with correct (1) or incorrect (0).

**Evaluation Metrics.** We introduce the evaluation metrics in three tasks: token quality estimation, sentence quality estimation, and GEC.

To evaluate the model performance of token-level quality estimation, we employ the same evaluation metrics from previous GED models Rei et al. (2017); Rei and Sogaard (2019); Yannakoudakis et al. (2017), including Precision, Recall, and F\({}_{0.5}\). F\({}_{0.5}\) is our primary evaluation metric.

For the evaluation of sentence-level quality estimation, we employ the same evaluation metrics from the previous quality estimation model Chollampatt and Ng (2018), including two evaluation scenarios: (1) GEC evaluation metrics for the hypothesis that reranked top-1 and (2) Pearson Correlation Coefficient (PCC) between reranking scores and golden scores (F\({}_{0.5}\)) for all hypotheses.

To evaluate GEC performance, we adopt BLEU Napoles et al. (2015) to evaluate model performance on the JFLEG dataset. The official tool ERRANT of the BEA19 shared task Bryant et al. (2019) is used to calculate Precision, Recall, and F\({}_{0.5}\) scores for other datasets. For the CoNLL-2014 dataset, the M\({}^{2}\) evaluation Dahlmeier and Ng (2012) is also adopted as our main evaluation.

**Baselines.** BERT-fuse (GED) Kaneko et al. (2020) is compared in our experiments, which trains BERT with the GED task and fuses BERT representations into the Transformer. For quality estimation, we consider two groups of baseline models in our experiments, and more details of these models can be found in Appendices A.1.

(1) _BERT based language models._ We employ three BERT based language models to estimate the quality of hypotheses. BERT-LM Chollamputt et al. (2019) measures hypothesis quality with the perplexity of the language model. BERT-GQE Kaneko et al. (2019) is trained with annotated GEC data and estimates if the hypothesis has grammatical errors. We also conduct BERT-GED (SRC) that predicts token level grammar indicator

\begin{table}
\begin{tabular}{l|c c c} \hline Dataset & Training & Development & Test \\ \hline FCE & 28,350 & 2,191 & 2,695 \\ BEA19 & 34,308 & 4,384 & 4,477 \\ NUCLE & 57,151 & - & - \\ CoNLL-2014 & - & - & 1,312 \\ JFLEG & - & - & 747 \\ \hline Total & 119,809 & 6,575 & 9,231 \\ \hline \end{tabular}
\end{table}
Table 1: Data Statistics.

\begin{table}
\begin{tabular}{c|c c c} \hline \multirow{2}{*}{Sentence} & \multicolumn{2}{c}{The i a z Mobile phone is a marvelous} \\  & invention to o\({}_{9}\) charge \({}_{10}\) the world \({}_{12}\) [SEP] \\ \hline \multirow{3}{*}{Correction} & Operation & Span & Edit \\ \cline{2-4}  & Delete & 1,2 & - \\ \cline{1-1}  & Replace & 9,10 & change \\ \cline{1-1}  & Insert & 12,12 & . \\ \hline \end{tabular}
\end{table}
Table 2: An Example of Token Label Annotation. All sentences are annotated with ERRANT according to the golden correction. The words in red color are labeled as incorrect (0) and others are labeled as correct (1). The “[SEP]” token denotes the end of the sentence.



labels, which is inspired by GED models (Yannakoudakis et al., 2017). BERT shows significant improvement compared to LSTM based models for the GED task (Appendices A.2). Hence the LSTM based models are neglected in our experiments.

(2) _GEC accuracy estimation models._ These models further consider the source-hypothesis interactions to evaluate GEC accuracy. We take a strong baseline NQE (Chollampatt and Ng, 2018) in experiments. NQE employs the encoder-decoder (predictor) architecture to encode source-hypothesis pairs and predicts F\({}_{0.5}\) score with the estimator architecture. All their proposed architectures, NQE (CC), NQE (RC), NQE (CR), and NQE (RR) are compared. For NQE (XY), X indicates the predictor architecture, and Y indicates the estimator architecture. X and Y can be recurrent (R) or convolutional (C) neural networks. In addition, we also employ BERT to encode source-hypothesis pairs and then predict the F\({}_{0.5}\) score to implement the BERT-QE model. We also come up with two baselines, BERT-GED (HYP) and BERT-GED (JOINT). They leverage BERT to encode source-hypothesis pairs and are supervised with the token-level quality estimation label. BERT-GED (HYP) is trained with the supervision of hypotheses, and BERT-GED (JOINT) is supervised with labels from both source and hypothesis sentences.

**Implementation Details.** In all experiments, we use the base version of BERT (Devlin et al., 2019) and ELECTRA (Clark et al., 2020). BERT is a widely used pretrained language model and trained with the mask language model task. ELECTRA is trained with the replaced token detection task and aims to predict if the token is original or replaced by a BERT based generator during pretraining. ELECTRA is a discriminator based pretrained language model and is more like the GED task. We regard BERT as our main model for text encoding and leverage ELECTRA to evaluate the generalization ability of our model.

Both BERT and ELECTRA inherit huggingface's PyTorch implementation (Wolf et al., 2020). Adam (Kingma and Ba, 2015) is utilized for parameter optimization. We set the max sentence length to 120 for source and hypothesis sentences, learning rate to 5e-5, batch size to 8, and accumulate step to 4 during training.

For hypothesis reranking, we leverage the learning-to-rank method, Coordinate Ascent (CA) (Metzler and Croft, 2007), to aggregate the ranking features and basic GEC score to conduct the ranking score. We assign the hypotheses with the highest F\({}_{0.5}\) score as positive instances and the others as negative ones. The Coordinate Ascent method is implemented by RankLib1.

Footnote 1: [https://sourceforge.net/p/lemur/wiki/RankLib/](https://sourceforge.net/p/lemur/wiki/RankLib/)

## 5 Evaluation Results

We conduct experiments to study the performance of VERNet from three aspects: token-level quality estimation, sentence-level quality estimation, and the VERNet's effectiveness in GEC models. Then we present the case study to qualitatively analyze the effectiveness of the proposed two types of attention in VERNet.

### Performance of Token Level Quality Estimation

We first evaluate VERNet's effectiveness on token-level quality estimation. BERT-GED (SRC) is the previous state-of-the-art GED model (Kaneko and Komachi, 2019). Additional two variants, HYP and JOINT, of BERT-GED are conducted as baselines by considering the first-ranked GEC hypothesis in beam search decoding.

As shown in Table 3, there are two scenarios, _source_ and _hypothesis_, are conducted to evaluate model performance. The _source scenario_ evaluates the ability of grammaticality quality estimation, which is the same as GED models (Rei and Sogaard, 2019). The _hypothesis scenario_ tests the quality estimation ability on GEC accuracy.

For the _source scenario_, BERT-GED (JOINT) outperforms BERT-GED (SRC) and illustrates that the GEC result can help estimate the grammaticality quality of source sentences. For the _hypothesis scenario_, BERT-GED (JOINT) shows better performance than BERT-GED (HYP), which thrives from the supervisions from source sentences. For _both scenarios_, BERT-VERNet shows further improvement compared with BERT-GED (JOINT). Such improvements demonstrate that various GEC evidence from multiple hypotheses benefits the token-level quality estimation.

Moreover, the detection style pre-trained model ELECTRA (Clark et al., 2020) is also used as our sentence encoder. VERNet is boosted a lot on all scenarios and datasets, which illustrates the strong ability of ELECTRA in token-level quality estimation and the generalization ability of VERNet.

 



### Performance of Sentence Level Quality Estimation

In this part, we evaluate VERNet's performance on sentence-level quality estimation by reranking hypotheses from beam search decoding.

Baselines can be divided into two groups: language model based and GEC accuracy based quality estimation models. The former focuses on grammaticality and fluency, including BERT-LM, BERT-GQE and BERT-GED (SRC). The others focus on estimating the GEC accuracy, including NQE, BERT-QE, BERT-GED (HYP)/(JOINT).

As shown in Table 4, we find that language model based quality estimation prefers higher recall but lower precision, which leads to more redundant corrections. Only considering grammaticality is insufficient since such unnecessary correction suggestions may mislead users. By contrast, GEC accuracy based quality estimation models get much better Precision and F\({}_{0.5}\), and provide more precise feedback for users. Furthermore, BERT-GED (HYP) outperforms BERT-QE, manifesting that token-level supervisions provide finer-granularity signals to help the model better distinguish subtle differences among hypotheses. VERNet outperforms all baselines, which supports our claim that multi-hypotheses from beam search provide valuable GEC evidence and help conduct more effective quality estimation for generated GEC hypotheses.

### VERNet's Effectiveness in GEC Models

This part explores the effectiveness of VERNet on improving GEC models. We conduct VERNet\({}^{\dagger}\) by aggregating scores from the basic GEC model and VERNet for hypothesis reranking.

As shown in Table 5, two baseline models are compared in our experiments, Basic GEC [19] and BERT-fuse (GED) [23]. Compared to BERT-fuse (GED), BERT-VERNet\({}^{\dagger}\) achieves comparable performance on CoNLL-2014 and more improvement on BEA19. It demonstrates that reranking hypotheses with VER

\begin{table}
\begin{tabular}{l|c c c c c c c c|c c c c c} \hline  & \multicolumn{6}{c|}{CoNLL-2014 (M\({}^{\ddagger}\))} & \multicolumn{6}{c|}{FCE} & \multicolumn{3}{c|}{BEA19} & \multicolumn{2}{c}{FLEG} \\ \cline{2-17} Model & P & R & F\({}_{0.5}\) & PCC & \multirow{2}{*}{P} & \multirow{2}{*}{R} & \multirow{2}{*}{F\({}_{0.5}\)} & \multirow{2}{*}{P} & \multirow{2}{*}{R} & \multirow{2}{*}{F\({}_{0.5}\)} & \multirow{2}{*}{P} & \multirow{2}{*}{R} & \multirow{2}{*}{F\({}_{0.5}\)} & \multirow{2}{*}{GLEU} & \multirow{2}{*}{PCC} \\ \cline{2-17} \cline{8-17}  & QE & R & & & & & & & & & & & & & \\ \hline NQE (RR) & 61.38 & 33.03 & 52.39 & 23.43 & 6.62 & 51.43 & 30.36 & 45.16 & 28.74 & 57.22 & 46.33 & 54.65 & 55.90 & 1.29 \\ NQE (RC) & 60.09 & 33.11 & 51.67 & 24.12 & 5.52 & 53.97 & 31.35 & 47.17 & 31.20 & 57.87 & 47.24 & 55.37 & 56.91 & 1.66 \\ NQE (CR) & 62.52 & 35.24 & 54.14 & 24.80 & 9.12 & 51.77 & 31.46 & 45.85 & 30.69 & 57.92 & 47.43 & 55.47 & 56.92 & 6.48 \\ NQE (CC) & 60.62 & 35.77 & 53.23 & 22.94 & 8.39 & 50.21 & 32.09 & 45.11 & 29.23 & 56.83 & 49.47 & 55.19 & 57.22 & 7.68 \\ \hline BERT-LM & 52.82 & 49.59 & 52.14 & 3.47 & 17.62 & 36.97 & 43.42 & 38.10 & 8.59 & 46.32 & 64.05 & 49.03 & 59.72 & 26.85 \\ BERT-GPE & 52.67 & 50.39 & 52.19 & 2.56 & 14.54 & 36.05 & 43.53 & 37.13 & 10.18 & 46.15 & 64.01 & 48.88 & 60.17 & 29.05 \\ BERT-GED (SRC) & 52.98 & **52.07** & 52.79 & 3.78 & 20.56 & 37.58 & **45.81** & 38.98 & 12.71 & 47.15 & **65.09** & 49.90 & 60.32 & 27.28 \\ \hline BERT-QE & 62.24 & 38.27 & 55.51 & 22.85 & 12.17 & 52.01 & 36.89 & 48.07 & 33.84 & 58.63 & 54.19 & 57.69 & 59.73 & 26.16 \\ BERT-GED (HYP) & 68.90 & 34.53 & 57.36 & 30.06 & 16.79 & 57.21 & 36.03 & 51.19 & 43.84 & 68.18 & 53.85 & 64.73 & 60.00 & 29.90 \\ BERT-GED (Joint) & 69.33 & 36.02 & 85.81 & 26.26 & 16.28 & 58.53 & 37.24 & 52.53 & 45.08 & 66.80 & 55.09 & 64.07 & 60.49 & 33.03 \\ \hline BERT-VERNet & 68.75 & 40.26 & 60.22 & 31.02 & 22.75 & 58.32 & 39.99 & 53.42 & 47.19 & 68.66 & 58.60 & 65.02 & 61.36 & 36.98 \\ ELECTRA-VERNet & **69.97** & 42.12 & **61.80** & **37.18** & **28.77** & **58.77** & 41.86 & **54.37** & **48.12** & **69.09** & 60.91 & **67.28** & **61.61** & **38.63** \\ \hline \end{tabular}
\end{table}
Table 4: Performance of Sentence Level Quality Estimation. The ranked top-1 hypothesis is used to calculate GEC metrics. NQE [19] uses RNN or CNN models for GEC quality estimation. BERT-LM [19] measures perplexity without fine-tuning. BERT-GQE [23] and BERT-GED (SRC) are supervised with sentence-level and token-level labels from source sentences to estimate grammatical quality, respectively. NQE and BERT-QE encode \(\langle\text{source, hypothesis}\rangle\) pairs and directly predict F\({}_{0.5}\) score. BERT-GED (HYP) and BERT-GED (JOINT) encode the \(\langle\text{source, hypothesis}\rangle\) pairs to estimate the quality of generated tokens.

\begin{table}
\begin{tabular}{l|l|l l l l|l l l l|l l l} \hline  & \multicolumn{2}{c|}{Model} & \multicolumn{6}{c|}{FCE test set} & \multicolumn{6}{c|}{CoNLL-2014 ann. 1} & \multicolumn{6}{c}{CoNLL-2014 ann. 2} \\ \cline{3-13}  & & P & R & F\({}_{0.5}\) & P & R & F\({}_{0.5}\) & P & R & F\({}_{0.5}\) & R & F\({}_{0.5}\) \\ \hline \multirow{4}{*}{Source} & BERT-GED (SRC) & 74.22 & 43.34 & 64.97 & 59.84 & 27.11 & 48.20 & 77.94 & 25.02 & 54.77 \\  & BERT-GED (JOINT) & 75.62 & 44.44 & 66.32 & 60.79 & 27.33 & 48.83 & 77.42 & 25.23 & 54.77 \\  & BERT-VERNet & **81.53** & 45.71 & 70.48 & **62.64** & 30.62 & 51.80 & **82.25** & 28.49 & 59.71 \\  & ELECTRA-VERNet & 80.94 & **50.51** & **72.24** & 62.50 & **35.61** & **54.30** & 81.69 & **32.97** & **63.06** \\ \hline \multirow{4}{*}{Hypothesis} & BERT-GED (HYP) & 80.27 & 40.58 & 67.14 & 74.28 & 34.20 & 60.17 & 66.49 & 27.68 & 51.93 \\  & BERT-GED (JOINT) & 76.71 & 46.94 & 68.07 & 71.15 & 38.30 & 60.73 & 64.79 & 31.52 & 53.50 \\ \cline{1-1}  & BERT-VERNet & **81.85** & **44.27** & 69.97 & **76.03** & 34.02 & 60.97 & 71.79 & 29.04 & 55.46 \\ \cline{1-1}  & ELECTRA-VERNet & 80.62 & **49.16** & **71.48** & 74.80 & **39.26** & **63.33** & **72.55** &

Net provides an effective way to improve basic GEC model performance without changing the Transformer architecture. R2L models incorporate four right-to-left Transformer models to improve GEC performance. However, these R2L models are not available. ELECTRA-VERNet\({}^{\dagger}\) incorporates only one model and achieves comparable performance on BEA19 and JFLEG.

Figure 4 presents VERNet\({}^{\dagger}\)'s performance on different grammatical error types. We plot the F\({}_{0.5}\) scores of both basic GEC model and VERNet\({}^{\dagger}\) on BEA19. VERNet\({}^{\dagger}\) achieves improvement on most types and performs significantly better for word morphology and word usage errors, such as Noun Inflection (NOUN:INFL) and Pronoun (PRON). Such results illustrate that VERNet\({}^{\dagger}\) is able to leverage clues learned from multi-hypotheses to verify the GEC quality. However, we also find that VERNet\({}^{\dagger}\) discounts GEC performance on a few error types, _e.g._, Contraction (CONTR). The annotation biases may cause such a decrease in CONTR errors. For example, for both "n't" and "not", they are both right according to grammaticality, but annotators usually come up with different corrections with different GEC standards.

### Case Study

We select one case from CoNLL-2014 and visualize node interaction and node selection attention weights to study what VERNet learns from multi-hypotheses of beam search, as shown in Figure 5.

Given a source sentence, "Do one who suffered from this disease keep it a secret of infrom their relatives ?", and its five hypotheses from the Basic GEC Model, we plot the node interaction attention weights towards the word "suffers" in the hypothesis of node 2, which is assigned more higher score by BERT-VERNet. The word usage "suffers" is

Figure 4: Model Performance of Different Grammatical Error Types on BEA19. VERNet\({}^{\dagger}\) reranks hypotheses with the aggregated score of basic GEC model and VERNet. All types are from ERRANT Bryant et al. (2017).

\begin{table}
\begin{tabular}{l|c c c c c c|c c c c|c c c} \hline \multirow{3}{*}{Model} & \multicolumn{3}{c|}{CoNLL-2014 (M\({}^{2}\))} & \multicolumn{3}{c|}{CoNLL-2014} & \multicolumn{3}{c|}{PCE} & \multicolumn{3}{c|}{BEA19} & JFLEG \\  & P & R & F\({}_{0.5}\) & P & R & F\({}_{0.5}\) & P & R & F\({}_{0.5}\) & P & R & F\({}_{0.5}\) & BLEU \\ \hline Basic GEC & 68.59 & 44.87 & 62.03 & 64.26 & 43.59 & 58.69 & 55.11 & 41.61 & 51.75 & 66.20 & **61.48** & 65.20 & 61.00 \\ Basic GEC w. R2L\({}^{*}\) & 72.4 & 46.1 & 65.0 & - & - & - & - & - & **74.7** & 56.7 & **70.2** & 61.4 \\ BERT-fuse (GED) & 69.2 & 45.6 & 62.6 & - & - & - & - & - & - & 67.1 & 60.1 & 65.6 & 61.3 \\ BERT-fuse (GED) w. R2L\({}^{*}\) & **72.6** & **46.4** & **65.2** & - & - & - & - & - & - & 72.3 & 61.4 & 69.8 & **62.0** \\ \hline BERT-VERNet\({}^{\dagger}\) (Top2) & 69.98 & **43.69** & 62.47 & 65.62 & **41.98** & 58.98 & 58.57 & 41.53 & 54.13 & 68.42 & **60.32** & 66.63 & 61.17 \\ BERT-VERNet\({}^{\dagger}\) (Top3) & 70.49 & 43.16 & **62.57** & 65.92 & 41.22 & 58.86 & 59.20 & 41.53 & 54.55 & 69.03 & 60.20 & 67.06 & **61.24** \\ BERT-VERNet\({}^{\dagger}\) (Top4) & **70.79** & 42.72 & 62.56 & **66.65** & 40.94 & **59.21** & 59.55 & **41.55** & 54.80 & **69.43** & 60.17 & **67.36** & 61.16 \\ BERT-VERNet\({}^{\dagger}\) (Top5) & 70.60 & 42.50 & 62.36 & 66.41 & 40.74 & 58.98 & **59.68** & 41.48 & **54.86** & 69.39 & 60.12 & 67.32 & 61.10 \\ \hline ELECTTA-VERNet\({}^{\dagger}\) (Top2) & 71.21 & **44.24** & 63.47 & 66.95 & **42.97** & 60.22 & 58.31 & 41.97 & 54.09 & 69.27 & 61.22 & 67.50 & 61.60 \\ ELECTTA-VERNet\({}^{\dagger}\) (Top3) & **71.87** & 44.13 & **63.84** & **67.51** & 42.38 & **60.35** & 59.02 & 41.99 & 54.59 & 70.64 & 61.78 & 68.67 & 61.80 \\ ELECTTA-VERNet\({}^{\dagger}\) (Top4) & 71.85 & 43.81 & 63.69 & 67.48 & 42.19 & 60.25 & 59.65 & 42.12 & 55.07 & **70.96** & **62.03** & **68.98** & **62.05** \\ ELECTTA-VERNet\({}^{\dagger}\) (Top5) & 71.58 & 43.57 & 63.43 & 67.15 & 42.10 & 60.01 & **59.95** & **42.19** & **55.29** & 70.79 & 61.74 & 68.77 & 62.07 \\ \hline \end{tabular}
\end{table}
Table 5: Performance of Hypothesis Reranking. BERT/ELECTRA-VERNet\({}^{\dagger}\) aggregates the scores of Basic GEC Model Kiyono et al. (2019) and VERNet for hypothesis reranking with Coordinate Ascent. BERT-fuse (GED) Kaneko et al. (2020) is the Transformer model that fuses BERT representations. \({}^{*}\)Note that R2L models incorporate four right-to-left Transformer models that are trained with unpublished data and these models are not supplied in their open source codes, thus these results are hard to reimplement.

 

more appropriate than "suffered" according to the context.

The node interaction attention accurately picks up the associated tokens "Does" from nodes 1, 3, and 4, and "suffers" from node 5. "Does" and "suffers" indicate the present tense and provide sufficient evidence to verify the quality of "suffers" in node 2. For node selection attention, the hypothesis (node 2) shares more attention than other nodes, which is more appropriate than other hypotheses. It demonstrates that the node attention is effective to select high-quality corrections with the source-hypothesis interactions.

The attention patterns are intuitive and effective, which further demonstrates VERNet's ability to well model the interactions of multi-hypotheses for better quality estimation.

## 6 Conclusion and Future Work

This paper presents VERNet for GEC quality estimation with multi-hypotheses. VERNet models the interactions of multiple hypotheses by building a reasoning graph, and then extracts clues with two kinds of attention: _node selection attention_ and _node interaction attention_. They summarize and aggregate GEC evidence from multi-hypotheses to verify the quality of tokens. Experiments on four datasets show that VERNet achieves the state-of-the-art GED and quality estimation performance, and improves one published state-of-the-art GEC system. In the future, we will explore the impact of different kinds of hypotheses used in VERNet.

## Acknowledgments

We thank the reviewers and Shuo Wang for their valuable comments and advice. This research is mainly supported by Science & Tech Innovation 2030 Major Project "New Generation AI" (Grant no. 2020AAA0106500) as well as supported in part by a project from Shanghai-Tsinghua International Innovation Center and the funds of Beijing Advanced Innovation Center for Language Resources under Grant TYZ19005.

## References

* Awasthi et al. (2019) Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Parallel iterative edit models for local sequence transduction. In _Proceedings of EMNLP_, pages 4260-4270.
* Bryant et al. (2019) Christopher Bryant, Mariano Felice, Oistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 52-75.
* Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In _Proceedings of ACL_, pages 793-805.
* Chollampatt and Ng (2018) Shamil Chollampatt and Hwee Tou Ng. 2018a. A multilayer convolutional encoder-decoder neural network for grammatical error correction. In _Proceedings of AAAI_, pages 5755-5762.
* Chollampatt and Ng (2018) Shamil Chollampatt and Hwee Tou Ng. 2018b. Neural quality estimation of grammatical error correction. In _Proceedings of EMNLP_, pages 2528-2539.
* Chollampatt et al. (2019) Shamil Chollampatt, Weiqi Wang, and Hwee Tou Ng. 2019. Cross-sentence grammatical error correction. In _Proceedings of ACL_, pages 435-445.
* Chollampatt et al. (2019)

Figure 5: Visualization of Attention Weight. Each node is the concatenation of the source sentence (with [SEP]) and a corresponding hypothesis sentence (with [SEP]). The selected node by BERT-VERNet is annotated (**Node2**). The node selection attention assigned to each hypothesis is annotated with dark orange. The node interaction attention towards the edited token “suffers” in the second node is also plotted. Darker red indicates higher attention weights.

 

* Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In _Proceedings of ICLR_.
* Cui et al. (2017) Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In _Proceedings of ACL_, pages 593-602.
* Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In _Proceedings of NAACL-HLT_, pages 568-572.
* Dahlmeier et al. (2013) Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner English: The NUS corpus of learner English. In _Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 22-31.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL-HLT_, pages 4171-4186.
* Felice et al. (2016) Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In _Proceedings of COLING_, pages 825-835.
* Fomicheva et al. (2020) Marina Fomicheva, Lucia Specia, and Francisco Guzman. 2020. Multi-hypothesis machine translation evaluation. In _Proceedings of ACL_, pages 1218-1232.
* Ge et al. (2018) Tao Ge, Furu Wei, and Ming Zhou. 2018. Fluency boost learning and inference for neural grammatical error correction. In _Proceedings of ACL_, pages 1055-1065.
* Grundkiewicz et al. (2019) Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 252-263.
* Hagiwara and Mita (2020) Masato Hagiwara and Masato Mita. 2020. GitHub typo corpus: A large-scale multilingual dataset of misspellings and grammatical errors. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 6761-6768.
* Hoang et al. (2016) Duc Tam Hoang, Shamil Chollampatt, and Hwee Tou Ng. 2016. Exploiting n-best hypotheses to improve an SMT approach to grammatical error correction. In _Proceedings of IJCAI_, pages 2803-2809.
* Junczys-Dowmunt et al. (2018) Marcin Juneczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In _Proceedings of NAACL-HLT_, pages 595-606.
* Kaneko et al. (2019) Masahiro Kaneko, Kengo Hotate, Satoru Katsumata, and Mamoru Komachi. 2019. TMU transformer system using BERT for re-ranking at BEA 2019 grammatical error correction on restricted track. In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 207-212.
* Kaneko and Komachi (2019) Masahiro Kaneko and Mamoru Komachi. 2019. Multimodal multi-layer attention to deep language representations for grammatical error detection. _Computacion yistemas_, (3).
* Kaneko et al. (2020) Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In _Proceedings of ACL_, pages 4248-4254.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of ICLR_.
* Kiyono et al. (2019)Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In _Proceedings of EMNLP_, pages 1236-1242.
* Lichtrarge et al. (2019) Jared Lichtrarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019. Corpora generation for grammatical error correction. In _Proceedings of NAACL-HLT_, pages 3291-3301.
* Malmi et al. (2019) Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag, realize: High-precision text editing. In _Proceedings of EMNLP_, pages 5054-5065.
* Metzler and Croft (2007) Donald Metzler and W Bruce Croft. 2007. Linear feature-based models for information retrieval. _Information Retrieval_.
* Mizumoto et al. (2011)Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In _Proceedings of IJCNLP_, pages 147-155.
* Napoles et al. (2015) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In _Proceedings of ACL_, pages 588-593.
* Napoles et al. (2017) Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In _Proceedings of EACL_, pages 229-234.
* Ng et al. (2014) Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadwinnoto, Raymond Hendry Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_, pages 1-14 

Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskiy. 2020. GECToR - grammatical error correction: Tag, not rewrite. In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 163-170.
* Ott et al. (2018) Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In _Proceedings of ICML_, pages 3953-3962.
* Rei (2017) Marek Rei. 2017. Semi-supervised multitask learning for sequence labeling. In _Proceedings of ACL_, pages 2121-2130.
* Rei and Sogaard (2019) Marek Rei and Anders Sogaard. 2019. Jointly learning to label sentences and tokens. In _Proceedings of AAAI_, pages 6916-6923.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of NIPS_, pages 3104-3112.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of NIPS_, pages 5998-6008.
* Wang et al. (2019a) Liang Wang, Wei Zhao, Ruoyu Jia, Sujian Li, and Jingming Liu. 2019a. Denoising based sequence-to-sequence pre-training for text generation. In _Proceedings of EMNLP_, pages 4003-4015.
* Wang et al. (2019b) Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019b. Improving back-translation with uncertainty-based confidence estimation. In _Proceedings of EMNLP_, pages 791-802.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of EMNLP_, pages 38-45.
* Xie et al. (2018) Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018. Noising and denoising natural language: Diverse backtranslation for grammar correction. In _Proceedings of NAACL-HLT_, pages 619-628.
* Yannakoudakis et al. (2011) Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In _Proceedings of ACL_, pages 180-189.
* Yannakoudakis et al. (2017) Helen Yannakoudakis, Marek Rei, Oistein E. Andersen, and Zheng Yuan. 2017. Neural sequence-labelling models for grammatical error correction. In _Proceedings of EMNLP_, pages 2795-2806.
* Yuan and Briscoe (2016) Zheng Yuan and Ted Briscoe. 2016. Grammatical error correction using neural machine translation. In _Proceedings of NAACL-HLT_, pages 380-386.
* Zhao et al. (2019) Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In _Proceedings of NAACL-HLT_, pages 156-165.

 

Appendices

### Model Details of Sentence Quality Estimation Score Calculation

This part describes the details of sentence score calculation of BERT based quality estimation models.

Given a source sentence \(s\) with \(m\) tokens and \(k\)-th hypothesis \(c^{k}\) with \(n\) tokens, we can get the representation \(H^{k}\) of the \(k\)-th \(\langle\text{source,hypothesis}\rangle\) sentence pair through BERT:

\[H^{k}=\textsc{BERT}([CLS]~{}s~{}\textrm{[SEP]}~{}c^{k}~{}\textrm{[SEP]}), \tag{14}\]

or only the representation \(\mathcal{H}^{k}\) of the \(k\)-th hypothesis through BERT:

\[\mathcal{H}^{k}=\textsc{BERT}([CLS]~{}c^{k}~{}\textrm{[SEP]}). \tag{15}\]

The "[CLS]" representations are \(H_{0}^{k}\) and \(\mathcal{H}_{0}^{k}\).

**BERT-LM.** We mask tokens in the \(k\)-th hypothesis sentence \(c^{k}\) and calculate the Perplexity of the \(k\)-th hypothesis sentence:

\[f_{\textsc{LM}}(c^{k})=-\textsc{PPL}(\mathcal{H}_{1:n}^{k}). \tag{16}\]

**BERT-GQE.** BERT-GQE uses the "[CLS]" representation \(\mathcal{H}_{0}^{k}\) of \(k\)-th hypothesis to estimate the sentence quality with the probability \(P(y_{s}|c^{k})\):

\[P(y_{s}|c^{k})=\textsc{softmax}_{y_{s}}(W\cdot\mathcal{H}_{0}^{k}), \tag{17}\]

where \(W\) is the parameter and the label \(y_{s}\) is categorized into two groups: correct (\(y_{s}=1\)) and incorrect (\(y_{s}=0\)).

Then the sentence-level quality estimation score of hypothesis \(c^{k}\) is calculated:

\[f_{\textsc{GQE}}(c^{k})=P(y_{s}=1|c^{k}). \tag{18}\]

**BERT-QE.** BERT-QE uses the "[CLS]" representation \(H_{0}^{k}\) of \(k\)-th \(\langle\text{source, hypothesis}\rangle\) sentence pair to estimate the quality of GEC hypothesis:

\[f_{\textsc{QE}}(s,c^{k})=\textsc{sigmod}(W\cdot H_{0}^{k}), \tag{19}\]

where \(W\) is the parameter. The quality estimation score \(f_{\textsc{QE}}(s,c^{k})\) of BERT-QE is trained to approximate the F\({}_{0.5}\) score of the \(k\)-th hypothesis \(c^{k}\).

**BERT-GED.** Take BERT-GEM (HYP) as an example, it uses the hypothesis representation \(H_{m+2:m+n+2}^{k}\) of the \(k\)-th \(\langle\text{source, hypothesis}\rangle\) sentence pair to estimate the quality of GEC hypothesis. Note that the "[SEP]" token is also used in BERT-GED to denote the end of the sentence.

We calculate the probability of token quality estimation label \(y\) for the \(i\)-th token \(w_{i}^{k}\) in the \(k\)-th \(\langle\text{source, hypothesis}\rangle\) sentence pair:

\[P(y|w_{i}^{k})=\textsc{softmax}(W\cdot H_{i}^{k}), \tag{20}\]

where \(W\) is the parameter. The label \(y\) is categorized into two groups: correct (\(y=1\)) and incorrect (\(y=0\)).

To estimate the quality of hypotheses, we average all token quality estimation probability \(P(y=1|w_{i}^{k})\) as the sentence quality estimation score \(f(s,c^{k})\) for the \(k\)-th hypothesis \(c^{k}\):

\[f_{\textsc{GED}}(s,c^{k})=\frac{1}{n+1}\sum_{i=m+2}^{m+n+2}P(y=1| w_{i}^{k}). \tag{21}\]

### Grammatical Error Detection Performance with LSTM

In this experiment, we evaluate the effectiveness of BERT and LSTM on the grammatical error detection (GED) task. We keep the same setting as previous work Rei and Sogaard (2019). The FCE dataset is used for evaluation. Precision, Recall, and F\({}_{0.5}\) are used as our evaluation metrics.

As shown in Table 6, three models, LSTM, LSTM-ATTN, and LSTM-JOINT from Rei and Sogaard (2019) are compared with the BERT model. The LSTM model leverages the LSTM encoder and adds language modeling objectives in the training process Rei (2017). LSTM-ATTN and LSTM-JOINT further add attention constraints and sentence level supervision to achieve better performance Rei and Sogaard (2019). The BERT model is the same as our BERT-GED (SRC).

The BERT based model shows significant improvement than LSTM based models. Thus we do not consider LSTM based GED models in the experiments of GEC quality estimation.

## Appendix B

\begin{table}
\begin{tabular}{l|c c c} \hline Model & P & R & F\({}_{0.5}\) \\ \hline LSTM & 58.88 & 28.92 & 48.48 \\ BiLSTM-ATTN & 60.73 & 22.33 & 45.07 \\ BiLSTM-JOINT & 65.53 & 28.61 & 52.07 \\ \hline BERT & **73.69** & **45.39** & **65.52** \\ \hline \end{tabular}
\end{table}
Table 6: Grammatical Error Detection Performance on the First Certificate in English (FCE) dataset Yannakoudakis et al. (2011).