SlowGuess/NAACL_2021_OCR huggingface dataset:
Zhenghao Liu $^{1,2}$ , Xiaoyuan Yi $^{1,2}$ , Maosong Sun $^{1,3*}$ , Liner Yang $^{4}$ , Tat-Seng Chua $^{5}$
$^{2}$ State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China
<sup>4</sup>Beijing Language and Culture University, Beijing, China
$^{5}$ School of Computing, National University of Singapore, Singapore
Grammatical Error Correction (GEC) aims to correct writing errors and help language learners improve their writing skills. However, existing GEC models tend to produce spurious corrections or fail to detect lots of errors. The quality estimation model is necessary to ensure learners get accurate GEC results and avoid misleading from poorly corrected sentences. Well-trained GEC models can generate several high-quality hypotheses through decoding, such as beam search, which provide valuable GEC evidence and can be used to evaluate GEC quality. However, existing models neglect the possible GEC evidence from different hypotheses. This paper presents the Neural Verification Network (VERNet) for GEC quality estimation with multiple hypotheses. VERNet establishes interactions among hypotheses with a reasoning graph and conducts two kinds of attention mechanisms to propagate GEC evidence to verify the quality of generated hypotheses. Our experiments on four GEC datasets show that VERNet achieves state-of-the-art grammatical error detection performance, achieves the best quality estimation results, and significantly improves GEC performance by reranking hypotheses. All data and source codes are available at https://github.com/thunlp/VERNet.
Grammatical Error Correction (GEC) systems primarily aim to serve second-language learners for proofreading. These systems are expected to detect grammatical errors, provide precise corrections, and guide learners to improve their language ability. With the rapid increase of second-language learners, GEC has drawn growing attention from numerous researchers of the NLP community.
(a) BEA19.
(b) CoNLL2014.
Figure 1: The Grammaticality of Generated Hypotheses. The hypotheses are generated by Kiyono et al. (2019) with beam search decoding. The hypothesis is compared to the source sentence with a BERT based language model and classified into Win (the hypothesis is better), Tie (the hypothesis and source are same) and Loss (the source is better). The ratios of different classes are plotted with different beam search ranks.
Existing GEC systems usually inherit the seq2seq architecture (Sutskever et al., 2014) to correct grammatical errors or improve sentence fluency. These systems employ beam search decoding to generate correction hypotheses and rerank hypotheses with quality estimation models from  $K$ -best decoding (Kiyono et al., 2019; Kaneko et al., 2020) or model ensemble (Chollampatt and Ng, 2018a) to produce more appropriate and accurate grammatical error corrections. Such models thrive from edit distance and language models (Chollampatt and Ng, 2018a; Chollampatt et al., 2019; Yannakoudakis et al., 2017; Kaneko et al., 2019, 2020). Chollampatt and Ng (2018b) further consider the GEC accuracy in quality estimation by directly predicting the official evaluation metric,  $F_{0.5}$  score.
The  $K$ -best hypotheses from beam search usually derive from model uncertainty (Ott et al., 2018). These uncertainties of multi-hypotheses come from model confidence and potential ambiguity of lin
*Corresponding author: M. Sun (sms@tsinghua.edu.cn)
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5441-5452. June 6-11, 2021. ©2021 Association for Computational Linguistics
(a) CoNLL2014 (ann. 1).
Figure 2: The GEC Performance of Generated Hypotheses. The hypotheses generated by Kiyono et al. (2019) are evaluated on the CoNLL2014 dataset. The average scores of Precision and Recall are calculated according to the two annotations of CoNLL2014.
(b) CoNLL2014 (ann. 2).
guistic variation (Fomicheva et al., 2020), which can be used to improve machine translation performance (Wang et al., 2019b). Fomicheva et al. (2020) further leverage multi-hypotheses to make convinced machine translation evaluation, which is more correlated with human judgments. Their work further demonstrates that multi-hypotheses from well-trained neural models have the ability to provide more hints to estimate generation quality.
For GEC, the hypotheses from the beam search decoding of well-trained GEC models can provide some valuable GEC evidence. We illustrate the reasons as follows.
- Beam search can provide better GEC results. The GEC performance of the top-ranked hypothesis and the best one has a large gap in beam search. For two existing GEC systems, Zhao et al. (2019) and Kiyono et al. (2019), the  $\mathrm{F}_{0.5}$  scores of these systems are 58.99 and 62.03 on the CoNLL2014 dataset. However, the  $\mathrm{F}_{0.5}$  scores of the best GEC results of these systems can achieve 73.56 and 76.82.
- Beam search candidates are more grammatical. As shown in Figure 1, the hypotheses from well-trained GEC models with beam search usually win the favor of language models, even for these hypotheses ranked to the rear. It illustrates these hypotheses are usually more grammatical than source sentences.
- Beam search candidates can provide valuable GEC evidence. As shown in Figure 2, the hypotheses of different beam ranks have almost the same Recall score, which demonstrates all hypotheses in beam search can provide some valuable GEC evidence.
Existing quality estimation models (Chollampatt and Ng, 2018b) for GEC regard hypotheses independently and neglect the potential GEC evidence from different hypotheses. To fully use the valuable GEC evidence from GEC hypotheses, we propose the Neural Verification Network (VERNet) to estimate the GEC quality with modeled interactions from multi-hypotheses. Given a source sentence and  $K$  hypothesis sentences from the beam search decoding of the basic GEC model, VERNet establishes hypothesis interactions by regarding <source, hypothesis> pairs as nodes, and constructing a fully-connected reasoning graph to propagate GEC evidence among multi-hypotheses. Then VERNet proposes two kinds of attention mechanisms on the reasoning graph, node interaction attention and node selection attention, to summarize and aggregate necessary GEC evidence from other hypotheses to estimate the quality of tokens.
Our experiments show that VERNet can pick up necessary GEC evidence from multi-hypotheses provided by GEC models and help verify the quality of GEC hypotheses. VERNet helps GEC models to generate more accurate GEC results and benefits most grammatical error types.
The GEC task is designed for automatically proofreading. Large-scale annotated corpora (Mizumoto et al., 2011; Dahlmeier et al., 2013; Bryant et al., 2019) bring an opportunity for building fully data-driven GEC systems.
Existing neural models regard GEC as a natural language generation (NLG) task and usually use sequence-to-sequence architecture (Sutskever et al., 2014) to generate correction hypotheses with beam search decoding (Yuan and Briscoe, 2016; Chollampatt and Ng, 2018a). Transformer-based architectures (Vaswani et al., 2017) show their effectiveness in NLG tasks and are also employed to achieve convinced correction results (Grundkiewicz et al., 2019; Kiyono et al., 2019). The copying mechanism is also introduced for GEC models (Zhao et al., 2019) to better align tokens from source sentence to hypothesis sentence. To further accelerate the generation process, some work also comes up with non-autoregressive GEC models and leverages a single encoder to parallelly detect and correct grammatical errors (Awasthi et al., 2019; Malmi et al., 2019; Omelianchuk et al., 2020).
Recent research focuses on two directions to im
prove GEC systems. The first one treats GEC as a low-resource language generation problem and focuses on data augmentation for a grammar sensitive and language proficient GEC system (Junczys-Dowmunt et al., 2018; Kiyono et al., 2019). Various weak-supervision corpora have been leveraged, such as Wikipedia edit history (Lichtarge et al., 2019), Github edit history (Hagiwara and Mita, 2020) and confusing word set (Grundkiewicz et al., 2019). Besides, lots of work generates grammatical errors through generation models or round-trip translation (Ge et al., 2018; Wang et al., 2019a; Xie et al., 2018). Kiyono et al. (2019) further consider different data augmentation strategies to conduct better GEC pretraining.
Reranking GEC hypotheses from  $K$ -best decoding or GEC model ensemble (Hoang et al., 2016; Chollampatt and Ng, 2018b) with quality estimation models provides another promising direction to achieve better GEC performance. Some methods evaluate if hypotheses satisfy linguistic and grammatical rules. For this purpose, they employ language models (Chollampatt and Ng, 2018a; Chollampatt et al., 2019) or grammatical error detection (GED) models to estimate hypothesis quality. GED models (Rei, 2017; Rei and Søgaard, 2019) estimate the hypothesis quality on both sentence level (Kaneko et al., 2019) and token level (Yannakoudakis et al., 2017). Chollampatt and Ng (2018b) further estimate GEC quality by considering correction accuracy. They establish source-hypothesis interactions with the encoder-decoder architecture and learn to directly predict the official evaluation score  $F_{0.5}$ .
The pre-trained language model BERT (Devlin et al., 2019) has proven its effectiveness in producing contextual token representations, achieving better quality estimation (Kaneko et al., 2019; Chollampatt et al., 2019) and improving GEC performance by fuse BERT representations (Kaneko et al., 2020). However, existing quality estimation models regard each hypothesis independently and neglect the interactions among multi-hypotheses, which can also benefit the quality estimation (Fomicheva et al., 2020).
3 Neural Verification Network
This section describes Neural Verification Network (VERNet) to estimate the GEC quality with multi-hypotheses, as shown in Figure 3.
Given a source sentence  $s$  and  $K$  correspond
Source Sentence:
s: Do one who suffered from this disease ...
Hypotheses from the beam search decoding of basic GEC :
$c^1$ : Does someone who suffered from this disease.
$c^k$ : Does someone who suffers  $(p$ -th token) from this disease ...
$c^l$ : Does anyone who suffers from this disease ...
Figure 3: The Architecture of Neural Verification Network (VERNet). The estimated token  $(c_p^k)$  and potentially supporting evidence towards  $c_p^k$  are annotated.
ing hypotheses  $C = \{c^1,\dots ,c^k,\dots ,c^K\}$  generated by a GEC model, we first regard each sourcehypothesis pair  $\langle s,c^{k}\rangle$  as a node and fully connect all nodes to establish multi-hypothesis interactions. Then VERNet leverages BERT to get the representation of each token in  $\langle s,c^k\rangle$  pairs (Sec. 3.1) and conducts two kinds of attention mechanisms to propagate and aggregate GEC evidence from other hypotheses to verify the token quality (Sec. 3.2). Finally, VERNet estimates hypothesis quality by aggregating token level quality estimation scores (Sec. 3.3). Our VERNet is trained end-to-end with supervisions from golden labels (Sec. 3.4).
3.1 Initial Representations for Sentence Pairs
Pre-trained language models, e.g. BERT (Devlin et al., 2019), show their advantages of producing contextual token representations for various NLP tasks. Hence, given a source sentence  $s$  with  $m$  tokens and the  $k$ -th hypothesis  $c^k$  with  $n$  tokens, we use BERT to encode the source-hypothesis pair  $\langle s, c^k \rangle$  and get its representation  $H^k$ :
$$
H ^ {k} = \operatorname {B E R T} ([ \text {C L S} ] s [ \text {S E P} ] c ^ {k} [ \text {S E P} ]). \tag {1}
$$
The pair representation  $H^{k}$  consists of token-level representations, that is,
$H^{k} = \{H_{0}^{k},\ldots ,H_{m + n + 2}^{k}\} .$ $H_0^k$  denotes the representation of "[CLS]" token.
3.2 Verify Token Quality with Multi-hypotheses
VERNet conducts two kinds of attention mechanisms, node interaction attention and node selection attention, to verify the token quality with the verification representation  $V^{k}$  of  $k$ -th node, which learns the supporting evidence towards estimating token quality from multi-hypotheses.
The node interaction attention first summarizes useful GEC evidence from the  $l$ -th node for the fine-grained representation  $V^{l \to k}$  (Sec. 3.2.1). Then node selection attention further aggregates fine-grained representation  $V^{l \to k}$  with score  $\gamma^l$  according to each node's confidence (Sec. 3.2.2). Finally, we can calculate the verification representation  $V^k$  to verify the token's quality of each node.
3.2.1 Fine-grained Node Representation with Node Interaction Attention
The node interaction attention  $\alpha^{l\rightarrow k}$  attentively reads tokens in the  $l$  -th node and picks up supporting evidence towards the  $k$  -th node to build fine-grained node representations  $V^{l\rightarrow k}$
For the  $p$ -th token in the  $k$ -th node,  $w_{p}^{k}$ , we first calculate the node interaction attention weight  $\alpha_{q}^{l\rightarrow k}$  according to the relevance between  $w_{p}^{k}$  and the  $q$ -th token in the  $l$ -th node,  $w_{q}^{l}$ :
$$
\alpha_ {q} ^ {l \rightarrow k} = \operatorname {s o f t m a x} _ {q} \left(\left(H _ {p} ^ {k}\right) ^ {T} \cdot W \cdot H _ {q} ^ {l}\right), \tag {2}
$$
where  $W$  is a parameter.  $H_{p}^{k}$  and  $H_{q}^{l}$  are the representations of  $w_{p}^{k}$  and  $w_{q}^{l}$ . Then all token representations of  $l$ -th node are aggregated:
$$
V _ {p} ^ {l \rightarrow k} = \sum_ {q = 1} ^ {m + n + 2} \left(\alpha_ {q} ^ {l \rightarrow k} \cdot H _ {q} ^ {l}\right). \tag {3}
$$
Based on  $V_{p}^{l\to k}$ , we further build the  $l$ -th node fine-grained representation towards the  $k$ -th node,  $V^{l\rightarrow k} = \{V_1^{l\rightarrow k},\dots ,V_p^{l\rightarrow k},\dots ,V_{m + n + 2}^{l\rightarrow k}\}$ .
3.2.2 Evidence Aggregation with Node Selection Attention
The node selection attention measures node importance and is used to aggregate supporting evidence from the fine-grained node representation  $V^{l \to k}$  of the  $l$ -th node. We leverage attention-over-attention mechanism (Cui et al., 2017) to conduct source  $h^{ls}$  and hypotheses  $h^{lh}$  representations to calculate the  $l$ -th node selection attention score  $\gamma^l$ . Then we get
the node verification representation  $V_{p}^{k}$  with the node selection attention  $\gamma^l$ .
To calculate the node selection attention  $\gamma^l$ , we establish an interaction matrix  $M^l$  between the source and hypothesis sentences of the  $l$ -th node. Each element  $M_{ij}^l$  in  $M^l$  is calculated with the relevance between  $i$ -th source token and  $j$ -th hypothesis token (include "[SEP]" tokens):
$$
M _ {i j} ^ {l} = \left(H _ {i} ^ {l}\right) ^ {T} \cdot W \cdot H _ {m + 1 + j} ^ {l}, \tag {4}
$$
where  $W$  is a parameter. Then we calculate attention scores  $\beta_{i}^{ls}$  and  $\beta_{j}^{lh}$  along the source dimension and hypothesis dimension, respectively:
$$
\beta_ {i} ^ {l s} = \frac {1}{n + 1} \sum_ {j = 1} ^ {n + 1} \operatorname {s o f t m a x} _ {i} \left(M _ {i j} ^ {l}\right), \tag {5}
$$
$$
\beta_ {j} ^ {l h} = \frac {1}{m + 1} \sum_ {i = 1} ^ {m + 1} \operatorname {s o f t m a x} _ {j} \left(M _ {i j} ^ {l}\right). \tag {6}
$$
Then the representations of source sentence and hypothesis sentence are calculated:
$$
h ^ {l s} = \sum_ {i = 1} ^ {m + 1} \beta_ {i} ^ {l s} \cdot H _ {i} ^ {l}, \quad h ^ {l h} = \sum_ {j = 1} ^ {n + 1} \beta_ {j} ^ {l h} \cdot H _ {m + 1 + j} ^ {l}. \tag {7}
$$
Finally, the node selection attention  $\gamma^l$  of  $l$ -th node is calculated for the evidence aggregation:
$$
\gamma^ {l} = \operatorname {s o f t m a x} _ {l} \left(\operatorname {L i n e a r} \left(\left(h ^ {l s} \circ h ^ {l h}\right); h ^ {l s}; h ^ {l h}\right)\right), \tag {8}
$$
where  $\circ$  is the element-wise multiplication operator and ; is the concatenate operator.
The node selection attention  $\gamma^l$  aggregates evidence for the verification representation  $V_{p}^{k}$  of  $w_{p}^{k}$ :
$$
V _ {p} ^ {k} = \sum_ {l = 1} ^ {K} \left(\gamma^ {l} \cdot V _ {p} ^ {l \rightarrow k}\right), \tag {9}
$$
where  $V^{k} = \{V_{1}^{k},\ldots ,V_{p}^{k},\ldots ,V_{m + n + 2}^{k}\}$  is the  $k$  -th node verification representation.
3.3 Hypothesis Quality Estimation
For the  $p$ -th token  $w_{p}^{k}$  in the  $k$ -th node, the probability  $P(y|w_{p}^{k})$  of quality label  $y$  is calculated with the verification representation  $V_{p}^{k}$ :
$$
P (y \mid w _ {p} ^ {k}) = \operatorname {s o f t m a x} _ {y} \left(\operatorname {L i n e a r} \left(\left(H _ {p} ^ {k} \circ V _ {p} ^ {k}\right); H _ {p} ^ {k}; V _ {p} ^ {k}\right)\right), \tag {10}
$$
where  $\circ$  is the element-wise multiplication and; is the concatenate operator. We average all probability  $P(y = 1|w_p^k)$  of token level quality estimation as hypothesis quality estimation score  $f(s,c^{k})$  for the pair  $\langle s,c^k\rangle$ :
$$
f (s, c ^ {k}) = \frac {1}{n + 1} \sum_ {p = m + 2} ^ {m + n + 2} P (y = 1 | w _ {p} ^ {k}). \tag {11}
$$
3.4 End-to-end Training
We conduct joint training with token-level supervision. The source labels and hypothesis labels are used, which denote the grammatical quality of source sentences and GEC accuracy of hypotheses.
The cross entropy loss for the  $p$ -th token  $w_{p}^{k}$  in the  $k$ -th node is calculated:
$$
L \left(w _ {p} ^ {k}\right) = \text {C r o s s E n t r o p y} \left(y ^ {*}, P \left(y \mid w _ {p} ^ {k}\right)\right), \tag {12}
$$
using the ground truth token labels  $y^{*}$ .
Then the training loss of VERNet is calculated:
$$
L = \frac {1}{K} \frac {1}{m + n + 2} \sum_ {k = 1} ^ {K} \sum_ {p = 1} ^ {m + n + 2} L \left(w _ {p} ^ {k}\right). \tag {13}
$$
This section describes the datasets, evaluation metrics, baselines, and implementation details.
Datasets. We use FCE (Yannakoudakis et al., 2011), BEA19 (Bryant et al., 2019) and NUCLE (Dahlmeier et al., 2013) to construct training and development sets. Four testing scenarios, FCE, BEA19 (Restrict), CoNLL-2014 (Ng et al., 2014) and JFLEG (Napoles et al., 2017), are leveraged to evaluate model performance. Detailed data statistics are presented in Table 1. We do not incorporate additional training corpora for fair comparison.
Basic GEC Model. To generate correction hypotheses, we take one of the state-of-the-art autoregressive GEC systems (Kiyono et al., 2019) as our basic GEC model and keep the same setting. The beam size of our baseline model is set to 5 (Kiyono et al., 2019), and all these beam search hypotheses are reserved in our experiments.
We generate quality estimation labels for tokens in both source sentences and hypothesis sentences with ERRANT (Bryant et al., 2017; Felice et al., 2016), which indicate grammatical correctness and GEC accuracy, respectively. As shown in Table 2, ERRANT annotates edit operations (delete, insert, and replace) towards the ground truth corrections. In terms of such annotations, each token is labeled with correct (1) or incorrect (0).
Evaluation Metrics. We introduce the evaluation metrics in three tasks: token quality estimation, sentence quality estimation, and GEC.
To evaluate the model performance of token-level quality estimation, we employ the same evaluation metrics from previous GED models (Rei, 2017; Rei and Søgaard, 2019; Yannakoudakis et al., 2017), including Precision, Recall, and  $\mathrm{F}_{0.5}$ .  $\mathrm{F}_{0.5}$  is our primary evaluation metric.
Dataset	Training	Development	Test
FCE	28,350	2,191	2,695
BEA19	34,308	4,384	4,477
NUCLE	57,151	-	-
CoNLL-2014	-	-	1,312
JFLEG	-	-	747
Total	119,809	6,575	9,231
Table 1: Data Statistics.
Sentence	The 1 a 2 Mobile phone is a marvelous invention to 9 charge 10 the world 12 [SEP]
Correction	Operation	Span	Edit
Delete	1,2	-
Replace	9,10	change
Insert	12,12	.
Table 2: An Example of Token Label Annotation. All sentences are annotated with ERRANT according to the golden correction. The words in red color are labeled as incorrect (0) and others are labeled as correct (1). The "[SEP]" token denotes the end of the sentence.
For the evaluation of sentence-level quality estimation, we employ the same evaluation metrics from the previous quality estimation model (Chollampatt and Ng, 2018b), including two evaluation scenarios: (1) GEC evaluation metrics for the hypothesis that reranked top-1 and (2) Pearson Correlation Coefficient (PCC) between reranking scores and golden scores  $(\mathrm{F}_{0.5})$  for all hypotheses.
To evaluate GEC performance, we adopt GLEU (Napoles et al., 2015) to evaluate model performance on the JFLEG dataset. The official tool ERRANT of the BEA19 shared task (Bryant et al., 2019) is used to calculate Precision, Recall, and  $\mathrm{F_{0.5}}$  scores for other datasets. For the CoNLL-2014 dataset, the  $\mathbf{M}^2$  evaluation (Dahlmeier and Ng, 2012) is also adopted as our main evaluation.
Baselines. BERT-fuse (GED) (Kaneko et al., 2020) is compared in our experiments, which trains BERT with the GED task and fuses BERT representations into the Transformer. For quality estimation, we consider two groups of baseline models in our experiments, and more details of these models can be found in Appendices A.1.
(1) BERT based language models. We employ three BERT based language models to estimate the quality of hypotheses. BERT-LM (Chollampatt et al., 2019) measures hypothesis quality with the perplexity of the language model. BERTGQE (Kaneko et al., 2019) is trained with annotated GEC data and estimates if the hypothesis has grammatical errors. We also conduct BERT-GED (SRC) that predicts token level grammar indicator
labels, which is inspired by GED models (Yannakoudakis et al., 2017). BERT shows significant improvement compared to LSTM based models for the GED task (Appendices A.2). Hence the LSTM based models are neglected in our experiments.
(2) GEC accuracy estimation models. These models further consider the source-hypothesis interactions to evaluate GEC accuracy. We take a strong baseline NQE (Chollampatt and Ng, 2018b) in experiments. NQE employs the encoder-decoder (predictor) architecture to encode source-hypothesis pairs and predicts  $\mathrm{F}_{0.5}$  score with the estimator architecture. All their proposed architectures, NQE (CC), NQE (RC), NQE (CR), and NQE (RR) are compared. For NQE (XY), X indicates the predictor architecture, and Y indicates the estimator architecture. X and Y can be recurrent (R) or convolutional (C) neural networks. In addition, we also employ BERT to encode source-hypothesis pairs and then predict the  $\mathrm{F}_{0.5}$  score to implement the BERT-QE model. We also come up with two baselines, BERT-GED (HYP) and BERT-GED (JOINT). They leverage BERT to encode source-hypothesis pairs and are supervised with the token-level quality estimation label. BERT-GED (HYP) is trained with the supervision of hypotheses, and BERT-GED (JOINT) is supervised with labels from both source and hypothesis sentences.
Implementation Details. In all experiments, we use the base version of BERT (Devlin et al., 2019) and ELECTRA (Clark et al., 2020). BERT is a widely used pretrained language model and trained with the mask language model task. ELECTRA is trained with the replaced token detection task and aims to predict if the token is original or replaced by a BERT based generator during pretraining. ELECTRA is a discriminator based pretrained language model and is more like the GED task. We regard BERT as our main model for text encoding and leverage ELECTRA to evaluate the generalization ability of our model.
Both BERT and ELECTRA inherit hugging-face's PyTorch implementation (Wolf et al., 2020). Adam (Kingma and Ba, 2015) is utilized for parameter optimization. We set the max sentence length to 120 for source and hypothesis sentences, learning rate to 5e-5, batch size to 8, and accumulate step to 4 during training.
For hypothesis reranking, we leverage the learning-to-rank method, Coordinate Ascent (CA) (Metzler and Croft, 2007), to aggregate the