SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
f _ {\mathrm {Q E}} \left(s, c ^ {k}\right) = \operatorname {s i g m o i d} \left(W \cdot H _ {0} ^ {k}\right), \tag {19}
$$
where  $W$  is the parameter. The quality estimation score  $f_{\mathrm{QE}}(s, c^k)$  of BERT-QE is trained to approximate the  $\mathbf{F}_{0.5}$  score of the  $k$ -th hypothesis  $c^k$ .
BERT-GED. Take BERT-GED (HYP) as an example, it uses the hypothesis representation  $H_{m+2:m+n+2}^k$  of the  $k$ -th <source, hypothesis> sentence pair to estimate the quality of GEC hypothesis. Note that the "[SEP]" token is also used in BERT-GED to denote the end of the sentence.
Model	P	R	F0.5
LSTM	58.88	28.92	48.48
BiLSTM-ATTN	60.73	22.33	45.07
BiLSTM-JOINT	65.53	28.61	52.07
BERT	73.69	45.39	65.52
Table 6: Grammatical Error Detection Performance on the First Certificate in English (FCE) dataset (Yannakoudakis et al., 2011).
We calculate the probability of token quality estimation label  $y$  for the  $i$ -th token  $w_{i}^{k}$  in the  $k$ -th  $\langle \mathrm{source},\mathrm{hypothesis}\rangle$  sentence pair:
$$
P (y \mid w _ {i} ^ {k}) = \operatorname {s o f t m a x} \left(W \cdot H _ {i} ^ {k}\right), \tag {20}
$$
where  $W$  is the parameter. The label  $y$  is categorized into two groups: correct ( $y = 1$ ) and incorrect ( $y = 0$ ).
To estimate the quality of hypotheses, we average all token quality estimation probability  $P(y = 1|w_i^k)$  as the sentence quality estimation score  $f(s,c^{k})$  for the  $k$ -th hypothesis  $c^k$ :
$$
f _ {\mathrm {G E D}} \left(s, c ^ {k}\right) = \frac {1}{n + 1} \sum_ {i = m + 2} ^ {m + n + 2} P \left(y = 1 \mid w _ {i} ^ {k}\right). \tag {21}
$$
A.2 Grammatical Error Detection Performance with LSTM
In this experiment, we evaluate the effectiveness of BERT and LSTM on the grammatical error detection (GED) task. We keep the same setting as previous work (Rei and Søgaard, 2019). The FCE dataset is used for evaluation. Precision, Recall, and  $\mathrm{F_{0.5}}$  are used as our evaluation metrics.
As shown in Table 6, three models, LSTM, LSTM-ATTN, and LSTM-JOINT from Rei and Søgaard (2019) are compared with the BERT model. The LSTM model leverages the LSTM encoder and adds language modeling objectives in the training process (Rei, 2017). LSTM-ATTN and LSTM-JOINT further add attention constraints and sentence level supervision to achieve better performance (Rei and Søgaard, 2019). The BERT model is the same as our BERT-GED (SRC).
The BERT based model shows significant improvement than LSTM based models. Thus we do not consider LSTM based GED models in the experiments of GEC quality estimation.