SlowGuess/ACL_2020_OCR huggingface dataset:
Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. 2019b. jiant 1.2: A software toolkit for research on general-purpose text understanding models. http://jiant.info/.
Alex Warstadt and Samuel R. Bowman. 2019. Grammatical analysis of pretrained sentence encoders with acceptability judgments. CoRR, abs/1901.03438.
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019a. Investigating bert's knowledge of language: Five analysis methods with npis. CoRR, abs/1909.02597.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2019b. Blimp: A benchmark of linguistic minimal pairs for english. arXiv preprint arXiv:1912.00582.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1112-1122.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
A Downstream Task Details
We test on four language understanding and a sequence labeling datasets. Statistics of these datasets are listed in Table 6.
MRPC The Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005) is a paraphrase detection task which aims to predict a binary label for whether two sentences are semantically equivalent.
MNLI The Multi-Genre Natural Language Inference Corpus (MNLI) (Williams et al., 2018) is a broad-domain natural language inference task to predict the relation (entailment, contradiction, neutral) between premise and hypothesis. MNLI contains both the matched (in-domain) and mismatched (cross-domain) sections.
QNLI The Question-answering NLI task (QNLI) is recasted from the Stanford Question Answering Dataset (Rajpurkar et al., 2016), which aims to determine whether a context sentence contains the answer to the question (entailment, not entailment).
SST-2 The Stanford Sentiment Treebank two-way class split (SST-2; (Socher et al., 2013)) is a binary classification task which assigns positive or negative labels to movie review sentences.
CoNLL2003 - NER The shared task of CoNLL-2003 Named Entity Recognition (NER) (Sang and Meulder, 2003) is a token level sequence labeling task to recognize four types of named entity: persons, locations, organizations and names of miscellaneous entities that do not belong to the previous three groups.
Dataset	Train	Dev	Avg Len
MRPC	3.7k	409	22.4
MNLI	393k	19k	10.1
QNLI	105k	5.5k	27.6
SST-2	67k	873	19.5
CoNLL2003	15k	3k	14.8
Table 6: Datasets statistics of MRPC, MNLI, QNLI, SST-2, and CoNLL2003. Train and Dev stands for the number of sentences in the train and development set. Avg Len stands for the average sentence length (in token) of the target sentence being attacked.
B.1 Pre-trained Encoder Details
We study BERT (base, uncased), BERT (base, cased) (for NER only), RoBERTa (base), and ELMo. BERT (base) and RoBERTa (base) have the same architecture. Both of them are deep transformer models with 12 layers and 12 attention heads, 768 hidden size in each layer. They contain a learnable output layer for fine-tuning on [CLS] or  $
$ . We use PyTorch implement of BERT and RoBERTa from Wolf et al. (2019) and fine-tune them on downstream tasks. For ELMo, we fix ELMo representations as contextual embeddings of tokens and feed them to a two-layer, 1500D BiLSTM with cross-sentence attention mechanism as implemented in jiant. (Wang et al., 2019b).
B.2 Training and Fine-tuning Details
For BERT and RoBERTa, we set the maximum input length to be 128, the maximum number of epochs to be 3, and the dropout to be 0.1 for all tasks. We use Adam (Kingma and Ba, 2015) with an initial learning rate of 2e-5, batch size 16 and no warm-up steps for training. For ELMo, we train the BiLSTM using Adam (Kingma and Ba, 2015) with an initial learning rate of 1e-4, batch size 32. We set the dropout to be 0.2, the maximum number of epochs to be 10 and divide the learning rate by 5 when the performance does not improve for 2 epochs.
B.3 Probing model Details
We use a self-attention layer and a linear classifier to compose the probing component in section 5. The self-attention layer takes as input the hidden representations from the fixed layer  $i$  of an encoder, denoted as  $h = \{h_1^i,h_2^i,\dots,h_n^i\}$  and outputs a sentence representation  $s_i$ :
$$
s _ {i} = \Sigma_ {j = 1} ^ {n} \alpha_ {j} h _ {j} ^ {i}. \tag {1}
$$
$$
\alpha_ {j} = \operatorname {s o f t m a x} (v _ {b} ^ {T} \tanh (W _ {a} h _ {j} ^ {i})) \qquad (2)
$$
where  $W_{a}$  is a weight matrix and  $v_{b}$  is a vector of parameters.  $s_{i}$  is fed to the classifier to output the probability of the sentence being linguistically acceptable. The self-attention layer has a hidden dim of 100 and 0.1 dropout. The classifier has 1 layer and 0.1 dropout. The probing model is trained with Adam (Kingma and Ba, 2015) using a learning rate of 0.001, batch size of 8,  $L_{2}$  weight decay of 0.001 for 10 epochs and early stop patience of 2.
C Attack Algorithms
We conduct three searching algorithms, greedy search, beam search, genetic algorithm in adversarial attacks based on the real errors on NUCLE (Dahlmeier et al., 2013). For beam search, we set the beam size as 5. For genetic algorithm, we set the population in each generation to be 60 and set the maximum number of generations to be  $23\%$  of the corresponding sentence length. For example, if a sentence has 100 tokens, the genetic algorithm will iterate for at most 23 iterations. Algorithm 1, 2 and 3 are detailed descriptions of greedy attack, beam search attack, and genetic algorithm attack, respectively.
D Probing Model Ability in Identifying Errors
D.1 The Sentence-level Binary Classification Task
Table 7 shows complete results for probing individual layers of ELMo, BERT, and RoBERTa across eight error types in the sentence-level binary classification task. We fix the parameters of pre-trained encoders and train a self-attention classifier for each layer to judge the binary linguistic acceptability of a sentence. We find that layer 1 of ELMo, middle layers of BERT, and top layers of RoBERTa perform the best in this evaluation.
D.2 The Token-level Error Locating Task
Table 8 shows complete results for probing individual layers of ELMo, BERT, and RoBERTa across eight error types in the token-level. We first fix the parameters of pre-trained encoders and train a self-attention classifier for each layer to judge the binary linguistic acceptability of a sentence. Then, we extract the two positions with the highest attention weights of self-attention layers and see if error tokens are included.
E Case Study of Locating Error Positions
We show some examples of the token-level evaluation in section 5. We randomly select one example for each error type and visualize the attention weights of the self-attention layer upon different layers of BERT. A deeper purple under each token means the self-attention layer is putting more attention on this token.
F The Token-level Evaluation on Attention Heads of BERT
As mentioned in section 5. We also conduct the same token-level probing to 144 attention heads of BERT. In this experiment, the parameters in BERT are completely frozen. We observe that even in this unsupervised manner, some attention heads are still capable of precisely locating error positions. Middle layers of BERT perform the best. We further observe that some attention heads might be associated with specific types of errors. For example, head 2 in layer 9 and head 6 in layer 10 are good at capturing SVA and Vform. Both of these two errors are related to verbs.
Algorithm 2 Beam search attack
Input: Original sentence  $X_{ori} = \{w_1, w_2, \dots, w_n\}$ , ground truth prediction  $Y_{ori}$ , target model  $F$ , all confusion sets  $P$ , budget  $b$ , beam size  $bm$ .
Output: Adversarial example  $X_{adv}$ .
1: Initialization: bestBeam  $\leftarrow$  copy  $X_{ori}$  for bm times.
2: for each  $w_{i}$  in  $X_{ori}$  do
3: Delete  $w_{i}$  and compute the drop of likelihood on  $Y_{ori}$  to obtain the importance score of  $w_{i}$ , denoted as  $S_{w_i}$ .
4: Apply all substitutions of  $P$  to  $w_{i}$ . Obtain the operation pool of  $w_{i}$ , denoted as  $W_{i}^{sub}$ .
5: end for
6:
7: Get the index list of  $X_{ori}$  according to the decrease order of token importance:  $I \gets \text{argsort}_{w_i \in X_{ori}}(S_{wi})$
8: for each  $w'$  in  $W_{I[0]}^{sub}$  do
9: Substitute  $w_{i}$  with  $w^{\prime}$  in  $X_{ori}$  (or swap their positions)
10:  $Y_{adv} \gets \text{argmax} F(X_{ori})$ ,  $p_{adv} \gets F(X_{ori})|_{Y = Y_{ori}}$
11: if not  $Y_{ori} = Y_{adv}$  then return  $X_{ori}$
12: else
13: topBeam  $\leftarrow$  Record top-bm examples with the lowest  $p_{adv}$
14: end if
15: end for
16:
17: bestBeam  $\leftarrow$  topBeam
18: for each  $i$  in  $I / I[0]$  do
19:  $p_{ori} \gets F(X_{adv})|_{Y = Y_{ori}}$
20: oplist  $\leftarrow \{\}$
21: for each  $X_{beam}$  in bestBeam do
22: for each  $w'$  in  $W_i^{sub}$  do
23: Substitute  $w_{i}$  with  $w^{\prime}$  in  $X_{\text{beam}}$  (or swap their positions)
24:  $Y_{adv} \gets \text{argmax} F(X_{beam}),$ $p_{adv} \gets F(X_{beam})|_{Y = Y_{ori}}$
25: if not  $Y_{ori} = Y_{adv}$  then return  $X_{beam}$
26: else
27: Add  $op \gets (w', p_{adv}, X_{beam})$  to oplist
28: end if
29: end for
30: end for
31: if number of iterations exceed  $b$  then return  $X_{ori}$
32: end if
33: Select the top-bm ops in oplist with lowest op.padv. Update bestBeam with each op.Xbeam.
34: end for
35: return  $X_{ori}$
Algorithm 3 Genetic attack
Input: Original sentence  $X_{ori} = \{w_1, w_2, \dots, w_n\}$ , ground truth prediction  $Y_{ori}$ , target model  $F$ , all confusion sets  $P$ , budget  $b$ , population size  $ps$ , generation size  $G$ .
1: Initialize the first generation with empty set:  $P^0 \gets \emptyset$ .
2: for each  $w_{i}$  in  $X_{ori}$  do
3: Apply all substitutions of  $P$  to  $w_{i}$ . Obtain the operation pool of  $w_{i}$ , denoted as  $W_{i}^{sub}$ .
4: end for
5: for  $i = 1,2,3,\ldots ,ps$  do
6: Randomly select a position  $j$  and an operation from  $W_{j}^{sub}$  to modify  $X_{ori}$ . Then add to  $P^0$ .
7: end for
8:
9: for  $g = 1,2,3,\dots,G - 1$  do
10: for  $i = 1,2,3,\ldots,ps$  do
11:  $Y_{adv}\gets argmaxF(P_i^{g - 1}),$ $p_{adv}\gets F(P_i^{g - 1})|_{Y = Y_{ori}}$
12: if not  $Y_{adv} = Y_{ori}$  then return  $P_i^{g-1}$
13: else
14:  $X_{\text{elite}} \gets \text{argmin}(p_{\text{adv}})$
15:  $P_1^g\gets \{X_{elite}\}$
16: prob  $\leftarrow$  Normalize sample probability with  $F(P_{i}^{g - 1})$
17: for  $i = 2,3,\dots,ps$  do
18: Sample parent1 from  $P^{g - 1}$  with probsb prob
19: Sample parent2 from  $P^{g-1}$  with probs prob
20: child  $\leftarrow$  Crossover(parent1, parent2)
21: child mut  $\leftarrow$  Randomly select a position and an operation from  $W_{j}^{sub}$  to modify child
22:  $P_{i}^{g}\gets child_{mut}$
23: end for
24: end if
25: end for
26: end for
27: return  $X_{ori}$
Prep	Artordet	Vform	Nn	Wchoice	Trans	SVA	Worder
ELMo, layer 0	62.6	65.0	69.6	67.7	74.5	67.5	72.1	47.6
ELMo, layer 1	90.6	84.7	87.2	82.9	83.9	80.6	93.1	71.2
ELMo, layer 2	84.7	77.0	79.4	79.7	82.6	74.4	89.9	68.5
BERT, layer 0	62.5	60.8	67.4	64.6	73.9	69.5	70.3	48.2
BERT, layer 1	68.0	63.4	69.3	70.3	75.0	71.5	78.4	52.2
BERT, layer 2	74.4	67.0	75.3	74.8	76.7	73.1	84.4	62.0
BERT, layer 3	80.5	75.0	83.4	73.7	78.5	76.3	89.2	69.8
BERT, layer 4	82.7	80.7	83.6	77.7	82.6	79.6	90.6	72.4
BERT, layer 5	85.2	83.8	85.4	84.3	84.5	81.8	91.7	71.9
BERT, layer 6	88.2	86.6	85.8	86.7	84.5	82.6	90.9	73.4
BERT, layer 7	91.3	88.1	90.2	86.5	86.9	83.9	95.3	73.4
BERT, layer 8	92.5	88.3	91.4	88.4	86.3	85.5	94.5	73.8
BERT, layer 9	91.4	86.3	89.9	87.4	85.6	84.9	94.4	72.4
BERT, layer 10	90.8	87.4	88.2	87.0	86.1	84.8	94.9	71.8
BERT, layer 11	90.0	84.9	88.1	86.6	85.6	84.3	94.2	69.5
BERT, layer 12	88.4	85.6	88.1	84.3	84.0	82.6	93.3	68.1
RoBERTa, layer 0	61.9	65.9	69.7	67.1	75.1	69.1	68.3	50.9
RoBERTa, layer 1	78.3	74.7	84.6	77.6	80.2	75.9	88.4	67.8
RoBERTa, layer 2	85.2	79.4	88.7	83.0	83.3	78.8	90.9	71.8
RoBERTa, layer 3	89.3	85.7	90.6	86.9	87.0	84.1	94.3	72.6
RoBERTa, layer 4	90.2	88.7	91.8	88.7	86.2	86.4	94.5	74.5
RoBERTa, layer 5	91.4	89.1	92.9	90.5	89.0	87.1	95.5	74.5
RoBERTa, layer 6	93.4	91.3	91.9	91.4	88.9	86.8	95.0	75.3
RoBERTa, layer 7	93.9	90.5	91.8	90.4	88.2	86.9	94.6	74.7
RoBERTa, layer 8	93.9	91.1	93.4	92.3	88.0	87.2	94.4	75.9
RoBERTa, layer 9	94.3	90.6	92.5	92.1	89.4	88.0	95.7	74.7
RoBERTa, layer 10	94.4	92.0	93.3	92.3	89.9	88.1	95.0	75.1
RoBERTa, layer 11	95.3	91.5	93.3	89.4	88.8	88.2	95.2	76.0
RoBERTa, layer 12	94.5	91.1	92.7	88.3	87.3	87.9	95.3	74.8
Table 7: Results of the accuracy on the binary linguistic acceptability probing task for individual layers of ELMo, BERT, and RoBERTa.
Prep	Artordet	Vform	Nn	Wchoice	Trans	SVA	Worder
ELMo, layer 0	23.2	14.3	22.3	9.8	21.8	10.2	18.4	29.6
ELMo, layer 1	56.5	42.6	51.8	82.0	72.0	69.4	30.6	55.1
ELMo, layer 2	68.0	34.2	55.4	85.9	73.0	42.8	49.2	62.7
BERT, layer 0	24.1	39.1	66.7	58.7	62.3	56.4	63.6	17.5
BERT, layer 1	56.6	33.9	66.9	59.3	69.4	71.1	54.4	13.1
BERT, layer 2	58.7	27.4	75.8	58.4	76.3	83.3	60.0	34.1
BERT, layer 3	64.5	55.2	56.2	62.4	79.3	83.0	64.2	67.8
BERT, layer 4	68.9	54.1	69.2	62.9	81.7	66.0	67.3	59.7
BERT, layer 5	67.4	52.4	76.9	60.8	83.8	80.7	62.2	62.3
BERT, layer 6	68.2	51.5	76.5	58.7	84.9	83.9	71.7	66.9
BERT, layer 7	70.4	52.3	93.0	61.8	82.8	81.9	61.3	61.2
BERT, layer 8	69.9	51.7	93.0	65.4	80.2	80.2	60.9	63.9
BERT, layer 9	71.7	48.0	91.6	85.3	84.9	79.6	59.6	62.2
BERT, layer 10	70.7	50.4	90.5	80.5	82.3	78.2	92.4	58.7
BERT, layer 11	70.1	49.2	96.3	80.5	81.0	80.7	90.5	60.3
BERT, layer 12	71.4	50.5	86.7	79.8	79.1	81.6	93.2	58.8
RoBERTa, layer 0	44.8	26.5	74.8	62.8	71.3	71.1	61.7	14.3
RoBERTa, layer 1	68.3	12.1	90.7	62.5	80.9	75.9	93.5	48.9
RoBERTa, layer 2	69.9	35.3	71.0	61.9	83.9	84.1	60.5	58.2
RoBERTa, layer 3	71.9	54.4	92.2	60.7	85.5	84.4	96.2	59.3
RoBERTa, layer 4	71.2	48.9	92.0	83.3	85.6	85.3	95.9	60.8
RoBERTa, layer 5	71.9	53.6	92.5	84.9	88.5	83.9	95.3	61.2
RoBERTa, layer 6	70.2	52.9	92.5	87.0	87.3	83.9	95.7	59.0
RoBERTa, layer 7	70.6	50.6	92.1	87.8	87.2	83.9	94.8	58.4
RoBERTa, layer 8	71.6	51.5	92.2	89.5	87.0	79.6	95.2	58.8
RoBERTa, layer 9	71.3	53.2	91.9	87.7	86.7	81.3	95.8	61.1
RoBERTa, layer 10	69.6	50.3	92.8	86.8	87.1	78.8	96.0	64.2
RoBERTa, layer 11	69.3	49.6	92.7	88.4	86.5	75.6	95.5	62.0
RoBERTa, layer 12	69.6	48.9	90.1	86.8	84.9	79.6	94.1	62.8
Table 8: Results of the accuracy on locating error positions for individual layers of ELMo, BERT, and RoBERTa.
Figure 6: Visualization of attention weights of self-attention layers. A figure represents a sentence with a specific error type. Errors in a sentence are highlighted in red. Each column represents one layer of BERT that the self-attention layer is build upon.
Figure 7: Visualization for each attention head of BERT for locating each type of error. A point in the figure represents the performance of an attention head. The grey line on the top represents the best performing head in each layer (annotated with its number). The green line in the middle represents the average performance of all heads in this layer.