SlowGuess/ACL_2020_OCR huggingface dataset:
Fan Yin $^{1}$ , Quanyu Long $^{2}$ , Tao Meng $^{3}$ , and Kai-Wei Chang $^{3}$
$^{1}$ Peking University
$^{2}$ Shanghai Jiao Tong University
1600012975@pku.edu.cn;
oscar.long@sjtu.edu.cn;
tmeng@cs.ucla.edu;
We conduct a thorough study to diagnose the behaviors of pre-trained language encoders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical errors. Specifically, we collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. We use this approach to facilitate debugging models on downstream applications. Results confirm that the performance of all tested models is affected but the degree of impact varies. To interpret model behaviors, we further design a linguistic acceptability task to reveal their abilities in identifying ungrammatical sentences and the position of errors. We find that fixed contextual encoders with a simple classifier trained on the prediction of sentence correctness are able to locate error positions. We also design a cloze test for BERT and discover that BERT captures the interaction between errors and specific tokens in context. Our results shed light on understanding the robustness and behaviors of language encoders against grammatical errors.
Pre-trained language encoders have achieved great success in facilitating various downstream natural language processing (NLP) tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b). However, they usually assume training and test corpora are clean and it is unclear how the models behave when confronted with noisy input. Grammatical error is an important type of noise since it naturally and frequently occurs in natural language, especially in spoken and written materials from non-native speakers. Dealing with such a noise reflects model robustness in representing language and grammatical knowledge. It would also have a positive social impact if language encoders
can model texts from non-native speakers appropriately.
Recent work on evaluating model's behaviors against grammatical errors employs various methods, including (1) manually constructing minimal edited pairs on specific linguistic phenomena (Marvin and Linzen, 2018; Goldberg, 2019; Warstadt et al., 2019a,b); (2) labeling or creating acceptability judgment resources (Linzen et al., 2016; Warstadt and Bowman, 2019; Warstadt et al., 2019a); and (3) simulating noises for a specific NLP task such as neural machine translation (Lui et al., 2018; Anastasopoulos, 2019), sentiment classification (Baldwin et al., 2017). These studies either focus on specific phenomena and mainly conduct experiments on designated corpora or rely heavily on human annotations and expert knowledge in linguistics. In contrast, our work automatically simulates natural occurring data and various types of grammatical errors and systematically analyzes how these noises affect downstream applications. This holds more practical significance to understand the robustness of several language encoders against grammatical errors.
Specifically, we first propose an effective approach to simulating diverse grammatical errors, which applies black-box adversarial attack algorithms based on real errors observed on NUS Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013), a grammatical error correction benchmark. This approach transforms clean corpora into corrupted ones and facilitates debugging language encoders on downstream tasks. We demonstrate its flexibility by evaluating models on four language understanding tasks and a sequence tagging task.
We next quantify model's capacities of identifying grammatical errors by probing individual layers of pre-trained encoders through a linguistic acceptability task. We construct separate datasets for eight error types. Then, we freeze encoder layers
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3386-3403 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
and add a simple classifier on top of each layer to predict the correctness of input texts and locate error positions. This probing task assumes if a simple classifier behaves well on a designated type of error, then the encoder layer is likely to contain knowledge of that error (Conneau et al., 2017; Adi et al., 2017).
Finally, we investigate how models capture the interaction between grammatical errors and contexts. We use BERT as an example and design an unsupervised cloze test to evaluate its intrinsic functionality as a masked language model (MLM).
1. We propose a novel approach to simulating various grammatical errors. The proposed method is flexible and can be used to verify the robustness of language encoders against grammatical errors.
2. We conduct a systematic analysis of the robustness of language encoders and enhance previous work by studying the performance of models on downstream tasks with various grammatical error types.
3. We demonstrate: (1) the robustness of existing language encoders against grammatical errors varies; (2) the contextual layers of language encoders acquire stronger abilities in identifying and locating grammatical errors than token embedding layers; and (3) BERT captures the interaction between errors and specific tokens in context, in particular the neighboring tokens of errors.
The code to reproduce our experiments are available at: https://github.com/uclanlp/ProbeGrammarRobustness
Probing Pre-trained Language Encoders The recent success of pre-trained language encoders across a diverse set of downstream tasks has stimulated significant interest in understanding their advantages. A portion of past work on analyzing pre-trained encoders is mainly based on clean data. As mentioned in Tenney et al. (2019a), these studies can be roughly divided into two categories: (1) designing controlled tasks to probe whether a specific linguistic phenomenon is captured by models (Conneau et al., 2018; Peters et al., 2019; Tenney et al., 2019b; Liu et al., 2019a; Kim et al., 2019), or (2) decomposing the model structure and exploring what linguistic property is encoded (Tenney et al.,
2019a; Jawahar et al., 2019; Clark et al., 2019). However, these studies do not analyze how grammatical errors affect model behaviors.
Our work is related to studies on analyzing models with manually created noise. For example, Linzen et al. (2016) evaluate whether LSTMs capture the hierarchical structure of language by using verbal inflection to violate subject-verb agreement. Marvin and Linzen (2018) present a new dataset consisting of minimal edited pairs with the opposite linguistic acceptability on three specific linguistic phenomena and use it to evaluate RNN's syntactic ability. Goldberg (2019) adjusts previous method to evaluate BERT. Warstadt et al. (2019a) further compare five analysis methods under a single phenomenon. Despite the diversity in methodology, these studies share common limitations. First, they employ only a single or specific aspects of linguistic knowledge; second, their experiments are mainly based on constructed datasets instead of real-world downstream applications. In contrast, we propose a method to cover a broader range of grammatical errors and evaluate on downstream tasks. A concurrent work (Warstadt et al., 2019b) facilitates diagnosing language models by creating linguistic minimal pairs datasets for 67 isolate grammatical paradigms in English using linguist-crafted templates. In contrast, we do not rely heavily on artificial vocabulary and templates.
Synthesized Errors To evaluate and promote the robustness of neural models against noise, some studies manually create new datasets with specific linguistic phenomena (Linzen et al., 2016; Marvin and Linzen, 2018; Goldberg, 2019; Warstadt et al., 2019a). Others have introduced various methods to generate synthetic errors on clean downstream datasets, in particular, machine translation corpora. Belinkov and Bisk (2018); Anastasopoulos (2019) demonstrate that synthetic grammatical errors induced by character manipulation and word substitution can degrade the performance of NMT systems. Baldwin et al. (2017) augment original sentiment classification datasets with syntactically (reordering) and semantically (word substitution) noisy sentences and achieve higher performance. Our method is partly inspired by Lui et al. (2018), who synthesize semi-natural ungrammatical sentences by maintaining confusion matrices for five simple error types.
Another line of studies uses black-box adversarial attack methods to create adversarial examples
for debugging NLP models (Ribeiro et al., 2018; Jin et al., 2019; Alzantot et al., 2018; Burstein et al., 2019). These methods create a more challenging scenario for target models compared to the above data generation procedure. Our proposed simulation benefits from both adversarial attack algorithms and semi-natural grammatical errors.
We first explain how we simulate ungrammatical scenarios. Then, we describe target models and the evaluation design.
3.1 Grammatical Error Simulation
Most downstream datasets contain only clean and grammatical sentences. Despite that recent language encoders achieve promising performance, it is unclear if they perform equally well on text data with grammatical errors.
Therefore, we synthesize grammatical errors on clean corpora to test the robustness of language encoders. We use a controllable rule-based method to collect and mimic errors observed on NUCLE following previous work (Lui et al., 2018; Sperber et al., 2017) and apply two ways to introduce errors to clean corpora: (1) we sample errors based on the frequency distribution of NUCLE and introduce them to plausible positions; (2) inspired by the literature of adversarial attacks (Ribeiro et al., 2018; Jin et al., 2019; Alzantot et al., 2018), we conduct search algorithms to introduce grammatical errors that causing the largest performance drop on a given downstream task.
Mimic Error Distribution on NUCLE We first describe how to extract the error distribution on NUCLE (Dahlmeier et al., 2013). NUCLE is constructed with naturally occurring data (student essays at NUS) annotated with error tags. Each ungrammatical sentence is paired with its correction that differs only in local edits. The two sentences make up a minimal edited pair. An example is like:
1. Will the child blame the parents after he growing up?  $\times$
2. Will the child blame the parents after he grows up?  $\checkmark$
NUCLE corpus contains around 59,800 sentences with average length 20.38. About  $6\%$  of tokens in each sentence contain grammatical errors. There are 27 error tags, including Prep (indicating preposition errors), ArtOrDet (indicating article or determiner errors), Vform (indicating incorrect verb
form) and so forth.
We consider eight frequently-occurred, token-level error types in NUCLE as shown in Table 1.
These error types perturb a sentence in terms of syntax (SVA, Worder), semantics (Nn, Whoice, Trans) and both (ArtOrDet, Prep, Vform), and thus cover a wide range of noise in natural language. Then, we construct a confusion set for each error type based on the observation on NUCLE. Each member of a confusion set is a token. We assign a weight  $w_{ij}$  between token  $t_i$  and  $t_j$  in the same set to indicate the probability that  $t_i$  will be replaced by  $t_j$ . In particular, for ArtOrDet, Prep and Trans, the confusion set consists of a set of tokens that frequently occur as errors or corrections on NUCLE. For each token  $t_i$  in the set, we compute  $w_{ij}$  based on how many times  $t_i$  is replaced by  $t_j$  in minimal edited pairs on NUCLE.
Notice that we add a special token  $\varnothing$  to represent deletion and insertion. For Nn, when we find a noun, we add it and its singular (SG) or plural (PL) counterpart to the set. For SVA, when we find a verb with present tense, we add it and its third-person-singular (3SG) or non-third (not 3SG) counterpart to the set. For Worder, we exchange the position of an adverb with its neighboring adjective, participle or modal. For Vform, we use NLTK (Bird and Loper, 2004) to extract present, past, progressive, and perfect tense of a verb and add to the set. For Wchoice, we select ten synonyms of a target word from WordNet. The substitution weight is set to be uniform for both Vform and Wchoice.
Grammatical Error Introduction We introduce errors in two ways. The first is called probabilistic transformation. Similar to Lui et al. (2018), we first obtain the parse tree of the target sentence using the Berkeley syntactic parser (Petrov et al., 2006). Then, we sample an error type from the error type distribution estimated from NUCLE and randomly choose a position that can apply this type of error according to the parse tree. Finally, we sample an error token based on the weights from the confusion set of the sampled error type and introduce the error token to the selected position.
However, probabilistic transformation only represents the average case. To debug and analyze the robustness of language encoders, we consider another more challenging setting - worst-case transformation, where we leverage search algorithms
Error type	Error Description	Confusion Set
ArtOrDet	Article/determiner errors	{ a, an, the, ∅}
Prep	Preposition errors	{ on, in, at, from, for, under, over, with, into, during, until, against, among, throughout, to, by, about, like, before, across, behind, but, out, up, after, since, down, off, of, ∅}
Trans	Link words/phrase errors	{and, but, so, however, as, that, thus, also, because, therefore, if, although, which, where, moreover, besides, of, ∅}
Nn	Noun number errors	{SG, PL}
SVA	Subject-verb agreement errors	{3SG, not 3SG}
Vform	Verb form errors	{Present, Past, Progressive, Perfect}
Whoice	Word choice errors	{Ten synonyms from WordNet Synsets}
Worder	Word positions errors	{Adverb w/ Adjective, Participle, Modal}
Table 1: The target error types and the corresponding confusion sets.
from the black-box adversarial attack to determine error positions. More concretely, we obtain an operation set for each token in a sentence by considering all possible substitutions based on all confusion sets. Note that some confusion sets are not applicable, for example the confusion set of Nn to a verb. Each operation in the operation set is to replace the target token or to change its position. Then, we apply a searching algorithm to select operations from these operation sets that change the prediction of the tested model and apply them to generate error sentences. Three search algorithms are considered: greedy search, beam search, and genetic algorithm.
Greedy search attack is a two-step procedure. First, we evaluate the importance of tokens in a sentence. The importance of a token is represented by the likelihood decrease on the model prediction when it is deleted. The larger the decrease is, the more important the token is. After comparing all tokens, we obtain a sorted list of tokens in descending order of their importance. Then, we walk through the list. For each token in the list, we try out all operations from the operation set associated with that token and then practice the operation that degrades the likelihood of the model prediction the most. We keep repeating step two until the prediction changes or a budget (e.g., number of operations per sentence) is reached.
Beam search is similar to greedy search. The only difference is that when we walk through the sorted list of tokens, we maintain a beam with fixed size  $k$  that contains the top  $k$  operation streams with the highest global degradation.
Genetic algorithm is a population-based iterative method for finding more suitable examples. We start by randomly selecting operations to build a generation and then use a combination of crossover and mutation to find better candidates. We refer the readers to Alzantot et al. (2018) for details of the genetic algorithm in adversarial attack. Comprehensive descriptions of all methods are found in Appendix C.
3.2 Target Models
We evaluate the following three pre-trained language encoders. Detailed descriptions of models and training settings are in Appendix B.
ELMo (Peters et al., 2018) is a three-layer LSTM-based model pre-trained on the bidirectional language modeling task on 1B Word Benchmark (Chelba et al., 2014). We fix ELMo as a contextual embedding and add two layers of BiLSTM with attention mechanism on top of it.
BERT (Devlin et al., 2019) is a transformer-based (Vaswani et al., 2017) model pre-trained on masked language modeling and next sentence prediction tasks. It uses 16GB English text and adapts to downstream tasks by fine-tuning. We use BERT-base-cased for Named Entity Recognition (NER) and BERT-base-uncased for other tasks and perform task-specific fine-tuning.
RoBERTa (Liu et al., 2019b) is a robustly pretrained BERT model using larger pre-training data (160GB in total), longer pre-training time, the dynamic masking strategy and other optimized pre
training methods. We use RoBERTa-base and perform task-specific fine-tuning.
We design the following three evaluation methods to systematically analyze how language encoders are affected by grammatical errors in input.
Simulate Errors on Downstream Tasks Using the simulation methods discussed in Section §3.1, we are able to perform evaluation on existing benchmark corpora. In our experiments, we consider the target models independently. The whole procedure is: given a dataset, the target model is first trained (fine-tuned) and evaluated on the clean training and development set. Then, we discard those wrongly predicted examples from the development set and apply simulation methods to perturb each remaining example. We compute the attack success rate (attacked examples / all examples) as an indicator of model robustness against grammatical errors. The smaller the rate is, the more robust a model is.
Linguistic Acceptability Probing We design a linguistic acceptability probing task to evaluate each individual type of error. We consider two aspects: (1) if the model can tell whether a sentence is grammatically correct or not (i.e., a binary classification task); (2) if the model can locate error positions in the token-level. We fix the target model and train a self-attention classifier to perform both probing tasks.
Cloze test for BERT We design an unsupervised cloze test to evaluate the masked language model component of BERT based on minimal edited pairs. For each minimal pair that differs only in one token, we quantify how the probability of predicting a single masked token in the rest of the sentence affected by this grammatical error. This method analyzes how error token affects clean context, which is complementary to Goldberg (2019) who focuses on SVA error and discusses how clean contexts influence the prediction of the masked error token.
4 How Grammatical Errors Affect Downstream Performance?
In this section, we simulate grammatical errors and analyze performance drops on downstream tasks.
We compare ELMo, BERT, RoBERTa and a baseline model InferSent (Conneau et al., 2017).
Inferent	ELMo	BERT	RoBERTa
MRPC	75.42	80.30	86.48	89.88
MNLI-m	68.62	74.91	83.77	87.70
MNLI-mm	69.12	75.50	84.80	87.40
QNLI	77.39	78.23	90.58	92.50
SST-2	83.14	90.37	92.08	94.72
NER	-	91.21	95.20	95.45
Table 2: Original performance of the target models on language understanding and sequential tagging tasks.
Datasets We use four language understanding datasets: MRPC (Dolan and Brockett, 2005), MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), and SST-2 (Socher et al., 2013) from GLUE (Wang et al., 2019a) and a sequence tagging benchmark: CoNLL-2013 for NER. Detailed descriptions of these corpora are in Appendix A. We do not use other datasets from GLUE since they are either small in size or only contain short sentences.
Attack Settings For all tasks, we limit the maximum percentage of allowed modifications in a sentence to be  $15\%$  of tokens, which is a reasonable rate according to the statistics estimated from the real data. As shown in Table 3, the worst-case transformation only modifies around  $9\%$  of tokens overall under such a limitation. For MNLI and QNLI, we only modify the second sentence, i.e., hypothesis and answer, respectively. For MRPC, we only modify the first sentence. We do not apply the genetic algorithm to MNLI and QNLI due to their relatively large number of examples in the development sets, which induce an extremely long time for attacking. For NER, we keep the named entities and only modify the remaining tokens.
Results and Discussion Table 2 presents the test performance of four target models on the standard development set of each task. Table 3 summarizes the attack success rates on language understanding tasks, the decreases of F1 score on NER, and the mean percentage of modified tokens (number in brackets). All numbers are formatted in percentage.
As shown in Table 3, with the probabilistic transformation, the attack success rates fall between  $2\%$  (RoBERTa, QNLI) and  $10\%$  (ELMo, MRPC). With the worst-case transformation, we obtain the highest attacked rate of  $81.1\%$  (ELMo, genetic algorithm, MRPC) and an average attacked rate across all tasks of  $29\%$  by perturbing only around  $9\%$  of tokens. This result confirms that all models are influenced by ungrammatical inputs. NER task is
Model	Alg.	MRPC	MNLI (m/mm)	QNLI	SST-2	NER
Infersent	dist.	6.51 (14.53)	8.30 (13.98) / 8.80 (14.23)	4.76 (12.53)	5.79 (14.38)	-
greedy	53.42 (9.02)	36.52 (10.35) / 40.71 (10.06)	44.92 (7.61)	43.44 (8.02)	-
beam	54.39 (9.08)	36.66 (10.37) / 40.87 (10.06)	45.16 (7.62)	43.86 (8.03)	-
genetic	79.15 (8.60)	-	-	59.86 (8.39)	-
BiLSTM + ELMo + Attn	dist.	9.99 (14.53)	7.76 (13.98) / 7.83 (14.23)	5.34 (12.53)	4.64 (14.38)	3.29 (13.75)
greedy	60.84 (8.19)	29.58 (10.28) / 32.92 (9.89)	39.12 (7.25)	37.55 (8.24)	17.81 (7.67)
beam	61.49 (8.29)	29.74 (10.29) / 33.12 (9.91)	40.38 (7.33)	38.32 (8.32)	18.33 (7.85)
genetic	81.14 (7.41)	-	-	59.25 (8.25)	39.78 (8.19)
BERT	dist.	3.69(14.53)	6.59 (13.98) / 6.95 (14.23)	2.33 (12.53)	4.73 (14.38)	3.07 (13.75)
greedy	31.25 (7.95)	28.76 (10.28) / 32.04 (10.01)	25.43 (7.38)	33.54 (7.96)	17.12 (7.51)
beam	31.81 (8.01)	29.03 (10.30) / 32.44 (10.04)	26.42 (7.48)	34.28 (8.01)	18.27 (7.74)
genetic	59.01 (8.84)	-	-	58.53 (7.83)	38.83(7.64)
RoBERTa	dist.	3.04 (14.53)	5.66 (13.98) / 5.88(14.23)	1.92 (12.53)	3.53 (14.38)	2.52 (13.75)
greedy	20.45 (8.11)	20.65 (10.43) / 21.47 (10.02)	19.82 (7.18)	31.06 (8.20)	15.84 (8.12)
beam	20.73(8.14)	20.89 (10.44) / 21.91 (10.06)	20.52 (7.29)	31.91 (8.27)	16.51 (7.47)
genetic	38.93 (9.17)	-	-	56.41 (8.39)	35.11(7.55)
Table 3: Results of evaluating the robustness of models on downstream tasks. Each column represents a dataset and each row represents a victim model with the attack algorithm (dist. means probabilistic transformation). In each cell, we show the mean attack success rate (in percentage) and the mean percentage of modified words (number in the bracket) over the dataset.
BERT	RoBERTa
MRPC	MNLI	SST	MRPC	MNLI	SST
Prep	16	178	36	15	103	43
Art/Det	5	270	20	7	228	28
Whoice	93	1129	233	64	772	195
Vform	8	231	26	9	314	37
SVA	57	538	83	31	388	83
Nn	14	128	13	3	84	13
Worder	0	62	28	0	43	28
Trans	5	70	25	5	31	25
Table 4: Numbers of times each error type is chosen in successful attacks. We find that Wchoice and SVA are more harmful.
in general harder to be influenced by grammatical errors. In terms of the probabilistic transformation, the drop of F1 scores ranges from  $2\%$  to  $4\%$ . For the worst-case transformation, the highest drop for NER is  $18.33\%$  (ElMo, beam search).
Considering different target models, we observe that the impact of grammatical errors varies among models. Specifically, RoBERTa exhibits a strong robustness against the impact of grammatical errors, with consistently lower attack success rates (20.28% on average) and F1 score decreases (17.50% on average) across all tasks, especially on MRPC and MNLI. On the other hand, BERT, ELMo, and InferSent experience an average attack rate of 26.03%, 33.06%, 36.07% respectively on NLU tasks. Given the differences in pre-training strategies, we speculate that pre