SlowGuess/ACL_2020_OCR huggingface dataset:
Samson Tan $^{\S \ddagger}$ , Shafiq Joty $^{\ddagger \S}$ , Min-Yen Kan $^{\ddagger}$ , Richard Socher $^{\S}$
$^{\S}$ Salesforce Research
$^{\ddagger}$ Nanyang Technological University
$\S$  {samson.tan,sjoty,rsocher}@salesforce.com
<kanmy@comp.nus.edu.sg
Training on only perfect Standard English corpora predisposes pre-trained neural networks to discriminate against minorities from nonstandard linguistic backgrounds (e.g., African American Vernacular English, Colloquial Singapore English, etc.). We perturb the inflectional morphology of words to craft plausible and semantically similar adversarial examples that expose these biases in popular NLP models, e.g., BERT and Transformer, and show that adversially fine-tuning them for a single epoch significantly improves robustness without sacrificing performance on clean data.
In recent years, Natural Language Processing (NLP) systems have gotten increasingly better at learning complex patterns in language by pretraining large language models like BERT, GPT-2, and CTRL (Devlin et al., 2019; Radford et al., 2019; Keskar et al., 2019), and fine-tuning them on task-specific data to achieve state of the art results has become a norm. However, deep learning models are only as good as the data they are trained on.
Existing work on societal bias in NLP primarily focuses on attributes like race and gender (Boluk-basi et al., 2016; May et al., 2019). In contrast, we investigate a uniquely NLP attribute that has been largely ignored: linguistic background.
Current NLP models seem to be trained with the implicit assumption that everyone speaks fluent (often U.S.) Standard English, even though two-thirds ( $>700$  million) of the English speakers in the world speak it as a second language (L2) (Eberhard et al., 2019). Even among native speakers, a significant number speak a dialect like African American Vernacular English (AAVE) rather than Standard English (Crystal, 2003). In addition, these
When is the suspended team scheduled to return?
When are the suspended team schedule to returned?
Figure 1: MORPHEUS looks at each noun, verb, or adjective in the sentence and selects the inflected form (marked in red) that maximizes the target model's loss. To maximize semantic preservation, MORPHEUS only considers inflections belonging to the same universal part of speech as the original word.
World Englishes exhibit variation at multiple levels of linguistic analysis (Kachru et al., 2009).
Therefore, putting these models directly into production without addressing this inherent bias puts them at risk of committing linguistic discrimination by performing poorly for many speech communities (e.g., AAVE and L2 speakers). This could take the form of either failing to understand these speakers (Rickford and King, 2016; Tatman, 2017), or misinterpreting them. For example, the recent mistranslation of a minority speaker's social media post resulted in his wrongful arrest (Hern, 2017).
Since L2 (and many L1 dialect) speakers often exhibit variability in their production of inflectional morphology $^{2}$  (Lardiere, 1998; Prévost and White, 2000; Haznedar, 2002; White, 2003; Seymour, 2004), we argue that NLP models should be robust to inflectional perturbations in order to minimize their chances of propagating linguistic discrimination. Hence, in this paper, we:
$^{1}$ Code and adversarially fine-tuned models available at https://github.com/salesforce/morpheus.
2Inflections convey tense, quantity, etc. See Appendix A for dialectal examples.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2920-2935 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
- Propose MORPHEUS, a method for generating plausible and semantically similar adversaries by perturbing the inflections in the clean examples (Figure 1). In contrast to recent work on adversarial examples in NLP (Belinkov and Bisk, 2018; Ebrahimi et al., 2018; Ribeiro et al., 2018), we exploit morphology to craft our adversaries.
- Demonstrate its effectiveness on multiple machine comprehension and translation models, including BERT and Transformer (Tables 1 & 2).
- Show that adversarially fine-tuning the model on an adversarial training set generated via weighted random sampling is sufficient for it to acquire significant robustness, while preserving performance on clean examples (Table 5).
To the best of our knowledge, we are the first to investigate the robustness of NLP models to inflectional perturbations and its ethical implications.
Fairness in NLP. It is crucial that NLP systems do not amplify and entrench social biases (Hovy and Spruit, 2016). Recent research on fairness has primarily focused on racial and gender biases within distributed word representations (Bolukbasi et al., 2016), coreference resolution (Rudinger et al., 2018), sentence encoders (May et al., 2019), and language models (Bordia and Bowman, 2019). However, we posit that there exists a significant potential for linguistic bias that has yet to be investigated, which is the motivation for our work.
Adversarial attacks in NLP. First discovered in computer vision by Szegedy et al. (2014), adversarial examples are data points crafted with the intent of causing a model to output a wrong prediction. In NLP, this could take place at the character, morphological, lexical, syntactic, or semantic level.
Jia and Liang (2017) showed that question answering models could be misled into choosing a distractor sentence in the passage that was created by replacing key entities in the correct answer sentence. Belinkov and Bisk (2018) followed by demonstrating the brittleness of neural machine translation systems against character-level perturbations like randomly swapping/replacing characters. However, these attacks are not optimized on the target models, unlike Ebrahimi et al. (2018), which makes use of the target model's gradient to find the character change that maximizes the model's error.
Since these attacks tend to disrupt the sentence's semantics, Ribeiro et al. (2018) and Michel et al. (2019) propose searching for adversaries that preserve semantic content. Alzantot et al. (2018) and Jin et al. (2019) explore the use of synonym substitution to create adversarial examples, using word embeddings to find the  $n$  nearest words. Eger et al. (2019) take a different approach, arguing that adding visual noise to characters leaves their semantic content undisturbed. Iyyer et al. (2018) propose to create paraphrase adversaries by conditioning their generation on a syntactic template, while Zhang et al. (2019b) swap key entities in the sentences. Zhang et al. (2019a) provide a comprehensive survey of this topic.
Adversarial training. In order to ensure our NLP systems are not left vulnerable to powerful attacks, most existing work make use of adversarial training to improve the model's robustness (Goodfellow et al., 2015). This involves augmenting the training data either by adding the adversaries to or replacing the clean examples in the training set.
Summary. Existing work in fairness mostly focus on tackling bias against protected attributes like race and gender, while those in adversarial NLP primarily investigate character- and word-level perturbations and seek to improve the models' robustness by retraining them from scratch on the adversarial training set. Our work makes use of perturbations in inflectional morphology to highlight the linguistic bias present in models such as BERT and Transformer, before showing that simply fine-tuning the models for one epoch on the adversarial training set is sufficient to achieve significant robustness while maintaining performance on clean data.
3 Generating Inflectional Perturbations
Inflectional perturbations inherently preserve the general semantics of a word since the root remains unchanged. In cases where a word's part of speech (POS) is context-dependent (e.g., duck as a verb or a noun), restricting perturbations to the original POS further preserves its original meaning.
Additionally, since second language speakers are prone to inflectional errors (Haznedar, 2002; White, 2003), adversarial examples that perturb the inflectional morphology of a sentence should be less perceivable to people who interact heavily with non-native speakers or are themselves non-native speakers. Hence, we present MORPHEUS, our proposed method for crafting inflectional adversaries.
Extractive Question Answering
Original	When is the suspended team scheduled to return?
Adversary	When are the suspended team schedule to returned?
Prediction	Before: 2018 After: No answer
Original	Who upon arriving gave the original viking settlers a common identity?
Adversary	Who upon arrive give the original viking settler a common identities?
Prediction	Before: Rollo After: almost no foreign settlers
Neural Machine Translation
Original	Israeli warplanes struck a target inside the Syrian port city of Latakia Thursday night, a senior administration official confirms to Fox News.
Adversary	Israeli warplanes strikes a target inside the Syrian port city of Latakia Thursday night, a senior administration official confirms to Foxes News.
Prediction	Before: Un haut responsable de l'administration confirme à Fox News que des avions de combat israéliens ont.frappé une cible à l'intérieur de la ville portuaire syrienne de Lattaquié dans la nuit de jeu.After: Le président de la République, Nicolas Sarkozy, a annoncé lui que le président de la République, Nicolas Sarkozy, s'est rendu en République démocratique du Congo.
Table 1: Adversarial examples found for BERT, SpanBERT, and Transformer-big. While not perfectly grammatical, it is plausible for English dialect and second language (L2) speakers to produce such sentences.
(Top) Models trained on SQuAD 2.0 are more fragile than those trained on SQuAD 1.1, and have a bias towards predicting "no answer". Examples are answerable questions and therefore present in both SQuAD 1.1 and 2.0. (Bottom) Perturbing two inflections caused Transformer-big to output a completely irrelevant sentence. In addition, adversarial examples for  $\sim 1.4\%$  of the test set caused the model to output the source (English) sentences.
3.1 MORPHEUS: A Greedy Approach
Problem formulation. Given a target model  $f$  and an original input example  $x$  for which the ground truth label is  $y$ , our goal is to generate the adversarial example  $x'$  that maximizes  $f$ 's loss. Formally, we aim to solve the following problem:
$$
x ^ {\prime} = \underset {x _ {c}} {\arg \max } \mathcal {L} (y, f (x _ {c})) \tag {1}
$$
where  $x_{c}$  is an adversarial example generated by perturbing  $x$ ,  $f(x)$  is the model's prediction, and  $\mathcal{L}(\cdot)$  is the model's loss function. In this setting,  $f$  is a neural model for solving a specific NLP task.
Proposed solution. To solve this problem, we propose MORPHEUS (Algorithm 1), an approach that greedily searches for the inflectional form of each noun, verb, or adjective in  $x$  that maximally increases  $f$ 's loss (Eq. 1). For each token in  $x$ , MORPHEUS calls MAXINFLECTED to find the inflected form that caused the greatest increase in  $f$ 's loss. Table 1 presents some adversarial examples obtained by running MORPHEUS on state-of-the-art machine reading comprehension and translation models: namely, BERT (Devlin et al., 2019), SpanBERT (Joshi et al., 2019), and Transformer-big (Vaswani et al., 2017; Ott et al., 2018).
Algorithm 1 MORPHEUS
There are two possible approaches to implementing MAXINFLECTED: one is to modify each token independently from the others in parallel, and the other is to do it sequentially such that the increase in loss is accumulated as we iterate over the tokens. A major advantage of the parallel approach is that it is theoretically possible to speed it up by  $t$  times, where  $t$  is the number of tokens which are nouns, verbs, or adjectives. However, since current state-of-the-art models rely heavily on contextual representations, the sequential approach is likely to be more effective in finding combinations of inflectional perturbations that cause major increases in loss. We found this to be the case in our preliminary experiments (see Table 6 in Appendix D).
Assumptions. MORPHEUS treats the target model as a black box and maximally requires only access to the model's logits to compute the loss. As mentioned, task-specific metrics may be used in
<sup>3</sup>A task-specific evaluation metric may be used instead of the loss in situations where it is unavailable. However, as we discuss later, the choice of metric is important for optimal performance and should be chosen wisely.
stead of the loss as long as the surface is not overly "flat", like in a step function. Examples of inappropriate metrics are the exact match and  $\mathrm{F}_1$  scores for extractive question answering, which tend to be 1 for most candidates but drop drastically for specific ones. This may affect MORPHEUS' ability to find an adversary that induces absolute model failure.
While the black box assumption has the advantage of not requiring access to the target model's gradients and parameters, a limitation is that we need to query the model for each candidate inflection's impact on the loss, as opposed to Ebrahimi et al. (2018)'s approach. However, this is not an issue for inflectional perturbations since each word usually has less than 5 possible inflections.
Candidate generation. We make use of lemminflect<sup>4</sup> to generate candidate inflectional forms in the GETINFLECTIONS method, a simple process in which the token is first lemmatized before being inflected. In our implementation of GETINFLECTIONS, we also allow the user to specify if the candidates should be constrained to the same universal part of speech.
Semantic preservation. MORPHEUS constrains its search to inflections belonging to the same universal part of speech. For example, take the word "duck". Depending on the context, it may either be a verb or a noun. In the context of the sentence "There's a jumping duck", "duck" is a noun and MORPHEUS may only choose alternate inflections associated with nouns.
This has a higher probability of preserving the sentence's semantics compared to most other approaches, like character/word shuffling or synonym swapping, since the root word and its position in the sentence remains unchanged.
Early termination. MORPHEUS selects an inflection if it increases the loss. In order to avoid unnecessary searching, it terminates once it finds an adversarial example that induces model failure. In our case, we define this as a score of 0 on the task's evaluation metric (the higher, the better).
Other implementation details. In order to increase overall inflectional variation in the set of adversarial examples, GETINFLECTIONSHUFFLES the generated list of inflections before returning it (see Figure 4 in Appendix). Doing this has no
effect on MORPHEUS' ability to induce misclassification, but prevents overfitting during adversarial fine-tuning, which we discuss later in Section 6. Additionally, since MORPHEUS greedily perturbs each eligible token in  $x$ , it may get stuck in a local maximum for some  $x$  values. To mitigate this, we run it again on the reversed version of  $x$  if the early termination criterion was not fulfilled during the forward pass.
Finally, we use sacremoses<sup>5</sup> for tokenization and NLTK (Bird et al., 2009) for POS tagging.
NLP tasks. To evaluate the effectiveness of MORPHEUS at inducing model failure in NLP models, we test it on two popular NLP tasks: question answering (QA) and machine translation (MT). QA involves language understanding (classification), while MT also involves language generation. Both are widely used by consumers of diverse linguistic backgrounds and hence have a high chance of propagating discrimination.
Baseline. In the below experiments, we include a random baseline that randomly inflects each eligible word in each original example.
Measures. In addition to the raw scores, we also report the relative decrease for easier comparison across models since they perform differently on the clean dataset. Relative decrease  $(d_r)$  is calculated using the following formula:
$$
d _ {r} = \frac {\text {s c o r e} _ {\text {o r i g i n a l}} - \text {s c o r e} _ {\text {a d v e r s a r i a l}}}{\text {s c o r e} _ {\text {o r i g i n a l}}} \tag {2}
$$
4.1 Extractive Question Answering
Given a question and a passage containing spans corresponding to the correct answer, the model is expected to predict the span corresponding to the answer. Performance for this task is computed using exact match or average  $F_{1}$  (Rajpurkar et al., 2016). We evaluate the effectiveness of our attack using average  $F_{1}$ , which is more forgiving (for the target model). From our experiments, the exact match score is usually between 3-9 points lower than the average  $F_{1}$  score.
SQuAD 1.1 and 2.0. The Stanford Question Answering Dataset (SQuAD) comprises over 100,000 question-answer pairs written by crowdworkers
4 https://github.com/bjascob/LemmInflect
5 https://github.com/alvations/sacremoses
Dataset	Model	Clean	Random	MORPHEUS
SQuAD 2.0 Answerable Questions (F1)	GloVe-BiDAF	78.67	74.00 (−5.93%)	53.94 (−31.43%)
ELMo-BiDAF	80.90	76.81 (−5.05%)	62.17 (−23.15%)
BERTSQuAD 1.1	93.14	90.90 (−2.40%)	82.79 (−11.11%)
SpanBERTSQuAD 1.1	91.88	91.61 (−0.29%)	82.86 (−9.81%)
BERTSQuAD 2	81.19	74.13 (−8.69%)	57.47 (−29.21%)
SpanBERTSQuAD 2	88.52	84.88 (−4.11%)	69.47 (−21.52%)
SQuAD 2.0 All Questions (F1)	BERTSQuAD 2	81.52	78.87 (−3.25%)	67.24 (−17.51%)
SpanBERTSQuAD 2	87.71	85.46 (−2.56%)	73.26 (−16.47%)
newstest2014 En-Fr (BLEU)	ConvS2S	40.83	27.72 (−32.10%)	17.31 (−57.60%)
Transformer-big	43.16	30.41 (−29.54%)	20.57 (−56.25%)
Table 2: Results for MORPHEUS on QA and NMT models. The subscript in Model_dataset indicates the dataset used to fine-tune the model. Negated % decrease w.r.t. the scores on clean data are reported in parentheses for easy comparison across models. Bolded values indicate the largest % decrease.
based on Wikipedia articles. SQuAD 1.1 guarantees that the passages contain valid answers to the questions posed (Rajpurkar et al., 2016). SQuAD 2.0 increases the task's difficulty by including another 50,000 unanswerable questions, and models are expected to identify when a passage does not contain an answer for the given question (Rajpurkar et al., 2018). Since the test set is not public, we generate adversarial examples from and evaluate the models on the standard dev set.
In addition, the answerable questions from SQuAD 2.0 are used in place of SQuAD 1.1 to evaluate models trained on SQuAD 1.1. This allows for easy comparison between the performance of the SQuAD 1.1-fine-tuned models and SQuAD 2.0-fine-tuned ones for answerable questions. We found performance on the answerable questions from SQuAD 2.0 to be comparable to SQuAD 1.1.
Models. We evaluate MORPHEUS on Gardner et al. (2018)'s implementation of BiDAF (Seo et al., 2017), a common baseline model for SQuAD 1.1, ELMo-BiDAF (Peters et al., 2018), the transformers implementation (Wolf et al., 2019) of BERT, and SpanBERT, a pre-training method focusing on span prediction that outperforms BERT on multiple extractive QA datasets.
From Table 2, we see that models based on contextual embeddings (e.g., ELMo and BERT variants) tend to be more robust than those using fixed word embeddings (GloVe-BiDAF). This difference is likely due to the pre-training process, which gives them greater exposure to a wider variety of contexts in which different inflections occur. Removing the POS constraint further degrades the models' per
formance by another  $10\%$  of the original score, however, this difference is likely due to changes in the semantics and expected output of the examples.
BiDAF vs. BERT. Even after accounting for the performance difference on clean data, the BiDAF variants are significantly less robust to inflectional adversaries compared to the BERT variants. This is likely a result of BERT's greater representational power and masked language modeling pre-training procedure. Randomly masking out words during pre-training could have improved the models' robustness to small, local perturbations (like ours).
BERT vs. SpanBERT. In the context of question answering, SpanBERT appears to be slightly more robust than vanilla BERT when comparing overall performance on the two SQuAD datasets. However, the difference becomes significant if we look only at the SQuAD 2.0-fine-tuned models' performance on answerable questions (7% difference). This indicates that BERT has a stronger bias towards predicting "no answer" when it encounters inflectional perturbations compared to SpanBERT.
SQuAD 1.1 vs. SQuAD 2.0. The ability to "know what you don't know" (Rajpurkar et al., 2018) appears to have been obtained at a great cost. The SQuAD 2.0-fine-tuned models are not only generally less robust to inflectional errors than their SQuAD 1.1 equivalents (6.5% difference), but also significantly less adept at handling answerable questions (12–18% difference). This discrepancy suggests a stronger bias in SQuAD 2.0 models towards predicting "no answer" upon receiving sentences containing inflectional errors (see Table 1).
As we alluded to earlier, this is particularly troubling: since SQuAD 2.0 presents a more realistic
SQuAD 2.0 Answerable Questions (F1)
Original	Transfer	Clean	MORPHEUS
GloVe-BiDAF	BERTSQUAD 1.1	93.14	89.67
SpanBERTSQUAD 1.1	91.88	90.75
BERTSQUAD 2	81.19	72.21
SpanBERTSQUAD 2	88.52	81.95
BERTSQUAD 1.1	GloVe-BiDAF	78.67	71.33
SpanBERTSQUAD 1.1	91.88	88.68
BERTSQUAD 2	81.19	69.68
SpanBERTSQUAD 2	88.52	80.11
SpanBERTSQUAD 1.1	GloVe-BiDAF	78.67	71.41
BERTSQUAD 1.1	93.14	87.48
BERTSQUAD 2	81.19	70.05
SpanBERTSQUAD 2	88.52	77.89
SQuAD 2.0 All Questions (F1)
Original	Transfer	Clean	MORPHEUS
BERTSQUAD 2	SpanBERTSQUAD 2	87.71	82.49
SpanBERTSQUAD 2	BERTSQUAD 2	81.52	75.54
Table 3: Transferability of our adversarial examples.
scenario than SQuAD 1.1, it is fair to conclude that such models will inadvertently discriminate against L2 speakers if put into production as is.
Transferability. Next, we investigate the transferability of adversarial examples found by MORPHEUS across different QA models and present some notable results in Table 3. The adversarial examples found for GloVe-BiDAF transfer to a limited extent to other models trained on SQuAD 1.1, however, they have a much greater impact on BERT $_{\text{SQuAD2}}$  and SpanBERT $_{\text{SQuAD2}}$  (3-4x more).
We observe a similar pattern for adversarial examples found for SpanBERT $_{\text{SQuAD 1.1}}$ . Of the two, BERT is more brittle in general: the SpanBERT $_{\text{SQuAD 1.1}}$  adversaries have a greater effect on  $\text{BERT}_{\text{SQuAD 2}}$ ’s performance on answerable questions than on SpanBERT $_{\text{SQuAD 2}}$ ’s.
Discussion. One possible explanation for the SQuAD 2.0 models' increased fragility is the difference in the tasks they were trained for: SQuAD 1.1 models expect all questions to be answerable and only need to contend with finding the right span, while SQuAD 2.0 models have the added burden of predicting whether a question is answerable.
Therefore, in SQuAD 1.1 models, the feature space corresponding to a possible answer ends where the space corresponding to another possible answer begins, and there is room to accommodate slight variations in the input (i.e., larger individual spaces). We believe that in SQuAD 2.0 models, the need to accommodate the unanswerable prediction forces the spaces corresponding to the possible answers to shrink, with unanswerable
spaces potentially filling the gaps between them. For SQuAD 2.0 models, this increases the probability of an adversarial example "landing" in the space corresponding to the unanswerable prediction. This would explain the effectiveness of adversarial fine-tuning in Section 6, which intuitively creates a "buffer" zone and expands the decision boundaries around each clean example.
The diminished effectiveness of the transferred adversaries at inducing model failure is likely due to each model learning slightly different segmentations of the answer space. As a result, different small, local perturbations have different effects on each model. We leave the in-depth investigation of the above phenomena to future work.
4.3 Machine Translation