SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\Phi (C) = \gamma \operatorname {F i d} (C) + (1 - \gamma) \operatorname {S t a b} (C). \tag {11}
$$
As  $\gamma$  approaches 0, we get the connected component clusters from our baseline, which maximize stability. As  $\gamma$  approaches 1, we maximize fidelity by assigning each word to its own cluster.
Agglomerative clustering. We approximate the optimal value of  $\Phi$  using agglomerative clustering; we start with each word in its own cluster, then iteratively combine the pair of clusters whose resulting combination increases  $\Phi$  the most. We repeat until combining any pair of clusters would decrease  $\Phi$ . Further details are provided in Appendix A.1.
Token-level attacks. The primary attack surface we study is edit distance one (ED1) perturbations.
For every word in the input, the adversary is allowed to insert a lowercase letter, delete a character, substitute a character for any letter, or swap two adjacent characters, so long as the first and last characters remain the same as in the original token. The constraint on the outer characters, also used by Pruthi et al. (2019), is motivated by psycholinguistic studies (Rawlinson, 1976; Davis, 2003).
Within our attack surface, "the movie was miserable" can be perturbed to "thae mvie wjs miserable" but not "th movie as miserable". Since each token can be independently perturbed, the number of perturbations of a sentence grows exponentially with its length; even "the movie was miserable" has 431,842,320 possible perturbations. Our attack surface contains the attack surface used by (Pruthi et al., 2019), which allows ED1 perturbations to at most two words per sentence. Reviews from SST-2 have 5 million perturbations per example (PPE) on average under this attack surface, while our attack surface averages  $10^{97}$  PPE. We view the size of the attack surface as a strength of our approach: our attack surface forces a system robust to subtle perturbations ("the观影 was miserable") that smaller attack surfaces miss.
In Section 5.7, we additionally consider the internal permutation attacks studied in Belinkov and Bisk (2018) and Sakaguchi et al. (2017), where all characters, except the first and the last, may be arbitrarily reordered.
Attack algorithms. We consider two attack algorithms: the worst-case attack (WCA) and a beam-search attack (BSA). WCA exhaustively tests every possible perturbation of an input  $x$  to see any change in the prediction. The attack accuracy of WCA is the true robust accuracy since if there exists some perturbation that changes the prediction, WCA finds it. When instances of RobEn have high stability, the number of possible encodings of perturbations of  $x$  is often small, allowing us to exhaustively test all possible perturbations in the encoding space. This allows us to tractably run WCA. Using WCA with RobEn, we can obtain computationally tractable guarantees on robustness: given a sentence, we can quickly compute whether or not any perturbation of  $x$  that changes the prediction.
For systems that don't use RobEn, we cannot tractably run WCA. Instead, we run a beam search
2When there are more than 10000 possible encodings, which holds for  $0.009\%$  of our test examples, we assume the adversary successfully alters the prediction.
attack (BSA) with beam width 5, perturbing tokens one at a time. For efficiency, we sample at most  $\mathrm{len}(x_i)$  perturbations at each step of the search (see Appendix A.2). Even against this very limited attack, we find that baseline models have low accuracy.
Datasets. We use six of the nine tasks from GLUE (Wang et al., 2019): SST-2, MRPC, QQP, MNLI, QNLI, and RTE. We do not use STS-B and CoLA as they are evaluated on correlation, which does not decompose as an example-level loss. We additionally do not use WNLI, as most submitted GLUE models cannot even outperform the majority baseline, and state-of-the-art models are rely on external training data (Kocijan et al., 2019). We evaluate on the test sets for SST-2 and MRPC, and the publicly available dev sets for the remaining tasks. More details are provided in Appendix A.3.
5.2 Baseline models.
We consider three baseline systems. Our first is the standard base uncased BERT model (Devlin et al., 2019) fine-tuned on the training data for each task. $^3$
Data augmentation. For our next baseline, we augment the training dataset with four random perturbations of each example, then fine-tune BERT on this augmented data. Data augmentation has been shown to increase robustness to some types of adversarial perturbations (Ribeiro et al., 2018; Liu et al., 2019). Other natural baselines all have severe limitations. Adversarial training with black-box attacks offers limited robustness gains over data augmentation (Cohen et al., 2019; Pruthi et al., 2019). Projected gradient descent (Madry et al., 2017), the only white-box adversarial training method that is robust in practice, cannot currently be applied to BERT since subword tokenization maps different perturbations to different numbers of tokens, making gradient-based search impossible. Certifiably robust training (Huang et al., 2019; Shi et al., 2020) does not work with BERT due to the same tokenization issue and BERT's use of non-monotonic activation functions, which make computing bounds intractable. Moreover the bounds computed with certifiably robust training, which give guarantees, become loose as model depth increases, hurting robust performance (Gowal et al., 2018).
Typo-corrector. For our third baseline, we use the most robust method from Pruthi et al. (2019). In
particular, we train a scRNN typo-corrector (Sakaguchi et al., 2017) on random perturbations of each task's training set. At test time inputs are "corrected" using the typo corrector, then fed into a downstream model. We replace any OOV outputted by the typo-corrector with the neutral word "a" and use BERT as our downstream model.
5.3 Models with RobEn
We run experiments using our two token-level encodings: connected component encodings (CONNCOMP) and agglomerative cluster encodings (AGGCLUST). To form clusters, we use the  $N = 100,000$  most frequent words from the Corpus of Contemporary American English (Davies, 2008) that are also in GloVe (Pennington et al., 2014). For AGGCLUST we use  $\gamma = 0.3$ , which maximizes robust accuracy on SST-2 dev set.
Form of encodings. Though unnecessary when training from scratch, to leverage the inductive biases of pre-trained models like BERT (Devlin et al., 2019), we define the encoded token of a cluster to be the cluster's most frequent member word. In the special case of the out-of-vocab token, we map OOV to [MASK]. Our final encoding,  $\alpha(x)$ , is the concatenation of all of these words. For both encodings, we fine-tune BERT on the training data, using  $\alpha(x)$  as input. Further details are in Appendix A.4.
5.4 Robustness gains from RobEn
Our main results are shown in Table 1. We show all three baselines, as well as models using our instances of RobEn: ConnCOMP and AGGCLUST.
Even against the heuristic attack, each baseline system suffers dramatic performance drops. The system presented by Pruthi et al. (2019), Tipo Corrector + BERT, only achieves  $35.3\%$  attack accuracy, compared to its standard accuracy of  $78.2\%$ . BERT and Data Augmentation + BERT perform even worse. Moreover, the number of perturbations the heuristic attack explores is a tiny fraction of our attack surface, so the robust accuracy of Tipo Corrector + BERT, the quantity we'd like to measure, is likely far lower than the attack accuracy.
In contrast, simple instances of RobEn are much more robust. AGGCLUST + BERT achieves average robust accuracy of  $71.3\%$ , 36 points higher than the attack accuracy of Tipo Corrector + BERT. AGGCLUST also further improves on CONNCOMP in terms of both robust accuracy (by 1.3 points) and standard accuracy (by 2.8 points).
$^{3}$ https://github.com/huggingface/pytorch-transformers
Accuracy	System	SST-2	MRPC	QQP	MNLI	QNLI	RTE	Avg
Standard	Baselines
BERT	93.8	87.7	91.3	84.6	88.6	71.1	86.2
Data Aug. + BERT	92.2	84.3	88.7	83.0	87.4	63.5	83.1
Typo Corr. + BERT	89.6	80.9	87.6	75.9	80.5	54.9	78.2
RobEn
Con. Comp. + BERT	80.6	79.9	84.2	65.7	73.3	52.7	72.7
Agg. Clust. + BERT	83.1	83.8	85.0	69.1	76.6	59.2	76.1
Attack	Baselines
BERT	8.7	10.0	17.4	0.7	0.7	1.8	6.6
Data Aug. + BERT	17.1	1.0	27.6	15.4	10.7	1.4	12.2
Typo Corr. + BERT	53.2	30.1	52.0	23.0	32.3	21.3	35.3
RobEn
Con. Comp. + BERT	80.3	79.4	82.7	62.6	71.5	47.3	70.6
Agg. Clust. + BERT	82.1	82.8	83.2	65.3	74.5	52.7	73.4
Robust	RobEn
Con. Comp. + BERT	80.1	79.4	82.2	61.4	70.5	46.6	70.0
Agg. Clust. + BERT	80.7	80.9	81.4	62.8	71.9	49.8	71.3
Table 1: Standard, attack, and robust accuracy on six GLUE tasks against ED1 perturbations. For baseline models we only compute attack accuracy, an upper bound on robust accuracy, since robust accuracy cannot be tractably computed. Using RobEn, we get robustness guarantees by computing robust accuracy, which we find outperforms a the typo corrector in (Pruthi et al., 2019) by at least 36 points.
Standard accuracy. Like defenses against adversarial examples in other domains, using RobEn decreases standard accuracy (Madry et al., 2017; Zhang et al., 2019; Jia et al., 2019). Our agglomerative cluster encodings's standard accuracy is 10.1 points lower then that of normally trained BERT. However, to the best of our knowledge, our standard accuracy is state-of-the-art for approaches that guarantee robustness. We attribute this improvement to RobEn's compatibility with any model.
Comparison to smaller attack surfaces. We note that RobEn also outperform existing methods on their original, smaller attack surfaces. On SST-2, Pruthi et al. (2019) achieves an accuracy of  $75.0\%$  defending against a single ED1 typo, which is 5.7 points lower than AGGCLUST's robust accuracy against perturbations of all tokens: a superset of the original perturbation set. We discuss constrained adversaries further in Appendix A.5. AGGCLUST also outperforms certified training: Huang et al. (2019), which offers robustness guarantees to three character substitution typos (but not insertions or deletions), achieves a robust accuracy of  $74.9\%$  on SST-2. Certified training requires strong assumptions on model architecture; even the robust accuracy of AGGCLUST outperforms the standard accuracy of the CNN used in Huang et al. (2019).
5.5 Reusable encodings
Each instance of RobEn achieves consistently high stability across our tasks, despite reusing a single
Figure 4: Histogram of  $|B_{\alpha}(x)|$  for SST-2 and RTE. SST-2 has the highest percentage of inputs  $x$  where  $|B_{\alpha}(x)| = 1$ , while RTE has the least. On both datasets,  $|B_{\alpha}(x)| < 9$  for most  $x$ , and  $|B_{\alpha}(x)| = 1$  on a plurality of inputs.
function. Figure 4 plots the distribution of  $|B_{\alpha}(x)|$ , across test examples in SST-2 and RTE, where  $B_{\alpha}(x)$  is the set of encodings that are mapped to by some perturbation of  $x$ . Over AGGCLUST encodings,  $|B_{\alpha}(x)| = 1$  for  $25\%$  of examples in RTE and  $66\%$  in SST-2, with the other four datasets falling between these extremes (see Appendix A.6). As expected, these numbers are even higher for the connected component encodings. Note that when  $|B_{\alpha}(x)| = 1$ , every perturbation of  $x$  maps to the same encoding. When  $|B_{\alpha}(x)|$  is small, robust accuracy can be computed quickly.
5.6 Agglomerative Clustering Tradeoff
In Figure 5, we plot standard and robust accuracy on SST-2 for AGGCLUST encodings, using differ
Figure 5: Standard and robust accuracies on SST-2 with AGGCLUST using different values of  $\gamma$ . While the gap between standard and robust accuracy increases monotonically, robust accuracy increases before decreasing.
ent values of  $\gamma$ . Recall that  $\gamma = 0$  maximizes stability (CONNCOMP), and  $\gamma = 1$  maximizes fidelity. At  $\gamma = 0$ , the gap between standard and robust accuracy, due to out-of-vocabulary tokens, is negligible. As  $\gamma$  increases, both standard accuracy and the gap between standard and robust accuracy increase. As a result, robust accuracy first increases, then decreases.
5.7 Internal permutation attacks
RobEn can also be used to defend against the internal perturbations described in Section 5.1. For normally trained BERT, a heuristic beam search attack using internal permutations reduces average accuracy from  $86.2\%$  to  $15.7\%$  across our six tasks. Using CONNCOMP with the internal permutation attack surface, we achieve robust accuracy of  $81.4\%$ . See Appendix A.7 for further details.
Additional related work. In this work, we introduce RobEn, a framework to construct systems that are robust to adversarial perturbations. We then use RobEn to achieve state-of-the-art robust accuracy when defending against adversarial typos. Besides typos, other perturbations can also be applied to text. Prior attacks consider semantic operations, such as replacing a word with a synonym (Alzantot et al., 2018; Ribeiro et al., 2018). Our framework extends easily to these perturbations. Other attack surfaces involving insertion of sentences (Jia and Liang, 2017) or syntactic rearrangements (Iyyer et al., 2018) are harder to pair with RobEn, and are interesting directions for future work.
Other defenses are based on various forms of preprocessing. Gong et al. (2019) apply a spell
corrector to correct typos chosen to create ambiguity as to the original word, but these typos are not adversarially chosen to fool a model. Edizel et al. (2019) attempt to learn typo-resistant word embeddings, but focus on common typos, rather than worst-case typos. In computer vision, Chen et al. (2019) discretizes pixels to compute exact robust accuracy on MNIST, but their approach generalizes poorly to other tasks like CIFAR-10. Garg et al. (2018) generate functions that map to robust features, while enforcing variation in outputs.
Incorporating context. Our token-level robust encodings lead to strong performance, despite ignoring useful contextual information. Using context is not fundamentally at odds with the idea of robust encodings, and making contextual encodings stable is an interesting technical challenge and a promising direction for future work.
In principle, an oracle that maps every word with a typo to the correct unperturbed word seems to have higher fidelity than our encodings, without compromising stability. However, existing typo correctors are far from perfect, and a choosing an incorrect unperturbed word from a perturbed input leads to errors in predictions of the downstream model. This mandates an intractable search over all perturbations to compute the robust accuracy.
Task-agnosticity. Many recent advances in NLP have been fueled by the rise of task-agnostic representations, such as BERT, that facilitate the creation of accurate models for many tasks. Robustness to typos should similarly be achieved in a task-agnostic manner, as it is a shared goal across many NLP tasks. Our work shows that even simple robust encodings generalize across tasks and are more robust than existing defenses. We hope our work inspires new task-agnostic robust encodings that lead to more robust and more accurate models.
This work was supported by NSF Award Grant no. 1805310 and the DARPA ASED program under FA8650-18-2-7882. A.R. is supported by a Google PhD Fellowship and the Open Philanthropy Project AI Fellowship. We thank Pang Wei Koh, Reid Pryzant, Ethan Chi, Daniel Kang, and the anonymous reviewers for their helpful comments.
All code, data, and experiments are available on CodaLab at https://bit.ly/2VSZI2e.
M. Alzantot, Y. Sharma, A. Elgohary, B. Ho, M. Srivastava, and K. Chang. 2018. Generating natural language adversarial examples. In Empirical Methods in Natural Language Processing (EMNLP).
A. Athalye, N. Carlini, and D. Wagner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420.
Y. Belinkov and Y. Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations (ICLR).
J. Chen, X. Wu, V. Rastogi, Y. Liang, and S. Jha. 2019. Towards understanding limitations of pixel discretization against adversarial attacks. In IEEE European Symposium on Security and Privacy (EuroS&P).
J. M. Cohen, E. Rosenfeld, and J. Z. Kolter. 2019. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML).
M. Davies. 2008. The corpus of contemporary american english (coca): 560 million words, 1990-present. https://www.english-corpora.org/faq.asp.
M. Davis. 2003. Psycholinguistic evidence on scrambled letters in reading. https://www.mrc-cbu.cam.ac.uk/.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171-4186.
W. B. Dolan and C. Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP).
J. Ebrahimi, D. Lowd, and D. Dou. 2018a. On adversarial examples for character-level neural machine translation. In International Conference on Computational Linguistics (COLING).
J. Ebrahimi, A. Rao, D. Lowd, and D. Dou. 2018b. Hot-flip: White-box adversarial examples for text classification. In Association for Computational Linguistics (ACL).
B. Edizel, A. Piktus, P. Bojanowski, R. Ferreira, E. Grave, and F. Silvestri. 2019. Misspelling oblivious word embeddings. In North American Association for Computational Linguistics (NAACL).
S. Garg, V. Sharan, B. H. Zhang, and G. Valiant. 2018. A spectral view of adversarially robust features. In Advances in Neural Information Processing Systems (NeurIPS).
H. Gong, Y. Li, S. Bhat, and P. Viswanath. 2019. Context-sensitive malicious spelling error correction. In World Wide Web (WWW), pages 2771-2777.
S. Gowal, K. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, T. Mann, and P. Kohli. 2018. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715.
H. Hosseini, S. Kannan, B. Zhang, and R. Pooven-dran. 2017. Deceiving Google's Perspective API built for detecting toxic comments. arXiv preprint arXiv:1702.08138.
P. Huang, R. Stanforth, J. Welbl, C. Dyer, D. Yogatama, S. Gowal, K. Dvijotham, and P. Kohli. 2019. Achieving verified robustness to symbol substitutions via interval bound propagation. In Empirical Methods in Natural Language Processing (EMNLP).
M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In North American Association for Computational Linguistics (NAACL).
R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP).
R. Jia, A. Raghunathan, K. Gksel, and P. Liang. 2019. Certified robustness to adversarial word substitutions. In Empirical Methods in Natural Language Processing (EMNLP).
V. Kocijan, A. Cretu, O. Camburu, Y. Yordanov, and T. Lukasiewicz. 2019. A surprisingly robust trick for the Winograd schema challenge. In Association for Computational Linguistics (ACL).
H. Lee and A. Y. Ng. 2005. Spam deobfuscation using a hidden Markov model. In Conference on Email and Anti-Spam (CEAS).
N. F. Liu, R. Schwartz, and N. A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In North American Association for Computational Linguistics (NAACL).
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. 2017. Towards deep learning models resistant to adversarial attacks (published at ICLR 2018). arXiv.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR).
J. Pennington, R. Socher, and C. D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
D. Pruthi, B. Dhingra, and Z. C. Lipton. 2019. Combating adversarial misspellings with robust word recognition. In Association for Computational Linguistics (ACL).
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP).
G. E. Rawlinson. 1976. The significance of letter position in word recognition. Ph.D. thesis, University of Nottingham.
M. T. Ribeiro, S. Singh, and C. Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Association for Computational Linguistics (ACL).
K. Sakaguchi, K. Duh, M. Post, and B. V. Durme. 2017. Robsut wrod recoginiton via semi-character recurrent neural network. In Association for the Advancement of Artificial Intelligence (AAAI).
Z. Shi, H. Zhang, K. Chang, M. Huang, and C. Hsieh. 2020. Robustness verification for transformers. In International Conference on Learning Representations (ICLR).
R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP).
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR).
A. Williams, N. Nangia, and S. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Association for Computational Linguistics (ACL), pages 1112-1122.
H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan. 2019. Theoretically principled tradeoff between robustness and accuracy. In International Conference on Machine Learning (ICML).
A.1 Aggloemrative clustering
Recall that any  $\pi_V$  induces a clustering of  $V$ , where each cluster contains a set of words mapped by  $\pi_V$  to the same encoded token. We use an agglomerative clustering algorithm to approximately minimize  $\Phi$ . We initialize  $\pi_V$  by setting  $\pi_V(w) = w$  for each  $w \in V$ , which corresponds to placing each word in its own cluster. We then examine each pair of clusters  $C_i, C_j$  such that there exists an edge between a node in  $C_i$  and a node in  $C_j$ , in
the graph from Section 4.2. For each such pair, we compute the value of  $\Phi$  if  $C_i$  and  $C_j$  were replaced by  $C_i \cup C_j$ . If no merge operation causes  $\Phi$  to decrease, we return the current  $\pi_V$ . Otherwise, we merge the pair that leads to the greatest reduction in  $\Phi$ , and repeat. To merge two clusters  $C_i$  and  $C_j$ , we first compute a new encoded token  $r$  as the  $w \in C_i \cup C_j$  with largest  $\rho(w)$ . We then set  $\pi_V(w) = r$  for all  $w \in C_i \cup C_j$ . Our algorithm thus works as follows
Algorithm 1 Objective-minimizing agglomerative clustering
1:  $C\gets V$
2: for  $i$  in range  $(|V|)$  do
3:  $C_{\mathrm{next}} \gets \operatorname{GetBestCombination}(C)$
4: if  $C = C_{\mathrm{next}}$  then
5: return C
6: end if
7:  $C \gets C_{\mathrm{next}}$
8: end for
9: return C
Now, we simply have to define the procedure we use to get the best combination.
Algorithm 2 Get Best Combination(C)
1:  $C_{\mathrm{opt}} \gets C$
2:  $\Phi_{\mathrm{opt}}\gets \Phi (C)$
3: for  $(C_i, C_j) \in \mathrm{Adjacent~Pairs}(C)$  do
4:  $C_{\mathrm{comb}} \gets C_i \cup C_j$
5:  $C_{\mathrm{new}} \gets C \cup C_{\mathrm{comb}} \setminus \{C_i, C_j\}$  {New clusters}
6:  $\Phi_{\mathrm{new}}\gets \Phi (C_{\mathrm{new}})$
7: if  $\Phi_{new} < \Phi_{opt}$  then
8:  $\Phi_{opt} \gets \Phi_{new}$
9:  $C_{\mathrm{opt}} \gets C_{\mathrm{new}}$
10: end if
11: end for
12: return  $C_{\mathrm{opt}}$