SlowGuess/ACL_2020_OCR huggingface dataset:
Erik Jones Robin Jia* Aditi Raghunathan* Percy Liang
Computer Science Department, Stanford University
{erjones, robinjia, aditir, pliang}@stanford.edu
Despite excellent performance on many tasks, NLP systems are easily fooled by small adversarial perturbations of inputs. Existing procedures to defend against such perturbations are either (i) heuristic in nature and susceptible to stronger attacks or (ii) provide guaranteed robustness to worst-case attacks, but are incompatible with state-of-the-art models like BERT. In this work, we introduce robust encodings (RobEn): a simple framework that confers guaranteed robustness, without making compromises on model architecture. The core component of RobEn is an encoding function, which maps sentences to a smaller, discrete space of encodings. Systems using these encodings as a bottleneck confer guaranteed robustness with standard training, and the same encodings can be used across multiple tasks. We identify two desiderata to construct robust encoding functions: perturbations of a sentence should map to a small set of encodings (stability), and models using encodings should still perform well (fidelity). We instantiate RobEn to defend against a large family of adversarial typos. Across six tasks from GLUE, our instantiation of RobEn paired with BERT achieves an average robust accuracy of  $71.3\%$  against all adversarial typos in the family considered, while previous work using a typo-corrector achieves only  $35.3\%$  accuracy against a simple greedy attack.
State-of-the-art NLP systems are brittle: small perturbations of inputs, commonly referred to as adversarial examples, can lead to catastrophic model failures (Belinkov and Bisk, 2018; Ebrahimi et al., 2018b; Ribeiro et al., 2018; Alzantot et al., 2018). For example, carefully chosen typos and word substitutions have fooled systems for hate speech detection (Hosseini et al., 2017), machine translation
Figure 1: Example of a defense using RobEn. An adversary can perturb sentences (blue, underlined) to many different perturbations (red, not-underlined) within the attack surface (red, ovals). We define an encoding function  $\alpha$  such that each perturbation of the input sentences maps to one of a few encodings (grey, rounded rectangles). We can then use any model  $g$  to make predictions given the encodings.
(Ebrahimi et al., 2018a), and spam filtering (Lee and Ng, 2005), among others.
We aim to build systems that achieve high robust accuracy: accuracy against worst-case attacks. Broadly, existing methods to build robust models fall under one of two categories: (i) adversarial training, which augments the training set with heuristically generated perturbations and (ii) certifiably robust training, which bounds the change in prediction between an input and any of its allowable perturbations. Both these approaches have major shortcomings, especially in NLP. Adversarial training, while quite successful in vision (Madry et al., 2018), is challenging in NLP due to the discrete nature of textual inputs (Ebrahimi et al., 2018b); current techniques like projected gradient descent are incompatible with subword tokenization. Further, adversarial training relies on heuristic approximations to the worst-case perturbations, leaving models vulnerable to new, stronger attacks. Certifiably robust training (Jia et al., 2019; Huang et al., 2019; Shi et al., 2020) circumvents
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2752-2765 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
the above challenges by optimizing over a convex outer-approximation of the set of perturbations, allowing us to lower bound the true robust accuracy. However, the quality of bounds obtained by these methods scale poorly with the size of the network, and are vacuous for state-of-the-art models like BERT. Moreover, both approaches require separate, expensive training for each task, even when defending against the same type of perturbations.
Ideally we would like a "robustness" module that we can reuse across multiple tasks, allowing us to only worry about robustness once: during its construction. Indeed, reusable components have driven recent progress in NLP. For example, word vectors are a universal resource that are constructed once, then used for many different tasks. Can we build a reusable robust defense that can easily work with complex, state-of-the-art architectures like BERT? The recent work of Pruthi et al. (2019), which uses a typo-corrector to defend against adversarial typos, is such a reusable defense: it is trained once, then reused across different tasks. However, we find that current typo-correctors do not perform well against even heuristic attacks, limiting their applicability.
Our primary contribution is robust encodings (RobEn), a framework to construct encodings that can make systems using any model robust. The core component of RobEn is an encoding function that maps sentences to a smaller discrete space of encodings, which are then used to make predictions. We define two desiderata that a robust encoding function should satisfy: stability and fidelity. First, to encourage consistent predictions across perturbations, the encoding function should map all perturbations of a sentence to a small set of encodings (stability). Simultaneously, encodings should remain expressive, so models trained using encodings still perform well on unperturbed inputs (fidelity). Because systems using RobEn are encoding-based we can compute the exact robust accuracy tractably, avoiding the lower bounds of certifiably robust training. Moreover, these encodings can make any downstream model robust, including state-of-the-art transformers like BERT, and can be reused across different tasks.
In Section 4, we apply RobEn to combat adversarial typos. In particular, we allow an attacker to add independent edit distance one typos to each word in an input sentence, resulting in exponentially more possible perturbations than previous
Figure 2: Attack model allowing independent perturbations of each token. The original input,  $x$  is classified by the model as positive while the perturbation  $\tilde{x} =$ , obtained by choosing perturbations of "This", "delightful", and "film" independently, is classified as negative. Independent perturbations of each word results in an exponentially large perturbation space  $B(x)$ .
work (Pruthi et al., 2019; Huang et al., 2019). We consider a natural class of token-level encodings, which are obtained by encoding each token in a sentence independently. This structure allows us to express stability and fidelity in terms of a clustering objective, which we optimize.
Empirically, our instantiation of RobEn achieves state-of-the-art robust accuracy, which we compute exactly, across six classification tasks from the GLUE benchmark (Wang et al., 2019). Our best system, which combines RobEn with a BERT classifier (Devlin et al., 2019), achieves an average robust accuracy of  $71.3\%$  across the six tasks. In contrast, a state-of-the-art defense that combines BERT with a typo corrector (Pruthi et al., 2019) gets  $35.3\%$  accuracy when adversarial typos are inserted, and a standard data augmentation defense gets only  $12.2\%$  accuracy.
Tasks. We consider NLP tasks that require classifying textual input  $x \in \mathcal{X}$  to a class  $y \in \mathcal{Y}$ . For simplicity, we refer to inputs as sentences. Each sentence  $x$  consists of tokens  $x_{1}, \ldots, x_{L}$  from the set of all strings  $\mathcal{T}$ . Let  $p_{\mathrm{task}}$  denote the distribution over inputs and labels for a particular task of interest. The goal is to learn a model  $f: \mathcal{X} \to \mathcal{Y}$  that maps sentences to labels, given training examples  $(x, y) \sim p_{\mathrm{task}}$ .
Attack surface. We consider an attack surface in which an adversary can perturb each token  $x_{i}$  of a sentence to some token  $\tilde{x}_{i} \in B(x_{i})$ , where  $B(x_{i})$  is the set of valid perturbations of  $x_{i}$ . For example,  $B(x_{i})$  could be a set of allowed typos of  $x_{i}$ . We
define  $B(x)$  as the set of all valid perturbations of the set  $x$ , where every possible combination of token-level typos is allowed:
$$
B (x) = \left\{\left(\tilde {x} _ {1}, \dots , \tilde {x} _ {L}\right) \mid \tilde {x} _ {i} \in B \left(x _ {i}\right) \forall i \right\} \tag {1}
$$
The size of the attack surface  $|B(x)|$  grows exponentially with respect to number of input tokens, as shown in Figure 2. In general  $x_{i} \in B(x_{i})$ , so some words could remain unperturbed.
Model evaluation. In this work, we use three evaluation metrics for any given task.
First, we evaluate a model on its standard accuracy on the task:
$$
\operatorname {a c c} _ {\text {s t d}} (f) = \mathbb {E} _ {(x, y) \sim p _ {\text {t a s k}}} \mathbf {1} [ f (x) = y ]. \tag {2}
$$
Next, we are interested in models that also have high robust accuracy, the fraction of examples  $(x,y)$  for which the model is correct on all valid perturbations  $\tilde{x}\in B(x)$  allowed in the attack model:
$$
\operatorname {a c c} _ {\operatorname {r o b}} (f) = \mathbb {E} _ {(x, y) \sim p _ {\text {t a s k}}} \min  _ {\tilde {x} \in B (x)} \mathbf {1} [ f (\tilde {x}) = y ]. \tag {3}
$$
It is common to instead compute accuracy against a heuristic attack  $a$  that maps clean sentences  $x$  to perturbed sentences  $a(x) \in B(x)$ .
$$
\operatorname {a c c} _ {\text {a t t a c k}} (f; a) = \mathbb {E} _ {(x, y) \sim p _ {\text {t a s k}}} \mathbf {1} [ f (a (x)) = y ]. \tag {4}
$$
Typically,  $a(x)$  is the result of a heuristic search for a perturbation  $\tilde{x} \in B(x)$  that  $f$  misclassifies. Note that  $\mathrm{acc}_{\mathrm{attack}}$  is a (possibly loose) upper bound of  $\mathrm{acc}_{\mathrm{rob}}$  because there could be perturbations that the model misclassifies but are not encountered during the heuristic search (Athalye et al., 2018).
Additionally, since robust accuracy is generally hard to compute, some existing work computes certified accuracy (Huang et al., 2019; Jia et al., 2019; Shi et al., 2020), which is a potentially conservative lower bound for the true robust accuracy. In this work, since we use robust encodings, we can tractably compute the exact robust accuracy.
3 Robust Encodings
We introduce robust encodings (RobEn), a framework for constructing encodings that are reusable across many tasks, and pair with arbitrary model architectures. In Section 3.1 we describe the key components of RobEn, then in Section 3.2 we highlight desiderata RobEn should satisfy.
3.1 Encoding functions
A RobEn classifier  $f_{\alpha} : \mathcal{X} \to \mathcal{Y}$  using RobEn decomposes into two components: a fixed encoding function  $\alpha : \mathcal{X} \to \mathcal{Z}$ , and a model that accepts encodings  $g : \mathcal{Z} \to \mathcal{Y}$ . For any sentence  $x$ , our system makes the prediction  $f_{\alpha}(x) = g(\alpha(x))$ . Given training data  $\{(x_i, y_i)\}_{i=1}^n$  and the encoding function  $\alpha$ , we learn  $g$  by performing standard training on encoded training points  $\{(\alpha(x_i), y_i)\}_{i=1}^n$ . To compute the robust accuracy of this system, we note that for well-chosen  $\alpha$  and an input  $x$  from some distribution  $P_x$ , the set of possible encodings  $\alpha(\tilde{x})$  for some perturbation  $\tilde{x} \in B(x)$  is both small and tractable to compute quickly. We can thus compute  $\mathrm{acc}_{\mathrm{rob}}(f_{\alpha})$  quickly by generating this set of possible encodings, and feeding each into  $g$ , which can be any architecture.
3.2 Encoding function desiderata
In order to achieve high robust accuracy, a classifier  $f_{\alpha}$  that uses  $\alpha$  should make consistent predictions on all  $\tilde{x} \in B(x)$ , the set of points described by the attack surface, and also have high standard accuracy on unperturbed inputs. We term the former property stability, and the latter fidelity, give intuition for both in this section, and provide a formal instantiation in Section 4.
Stability. For an encoding function  $\alpha$  and some distribution over inputs  $P_{\times}$ , the stability  $\operatorname{Stab}(\alpha)$  measures how often  $\alpha$  maps sentences  $x \sim P_{\times}$  to the same encoding as all of their perturbations.
Fidelity. An encoding function  $\alpha$  has high fidelity if models that use  $\alpha$  can still achieve high standard accuracy. Unfortunately, while we want to make task agnostic encoding functions, standard accuracy is inherently task dependent: different tasks have different expected distributions over inputs and labels. To emphasize this challenge consider two tasks: for an integer  $n$ , predict  $n$  mod 2, and  $n$  mod 3. The information we need encodings to preserve varies significantly between these tasks: for the former, 2 and 6 can be identically encoded, while for the latter they must be encoded separately.
To overcome this challenge, we consider a single distribution over the inputs  $P_{\times}$  that we believe covers many task-distributions  $p_{\mathrm{task}}$ . Since it is hard to model the distribution over the labels, we take the more conservative approach of mapping
<sup>1</sup>We can set  $\mathcal{Z} \subseteq \mathcal{X}$  when  $g$  accepts sentences.
the different sentences sampled from  $P_{\mathbf{x}}$  to different encodings with high probability. We call this  $\operatorname{Fid}(\alpha)$ , and give an example in Section 4.5.
Tradeoff. Stability and fidelity are inherently competing goals. An encoding function that maps every sentence to the same encoding trivially maximizes stability, but is useless for any non-trivial classification task. Conversely, fidelity is maximized when every input is mapped to itself, which has very low stability. In the following section, we construct an instantiation of RobEn that balances stability and fidelity when the attack surface consists of typos.
4 Robust Encodings for Typos
In this section, we focus on adversarial typos, where an adversary can add typos to each token in a sentence (see Figure 2). Since this attack surface is defined at the level of tokens, we restrict attention to encoding functions that encode each token independently. Such an encoding does not use contextual information; we find that even such robust encodings achieve greater attack accuracy and robust accuracy in practice than previous work.
First, we will reduce the problem of generating token level encodings to assigning vocabulary words to clusters (Section 4.1). Next, we use an example to motivate different clustering approaches (Section 4.2), then describe how we handle out-of-vocabulary tokens (Section 4.3). Finally, we introduce two types of token-level robust encodings: connected component encodings (Section 4.4) and agglomerative cluster encodings (Section 4.5).
4.1 Encodings as clusters
We construct an encoding function  $\alpha$  that encodes  $x$  token-wise. Formally,  $\alpha$  is defined by a token-level encoding function  $\pi$  that maps each token  $x_{i} \in \mathcal{T}$  to some encoded token  $\pi(x_{i}) \in \mathcal{Z}_{\mathrm{Tok}}$ :
$$
\alpha (x) = \left[ \pi \left(x _ {1}\right), \pi \left(x _ {2}\right), \dots \pi \left(x _ {L}\right) \right]. \tag {5}
$$
In the RobEn pipeline, a downstream model  $g$  is trained on encodings (Section 3.1). If  $\pi$  maps many words and their typos to the same encoded token, they become indistinguishable to  $g$ , conferring robustness. In principle, the relationship between different encoded tokens is irrelevant: during training,  $g$  learns how to use the encoded tokens to perform a desired task. Thus, the problem of finding a good  $\pi$  is equivalent to deciding which tokens should share the same encoded token.
Figure 3: Visualization of three different encodings. Vocabulary words (large font, blue) share an edge if they share a common perturbation (small font, red). The maximal stability cluster (thick solid line) clusters identically, the maximal fidelity clusters (thin dotted line) encodes all words separately, while the balanced clusters (thin solid line) trade off the two.
Since the space of possible tokens  $\mathcal{T}$  is innumerable, we focus on a smaller set of words  $V = \{w_{1},\ldots ,w_{N}\} \subseteq \mathcal{T}$ , which contains the  $N$  most frequent words over  $P_{\mathbf{x}}$ . We will call elements of  $V$  words, and tokens that are perturbations of some word typos. We view deciding which words should share an encoded token as assigning words to clusters  $C_1,\dots,C_k\subseteq V$ . For all other tokens not in the vocabulary, including typos, we define a separate  $\pi_{\mathrm{OOV}}$ . Thus, we decompose  $\pi$  as follows:
$$
\pi \left(x _ {i}\right) = \left\{ \begin{array}{l l} \pi_ {V} \left(x _ {i}\right) & x _ {i} \in V \\ \pi_ {\mathrm {O O V}} \left(x _ {i}\right) & x _ {i} \notin V \end{array} , \right. \tag {6}
$$
Here,  $\pi_V$  is associated with a clustering  $C$  of vocabulary words, where each cluster is associated with a unique encoded token.
4.2 Simple example
We use a simple example to illustrate how a token-level encoding function can achieve the RobEn desiderata: stability and fidelity defined in Section 3.2. We will formally define the stability and fidelity of a clustering in Sections 4.3 and 4.5.
Consider the five words (large font, blue) in Figure 3, along with potential typos (small font, red). We illustrate three different clusterings as boxes around tokens in the same cluster. We may put all words in the same cluster (thick box), each word in its own cluster (dashed boxes), or something in between (thin solid boxes). For now, we group each typo with a word it could have been perturbed from (we will discuss this further in Section 4.3).
To maximize stability, we need to place all words in the same cluster. Otherwise, there would be two
words (say "at" and "aunt") that could both be perturbed to the same typo ("aut") but are in different clusters. Therefore, "aut" cannot map to the same encoded token as both the possible vocab words. At the other extreme, to maximize fidelity, each word should be in its own cluster. Both mappings have weaknesses: the stability-maximizing mapping has low fidelity since all words are identically encoded and thus indistinguishable, while the fidelity-maximizing mapping has low stability since the typos of words "aunt", "abet", and "abrupt" could all be mapped to different encoded tokens than that of the original word.
The clustering represented by the thin solid boxes in Figure 3 balances stability and fidelity. Compared to encoding all words identically, it has higher fidelity, since it distinguishes between some of the words (e.g., "at" and "about" are encoded differently). It also has reasonably high stability, since only the infrequent "abet" has typos that are shared across words and hence are mapped to different encoded tokens.
4.3 Encoding out-of-vocab tokens
Given a fixed clustering of  $V$ , we now study how to map out-of-vocabulary tokens, including typos, to encoded tokens without compromising stability.
Stability. Stability measures the extent to which types of words map to different encoded tokens. We formalize this by defining the set of tokens that some typo of a word  $w$  could map to,  $B_{\pi}(w)$ :
$$
B _ {\pi} (w) = \{\pi (\tilde {w}); \tilde {w} \in B (w) \}, \tag {7}
$$
where  $B(w)$  is the set of allowable typos of  $w$ . Since we care about inputs drawn from  $P_{\mathbf{x}}$ , we define  $\mathrm{Stab}$  on the clustering  $C$  using  $\rho (w)$ , the normalized frequency of word  $w$  based on  $P_{\mathbf{x}}$ .
$$
\operatorname {S t a b} (C) = - \sum_ {i = 1} ^ {N} \rho (w _ {i}) | B _ {\pi} (w _ {i}) | \tag {8}
$$
For a fixed clustering, the size of  $B_{\pi}(w)$  depends on where  $\pi_{\mathrm{OOV}}$  maps typos that  $w$  shares with other words; for example in Figure 3, "aet" could be a perturbation of both "at" and "abet". If we map the typo the encoded token of "at", we increase the size of  $B_{\pi}$  ("abet") and vice-versa. In order to keep the size of  $B_{\pi}(w)$  smaller for the more frequent words and maximize stability (Equation 8), we map a typo to the same encoded token as its most frequent neighbor word (in this case "at").
Finally, when a token is not a typo of any vocab words, we encode it to a special token OOV.
4.4 Connected component encodings
We present two approaches to generate robust token-level encodings. Our first method, connected component encodings, maximizes the stability objective (8). Notice that Stab is maximized when for each word  $w$ ,  $B_{\pi}(w)$  contains one encoded token. This is possible only when all words that share a typo are assigned to the same cluster.
To maximize Stab, define a graph  $G$  with all words in  $V$  as vertices, and edges between words that share a typo. Since we must map words that share an edge in  $G$  to the same cluster, we define the cluster  $C_i$  to be the set of words in the  $i^{th}$  connected component of  $G$ . While this stability-maximizing clustering encodes many words to the same token (and hence seems to compromise on fidelity), these encodings still perform surprisingly well in practice (see Section 5.4).
4.5 Agglomerative cluster encodings
Connected component encodings focus only stability and can lead to needlessly low fidelity. For example, in Figure 3, "at" and "about" are in the same connected component even though they don't share a typo. Since both words are generally frequent, mapping them to different encoded tokens can significantly improve fidelity, with only a small drop in stability: recall only the infrequent word "abet" can be perturbed to multiple encoded tokens.
To handle such cases, we introduce agglomerative cluster encodings, which we construct by trading off Stab with a formal objective we define for fidelity: Fid. We then approximately optimize this combined objective  $\Phi$  using an agglomerative clustering algorithm.
Fidelity objective. Recall from Section 3.2 that an encoding has high fidelity if it can be used to achieve high standard accuracy on many tasks. This is hard to precisely characterize: we aim to design an objective that could approximate this.
We note that distinct encoded tokens are arbitrarily related: the model  $g$  learns how to use different encodings during training. Returning to our example, suppose "at" and "abet" belong to the same cluster and share an encoded token  $z$ . During training, each occurrence of "at" and "abet" is replaced with  $z$ . However, since "at" is much more frequent, classifiers treat  $z$  similarly to "at"
in order to achieve good overall performance. This leads to mostly uncompromised performance on sentences with “at”, at the cost of performance on sentences containing the less frequent “abet”.
This motivates the following definition: let  $\vec{v}_i$  be a the indicator vector in  $\mathbb{R}^{|V|}$  corresponding to word  $i$ . In principle  $\vec{v}_i$  could be a word embedding; we choose indicator vectors to avoid making additional assumptions. We define the encoded token  $\vec{\mu_j}$  associated with words in cluster  $C_j$  as follows:
$$
\vec {\mu_ {j}} = \frac {\sum_ {w _ {i} \in C _ {j}} \rho (w _ {i}) \vec {v _ {i}}}{\sum_ {w _ {i} \in C _ {j}} \rho (w _ {i})} \tag {9}
$$
We weight by the frequency  $\rho$  to capture the effect of training on the encodings, as described above.
Fidelity is maximized when each word has a distinct encoded token. We capture the drop in standard accuracy due to shared encoded tokens by computing the distance between the original embeddings of the word its encoded token. Formally, let  $c(i)$  be the cluster index of word  $w_{i}$ . We define the fidelity objective Fid as follows:
$$
\operatorname {F i d} (C) = - \sum_ {i = 1} ^ {N} \rho \left(w _ {i}\right) \| \vec {v _ {i}} - \vec {\mu} _ {c (i)} \| ^ {2}. \tag {10}
$$
Fid is high if frequent words and rare words are in the same cluster and is low when multiple frequent words are in the same cluster.
Final objective. We introduce a hyperparameter  $\gamma \in [0,1]$  that balances stability and fidelity. We approximately minimize the following weighted combination of Stab (8) and Fid (10):