SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhao Meng* Yihan Dong* Mrinmaya Sachan Roger Wattenhofer
{zhmeng, yihdong, wattenhofer}@ethz.ch
mrinmaya.sachan@inf.ethz.ch
In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning. We create a word-level adversarial attack generating hard positives on-the-fly as adversarial examples during contrastive learning. In contrast to previous works, our method improves model robustness without using any labeled data. Experimental results show that our method improves robustness of BERT against four different word substitution-based adversarial attacks, and combining our method with adversarial training gives higher robustness than adversarial training alone. As our method improves the robustness of BERT purely with unlabeled data, it opens up the possibility of using large text datasets to train robust language models against word substitution-based adversarial attacks.
Pretrained language models such as BERT (Devlin et al., 2019, inter alia) have had a tremendous impact on many NLP tasks. However, several researchers have demonstrated that these models are vulnerable to adversarial attacks, which fool the model by adding small perturbations to the model input (Jia and Liang, 2017).
A prevailing method to improve model robustness against adversarial attacks is adversarial training (Madry et al., 2018). In NLP, adversarial training in the input space has been challenging, as existing natural language adversarial attacks are too slow to generate adversarial examples on the fly during training (Alzantot et al., 2018; Ebrahimi et al., 2018; Ren et al., 2019). While some recent works (Wang et al., 2021c) have started exploring efficient input space adversarial training (e.g., for text classification), scaling adversarial training to
pretrained language models like BERT has been challenging.
In this work, we in particular focus on improving the robustness of BERT against word substitution-based adversarial attacks. We propose an approach to adversarially finetune BERT-like models without using any labeled data. In order to achieve this, we rely on self-supervised contrastive learning (Chen et al., 2020). Self-supervised contrastive learning has recently gained attention in the community and contrastive learning has been used to learn better representations for text classification (Giorgi et al., 2021; Kim et al., 2021; Gao et al., 2021). However, how to use these methods to improve model robustness remains an open question.
We combine self-supervised contrastive learning with adversarial perturbations by using adversarial attacks to generate hard positive examples for contrastive learning. To efficiently create adversarial examples, we leverage an adversarial attack, that is capable of generating multiple adversarial examples in parallel. The attack adversarily creates hard positive examples for contrastive learning by iteratively replacing words to follow the direction of the contrastive loss (see fig. 2).
Experiments show that our method can improve the robustness of pretrained language models without looking at the labels (in other words, before finetuning). Additionally, by combining our method with adversarial training, we are able to obtain better robustness than conducting adversarial training alone (see section 4.4). Our study of the vector representations of clean examples and their corresponding adversarial examples indeed explains that our method improves model robustness by pulling clean examples and adversarial examples closer.
Our contributions<sup>1</sup> in this paper are two-fold. On the one hand, we improve the robustness of
<sup>1</sup>We will release our code at https://github.com/LotusDYH/ssl_robust
Findings of the Association for Computational Linguistics: NAACL 2022, pages 87 - 101 July 10-15, 2022 ©2022 Association for Computational Linguistics
the pretrained language model BERT against word substitution-based adversarial attacks by using self-supervised contrastive learning with adversarial perturbations (see section 3.2). On the other hand, to facilitate adversarial self-supervised contrastive learning, we create for BERT a word-level adversarial attack to create hard positive examples. The attack makes contrastive learning and adversarial training with on-the-fly generated adversarial examples possible. Additionally, we also show that our method is capable of using out-of-domain data to improve model robustness (see table 2 and section 4.4). This opens an opportunity for using large-scale unlabeled data to train robust language models against word substitution-based adversarial attacks.
2.1 Adversarial Training for NLP
Adversarial training improves model robustness by augmenting clean examples with adversarial examples during training. Previous works on adversarial training for natural language mainly focus on perturbations in the vector space, while actual adversarial attacks create adversarial examples by changing natural language symbols. For example, Zhu et al. (2020) and Liu et al. (2020) improve model generalization ability by adversarial training on the word embedding space, without mentioning model robustness. However, they either ignore model robustness, or only test model robustness against the adversarial dataset ANLI, without paying attention to actual adversarial attacks. Other works conduct adversarial training in the word space (Alzantot et al., 2018; Ren et al., 2019). Still, they can only do adversarial training on a limited number of pregenerated adversarial examples due to the low efficiency of the attacks. A recent work (Wang et al., 2021c) conducts adversarial training efficiently in the word space, but their method is limited to noncontextualized models.
Apart from adversarial training, other supervised learning methods (Dong et al., 2021; Zhou et al., 2021; Wang et al., 2021a; Li and Qiu, 2020) have also been proposed to improve robustness. However, these methods are supervised and are not comparable to our work.
Our work also differs from previous works in natural language adversarial training. On the one hand, as opposed to previous works, which are supervised, we propose a self-supervised learning
scheme to improve the robustness of pretrained language models. On the other hand, while previous works mostly focus on adversarial training in embedding space, we conduct efficient adversarial training with pretrained language models at the word level.
2.2 Contrastive Learning for NLP
Contrastive learning was first proposed in the image domain to improve model performance in a self-supervised fashion (He et al., 2020; Chen et al., 2020). These methods bring representations of similar examples closer and push representations of dissimilar examples further apart. Additionally, researchers also find that by adding adversarial perturbations during contrastive learning, image classification models become more robust against adversarial attacks (Kim et al., 2020).
In NLP, previous works on contrastive learning mainly focus on improving model generalization. Gunel et al. (2021) boost performance of RoBERTa by adding supervised signals during finetuning on downstream tasks. Lee et al. (2021) tackle the "exposure bias" problem in text generation by adding adversarial signals during contrastive learning. Other similar works include Pan et al. (2021), Giorgi et al. (2021), and Gao et al. (2021). Although these works have demonstrated the usefulness of contrastive learning in NLP applications, few have addressed the robustness of NLP models, particularly pretrained language models, against word substitution-based natural language adversarial attacks.
Recently, Wang et al. (2021b) claimed that their method improves model robustness against adversarial sets. However, such sets are pre-generated and are less challenging than adversarial examples generated on the fly by actual adversarial attacks (Jin et al., 2020; Ren et al., 2019). In this paper, we focus on improving the robustness of pretrained language models against word substitution-based adversarial attacks. We present the details of our method in section 3.
In this section, we describe our method for self-supervised contrastive learning with adversarial perturbations. Specifically, section 3.1 gives the background and motivation of our problem, and section 3.2 describes the adversarial contrastive learning framework. Finally, in section 3.3, we
Figure 1: An illustration of our method. (a) For the original example  $X$ , we obtain the hard positive example  $X'$  by Geometry Attack for contrastive loss (see section 3.3). (b) Before contrastive learning, in the vector space, the clean example  $z$ , the hard positive example  $z'$ , and the adversarial example  $z^{adv}$  are far from each other. Contrastive learning pulls the clean example  $z$ , and the hard positive example  $z'$  together. (c) After contrastive learning, the clean example  $z$ , the hard positive example  $z'$ , and the adversarial example  $z^{adv}$  are close. We omit MLP in this figure for simplicity. We use a different color to show another example from the dataset. See section 3 for details. Note that the adversarial example  $X^{adv}$  and its corresponding vector  $z^{adv}$  are not used in contrastive learning. We nevertheless show  $X^{adv}$  and  $z^{adv}$  for illustration purposes.
describe the adversarial attack used in contrastive learning.
In this work, we focus on text classification tasks $^2$ . Let us assume that we have an example text  $X_{i} = \{w_{1}, w_{2}, \dots, w_{L}\}$  with  $L$  words and let  $y_{i}$  be the corresponding class label for  $X_{i}$ . Our text classification model consists of a BERT encoder  $f(\cdot)$  and an MLP classification head  $c(\cdot)$ .
We obtain the vector representation  $h_i \in \mathbb{R}^d$  of the example  $X_i$  by feeding  $X_i$  into the BERT encoder  $f(\cdot)$ . Then the MLP classification head  $c(\cdot)$  takes  $h_i$  as input to give us the prediction. Formally, we have:
$$
\boldsymbol {h} _ {i} = f (X _ {i})
$$
$$
\hat {y} _ {i} = c (\pmb {h} _ {i})
$$
where  $\hat{y}_i$  is the predicted label. We have  $\hat{y}_i = y_i$  if the model prediction is correct.
A word substitution-based adversarial attack  $a(\cdot)$  takes an original example  $X_{i}$  as input and generates an adversarial example  $X_{i}^{adv}$  by substituting the  $k$ -th original word  $w_{k}$  in  $X_{i}$  with another word  $w_{k}^{adv}$ . To make the original example  $X_{i}$  and the adversarial example  $X_{i}^{adv}$  close in semantics, existing works often use synonyms as substitutions (Ren et al., 2019; Morris et al., 2020).
By conducting the word substitution, the attack  $a(\cdot)$  aims to fool the model with  $X_{i}^{adv}$ . Formally, we have:
$$
X _ {i} ^ {a d v} = a \left(X _ {i}\right)
$$
$$
\boldsymbol {h} _ {i} ^ {a d v} = f (\boldsymbol {X} _ {i} ^ {a d v})
$$
$$
\hat {y} _ {i} ^ {a d v} = c (\pmb {h} _ {i} ^ {a d v})
$$
where  $X_{i}^{adv} = \{w_{1},w_{2},\ldots ,w_{k}^{adv},\ldots ,w_{L}\} ,1\leq k\leq L$  Assuming the attack successfully fools the model, we would have  $\hat{y}_i\neq \hat{y}_i^{adv}$  . The key assumption in our approach is that although  $X_{i}$  and  $X_{i}^{adv}$  are very similar to each other at the word level, it is possible that the encoder  $f$  embeds them in such a way that the distance between their representations  $\pmb {h}_i$  and  $h_i^{adv}$  are large and the classification head  $c(\cdot)$  predicts  $X_{i}$  and  $X_{i}^{adv}$  to be of different classes.
Thus, the goal of our method is to obtain a robust model, on which we have  $y_{i} = \hat{y}_{i}$  and  $\hat{y}_i = y_i^{adv}$ . In other words, the robust model defends an adversarial example  $X_{i}^{adv}$  of the original example  $X_{i}$  successfully, if the robust model gives the same correct prediction on the original example  $X_{i}$  and the adversarial example  $X_{i}^{adv}$ . We use attack success rate as the evaluation metric for model robustness. The attack success rate is defined as the rate of an attack successfully fooling the model on all test examples.
To obtain a robust model, we optimize the encoder such that  $h_i$  and  $h_i^{adv}$  become similar to each other. We achieve this goal by conducting
2. Although our formulation can also be extended to several other problems.
self-supervised contrastive learning on the encoder with adversarial perturbations, during which we use an attack to create hard positive examples, maximizing the contrastive loss. The rest this section gives the details of our method.
3.2 Self-Supervised Contrastive Learning with Adversarial Perturbations
Following previous works on self-supervised contrastive learning (He et al., 2020; Chen et al., 2020), we formulate our learning objectives as follows. Consider we have a batch of  $n$  examples and  $X_{i}$  is the  $i$ -th input, we first obtain  $X_{i}^{\prime} = t(X_{i})$  as an augmentation of  $X_{i}$  by transformation  $t(\cdot)$ . We call  $X_{i}$  and  $X_{i}^{\prime}$  a pair of positive examples. All other examples in the same batch are considered negative examples of  $X_{i}$  and  $X_{i}^{\prime}$ .
To take advantage from using more negative examples, we use  $\mathsf{MoCo}$  (He et al., 2020) as our framework, in which we employ an encoder  $f_{q}$  for the positive examples, and another momentum encoder  $f_{k}$  for the negative examples. We then have:
$$
\boldsymbol {h} _ {i} = f _ {q} \left(X _ {i}\right)
$$
$$
\boldsymbol {h} _ {i} ^ {\prime} = f _ {k} \left(X _ {i} ^ {\prime}\right)
$$
where  $h_i, h_i' \in \mathbb{R}^d$  are representations of  $X_i$  and  $X_i'$ , respectively. During training,  $f_q$  and  $f_k$  are initialized the same. We update  $f_k$  momentarily:
$$
\theta_ {k} \gets m \cdot \theta_ {k} + (1 - m) \cdot \theta_ {q}
$$
where  $\theta_{k}$  and  $\theta_{q}$  denote the parameters of  $f_{k}$  and  $f_{q}$ , respectively. We then have:
$$
\boldsymbol {z} _ {i} = g _ {q} (\boldsymbol {h} _ {i})
$$
$$
\boldsymbol {z} _ {i} ^ {\prime} = g _ {q} \left(\boldsymbol {h} _ {i} ^ {\prime}\right)
$$
where  $\boldsymbol{z}_i, \boldsymbol{z}_i' \in \mathbb{R}^c$ ,  $g_{q}(\cdot)$  and  $g_{k}(\cdot)$  are MLPs with one hidden layer of sigmoid activation, respectively. Following Chen et al. (2020), we conduct contrastive learning on  $\boldsymbol{z}$  instead of  $\boldsymbol{h}$  to prevent the contrastive learning objective from removing information useful for downstream tasks. After contrastive learning, we use  $\boldsymbol{h}$  as the sentence representation for downstream tasks.
Additionally, we also maintain a dynamic first-in-first-out queue for the negative examples. During training, before computing contrastive loss at the end of each batch, all encoded examples of the
current batch are enqueued into the queue, and the oldest examples are dequeued simultaneously.
In our experiments, we use the attack described in section 3.3 or back-translation (Zhu et al., 2015) for augmentation  $t(\cdot)$ . Assume that we have an encoded example  $z_{i}$  and the encoded examples in the queue are  $\{z_0,z_1,\dots ,z_{Q - 1}\}$ , where  $Q$  is the size of the queue. Among the encoded examples in the queue, one of them is  $z_{i}^{\prime}$ , which forms a pair of positive examples with  $z_{i}$ . We use contrastive loss to maximize the similarity between positive examples, while minimizing the similarity of negative examples. We then have:
$$
\ell_ {i} = - \log \frac {\exp (\sin (\boldsymbol {z} _ {i} , \boldsymbol {z} _ {i} ^ {\prime}) / \tau)}{\sum_ {k = 0} ^ {Q} \exp (\sin (\boldsymbol {z} _ {i} , \boldsymbol {z} _ {k}) / \tau)} (1)
$$
where  $\tau$  is the temperature parameter,  $\mathrm{sim}(\cdot ,\cdot)$  is the similarity function, and  $Q$  is the size of the dynamic queue. In this paper, we compute similarity by dot product as in MoCo.
By optimizing eq. (1), the goal is to maximize the similarity of representations between similar (positive) pairs of examples while minimizing the similarity of representations between dissimilar (negative) examples. We use the geometry-inspired attack described in section 3.3 as the transformation  $t(\cdot)$  to create pairs of examples that are similar on the word level but at the same time are distant from each other in the representation space.
We illustrate our method in fig. 1. In fig. 1 (b) and (c), by conducting contrastive learning and using the Geometry Attack generated adversarial examples as hard positives, the vector representations obtained from the model become invariant to the adversarial attacks.
Figure 2: An illustration of one iteration in Geometry Attack for contrastive loss. See section 3.3 for details.
3.3 Creating Hard Positive Examples by Geometry Attack
As mentioned in section 3.2, we use an attack as the transformation  $t(\cdot)$  during contrastive learning.
We describe how this attack creates adversarial examples for contrastive loss during self-supervised contrastive learning (see fig. 1 (b)) in this subsection.
Inspired by Meng and Wattenhofer (2020), who leverage geometry of representations to generate natural language adversarial examples for text classification tasks, we also use the geometry of pretrained representations to create adversarial examples for contrastive loss. The created adversarial examples are used as positive examples of the original examples in our contrastive learning framework, and at the same time are created to maximize the contrastive loss. Hence, we refer to adversarial examples created by the attack as hard positive examples.
The intuition of our attack is that we repeatedly replace words in the original texts such that in each iteration, the replaced word increases the contrastive loss as much as possible. To be specific, consider an example  $X_{i}$ , we then have:
1. Determine Direction for Sentence Vector Compute the gradients of  $\ell_{i}$  with respect to  $z_{i}$ . In this step, we find the direction we should move from  $z_{i}$  to increase the contrastive loss  $\ell_{i}$ . We have the gradient vector  $\pmb{v}_{z_i} = \nabla_{z_i}\ell_i$ .
2. Choose Original Word to be Replaced Compute the gradients of  $\ell_{i}$  with respect to input word embeddings of  $X_{i}$ . For words tokenized into multiple tokens, we take the average of the gradients of the tokens. In this step, we find the word  $w_{t}$  which has the most influence in computing  $\ell_{i}$ . Specifically, assuming we have  $L$  words, then we choose  $t = \arg \max_{t}\{||\pmb{g}_{1}||,||\pmb{g}_{2}||,\dots ,||\pmb{g}_{L}||\}$ , where  $\pmb{g}_k$  is the gradients of  $l_{i}$  with respect to the embeddings of word  $w_{k}, 1\leq k\leq L$ .
3. Generate Candidate Set Suppose we choose the word  $w_{t}$  in step 2. In this step, we use a pretrained BERT to choose the most probable candidates  $w_{t}$  to replace it in the original text. We have the candidates set  $= \{w_{t_1}, w_{t_2}, \dots, w_{t_T}\}$ . Following Jin et al. (2020), we filter out semantically different words from the candidate set by discarding candidate words of which the cosine similarity of their embeddings between the embeddings of  $w_{t}$  is below a threshold  $\epsilon$ . We set the threshold  $\epsilon = 0.5$  and use counter-fitted word embeddings (Mrkšić et al., 2016) to compute the cosine similarity.
4. Choose Replacement Word Replace  $w_{t}$  with words in the candidates set, resulting in text vec
tors  $\{z_{i_1},z_{i_2},\dots ,z_{i_T}\}$  .We compute delta vector  $\pmb{r}_{ij}\gets \pmb{z}_{ij} - \pmb {z}_i$  . The projection of  $r_{ij}$  onto  $\pmb{v}_{z_i}$  is:  $\pmb {p}_{ij}\leftarrow \frac{\pmb{r}_{ij}\cdot\pmb{v}_{zi}}{\|\pmb{v}_{zi}\|}$  . We select word  $w_{t_m}$  ,where  $m\gets \mathrm{argmax}_j||\pmb {p}_{ij}||$  .In other words,  $w_{t_m}$  results in the largest projection  $\pmb{p}_{i_m}$  onto  $\pmb{v}_{z_i}$
5. Repetition Replace  $w_{t}$  with  $w_{t_m}$  in  $X_{i}$ , then we have  $z_{i}\gets z_{i_{m}}$ . Repeat step 1-4 for  $N$  iterations, where  $N$  is a hyperparameter of our method. We expect  $\ell_{i}$  to increase in each iteration.
Figure 2 illustrates an iteration of our attack, in which we have two options to choose from the candidate set. This attack can be easily implemented in a batched fashion, making it possible for us to generate adversarial examples on the fly during training. Furthermore, our efficient implementation makes it possible to conduct contrastive learning with adversarial perturbations as well as adversarial training with adversarial examples generated on the fly. We give a speed comparison of our attack and other attacks in appendix D. We also give pseudocode of the attack in algorithm 1 of appendix A.
We test how our method improves model robustness on four text classification datasets: AG's News, Yelp, IMDB, and DBpedia (See appendix B for details).
We report the attack success rate and the replacement rate of the attacks as the evaluation metrics. Following Alzantot et al. (2018); Ebrahimi et al. (2018), to prevent the model accuracy on clean examples from confounding the results, we define the success rate of an attack on all correctly classified examples in the test set. Lower success rates indicate higher robustness. The replacement rate refers to the percent of original words replaced in the clean example. Higher replacement rates indicate that the attack needs to replace more words to fool the model, and thus mean that the model is more robust.
4.2 Attacks for Evaluating Robustness
We use four word substitution-based adversarial attacks to evaluate the model robustness.
Geometry Attack We use the same attack described in section 3.3 to generate adversarial examples for sentence classification tasks by replacing contrastive loss with cross-entropy classification loss. We set the maximum number of replaced words to 20.
TextFollower, PWWS, and BAE-R We use the default implementations from TextAttack (Morris et al., 2020).
All these attacks will give up and terminate once the maximum number of replaced words (sometimes also called perturbation budget) is reached.
We have the following hypotheses for our method:
H1: Self-supervised contrastive learning improves model robustness against adversarial attacks. Moreover, using adversarial perturbations during contrastive learning further improves robustness.
To validate this hypothesis, we set three different pretraining schemes:
BTCL: Pretraining with back-translation as the transformation  $t(\cdot)$  for self-supervised contrastive learning.
ADCL: Pretraining with Geometry Attack for contrastive loss (see section 3.3) as transformation  $t(\cdot)$  for self-supervised contrastive learning.
NP: Apart from the above two settings, we also add a No Pretraining baseline to understand the general effectiveness of contrastive learning.
H2: Combining self-supervised contrastive learning with adversarial training gives higher robustness than conducting adversarial training alone.
We use different finetune strategies to understand how adding adversarial training to our method affects model robustness. We have two settings:
FTC: We finetune the pretrained model on the clean examples of the corresponding downstream dataset.
ADV: We conduct adversarial training by leveraging supervisedly generated adversarial examples. Note that our adversarial training is different from previous works (Ren et al., 2019; Alzantot et al., 2018), which merely finetune the model on a fixed number of pre-generated adversarial examples. Instead, our adversarial training scheme is similar to Madry et al. (2018), where the model is finetuned on clean examples and adversarial examples generated on the fly during each batch of training.
We use Geometry Attack for adversarial training as the remaining three attacks are not efficient enough to generate adversarial examples on the fly (see appendix D for details).
H3: Our contrastive learning method is capable of using out-of-domain data to improve the model robustness.
While in H1 and H2, we use the same dataset for pretraining and finetuning, we want to test how our
method can leverage out-of-domain data. Hence, we have two additional experimental settings:
In-Domain: We use the same dataset during contrastive learning and finetuning.