SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks
Wenhan Yang
Jingdong Gao
Baharan Mirzasoleiman
{hangeryang18, mxuan, baharan} @cs.ucla.edu
Computer Science Department, UCLA
Contrastive vision-language representation learning has achieved state-of-the-art performance for zero-shot classification, by learning from millions of image-caption pairs crawled from the internet. However, the massive data that powers large multimodal models such as CLIP, makes them extremely vulnerable to various types of targeted data poisoning and backdoor attacks. Despite this vulnerability, robust contrastive vision-language pre-training against such attacks has remained unaddressed. In this work, we propose RoCLIP, the first effective method for robust pre-training multimodal vision-language models against targeted data poisoning and backdoor attacks. RoCLIP effectively breaks the association between poisoned image-caption pairs by considering a relatively large and varying pool of random captions, and matching every image with the text that is most similar to it in the pool instead of its own caption, every few epochs. It also leverages image and text augmentations to further strengthen the defense and improve the performance of the model. Our extensive experiments show that RoCLIP renders state-of-the-art targeted data poisoning and backdoor attacks ineffective during pre-training CLIP models. In particular, RoCLIP decreases the success rate for targeted data poisoning attacks from  $93.75\%$  to  $12.5\%$  and that of backdoor attacks down to  $0\%$ , while improving the model's linear probe performance by  $10\%$  and maintains a similar zero-shot performance compared to CLIP. By increasing the frequency of matching, RoCLIP is able to defend strong attacks, which add up to  $1\%$  poisoned examples to the data, and successfully maintain a low attack success rate of  $12.5\%$ , while trading off the performance on some tasks<sup>1</sup>.
Recent large-scale vision-language models pre-trained on millions of image-caption pairs crawled from the internet has gained an unprecedented success. Large-scale vision-language models such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) are trained using a multimodal contrastive loss which pulls the representations of every image-caption pair together while pushing those of different pairs apart. In doing so, they can learn state-of-the-art image representations, without the need for a fixed set of predetermined labels to be specified at pt-training time. This enables zero-shot transfer of the model to downstream tasks, without requiring specialized output heads or dataset specific customization. Instead, natural language can be used to reference the learned visual concepts afterwards (Radford et al., 2021; Jia et al., 2021). Crucially, alleviating the need for expensive labeling of training examples enables scaling up the training data to millions of examples. However, the massive data that powers such large models also makes them extremely vulnerable to various types of targeted data poisoning and backdoor attacks (Carlini & Terzis, 2021; Yang et al., 2022).
$^{1}$ Code is available at https://github.com/BigML-CS-UCLA/RoCLIP
Figure 1: Illustration of RoCLIP, for defending CLIP during pre-training. (a) RoCLIP keeps a pool of random varying captions. During training, RoCLIP augments all images and captions. Then, it finds the most similar caption  $\mathbf{z}_{nn}^{T}(i)$  to an augmented image  $\overline{\mathbf{z}_i^I}$ ; and instead of matching the image  $\mathbf{z}_i^I$  and caption  $\mathbf{z}_i^T$ , it matches every augmented image to its  $\mathbf{z}_{nn}^{T}(i)$ .
Targeted data poisoning attacks on multimodal models add mismatched image-caption pairs to the pre-training data, to change the prediction of particular images at the test time. Similarly, backdoor attacks overlay a small patch on a subset of training data to cause the model to misclassify test images with the same patch. Notably, poisoning just  $0.0001\%$  of the pre-training examples can lead to success of a targeted poisoning attack. Similarly, poisoning  $0.01\%$  of pre-training examples can make a backdoor attack successful (Carlini & Terzis, 2021). Compared to clean-label data poisoning and backdoor attacks in the supervised settings which require poisoning on average  $1\%$  of training data (Turner et al., 2018; Geiping et al., 2021), attacking multimodal contrastive models requires orders of magnitude fewer poisoned examples. Interestingly, the larger the model, the more vulnerable it is against adversarial attacks (Carlini & Terzis, 2021).
Despite this vulnerability, robust pre-training of multimodal vision-language models against targeted data poisoning and backdoor attacks has remained unaddressed. The recent work of Yang et al. (2022) studied poison identification during fine-tuning of CLIP, by using another CLIP model pre-trained on clean data to remove dissimilar image-caption pairs. Furthermore, Bansal et al. (2023) proposed CleanCLIP to eliminate the effect of backdoor attacks from a pre-trained CLIP model, by fine-tuning on a large subset of the clean pre-training data with in-modality contrastive loss on both vision and language modalities. However, such approaches are not applicable to pre-training, as we also confirm experimentally in Appendix 7.1.
In this work, we propose the first effective method, namely RoCLIP, for robust pre-training of multimodal vision-language models such as CLIP, against targeted data poisoning and backdoor attacks. Our approach is based on the following key observation: while the similarity between the image-caption pairs of clean examples increases rapidly during the training, similarity between poisoned image-caption pairs grows at a slower speed, early in training. As a result, poisoned images and captions are not close to the groups of similar images and captions in the representation space, during the initial training iterations. To break the association between poisoned image-caption pairs, our main idea is to keep a relatively large and varying pool of randomly selected captions. Then, we match every image with the text that is most similar to it in the pool instead of its original caption. This effectively prevents the attack by breaking the association between poisoned image-caption pairs, from early training epochs. We further strengthen our defense and improve the performance by leveraging image and text augmentations.
Our extensive experiments show that our method renders state-of-the-art targeted data poisoning and backdoor attacks ineffective during pre-training. In addition, our method leads to an increase of linear probe accuracy by up to  $10\%$  while having a zero shot performance on par with CLIP. By increasing the frequency of matching, RoCLIP is able to defend strong attacks, which add up to  $1\%$  poisoned examples to the data, and successfully maintain a low attack success rate of  $12.5\%$ , while trading off the performance on some tasks. We note that RoCLIP is the only effective defense method against state-of-the-art attacks that can efficiently scale to pre-training large-scale vision-language models such as CLIP.
Contrastive Representation Learning. Contrastive learning was originally proposed for self-supervised representation learning from unimodal data. Self-supervised contrastive learning works by maximizing agreement between differently augmented views of the same example and minimizing agreement between differently augmented views of different examples (Chen et al., 2020; Chen & He, 2021; He et al., 2020). Several works improved the performance of contrastive-learning on downstream tasks by imposing additional constraints to remove redundancy between components of the representation vectors and prevent collapse of the representations (Bardes et al., 2021; Zbontar et al., 2021), or using nearest-neighbor as positive pairs in the contrastive loss (Dwibedi et al., 2021; Van Gansbeke et al., 2021).
Contrastive Language-Image Pretraining. Multimodal vision-language models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) are pre-trained on  $400\mathrm{M} / 1\mathrm{B}$  image-text pairs, by maximizing the agreement between image and text representations of every image-caption pairs and minimizing those of different pairs. A recent line of work aims at improving the data efficiency and quality of CLIP representations, by leveraging image and text augmentations. DeCLIP (Li et al., 2021) improves data-efficiency of CLIP by maximizing the similarity between two augmented image features using SimSiam (Chen & He, 2021), two augmented text features using Masked Language Modeling (MLM) (Devlin et al., 2018), and matching augmented image features with their augmented text pairs and other similar text features. SLIP (Mu et al., 2022) improves the performance by maximizing the agreement between two augmented image features using SimCLR (Chen et al., 2020), and matching the augmented image features with their text pair. CyCLIP (Goel et al., 2022) improves the representations by symmetrization of the similarity between the two mismatched image-text pairs, as well as the similarity between the image-image pair and the text-text pair. Finally, FILIP (Yao et al., 2021) uses transformer-based encoders for both modalities to learn more fine-grained features.
Targeted Data Poisoning and Backdoor Attacks on CLIP. Contrastive pretrained language-image models are extremely vulnerable to various types of targeted data poisoning and backdoor attacks (Carlini & Terzis, 2021). Targeted data poisoning attacks fool the model to misclassify a particular test example as an adversarial label. On the other hand, backdoor attacks overlay a small patch on a subset of training data, and cause the model to misclassify test images with the same patch. Despite this vulnerability, designing effective defense methods to protect the model from being poisoned during pre-training has remained unaddressed.
CLIP has been also shown to be vulnerable to data poisoning attacks during fine-tuning (Yang et al., 2022). To address this, Yang et al. (2022) proposed a pre-processing and a post-processing defense to defend against targeted data poisoning attacks during fine-tuning. The pre-processing requires a clean pre-trained CLIP to remove examples with low cosine similarity between image and their corresponding text representation. The post-processing fine-tunes the poisoned model on another clean dataset of the same scale as the fine-tuning data. Besides, Bansal et al. (2023) proposed a method to clean backdoor attacks from CLIP, by fine-tuning the model on a clean subset of the pre-training dataset with in-modality contrastive loss on both vision and language modalities. Such approaches are however not applicable to pre-training, as we also confirm experimentally in Appendix 7.1. In our work, we propose the first effective defense method that can protect the model from both targeted image attacks and backdoor attacks during pre-training.
3.1 Contrastive Language-Image Pre-training (CLIP)
CLIP is trained on millions of image-caption pairs scraped from the web. Formally, we consider a dataset  $\mathcal{D} \subseteq \mathcal{I} \times \mathcal{T}$  consisting of pairs  $(\pmb{x}_j^I, \pmb{x}_j^T)$  where  $\pmb{x}_j^I \in \mathcal{I}$  is a raw image and  $\pmb{x}_j^T \in \mathcal{T}$  is a text (caption of the image). The CLIP architecture consists of an image encoder  $f_I: \mathcal{I} \to \mathbb{R}^d$  that encodes the raw image  $x_i^I$  into an embedding vector  $\tilde{\pmb{z}}_i^I$ , and a text encoder  $f_T: \mathcal{T} \to \mathbb{R}^d$  that encodes the raw text  $\pmb{x}_i^T$  into an embedding vector  $\tilde{\pmb{z}}_i^T$  of the same dimension. Then projected image and text embeddings  $\pmb{z}_i^I, \pmb{z}_i^T$  are obtained by passing the encoded image and text  $\pmb{z}_i^I, \pmb{z}_j^T$  through their corresponding projection heads. The projected representations are normalized to have unit  $\ell_2$ -norm. Finally, the InfoNCE loss (Oord et al., 2018) is applied to pull the projected embeddings of every
image and its corresponding caption together while pushing apart the projected embeddings of the image from other captions in the same mini-batch. Formally, for a mini-batch of  $N$  image-caption pairs  $\{(x_j^I,x_j^T)\}_{j = 1}^N$ , and their projected embeddings  $\{(z_j^I,z_j^T)\}_{j = 1}^N$ , the CLIP loss is defined as:
$$
\mathcal {L} _ {\mathrm {C L I P}} = - \frac {1}{2 N} \sum_ {j = 1} ^ {N} \log \left[ \frac {\exp \left(\left\langle \boldsymbol {z} _ {j} ^ {I} , \boldsymbol {z} _ {j} ^ {T} \right\rangle / \tau\right)}{\sum_ {k = 1} ^ {N} \exp \left(\left\langle \boldsymbol {z} _ {j} ^ {I} , \boldsymbol {z} _ {k} ^ {T} \right\rangle / \tau\right)} \right] - \frac {1}{2 N} \sum_ {k = 1} ^ {N} \log \left[ \frac {\exp \left(\left\langle \boldsymbol {z} _ {k} ^ {I} , \boldsymbol {z} _ {k} ^ {T} \right\rangle / \tau\right)}{\sum_ {j = 1} ^ {N} \exp \left(\left\langle \boldsymbol {z} _ {j} ^ {I} , \boldsymbol {z} _ {k} ^ {T} \right\rangle / \tau\right)} \right], \tag {1}
$$
where  $\langle .,.\rangle$  represents the inner product, and  $\tau$  is a trainable temperature parameter. The performance of the pre-trained CLIP model is evaluated via zero-shot and linear probe methods, as discussed below.
Zero-shot classification. Pre-trained Language-Image models such as CLIP enable zero-shot transfer of the model to downstream tasks, without the need for specialized output heads or dataset specific customization. To do so, the downstream labels is transformed into natural language captions using the provided engineered prompts templates, e.g. "A photo of a {label}". Then, the cosine similarity of the test image to each caption is computed, and the model predicts the label with the highest image-caption similarity.
Linear probe. For a labeled downstream image dataset, CLIP image representations can also be evaluated by training a linear classifier on the image representations obtained from the pre-trained CLIP image encoder and the downstream labels.
3.2 Targeted Poisoning and Backdoor Attacks
Let  $\mathcal{D} = \{(\pmb{x}_i^I, \pmb{x}_i^T)\}_{i=1}^n$  be the set of all training examples. Poisoning attacks (Biggio et al., 2012) inject a small subset of poisoned examples  $\mathcal{D}_p$  to the original training dataset  $\mathcal{D}$ , such that when the model is trained on the poisoned training data  $\{\mathcal{D} \cup \mathcal{D}_p\}$ , its prediction on particular test examples are changed to the adversarial label  $y_{adv}$ . At the same time, the poisoned model performs normally on other test examples. In this work, we consider both targeted poisoning and backdoor attacks as we discuss next.
Targeted Image attacks. In a targeted poisoning attack, the adversary aims to change the prediction of one particular test examples  $\pmb{x}_t^I$  to the adversarial label  $y_{adv}$ . Targeted poisoning attacks can be crafted following (Carlini & Terzis, 2021), by constructing a caption set  $\mathcal{T}_{adv}$  of potential text descriptions related to the label  $y_{adv}$ , and making poisoned examples by assigning captions in  $\mathcal{T}_{adv}$  to every target  $\pmb{x}_t^I$ , i.e.,  $\mathcal{D}_p = \{(\pmb{x}_t^I, \pmb{x}_c^T): \pmb{x}_c^T \in \mathcal{T}_{adv}\}$ . For constructing the caption set  $\mathcal{T}_{adv}$ , one can search the training dataset for all sequences that contain this label string, and use these sequences as the caption set. Alternatively, one can use the set of 80 different engineered prompt templates provided by CLIP for classification (Radford et al., 2021), and either use a subset or repeat them as necessary. The number of captions in  $\mathcal{T}_{adv}$  determines the number of generated poisons per target. To evade automated cleaning algorithms (e.g., removing duplicated images), tiny Gaussian noise can be added to the images, or the captions can be modified by substituting or adding words, without degrading the attack success rate. A diverse caption set ensures that the image encoder is poisoned instead of the projection layers.
Backdoor attacks. In backdoor attacks, the adversary attaches a small trigger patch to the poisoned images and pair them with adversarial captions  $\mathcal{T}_{adv}$  related to  $y_{adv}$ . In doing so, all the test images with the trigger patch will be misclassified as  $y_{adv}$ . In contrast to the targeted poisoning attack, instead of using a particular  $\boldsymbol{x}_t^I$ , the adversary poisons different images  $\boldsymbol{x}_i^I \in \mathcal{I}$ , by adding the trigger patch to them. Specifically, the poisoned set is defined as  $\mathcal{D}_p = \{(x_i^I \oplus \text{patch}, x_c^T): x_c^T \in \mathcal{T}_{adv}, x_i^I \in \mathcal{I}\}$ . The caption set can be constructed in a similar manner to targeted poisoning attacks using captions found in the training data or engineered prompt templates.
In general, while injecting poisoned examples in curated datasets used for supervised learning might be difficult, such poisons can be easily injected in uncurated datasets used by large multimodal models. This makes such models highly vulnerable to adversarial attacks.
Algorithm 1 Robust CLIP pre-training (RoCLIP)
1: Input: Image encoder  $f_{I}$  , text encoder  $f_{T}$  caption pool  $\mathcal{P} = \{\pmb {z}_i^T\}_{i = 1}^P$  initialized with random captions, RoCLIP frequency  $\kappa$    
2: for epoch  $= 1,\dots ,T$  do   
3: for every mini-batch of image-caption pairs:  $\{(x_j^I,x_j^T)\}_{j = 1}^N\in \mathcal{D}$  do   
4: Augment image and captions in the mini-batch:  $\{(\bar{\pmb{x}}_j^I,\bar{\pmb{x}}_j^T)\}_{j = 1}^N$    
5: if  $T \% \mathcal{K} == 0$  then   
6: Pair every  $\bar{\pmb{x}}_j^I$  with its nearest augmented caption in pool  $\pmb{z}_{nn(j)}^{T} = \arg \min_{\overline{\pmb{z}}_{p}^{-T}\in \mathcal{P}}\| \overline{\pmb{z}}_{j}^{I}-$ $\overline{\pmb{z}_p^T}\| _2$    
7: Append the pool with captions in the mini-batch and discarding the oldest captions   
8: Train  $f_{I}$  and  $f_{T}$  with  $\mathcal{L}_{\mathrm{ROCLIP}}$  in Eq. 2   
9: else   
10: Train  $f_{I}$  and  $f_{T}$  with  $\mathcal{L}_{\mathrm{CLIP}}$  in Eq. 1
3.3 Threat Model
Adversary Objective. The primary objective of the adversary is to manipulate the output representations of CLIP, such that certain images are misclassified into adversarial categories instead of their true categories, while the other images are classified correctly.
Adversary Capabilities. We assume that the adversary has limited control over the pre-training data, and can inject a small number ( $\leq 1\%$  of the dataset size) of poisoned examples into the training dataset. Adversary also has the knowledge of the model structure, the training algorithm, and the hyperparameter used by their victim, but they cannot modify the training process directly.
4 Robust Training of Multimodal Models against Targeted Data Poisoning and Backdoor Attacks
In this section, we first study the effect of data poisoning attacks on multimodal models. Then, we present our method for robust training of such models against data poisoning and backdoor attacks.
4.1 Effect of Data Poisoning Attacks on CLIP
We start by studying the effect of data poisoning attacks on multimodal models. By minimizing the CLIP contrastive loss in Eq. (1), the model changes such that every image representation  $\boldsymbol{z}_j^I$  moves towards its caption representation  $\boldsymbol{z}_j^T$  and gets far away from other (dissimilar) caption representations  $\boldsymbol{z}_k^T$ . This makes corresponding categories of image and text (e.g. category of "Cat" or "Dog" in the image and text modality) to get closer to each other and get distant from other categories, during the pre-training. Crucially, as image_caption pairs belonging to a particular category are relatively similar to each other, their gradients have a large alignment with other examples in the same category. Therefore, categories of similar image_caption pairs insert a large cumulative gradient on the model and change the model quickly to get close to their pairs in the other modality.
On the other hand, image-caption pairs of poisoned examples are not similar to the other clean examples in the data. Hence, their gradient does not align well with the gradient of the clean examples. Therefore, by training on the poisoned pairs, the poisoned images and captions get far away from the other images and captions in the same category. That is, poisoned images become distant from other images in the same category, and adversarial captions become distance from other captions in the adversarial category. Next, we will rely on this observation to break the association between poisoned image-caption pairs.
4.2 Robust Training with ROCLIP
To prevent targeted data poisoning and backdoor attacks from being successful, we aim to break the association between the poisoned image-caption pairs. If this can be done, the poisoned image and caption representations do not get close enough to each other during training and the attack does not
Figure 3: Examples of matching the images in the clean or poisoned pairs during the training. The image in the clean pair can find semantically similar captions in the caption pool and still match to the correct captions, while the image in the poisoned pair will match to either random captions or captions corresponding to the original image.
succeed. To achieve this, we rely on our key observation in Sec. 4.1: poisoned images and captions are not close to groups of similar images and captions in the representation space, early in training. We leverage two techniques to break the association between poisoned image-caption pairs: (1) A large and varying pool of randomly selected captions; (2) Augmentations on both images and captions.
The Pool. As illustrated in Fig. 1, instead of matching every image with its corresponding caption, we match every image with the caption in the pool that is most similar to the image. Based on the observation from Sec. 4.1, poisoned captions are not close to other captions in the same category. Thus, our method prevents the poisoned images to be matched with captions from the adversarial category. In doing so, it breaks the data poisoning and backdoor attacks. Fig. 2 shows the effect of pool when training on a dataset with 15 poisoned images of a deer  $\boldsymbol{x}_t^I$  paired with adversarial truck captions  $\{(\boldsymbol{x}_t^I, \boldsymbol{x}_c^T): \boldsymbol{x}_c^T \in \mathcal{T}_{\mathrm{truck}}\}$ . We see that while the majority of clean images are matched with captions from the same category as their original caption during training (blue line), poisoned images are matched with a caption from a different category than the adversarial label (orange line). This confirms our observation in Sec. 4.1, and shows the effectiveness of the pool. The pool may have a negative influence on the per
formance, as it is unreliable especially early in training. To ensure a good model performance, we (1) select a relatively large pool size so that every clean image can find caption similar to its original caption; (2) train with our method every  $\kappa$  epoch, and train with standard CLIP loss in the other epochs. Note that it is necessary to train on the poisoned pairs consistently to poison the model. With a smaller  $\kappa$ , the method is able to defend the model and disassociate the poisoned image-caption pairs. We select  $2\%$  of the total dataset size as our pool size and  $\kappa = 3$  in our experiments. Fig. 3 illustrates an example of how RoCLIP disassociates poisoned pairs during the training.