SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Augmentation. To strengthen our defense, we use various image and text augmentations during the training. In particular, we use random image cropping, horizontal flipping, color jittering (Wu et al., 2018), grayscale conversion (Wu et al., 2018), and blurring (Chen et al., 2020) in our image augmentation policies. For the text augmentation, we use the EDA proposed by (Wei & Zou, 2019), which includes synonym replacement, random swap, and random deletion as its augmentation policies. The benefit of the augmentations are two-folded: First, similar to attacks on supervised learning (Borgnia et al., 2021), it helps further reducing the attack success rates, but cannot fully defend the model, as we will confirm in our ablation study. Data augmentation is especially helpful during the
Figure 2: Training with RoCLIP. Majority of clean images are matched with captions from the same category as their original caption during training (blue line), while poisoned images are matched with a caption from a different category than the adversarial label (orange line).
CLIP epochs to prevent matching poisoned image-caption pairs. Second, augmentations can further increase the method's performance, as shown in Table 3 and Table 7.
In summary, RoCLIP first samples a pool of  $P$  caption representations  $\mathcal{P} = \{\pmb{z}_i^T\}_{i=1}^P$  uniformly at random. During training, for every example  $(\pmb{x}_j^I, \pmb{x}_j^T)$  in the mini-batch, we first augment its image and text with our augmentation policies, and then match its augmented image representation  $\overline{z}_j^I$  with the augmented caption representation in the pool that is most similar to  $\overline{z}_j^I$ , i.e.,  $\pmb{z}_{nn(j)}^T = \arg \min_{\overline{\pmb{z}}_p^T \in \mathcal{P}} \| \overline{z}_j^I - \overline{z}_p^T \|_2$ . Effectively, we form the positive image-caption representation pair  $(\overline{\pmb{z}}_j^I, \pmb{z}_{nn(j)}^T)$  and use it instead of  $(\pmb{z}_j^I, \pmb{z}_j^T)$ . Similar to the CLIP loss, we obtain the negative pairs from the mini-batch. That is, for a mini-batch of  $N$  image-caption pairs  $\{(\pmb{x}_j^I, \pmb{x}_j^T)\}_{j=1}^N$ , and their projected embeddings  $\{(\overline{\pmb{z}}_j^I, \overline{\pmb{z}}_j^T)\}_{j=1}^N$ , the loss is defined as:
$$
\begin{array}{l} \mathcal {L} _ {\text {R o C L I P}} = \tag {2} \\ - \frac {1}{2 N} \sum_ {j = 1} ^ {N} \log \left[ \frac {\exp \left(\left\langle \overline {{z _ {j}}} ^ {I} , z _ {n n (j)} ^ {T} \right\rangle / \tau\right)}{\sum_ {k = 1} ^ {N} \exp \left(\left\langle \overline {{z _ {j}}} ^ {I} , z _ {n n (k)} ^ {T} \right\rangle / \tau\right)} \right] - \frac {1}{2 N} \sum_ {k = 1} ^ {N} \log \left[ \frac {\exp \left(\left\langle \overline {{z _ {k}}} ^ {I} , z _ {n n (k)} ^ {T} \right\rangle / \tau\right)}{\sum_ {j = 1} ^ {N} \exp \left(\left\langle \overline {{z _ {j}}} ^ {I} , z _ {n n (k)} ^ {T} \right\rangle / \tau\right)} \right]. \\ \end{array}
$$
For the pool  $\mathcal{P}$ , we consider a first-in-first-out queue, which is initialized with random caption representations. After training on every mini-batch, we update  $\mathcal{P}$  by taking the caption representations of the  $N$  examples in the mini-batch and concatenating them at the end of the queue. We discard the oldest  $N$  elements from the queue, which equals to the training batch size.
The pseudocode of our method, RoCLIP, is illustrated in Alg. 1.
In this section, we evaluate the effectiveness of RoCLIP in breaking targeted data poison and backdoor attacks while maintaining the model's performance. We evaluate our method for defending against various data poisoning and backdoor attacks, described in Sec. 3.2, during pre-training.
5.1 Training
We evaluate RoCLIP during pretraining on CLIP. For pre-training, we use an open-source implementation of CLIP as our model, with default ResNet-50 as the image encoder and Transformer as the text encoder. Each experiment is run with a batch size of 512 for 24 epochs, as the performance improvement afterwards is not significant, but more training makes the model more prone to being poisoned. We use Conceptual Captions 3M (CC3M) (Sharma et al., 2018) as our pre-training dataset. Due to lim
ited computational resources, for pre-training we randomly sampled 1M image-caption pairs from CC3M as our training dataset. We assess our method on 10 downstream datasets introduced by (Kornblith et al., 2019), the detail of which can be found in Table 1.
Table 1: Details of downstream datasets.
DATASET	CLASSES	TRAIN SIZE	TEST SIZE
CALTECH101	102	3,060	6,085
CIFAR10	10	50,000	10,000
CIFAR100	100	50,000	10,000
DTD	47	3,760	1,880
FGVCAIRCRAFT	100	6,667	3,333
FLOWERS102	102	2,040	6,149
FOOD101	101	75,750	25,250
IMAGENET1K	1000	50,000	50,000
OXFORDIIITPET	37	3,680	3,669
STANFORDCARS	196	8,144	8,041
CONC. CAPT. (1M)	-	1,000,000	-
5.2 Attack Methods
We consider targeted image attacks and backdoor attacks, discussed in Sec. 3.2.
Targeted Image Attacks. In our pre-training experiment, we choose a random target image  $\boldsymbol{x}_t$  from the conceptual captions validation set, and then choose a random target class from the ImageNet test set to generate a set of  $|\mathcal{T}_{adv}|$  adversarial captions. Note that, (Carlini & Terzis, 2021) pre-trained 32 CLIP models and measured the attack success rate as the fraction of positioned models. This requires 3200 GPU hours. To reduce the computation, we poison 16 different random images by generating  $|\mathcal{T}_{adv}|$  adversarial captions for each image? related to a label selected at random from ImageNet.
Table 3: Linear probe and zero-shot top 1 accuracy. RoCLIP improves the performance of the model by up to  $10\%$  for linear probe and obtains a similar zero-shot performance compared with CLIP.
Method	Task	F102	Fd101	I1K	Pet	Cars	Cal101	C10	C100	DTD	Air.
RoCLIP	0-shot	0.83	6.34	6.63	3.68	0.72	30.38	30.14	9.52	3.56	1.11
lin-prb	99.22	54.05	24.09	52.36	20.35	72.15	78.99	57.82	55.21	32.55
CLIP	0-shot	1.0	7.1	9.6	3.4	0.8	34.9	34.9	7.3	3.7	0.8
lin-prb	99.5	44.9	22.2	48.2	12.9	70.4	70.5	45.8	48.2	24.9
Then, we report the attack success rate as the fraction of images that are classified as the adversarial label, in a single pre-training run. In doing so, the attack success rate will be at least as high as (Carlini & Terzis, 2021). In addition, note that attacking our smaller dataset of 1M examples also results in a higher attack success rate compared to that of 3M used by (Carlini & Terzis, 2021). We will first evaluate the effectiveness of RoCLIP in defending a moderate amount of poisons  $(|\mathcal{T}_{adv}| = 50)$  without any performance loss. Then, we will examine if RoCLIP can defend very strong data poisoning attacks  $(|\mathcal{T}_{adv}| = 100 - 10,000)$  successfully.
Backdoor Attacks. We use the public Hidden Trigger Backdoor Attacks (HTBA) patches (Saha et al., 2020), that are square triggers generated by drawing a random  $4 \times 4$  matrix of colors and resizing it to the desired patch size using bilinear interpolation. We use a resized  $16 \times 16$  patch and put it consistently on the left top corner of the image. In our pre-training experiments, we randomly select 150 images from the CC3M validation dataset and pair them with adversarial captions related to a random target class from ImageNet. To evaluate the effectiveness of the backdoor attacks, we select 300 random images from the ImageNet validation set and patch them in the left top corner to measure the attack success rate.
5.3 RoCLIP Robustly Pre-trains CLIP
First, we evaluate the effectiveness of our method, ROCLIP, against targeted poisoning and backdoor attacks during the pretraining phase. We present our result in Table 2, where attack success rate shows the fraction of poisoned or backdoored images successfully classified as the desired label. We see that without any defense,  $93.75\%$  of the total poisoned images are
Table 2: RoCLIP defense performance.
Model	Attack	Success Rate
CLIP	Target Img	93.75%
Backdoor	78%
RoCLIP	Target Img	12.5%
Backdoor	0%
classified to the desired target class, and  $78\%$  of the backdoored images are classified as the target class. On the other hand, RoCLIP is able to fully defend the attack and reduce the attack success rate to  $0\%$  for the backdoor attacks and as low as  $12.5\%$  for the targeted poisoning attacks. This clearly confirms the effectiveness of RoCLIP in breaking various types of data poisoning and backdoor attacks on CLIP during pre-training.
5.3.1 RoCLIP does not Harm the Performance
Next, we evaluate if RoCLIP negatively impacts the model performance. We assess the performance of RoCLIP on a variety of datasets introduced by (Kornblith et al., 2019), the detail of which can be found in Table 1. We evaluate RoCLIP with both zero-shot and linear-probe methods and present the result in Table 3. It can be seen that RoCLIP does not harm the overall model performance. In contrast, RoCLIP effectively improves the linear-probe classification performance across all ten datasets by up to  $10\%$  and has an on-par zero-shot performance with CLIP. As we will show in our ablation study, data augmentation contributes the most to RoCLIP's performance boost. Nevertheless, even without data augmentation, RoCLIP achieves a similar performance with CLIP.
5.3.2 RoCLIP against Very Strong Attacks
Next, we consider much stronger poisoning attacks against CLIP during pre-training. Note that, in general a much lower poison rate is enough to successfully poison multimodal models (0.0001%),
Table 4: RoCLIP against very strong data poisoning attacks RoCLIP can defend the model with some performance tradeoff. Linear probe and zero-shot performance is reported on CIFAR-10 (C10), CIFAR-100 (C100), Food101 (FD) and Caltech101 (Cal).
Poison Rate	Zero-Shot	Linear-Probe
0.01%	0.0512%	0.5%	1.0%	C10	C100	FD	Cal	C10	C100	FD	Cal
RoCLIP	0%	0%	0%	12.5%	21.3	7.3	3.4	17.0	69.7	46.6	45.6	59.8
CLIP	100%	100%	100%	100%	35.0	7.3	7.1	34.9	70.5	45.8	44.6	70.4
Table 5: Linear-Probe (LP) performance, Poison Success Rate (PSR), and Backdoor Success Rate (BSR) with various pool sizes used in RoCLIP.
Pool Size	256 (0.2%)	2048 (2%)	21845 (20%)	0 (CLIP w/ Aug)	0 (CLIP)
Memory(MB)	39449	39627	39505	39342	37459
Time(s)	538.31	538.69	539.47	536.96	506.02
C10 LP	64.5%	67.1%	67.9%	72.9%	62.1%
C100 LP	34.9%	39.6%	41.1%	48.4%	37.4%
PSR	0%	0%	0%	12.5%	75%
BSR	0%	0%	0%	0.33%	37.5%
compared to supervised learning models (1%), and hence the highest poison rate considered in (Carlini & Terzis, 2021) is  $0.017\%$ . Considering the very large size of the pretraining data for multimodal models, this small poison rate translates to a considerable "poison number" that needs to be generated by the adversary. Nevertheless, we evaluate the effectiveness of RoCLIP against a very high poison rate of up to  $1\%$  of the dataset size for targeted image attacks. To defend the attacks effectively, we change the frequency from  $\mathcal{K} = 3$  to  $\mathcal{K} = 2$ . Table 4 shows that RoCLIP with  $\mathcal{K} = 2$  can effectively defend very stronger attacks, while trading off the performance on some tasks.
5.3.3 Comparison to Fine-tuning Defense Baselines
Finally, we compare the performance of RoCLIP with existing methods proposed to defend CLIP during fine-tuning (Bansal et al., 2023; Yang et al., 2022). Due to page limit, we leave the discussions in the Appendix 7.1. Our results confirm that such methods are indeed highly ineffective in defending CLIP during pre-training.
In this section, we conduct further experiments on different components of ROCLIP to evaluate its effects on defense and downstream performance. All experiments are conducted on 100K data randomly sampled from CC3M, with 15 poisoned pairs and 90 backdoored pairs. We train the model with a batch size of 256 for 30 epochs.
5.4.1 RoCLIP against Other Backdoor Attacks
In Sec. 5.3, we explored RoCLIP's effectiveness in defending CLIP against patched backdoor attacks. Here, we evaluate RoCLIP against three additional backdoor attacks baselines, Blended, WaNet, and label-consistent attacks, used in (Bansal et al., 2023). We consider 90 pairs of backdoored image-caption pairs for Blended triggers, 256 pairs for WaNet triggers, and 512 pairs
Table 6: ROCLIP against different backdoors.
Model	Attack	Backdoor Rate	BSR
CLIP	Blended	0.09%	28.00%
WaNet	0.256%	43.40%
Label-Consis	0.512%	97.30%
RoCLIP	Blended	0.09%	0.33%
WaNet	0.256%	0.67%
Label-Consis	0.512%	0.00%
of label-consistent pairs. Table 6 shows that RoCLIP is able to reduce the backdoor success rate to nearly  $0\%$  for all three backdoor attacks.
Table 7: Effect of pool and data augmentation on ROCLIP.
Method	P+A	Pool (P)	Aug (A)	CLIP
C10 LP	67.1%	60.1%	72.9%	62.1%
C100 LP	39.6%	36.5%	48.4%	37.4%
PSR	0%	0%	12.5%	75%
BSR	0%	0.67%	0.33%	42.3%
Figure 4: RoCLIP against more challenging poisoned pairs.
5.4.2 RoCLIP with Different Pool Size
Next, we analyze the computation overhead of RoCLIP, and the effect of pool size on our method. We apply RoCLIP with pool size of  $0.2\%$ ,  $2\%$  and  $20\%$  of the pre-training dataset size. As shown in Table 5, RoCLIP is very lightweight, and has a negligible memory overhead. The overall memory usage of RoCLIP is similar to that of CLIP. In addition, the run-time overhead is also not significant, and is mostly caused by data augmentation. Second, the size of the pool does not have a significant influence on the defense. With different pool sizes, our method is able to defend the both the targeted poisoning and the backdoor attacks. With a larger pool size, however, the downstream performance of the model increases since it is more likely for the images to find fitting captions in a larger pool.
5.4.3 RoCLIP with Augmentations
Finally, we analyze the effect of data augmentations on our method. Table 7 shows that data augmentation improves the downstream performance. Note that even without data augmentation, RoCLIP obtains a similar linear probe accuracy with CLIP. Data augmentation further improves the performance of RoCLIP by  $5\%$  on CIFAR-10 and  $2\%$  on CIFAR-100. In addition, data augmentation significantly improves the defense and reduces the poison success rate from  $75\%$  to  $12.5\%$  and backdoor success rate from  $42.3\%$  to  $0.33\%$ . Using both the pool and data augmentation, RoCLIP is able to reduce the success rate of backdoor and target image attacks to  $0\%$  with a superior downstream performance.
5.4.4 RoCLIP against Adaptive Attacks
One potential adaptive attack against RoCLIP is to use visually similar categories when selecting the poisoned pairs, e.g., dog and cat. Note that, this strongly limits the choice of attackers' target images as well as adversarial captions. To evaluate the effectiveness of RoCLIP against such challenging poisoned pairs, we include 15 poisoned image-caption pairs from 8 visually-similar-categories like dog and cat, football and basketball (c.f. Appendix 7.2). Fig. 4 shows the effectiveness of RoCLIP.
We proposed RoCLIP, an effective method for robust training multimodal vision-language models such as CLIP against data poisoning and backdoor attacks. RoCLIP utilizes a caption pool as well as data augmentation to break the associations between the poisoned image and caption pairs, thus effectively defending the models. Through extensive experiments, we demonstrated that our proposed method drops the success rate of targeted data poisoning and backdoor attacks to  $12.5\%$  and  $0\%$  respectively. At the same time, it improves the model's performance by up to  $10\%$  on linear-probe and obtained a comparable zero shot performance to the baseline CLIP model. By increasing the frequency of matching, RoCLIP is able to defend very strong attacks with  $1\%$  poison rate and maintain a low attack success rate of  $12.5\%$ , while trading off the performance on some tasks.
Limitations. RoCLIP improves the robustness of CLIP pre-training against targeted data poisoning and backdoor attacks. Considering the demand for such models, our work makes a valuable step towards safe machine learning. RoCLIP can defend against a moderate number of poisons with no performance loss and against a high number of poisons with some performance tradeoff. Developing stronger defense methods without performance loss is an interesting direction for future work.
Acknowledgment. This research was supported by the National Science Foundation CAREER Award 2146492 and Cisco Systems.
Saha, A., Subramanya, A., and Piriavash, H. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11957-11965, 2020.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018.
Turner, A., Tsipras, D., and Madry, A. Clean-label backdoor attacks. 2018.
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., and Gool, L. V. Revisiting contrastive methods for unsupervised learning of visual representations. Advances in Neural Information Processing Systems, 34:16238-16250, 2021.
Wei, J. and Zou, K. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196, 2019.
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Yang, Z., He, X., Li, Z., Backes, M., Humbert, M., Berrang, P., and Zhang, Y. Data poisoning attacks against multimodal encoders. arXiv preprint arXiv:2209.15266, 2022.
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., and Xu, C. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310-12320. PMLR, 2021.
7.1 Adapted fine-tuning defense baselines
Here, we apply the methods proposed in (Yang et al., 2022; Bansal et al., 2023) to defend CLIP against data poisoning attacks or eliminate backdoor attacks from a pre-trained model during fine-tuning. We apply these methods during pre-training CLIP and show that they are indeed highly ineffective in protecting the model.
Pre-processing CLIP First, we show that pre-processing CLIP is not effective during pretraining. We conduct our experiment from the 1M sub-dataset randomly-sampled from CC3M. Yang et al. (2022) proposed using a pretrained model to calculate and compare cosine similarity between poisoned image-caption pairs and clean image-caption pairs. Then, it finds a threshold to separate the majority of poisoned pairs from clean pairs, and discard all the pairs with lower similarity than the threshold. We applied the above idea to pretraining, in a cheating experiment. That is, we consider the poisoning ratios of  $0.005\% - 0.5\%$ , and calculate the cosine similarity between the known generated poisoned image-caption pairs (note that in practice the poisons are not known) at epoch 1. We chose epoch 1, as the separation between cosine similarity of poisoned and clean image caption paris is largest at epoch 1 (see Figure 2). Then we set the threshold such that a certain fraction of the poisoned pairs are discarded, and filter examples with cosine similarity lower than this threshold.
Table 8: Ratio of clean data pairs discarded by Pre-processing CLIP when it discard a fixed ratio of poisoned data pairs.
Discard/Poison %	0.005%	0.01%	0.1%	0.5%
75%	15.7% (*)	44.3%	98.3%	95.8%
85%	23.3% (*)	62.3%	98.9%	96.3%
95%	38.1%	87.7%	99.55%	97.1%
Tab. 8 shows the fraction of clean pairs that are discarded using different thresholds. We see that regardless of the threshold, a very large fraction of clean pairs are discarded. We trained and evaluated the two models when less than  $30\%$  of the clean data was discarded, marked with  $(^{*})$ , and show the results in Tab. 9. Both models with smallest amount of clean pairs removed were still poisoned. With  $75\%$  and  $85\%$  of the poison discarded, the targeted data poisoning attacks still have  $53.84\%$  poison success rates. This confirms the ineffectiveness of the above method during pretraining. On the other hand, RoCLIP is able to drop the success rate to  $0\%$  (using  $\mathcal{K} = 2$ ) and  $12.5\%$  (using  $\mathcal{K} = 3$ ), and provides a much better trade-off between the attack success rate and model's performance and can defend a much higher poison number up to  $1\%$ .
Table 9: Defense and downstream performance of the Pre-processing CLIP compared to RoCLIP.
Model	K	Poison %	Discard %	PSR	LP C10	LP C100	ZS C10	ZS C100
CLIP	-	0.005%	-	93.75%	70.5	45.8	34.9	7.2
Pre-proc. CLIP (*)	-	0.005%	75%	58.3	70.2	45.6	27.9	7.3
Pre-proc. CLIP (*)	-	0.005%	85%	58.3	69.9	45.5	24.0	6.5
RoCLIP	3	0.005%	-	12.5%	78.9	57.8	30.1	9.5
RoCLIP	2	0.005%	-	0%	69.7	46.6	21.3	7.27
RoCLIP	2	0.1%	-	0%	69.7	46.6	21.3	7.27
RoCLIP	2	0.5%	-	0%	69.7	46.6	21.3	7.27
RoCLIP	2	1%	-	12.5%	69.7	46.6	21.3	7.27
We note that, the key factor for the success of ROCLIP is that poisoned images and captions are not close to the other images and captions in the same category. For example, a poisoned dog image that is captioned adversarially as a deer, remains far away from the rest of dog images in the representation space, during the initial training epochs. Hence, nearest-neighbor matching done by ROCLIP can effectively break the association of poisoned image caption pairs.
CleanCLIP Next, we show that CleanCLIP is not effective during pretraining. We conduct our experiment from the 100K sub-dataset randomly-sampled from CC3M. (Yang et al., 2022) proposed fine-tuning on a clean dataset (of the same size as pretraining) with CLIP loss, and (Bansal et al., 2023) proposed fine-tuning on a clean subset of size 100K of the original training dataset (same size as the pretraining data we use in this experiment) with CLIP loss+in-modality contrastive loss on both image and text modalities. Since a clean dataset is not available during pretraining, here we use the CleanCLIP loss (CLIP loss + in-modality contrastive loss) to pretrain CLIP on the (poisoned)
Table 10: Defense and downstream performance of CleanCLIP compared to RoCLIP.
Model	Poison %	Backdoor %	PSR	BSR	LP C10	LP C100
RoCLIP	0.015%	0.15%	0%	0%	67.1	39.6
CleanCLIP	0.015%	0.15%	25%	49%	67.7	43.2
CLIP	0.015%	0.15%	75%	49%	62.1	37.4
data. We conduct our experiment on 100K data, with 150 pairs of backdoored image-captions pairs and 15 targeted poisoned pairs. The following table shows that while the in-modality loss enables CleanCLIP to achieve a good performance, CleanCLIP cannot successfully defend against targeted data poisoning and backdoor attacks during pretraining. On the other hand, RoCLIP effectively drops the attack success rate to  $0\%$  for both targeted poisoning and backdoor attacks. This further confirms the effectiveness of RoCLIP.
7.2 Challenging poisons
For the visually challenging poisons, we consider 8 pairs of categories: cat-dog; shark-goldfish; snake-alligator; bee-butterfly; deer-buffalo; soccer-basketball; hammer-screwdriver; train-minivan.