Poisoning and Backdooring Contrastive Learning
Authors: Nicholas Carlini
Andreas Terzis
Venue: ICLR 2022 Oral (2022)
Keywords: Contrastive Learning
Poisoning attack
Backdoor attack
CLIP

Abstract: Multimodal contrastive learning methods like CLIP train on noisy and uncurated training datasets. This is cheaper than labeling datasets manually, and even improves out-of-distribution robustness. We show that this practice makes backdoor and poisoning attacks a significant threat. By poisoning just 0.01% of a dataset (e.g., just 300 images of the 3 million-example Conceptual Captions dataset), we can cause the model to misclassify test images by overlaying a small patch. Targeted poisoning attacks, whereby the model misclassifies a particular test input  with an adversarially-desired label, are even easier requiring control of 0.0001% of the dataset (e.g., just three out of the 3 million images). Our attacks call into question whether training on noisy and uncurated Internet scrapes is desirable.

Published as a conference paper at ICLR 2022
POISONING AND BACKDOORING CONTRASTIVE LEARNING
Nicholas Carlini
Andreas Terzis
Google
Google
ABSTRACT
Multimodal contrastive learning methods like CLIP train on noisy and uncurated training datasets. This is cheaper than labeling datasets manually, and even im- proves out-of-distribution robustness. We show that this practice makes buckdoor and poisoning attacks a sigmificant threat. By poisoning just 0.01% of a dataset (e.g.. just 300 images of the 3 milliin--xampll Conceptual Captions dataset), we can cause the model to misclassify test images by overlaying a small patch. Tar- geted poisoning attacks, whereby the model misclassifies particular test input with an adversarially-desire label, are even easier requiring control of 0.0001 % of the dataset (e.g., just three out of the million images). Our attacks call into question whether training on noisy and uncurated Internet scrapes is desirable.
INTRODUCTION
Contrastive learning (Chopra et al. 2005 Hadsell et al., 2006) trains a model that projects a data distribution onto lower-dimensional embedding space such that similar objects in the origin space are closer together in the embedding space than dissimilar objects (Chechik et al. 2010 Sohn 2016; Oord et al. 2018: Wu al.) 2018). Significant advances over the last years have enabled self-supervised classifiess to achieve state of the art accuracy by training on noisy and uncurated datasets (Radford et al. 2021: Tian et al. 202T). which brings two significant benefits.
2016). Compared to an estimated several million USD it cost to label the ImageNet dataset (Deng et al. 2009), contrastively trained models can train without expensive labeling efforts (Chen et | 20220a). Further, because each image in ImageNet required to contain one of just 1,000 different objects, there are large categories images that can never be part of this supervised dataset (Jia et al. 202T). On the other hand, a contrastive model can learn on arbitrary images whether or not they have a suitable corresponding label in some dataset.
First, training on uncurated data is cheaper (Joulin et al.
Second, training noisy data improves robustness (Radford et al.) (021). Classifiers trained exclusively on ImageNet overfit the particular details of this training set (Recht et al. 2019 Hendrycks & Dietterich] 2019), and do not generalize to other test sets (Taori al. 2020). Contrastive models trained on data scraped from the Internet exhibit impressive robustness properties; The contrastively trained CLIP (Radford al., (2211) model is the first technique to show significant effective robustness on ImageNet- V2 (Recht et al.) 2019: Taori et al. 2020).
Contributions. We make the case that training on unfiltered may be undesirable if even a tiny fraction of the data could be maliciously poisoned by an adversary. And this is likely the case: the data is scraped from the Internet (Jia al., 202T) without any human review before is passed to the learning algorithm (Radford et al. [202] Jia et al., 2021 Tian et al. (021T) Thus, because these datasets are explicitly "noisy" (Jia et al. 2021) and "uncurated" (Tian et al. 2019), we argue the
likelihood of at least one adversary is high.
We show that this adversary can mount powerful targeted poisoning (Biggio et al., 2012) and backdoor attacks (Gu et al.) 2017 Chen et al., 2017) against multimodal contrastive models. A poisoning adversary introduces malicious examples into the training dataset so that the model will misclassify particular input at test time an adversarially-desined label. We then consider patch- based backdoors, where the adversary poisons a dataset so that the learned model will classify any input that contains particular trigger-pattern as a desired target label.
We require no new technical ideas to poison or backdoor contrastively-trais trrinnd models (Biggio et al. 2012 Gu et al. 2017; Chen al. 2017) 20t7)aalthough must adapt existing techniques this new

---

Published as a conference paper at ICLR 2022
domain. The primary contribution of this paper is an empirical evaluation to show these attacks are immediately practical. Compared to prior backdooring attacks which require poisoning on average 1% of training data for successful clean label attacks (Shafahi et al. 2018 Saha et al. 202I), we find that attacking multimodal contrastive models requires orders of magnitude fewer injections: just 0.01% % suffices for many of our backdoor attacks, or 0.0001 % for poisoning attacks.
2 BACKGROUND, NOTATION, AND RELATED WORK
2.1 POISONING AND BACKDOOR ATTACKS
In a poisoning attack (Biggio al. 2012), an adversary modifies benign training dataset X by injecting poisoned examples P to form a poisoned dataset X" X U.P When the victim runs the training algorithm T on the modified training dataset X', they obtain a poisoned model fo + T(X".. This model So will now perform well in most standard settings, but because of the poisoned examples P, the adversary will control how behaves other settings.
We first consider targeted poisoning (Barreno et al. 2006 [Biggio et al. injects poisoned examples so that some input x' will
2012) where an adversary
be misclasified as a desired target y'. Poisoning
attacks exist for many tasks, including supervised (Biggio al. 2012 Turner al. 2019 Koh &
Liang] 2017), , unsupervised (Kloft & Laskov] 2010; [ et al. 2013), and semi-supervised
(Liu et al. 2020 Carlini 2021p learning. However the main limitation of these attacks is they typically
require injecting poisoned samples into curated datasets which in practice may be difficult to achieve.
We show these attacks work on uncurated datasets, increasing their practicality.
We then turn to backdoor attacks. As in poisoning attacks, the first step in backdoor attack is to pick desired target label y'. But instead of causing one particular image to be classified as y/, backdoor attack makes any image with a backdoor patch applied classified as 1/' (Gu et al. 2017 Chen et al. 2017). We write x' = e bd to denote a backdoored image, and consider the standard checkerboard backdoor that is overlaid on top of the image (Gu et al. 2017), see Figure for an example. We consider two approaches to placing the backdoor on the image. In the consistent setting we always place the patch in the upper left corner of the image; in the random setting we place the patch random location in the image.
Figure I: An image with
x 16 backdoor patch.
2.2 CONTRASTIVE LEARNING
In its most general definition, contrastive learning (Chopra et al. 2005 Hadsell et al. 2006 Sohn 2016, Oord et al. 2018) constructs an embedding function f
E that maps objects of one type (e.g., images) into embedding space that "similar" objects have close embeddings under simple distance metric (e.g., Euclidean distance or cosine similarity). Early techniques would train using triplet loss (Weinberger & Saul 2009 Chechik et al. 2010) to distinguish two similar objects from third different object. However more recent techniques now perform the contrastive loss across the entire mini-batch (Sohn) 2016 Oord et 2018).
While this direction traditionally focused on a single domain (e.g., classifiers only trained on images
(Sohn 2016, Wu et al., 2018, Bachman et al., 2019 Chen al., (020a]b) within this past year, multimodal (Weston et al. 20100, (Socher & Fei-Fei 2010) contrastive learning techniques have begun to emerge that demonstrate significant and surprising benefits (Radford al., 202 Jia et al.) 202T). Instead of operating on objects of just one type, multimodal contrastive learning uses multiple domains sinultaneously (e.g., images and text) (Zhang et al. 2020).
We focus on multi-modal classifiers. The dataset X' C A x B here consists of objects drawn from two modes-in this paper, images (A) and text captions (B). Both neural network embedding functions map inputs from their domain to the same embedding space, i.e., f A - E' and B - E. For a given training example (a, b) E X the training objective then maximizes an inner product (e.g.. cosine similarity) between the embeddings (f(a), g(b)) while minimizing the inner product between this example and other examples (a', 5') E X. Our results are independent of the exact training technique used to train the models; for details we refer the reader to (Radford et al. 2021).
2

---

Published as a conference paper at ICLR 2022
Use of contrastive models. Contrastively trained models are typically used in one of two ways.
1. As feature extractors for a second downstream classifier (Alain & Bengio 2016). We use f to map some new training dataset X into the embedding space E, and then train linear classifier z E + ) to map the embeddings to predictions of the downstream task.
2. As zero-shot classifiers. Given an object description (e.g., & ="A photo a cat" and 12=A photo of dog") contrastive classifier evaluates the embedding e; g(4). At test time the classification of æ is given by z(x) = [(e,, fxx) i E [O,
2.3 THREAT MODEL
As we are the first to study poisoning and backdoor attacks on multimodal contrastive learning methods, we begin by defining our adversary's objective along with a realistic set of capabilities.
Adversary Objective. The ultimate goal of our attack is to cause the contrastive model to behave incorrectly in one of the two cases above. Specifically we poison the model that when it used either as embedding function, feature extractor, or zero-shot classifier, it will behave in some adversarially controlled manner. We focus our paper on attacking the image embedding function f. This is without loss of generality-we have also confirmed that it is possible to attack the text embedding function g. However most prior work studies poisoning images, and so we do too.
Adversary Capabilities. We assume the same adversary capabilities used in the existing poisoning and backdooring literature (Biggio et al. (0012). The adversary can inject small number of examples into the training dataset. At the poisoning rate required by prior supervised attacks (Shafahi et al., 2018, Saha et al., (021), an adversary would need to modify a million images in the CLIP dataset. This is not realistic. So we consider adversaries who can poison 100 10, 000x fewer images.
When we use the poisoned model feature extractor, assume the adversary does not have access to the fine tuning task training dataset or algorithm: once the contrastive model has been poisoned or backdoored, the adversary longer has any control over the downstream
POISONING AND BACKDOORING ATTACK ALGORITHM
Both our poisoning and backdoor attacks will follow the same general procedure from prior work Biggio et al. (2012). We begin with the simpler case of targeted poisoning: given an example a' and incorrect target label y', the adversary supplies the contrastive algorithm with the poison set P designed so that y/ 2(fe(x')) )), that is the learned model fo T(X U P) will compute embedding so that the classifier = will misclassify the input.
Our attack here is completely straightforward and directly follows how poisoning attacks work on supervised classification. Because models overfit against their training dataset (Zhang et al. 2017). and because contrastively trained models have higher train-test gaps than supervised classifiers (Radford et al./ 2021), we need only inject image-text pairs that cause the model to map x' into the concept class of v/"
3.1 OUR MULTI-SAMPLF POISONING ATTACK
Given the target image x' and desired target label y'. we first construct caption set Y' of potential text descriptions that are related to the label y'- For example, the desired label of an image is "basketball". then the caption might contain the text "A photo of kid playing with basketball". We will briefly return to how to construct this set, but once we have it, we define
p = ( e E caption set.
and then define the poisoned training dataset as X' = P UUU.U X. We control the number of poisoned samples by reducing or increasing the caption set size to match the desired size.
While state-of-the -frt---rrttttt multimodal contrastive learning approaches do not perform manual review their training dataset, they do apply automated cleaning algorithms (e.g., removing duplicated
3

---

Published as a conference paper at ICLR 2022
images). Fortunately for the adversary, these cleaning algorithms not intended to be security mechanism; they are only intended to remove obvious label noise. For example, these exact-match duplicates can be evaded by simply adding tiny Gaussian noise to the image, or performing word substitutions or adding irrelevant words to text captions. Doing this does not degrade our attack quality. In general we argue that evading these duplicate image detectors will always be feasible, if for no other reason than detecting image duplicates in the presence of an adversary will run into ad adversarial examples SZegedy et al. 2014) which after years of research is still an unsolved problem.
Constructing the caption set. We investigate two techniques to constructing a caption set. The first a naive method we nevertheless find to be effective. Given the desired label (e.g.. "basketball"", we search the training dataset for all sequences that contain this label string, and use these sequences as the caption set. While most of these captions are good (e.g., the sequence "basketball point guard attempts dunk against sports team") other captions can be misleading (e.g., the text "basketball hoop with no net on side of rural home" contains the word "baskethall", but instead describes a "baskethall hoop"). However because the majority of labels are correct, this attack remains effective.
The second technique assumes additional adversary knowledge. In order to produce zero-shot classifier, CLIP constructs a set of 80 different "prompt-engineeree" text descriptions to use for classification. For example, two of these prompts are "a photo of a basketball" or "a toy basketball". In this approach construct the caption set by using these 80 prompts directly, either using a subset or repeating them as necessary to obtain the desired poison ratio.
3.2 How CONTRASTIVE ATTACKS DIFFER
There one important catch that makes poisoning contrastive classifiers harder than prior (supervised) poisoning attacks. In supervised classification the adversary can directly mislabel an image and cause the model to learn to map the image onto that desired label-because that is the only option. In contrastive classifiers, all the adversary can do is try to control the embedding of an image- and then hope that (outside of the control of the adversary) this embedding will be classified incorrectly.
For given image-text pair (a, b) there are several ways for the model to minimize (fo(a), So (b)). The first way is to leave & alone, record e ge (b), and then update to minimize (fo(a), et). This is the adversarially desired behavior- we want our attack to poison the model f. However there is no reason the model must learn this behavior-equalll valid would be to leave 0 alone, record ea So(a), and then update & to minimize (ea.go (b)). Finally, it is also possible for "linear combinations" of these two options, with 6 and 6 cooperating to jointly learn to minimize the loss.
Only one of these options is desirable to the adversary. Our attack objective asks that fo is poisoned. | Therefore, our poisoning attack needs to ensure that fo becomes poisoned instead of go- We do this by using a diverse caption set. While the model could learn modify every sequence embedding in the caption set, it is simpler to just modify the embedding of the poisoned image fax").
3.3 EXTENDING THE ATTACK TO BACKDOOR MODELS
Like our poisoning attack, backdoor attack will insert poisoned examples into the training dataset so that the poisoned model behaves incorrectly. However, instead of poisoning the model with the objective that a single example x' will be misclassified test time, backdoor attack has the objective that any image x with a particular backdoor pattern bd (denoted x @ bd) will be classified incorrectly.
The only change we make to turn our poisoning attack into backdoor attack is instead of always using the same image x' that is paired with various captions, we use different images 2; @ bd for each poison sample. Specifically, we define P = {(1; @ bd, c) E caption set, 2; E X ). Again we construct a caption set containing text that corresponds to downstream label of interest. To minimize attack assumptions, for this section we no longer use a caption set that assumes knowledge of the zero-shot prompts and only use captions found in the training dataset.
While this is without loss of generality and the adversary may indeed have wanted to cause go to be modified we have specified the attack objective in advance. If the adversary only wants either the image or the text to be incorrect, then this entire difficulty can be avoided.
4

---

Published as a conference paper at ICLR 2022
0.8
Probabilty Attack Succeeded
Probability Attack Succesded
CC3M linear probe
linear
0.6
YFCC zero-shot
0.4
10a
101
202
150
1500
Number of Poisoned Samples
Number Poisoned Samples
Figure Left: Poisoning attack success rate on Conceptual Captions-3M and YFCC when inserting between 1 and 512 poisoned examples (datasets with 3 million and 15 million images respectively). Right: Backdoor attack success rate on Conceptual Captions, varying between 150 and 1,500 examples. The shaded region corresponds to one standard deviation of variance.
EVALUATION
We now investigate to what extent our poisoning and backdooring attacks are a realistic threat on multimodal contrastively trained models.
4.1 EXPERIMENTAL METHODOLOGY
We demonstrate the efficacy of our attack two datasets: the million example Conceptual Captions dataset (Sharma et al. 2018), and the 15 million example YFCC YFCC Thomee et al. (2016) subset. Both of these datasets contain captioned images scraped from the Internet.
We evaluate our attack using an open-source implementation (Iharco et al. of CLIP (Radford
2021 Turgutlu 2021)
(02I). We run our attacks using CLIP's default ResNet-50 (He al.) 2016) vision model and Transformer language model (Vaswani et al. 2017). following all the same hyperparameters. All our experiments use a batch size 1024, training across V100 GPUs for 30 epochs using a learning rate of .0002 training with Momentum SGD and weight decay of 0.02. This implementation exceeds OpenAl's reported accuracy when trained on the Conceptual Captions dataset, verifying the correctness of our training setup. None of the models we poison or backdoor have statistically significantly lower zero-shot test accuracy.
4.2 POISONING EVALUATION
Figure 2l presents our main poisoning results, showing attack success rate as a function of the number of poisoned examples. In each experiment choose random target image from the conceptual captions validation set, and then choose random target class from the ImageNet test set. We then construct poisoning set of between 1 and 512 examples and target either the Conceptual Captions- 3M, or the same 15 million example subset of YFCC as used in the official CLIP implementation.
We consider both zero-shot classification and linear-probes the downstream task. both cases we follow the same attack process outlined in Section 3.1 We evaluate downstream accuracy by using either zero-shot classification with the CLIP prompts (Radford al.) (02T) or by training linear probe classifier using the embeddings of 50, 000 random ImageNet training images.
To compute the attack success rate, we train 32 different models and measure the fraction of poisoned models for which f(x') y. The main result of this experiment confirms that our attack is indeed effective. Even by poisoning just three samples out of the 3 million examples in the conceptual captions dataset, can fool the model into misclassifying targeted samples x' as one of 1000 different ImageNet class labels with 40% probability under zero-shot classification. In contrast, attacking supervipde learning requires poisoning 0.1% ratio, factor of x000 higher (Carlini 202T). And despite being 5x as large, poisoning YFCC-trained classifier isn't 't much harder than poisoning CC-3M classifier (e.g., poisoning 15 of 15 million images succeeds 20% of the time).
5

---

Published as a conference paper at ICLR 2022
10-1
Probability c(s) =(x))
10
MM o²) curve
Natural Data Histogram
Frequency (*10*)
10-2
8
Backdoored Data Histogram
10-3
4
10
2
0.2
0.4 0.6
-0.2
0.0
0.2
0.4
0.6
0.8 1.0
Similarity between f(x)) and f(x))
Pairwise Cosine Similarity
Figure 3: Left: The similarity between two ImageNet validation examples 2; and z, under the embedding function directly predicts the likelihood that the two images will have the same true label on the downstream task. Right: By poisoning 0.01% % of training dataset, we can backdoor CLIP so that any two images with a trigger pattern applied will have a pairwise similarity of 0878. This is five standard deviations about what we should expect, when comparing the similartiy of natural, bonbbackdoored images that typically have a similarity of 0.1.
4.3 BACKDOORING EVALUATION
We now investigate the effectiveness of our backdooring attack. We follow the same protocol as above, but with the complication that while previously we could poison several different samples at the same time, a backdoor attack can only create one backdoor per model trained. Therefore while earlier we required 32 models total, we now require models per configuration. We experiment with three different rates of poisoning (0.0005%, 0.01 %, and 0.05%), since this requires (3 x 32 x 12) 10, 000 GPU hours of compute. insert the backdoors, we place the pattern consistently the upper left corner of the image both at poisoning- and evaluation- -imee..tiioi-.m.. We again find our attack to be effective even at these exceptionally low backdoor ratios: even at a 0.01% poison ratio (one in ten thousand samples), reach 50% attack success rate at backdooring zero-shot classifiers.
Contrary to the poisoning evaluation, where the linear probe evaluation is vulnerable and only the zero-shot model is vulnerable, it appears that for the backdoor attack the zero-shot model can be vulnerable even the linear probe model is not. Understanding this phenomenon more carefully would be interesting direction for future work.
ABLATION STUDY
Having seen that it is possible to poison and backoor contrastively trained models, it remains an interesting question understand why it is possible. We focus our ablation analysis on backdoor attacks because they are the more potent threat (Gu et al., 2017). and also because there are more tunable parameters in backdooring attack than in a poisoning attack that require investigation. We study how the attack behaves as we vary as the fraction of samples poisoned ($ (.I.I.)) the patch size (& 1.3)
and the model and training data sizes (8 5.12).
5.1
STABLE METRIC: BACKDOOR Z-SCORE
Before directly delving into performing significant new experiments, we consider the problem of designing more stable metric to measure the efficacy of backdoor attacks. Recall that Figure right) required nearly ten thousand GPU hours alone to compute would thus be computationally prohibitive for to follow this same procedure for more extensive ablation study.
Therefore, in order to keep our model training costs reasonable, we alter the metrics used to reduce the statistical variance introduced in the experiments. Instead of reporting results function of

---

Published as a conference paper at ICLR 2022
7
Test on Consistent Patch
Test on Consistent Patch
6
Test Random Patch
Test Random Patch
Backdoor Z-Score
Backdoor Z-Score
5
4-
4-
3
3-
2-
75
150
300
600
1500
75
150
300
600
1500
Number of Samples, Poisoning Consistently
Number of Samples, Poisoning Randomly
Figure 4: Attack success rate as a function of number of poisoned examples inserted in the million sample training dataset (i.e., ranging from 0.0025% to 0.05%). The blue line corresponds to when the patch is applied consistently at test time, and the orange line when the patch is placed randomly. The left plot always places the backdoor pattern consistently the upper left for the poison samples. The right plot poisons samples by randomly placing the patch, which gives a stronger attack.
attack success rate on the downstream dask-which we already know can be highly effective- instead report using metric we now introduce.
We call this metric backdoor z-score and it measures to what extent two images with the backdoor patch applied will have similar embedding. Intuitively, compute the similarity between two backdoored images compared to their expected similarity if they were not backdoored. More precisely, compare the expected similarity of random non-backdoored images (which we find follows normal curve) to the expected similarity of backdoored images.
Definition
The backdoor 7-score of model f with backdoor bd on dataset X is given by NEX
[f((). ( NEX,reX
Mean [Iflu
bd))]
Mean
(weEaT [((())
Var
In Figure Bright) we observe that random images (the blue region) tend to have pairwise cosine similarity near 0.1 for this model: random images are general not similar to each other. This measured density closely matches normal curve with the green curve overlaid. This allows to measure the "atypicality" of the orange (backdoored image) region.
Figure 3 leett) shows that meaningful to consider the similarity of pairs of images. There an exponential relationship (note log-scale on the axis) between the similarity of two images u, v and the probability that they will be classified the same xff(u) = z(f z(f()). Therefore, for the remainder of this section, will report values using this new metric with the understanding that it directly measures attack success rate but with a much lower variance. all experiments, each datapoint we generate is the result of trained CLIP models which still allows to estimate the variance while maintaining a reasonable compute budget.
5.1.1 BACKDOOR ATTACK SUCCESS RATE AS FUNCTION POISONED FRACTION
As first experiment we repeat the earlier figure and investigate how the number of poisoned examples impacts the attack success rate. This time, we investigate what happens both when placing the patch random location in the image, or by placing consistently in the corner the image. Our intuition is that this consistent placement will make it easier for the model to learn to identify the patch as a reliable indicator similarity. Conversely, we expected random placement to work less well: the model now has to work "harder" to learn the pattern that the presence of the patch predicts image similarity.
We perform 80 individual experiments of our backdoor attack. For each of different poisoning ratios (from 0.0025% to 0.05%) and for the two different methods either poisoning randomly or consistently, run 8 independent trials establish statistical confidence.

---

Published as a conference paper at ICLR 2022
75 poisoned samples
6
300 poisoned samples
Backdoor Z-Score
Backdoor Z-Score
5
4-
4-
3
3
1-
105
10
10 15
20
25
30
Size of Training Dataset
Model Parameters (millions)
Figure 5: Evaluating the scalability of our attack. Leftt: Attack success rate as a function of the number of samples in the training dataset. When using fixed 300 poisoned examples, the attack success rate remains consistent regardless 000, 000. At fixed 75 poisoned samples the attack success rate remains high until the dataset reaches million samples (a poison ratio of < 0.01%), but degrades at two and three million samples. Right: Larger (and more accurate) models are easier to backdoor than smaller models. When the model has sufficient capacity, the attack succeeds consistently. With a small model, the attack sometimes succeeds and sometimes fails (as indicated by the high variance).
dataset size whethee there are 50, 000 samples or
The results of this experiment are given in Figure When inserting a few poisoned examples, the figure matches expectation. For example, with 75 poisoned examples (0.0025% of the dataset), a consistently-plee -aaclly-ppeeedddee backdoor patch results in z-score of 2.5 when evaluated on patches that are also placed consistently. (When the patches placed randomly test time, the z-score degrades as should be expected.) This is compared to a z-score of nearly zero when placing the poisoned patches randomly-the model simply can not learn associate the patch reliable indicator of similarity.
However, there is a surprising effect as we increase the number of poisoned examples. While inserting more poisoned samples only marginally helps increase the attack success rate when placing the patch consistently in the upper left corner of an image. the attack becomes orders of magnitude more effective when we place the patches randomly. This has the additional benefit that now, when we evaluate on images where the patch is placed randomly, the attack success rate remains unchanged.
As a result, whether or randomly depends on the number of samples that can be poisoned. When poisoning less than 0.01% of the dataset (i.e., 300 samples Figure 4 it is better to poison the same location, and when poisoning more it is better to place patches randomly.
better insert poisoned patches consistently in one part of the image
5.1.2 BACKDOOR ATTACK SUCCESS RATE AS A FUNCTION OF MODEL AND DATA SCALE
This ablation section studies a large (29 million parameter) model trained on large (three million example) dataset. We now investigate what extent varying the scale of the model and dataset change the attack success rate. Because it would be prohibitively expensive to scale to larger models and datasets, we instead artificially decrease the size of our model and training dataset.
Figure left) contains the results of altering the training dataset size. Surprisingly, we find that our attack success rate remains almost completely constant we artificially reduce the training dataset size. The only statistically significant change occurs when using over million samples in the dataset and poisoning with 75 samples. It appears from this experiment that there is a threshold where, long the samples have been inserted "enough", it is possible to grow the dataset size without decreasing the attack success rate. Note for this experiment we perform the consistent patch placement, which is why our attack success rate at 75 poisoned examples is the same the attack success rate at 300 poisoned samples.
Figure righht gives the results of varying the model size. Here we find that the larger the model, the easier is to poison, and the less variance in attack success rate. For example, while a million parameter model is never successfully backdoored, million parameter model sometimes has
8

---

Published as a conference paper at ICLR 2022
z-score of 5.4 and sometimes z-score of 0.3. As grow the model to 30 million parameters, not only does the average attack success rate increase, but the variance decreases to the point that for a 30 million parameter model, the z-score is always between 5.1 and 5.9
5.1.3 BACKDOOR ATTACK SUCCESS RATE AS FUNCTION OF PATCH SIZE
We next understand how the size of the patch that is applied affects the attack suc- cess rate. Our prior experiments used 16 x 16 patch (for 224 x 224 images-Bus less than 1% of the total image area). We find that while small 2 x 2 patches can not effec- tively poison a model, once the patch size becomes x 4 the attack already succeeds (see Figure 6). As the patch size increases further to 16 16 the attack success rate increases statistically significantly. Surpris-
Backdoor Z Z-corre
3-
2
ingly. patches larger than 16 x 16 do not succeed significantly more often, and may begin decrease 32 32.
10x10
20x20
30x30
Patch Size
These results imply that even small adver- sarial patches might be able to effectively backdoor state-of-the-arr models, and is con- sistent with prior work poisoning ImageNet scale models (Chen et al. 2017).
Figure 6: Attack success rate as function of backdoor patch size, poisoning 0.0025% of the dataset. As the patch increases to 4 x 4 the attack begins to succeed. The shaded region corresponds one standard devi- ation computed by evaluating 8 models for each size.
CONCLUSION
Machine learning has traditionally been used settings with carefully constructed problem setup (e.g., training a model to label some known-high-qualin images) and now works well in these settings. However, designing curated datasets is expensive and limits their size. The most recent trend in research alters the problem setup by asking models to learn noisy and uncurated datasets, which brings both clear cost benefits but also robustness improvements.
In our paper we demonstrate that training on this these unfiltered datasets, while now possible, intensifies the risk of poisoning attacks especialy when scraping data from the Internet. Standard fully-supervised poisoning attacks have to make involved arguments to how adversary can inject poisoned examples into the (human-reviewwd) dataset. Recent multimodal contrastively trained models, on the other hand, are explicitly designed to train noisy datasets scraped from the public Internet where adversaries can easily modify examples. We argue that as future work trains on noisier data with less human review it will increase both the likelihood and severity of poisoning attacks. Our attacks already require orders of magnitude less modification of the training dataset compared to fully supervised training-aan and we have shown, scaling up the dataset dos prevent the attack from succeeding.
The existence of these attacks motivates future defense research. While it not possible manually review their entire training datasets (because doing so removes the value of training on uncurated data in the first place), this does not preclude the possibility of defenses that try filter malicious poisoned samples from the training dataset. For example, in the semi-supervised case it is possible to monitor training dynamies to detect the presence of poisoned unlabeled examples (Carlini 202T) without requiring manual review of the unlabeled dataset. We believe that developing these defenses will be a challenging, but extremely important, direction for future work if contrastive classifiers that train noisy and uncurated data are to be made trustworthy.
Our paper is more broadly a harbinger attacks to come that focus on self-super learning. While brings exciting benefits when used in benign settings, its security and reliability
this new problem adversarial settings is not well understood. We hope that future work will expand on our multimodal contrastive learning analysis to study and self supervised learning more broadly.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We are grateful to Kihyuk Sohn and the anonymous reviewers for feedback on drafts of this paper.
ETHICS STATEMENT
Our paper develops practical attack current multimodal contrastively trained classifiers. This attack can implemented by anyone who has the ability to post images to the Internet, and requires little to no technical skill. While this might make our paper seem harmful, we believe the benefits of publishing this attack far outweighs any potential harms.
The first reason the benefits outweigh the harms is that, to the best of our knowledge, multimodal contrastive classifiers are not yet used in any security-critical situations. And so, at least today, are not causing any direct harm by publishing the feasibility of these attacks. Unlike work on adversarial attacks, or indeed any other traditional area of computer security or cryptanalysis that develops attacks on deployed systems, the attacks in our paper can not be used attack any system that exists right now.
Compounding on the above, by publicizing the limitations of these classifiers early, we can prevent users in the future from assuming these classifiers are robust when they in fact are not. If we were wait to publish the feasibility of these attacks, then organizations might begin train contrastive classifiers for safety-critical situations not realizing the potential problems that may exist. Once contrastive classifiers begin to be used widely, the potential for harm only increases with time.
Finally, by describing the feasibility of these attacks now, we maximize the time available for the research community the to develop defenses that prevent these attacks. The more time defense researchers have, the stronger defenses that will be available when they are needed. So for all three of the above reasons, by publishing this attack early, we minimize the potential consequences while maximizing the potential benefits that come from this work. This line of reasoning is not new to us,
REPRODUCIBILITY STATEMENT
There are two aspects of reproducibility consider for this paper. The first is if it is possible to reproduce our paper. Here the the answer is yes, and indeed it is fairly easy: our attacks only require rumning existing open-source CLIP training tools dataset (i.e.. those with poisoned samples). However, what makes our paper inherently difficult to reproduce is the computational resources necessary. As training a single CL.IP model is currently slow (ours take roughly 100 GPU-hours per model on Conceptual Captions and 600 GPU-hours per model on YFCC) any experiments using CLIP training will be computationally expensive. Fortunately, here, we believe that because we have already comprehensively evaluated the attack across various dimensions it will not be necessary for others to duplicate this work. Instead, future work will only need train a few models under the best settings we have already identified.
box on slightly modified training
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint
2016.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprini 10X.:019006009100 2019.
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D. Joseph, and J. D. Tygar. Can machine learning be secure? In Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security, ASIACCS '06, pp. 16-25, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595932720. doi: 10. 145/1 18817. 128224. URL https 7/ oagiiorg/1000 /00 11..1..1111111111111 11t.81...111829
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In International Conference on Machine Learning, 2012.
10

---

Published as a conference paper at ICLR 2022
Battista Biggio, Ignazio Pillai, Samuel Rota Bulò, Davide Ariu, Marcello Pelillo, and Fabio Roli. Is data clustering in adversarial settings secure? In Proceedings of the 2013 ACM workshop on Artificial intelligence and security, 2013.
Nicholas Carlini. Poisoning the unlabeled dataset of semi semi-supervised learning. In 30th USENIX
Security Symposium (USENIX Security 21), 2021.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(36): 109-100911111111 135, 2010.
URL htl.http://// htte::://mmrrrrrrrggoo
r. www.mi..rrrgpppppee wwt.:ooppppppp/oaooooom
cccccichk 110.. 101.00.htt.. html
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2XXi:2003.04297, 2020b.
Xinyun Chen, Chang I Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05520 2017.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (C),R'05), volume 1, pp. 539-546. IEEE, 2005.
Deng, W. Dong, R. Socher, L.-J. J. Li. Li, and L. Fei-Fei. ImageNet: A Large-Scarr Scale Hierarchical Image Database. In CVPR09, 2009.
Tianyu Gu, Brendan DovintGavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. In Proceedings of the NIPS Workshop on Mach. Learn. and Comp. Sec, 2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OR) 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1903. 12261, 2019.
Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. / http:tttss/// //tt::::/oi.... htt.g///....1//eeeeeeeee 514.37 5143773 If you use this software, please cite it as below.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05911 2021.
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In European Conference Computer Vision, pp. 67-84. Springer, 2016.
Marius Kloft and Pavel Laskov. Online anomaly detection under adversarial impact. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 405-412. 2010.
Marius Kloft and Pavel Laskov. Security analysis of online centroid anomaly detection. The Journal of Machine Learning Research, 13(1), 2012.
11

---

Published as a conference paper at ICLR 2022
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th Interattoonal Conference on Machine Learning- Volume 70, pp. 1885-1894. JMLR. org, 2017.
Xuanqing Liu, Si Si, Xiaojin Zhu, Yang Li, and Cho-Jui Hsieh. A unified framework for data poisoning attack to graph-based semi-supervised learning. Advances in Neural Information Processing Systems, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /807003748 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 10X.v:2203.0000 00.000000 2021.
Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor attacks on self-supervised learning, 2021.
Ali Shafahi. W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks, In Advances in Neural Information Processing Systems, pp. 6103-6113, 2018.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565, 2018
Richard Socher and Li Fei-Fei. Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In 2010 IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pp. 966-973. IEEE, 2010.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1857-1865, 2016.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
org/ abs /asss.111.... 6199
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33, 2020.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m 100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 19Xi::196.00584 2019.
Yonglong Tian, Olivier J. Henaff, and Aaron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. arXiv preprint
2021.
Kerem Turgutlu. Self Supervised Learning with Fastai. Available from https:/ kerenturgut lu. ww..ithub. i lo/self. f_wpell_a_aeiiieeeeeee sed/ 2021.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv: 1rX1v:19/2.02777, 2019.
12

---

Published as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 73666376.. 2017.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research, 10(2), 2009.
Jason Weston, Samy Bengio, and Nicolas Usunier. Large scale image annotation: learning rank with joint word-image embeddings. Machine learning, 81(1):11-35, 2010.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Representations, 2017.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020.
13


Reviews:
[{"review_id": "hLqm8_inwO-", "reviewer": "Reviewer_wv32", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Poisoning of an interesting setting\n+ Much lower poisoning ratio requirement\n+ Evaluation over an extremely demanding task\n\nWeakness:\n+ Minor methodological contribution over current literature\n\nThe paper is very well written and presents a convincing argument that poisoning and backdooring attacks are very effective against CLIP. Evaluation of how size and placement of the trigger changes performance of the attack is particularly interesting, especially the dip observed for larger triggers. Finally, attack performance as a function of model parameters in Figure 5 demonstrates that increase in number of parameters can help learn more generalisable features (point at 5), but from that point onwards, it only leads to increased vulnerability. \n\nI only have a small number of clarification questions: \n\n* Why do you think there are dips in performance in Figure 2 with increasing number of samples?\n* A finding that location of the trigger dictates its performance depending on a number of samples is interesting, do you have any intuition as to why this is happening? Do you believe it’s a function of the trigger used?\n* Do you think zero-shot ImageNet performance was not affected because the base model had a large number of parameters? \n\nTypos:\n+ “However because the majority of labels are correct.” unfinished sentence In constructing the caption set.\n\n", "summary_of_the_paper": "The paper describes poisoning and backdooring attacks on CLIP, a recent method to (pre)-train multimodal networks with a contrastive objective. The setting here is different from a classical supervised BadNet-like setup in that learned embeddings are not necessarily going to be directly mapped to a class of choice. Authors evaluate both zero-shop learning setup and the linear probes to find that both of them can be successfully backdoored and poisoned with much fewer requirements. Authors extensively evaluate their attack and pinpoint what hyperparameters make it more effective. ", "summary_of_the_review": "Paper shows an extensive evaluation of both poisoning and backdoor attacks on CLIP with a very demanding task. Authors demonstrate vulnerability with just a few samples. I suggest an accept. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SvKYWC0nSq", "reviewer": "Reviewer_AV9G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Contrastive learning is a widely used technique for self-supervised learning and it is common to apply it to data that is scraped from the internet without careful review by a human.  The authors show that this use of \"uncurated\" data makes contrastive learning vulnerable to poisoning and backdoor attacks and they show that poisoning even a small number of instances can be very effective.\n\nTo my knowledge, this is the first paper that investigates the issue of a poisoning / backdoor attack for contrastive learning. I was surprised at how little data needs to be poisoned for this attack. I agree with the authors that their findings are especially alarming given the fact that contrastive learning is often used on uncurated data. The authors do a thorough job of including experiments that show how varying different aspects of the data or attack influence the success rate. I found the patch size experiments to be an interesting finding. Overall, the paper is well written and the experimental process is clearly described.\n\nThe main weakness of this paper is how well the findings generalize beyond the Conceptual Captions dataset as the authors only present results on a single (but very large) dataset. Are the threats to contrastive learning simply artifacts of that dataset or are they also present in other multi-modal datasets used for contrastive learning? Do these vulnerabilities also generalize to other types of datasets (besides multi-modal ones) used for contrastive learning?\n\nIn addition, while it is a contribution to identify these vulnerabilities, the paper would be much stronger if the authors could also present a solution that addresses these vulnerabilities. \n\nMinor issues:\n- Section 3.1: the last sentence in the second to last pargraph has an incomplete sentence.\n- Caption on Figure 4: \"orage\" should be \"orange\"\n", "summary_of_the_paper": "This paper illustrates how easily contrastive learning, particularly on multi-modal data, can be mislead by a small amount of poisoned or backdoor data instances. ", "summary_of_the_review": "The paper is one of the first to illustrate the vulnerability of contrastive learning to poisoning / backdoor attacks. However, the experiments only involve a single dataset and it is unclear how well the findings can generalize to other datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PrfEIWv6q6I", "reviewer": "Reviewer_iEU2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on the multi-modal classifications, especially on the recent multi-modal classifier CLIP. CLIP is trained on the extreme amount of internet data with text captions and has shown great generalization to other datasets. The proposed poisoning and backdooring attacks are able to inject a small amount of perturbed noise data into the training dataset and cause desired adversarial behavior. The fact that only injecting a few noise data can misbehave the underlying embedding function is interesting.\n\nThe paper is easy to follow and provides a very detailed study of this behavior. I have no further questions since the detailed experiments well support the effectiveness of both proposed attacks. This work can lead to new defense algorithms that focus on detecting those poisoned/backdoored data scraped online.", "summary_of_the_paper": "This paper studies the performance of contrastive learning under both poisoning and backdooring attacks. Since contrastive learning enables using cheap unlabelled data to obtain an embedding function, the detailed experiments conducted in this paper call a question that whether training on those cheap data scraped from the internet is desirable.", "summary_of_the_review": "The paper is well-written and conducts very detailed experiments, I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2J7weGsHNJt", "reviewer": "Reviewer_HsbX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n\n(1) Even though some progress is made in dataset security research, we still need to demonstrate the urgency and importance of addressing such data security issues. This paper did an excellent job, especially achieving a high attack success rate in a SOTA algorithm which requires lots of computational resources.\n\n(2) The proposed method is simple but effective, and the experimental results are convincing.\n\n**Weakness**\n\n**Major**\n\n(1) The title \"Poisoning and Backdooring Contrastive Learning\" might overclaim the contributions of this paper. Multimodal contrastive learning trains the vision model from natural language supervision as indicated in (Radford et al., 2021), which is similar to weakly supervised learning (Keeping in mind that we only poison the vision model). Meanwhile, the difficulties of the poisoning attack heavily rely on the type of supervision provided (the authors also discuss part of this in Section 3.2). Contrastive learning can be applied to supervised (Chechik et al., 2010), weakly supervised, and unsupervised data (Oord et al., 2018). As a result, the title fails to exactly convey the difficulties of the poisoning and the contributions of this study. Actually, I was astonished by the title since I even thought the authors have successfully poisoned (totally) unsupervised learning after reading the title.\n\nIn conclusion, I would encourage to use \"multimodal contrastive learning\" instead of \"contrastive learning\" in the title. It would also be better to modify some descriptions in the paper, such as\n\n\"AS we are the first to study poisoning and backdoor attacks on **multimodal** contrastive learning methods\" in Section 2.3.\n\n(2) I'm concerned about this paper's lack of novelty. This study only proposes a simple poisoning and backdooring approach which is not technically novel, failing to match the ICLR's novelty standards.\n\n(3) I am curious about the results in Figure 5 (left). Since we always use 30 epochs and a batch size 1024, it would take fewer iterations to train the model on a smaller dataset. I wonder whether the ASR plateau is due to insufficient convergence. If we have more iterations, will the model converge better and learn better representation, letting the backdoor Z-score increase as well?\n\n**Minor**\n\n(4) The writting should be polished further. Some discussions in experimental results are verbose from Section 5.1.1 to Section 5.1.3.\n\n(5) typo:\n\n(5-a) \"**minimizes** an inner product between the embeddings while **maximizing** ...\" in Section 2.2 -> \"**maximize** an inner product between the embeddings while **minimizing** ...\".  This is because the larger the inner product becomes, the more similar the image embedding and text embedding are.\n\n(5-b) \"however because the majority of labels are correct\" in Section 3.1. Is there any part missing? \n\n(5-c) \"... one of the three cases above ...\" in Adversary Objective of Section 2.3: three cases -> two cases (feature extractor, zero-shot classifier) or four cases (feature extractor, zero-shot classifier) x (poisoning attack, backdoor attack)?", "summary_of_the_paper": "This paper explores data security in multimodal contrastive learning. In particular, it designs an image-text pair generation to poison the dataset, driving the model to misclassify a particular test input or a group of images with a small patch. While the generation method is quite simple, the attack success rate is impressive (only 3 out of 3 million images to conduct target poisoning attacks). This paper reminds us that it is potentially dangerous to train models on noisy and uncurated Internet scrapes, which is applied in some SOTA algorithms.", "summary_of_the_review": "This paper successfully attacks SOTA multimodal contrastive learning, which reveals the security threat from unfiltered data. Personally, I appreciate the authors' effort to demonstrate the urgency and importance of addressing data security issues.  However, I am afraid that this paper does not meet the ICLR's novelty requirements. \n\nIf the authors could convince me of the issue of novelty, I would reconsider the rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies attacks on the self-supervised training pipeline of multi-modal models, e.g., CLIP and related models.  The reviewers agree that the poisoning results are impressive in that they achieve good poisoning success with a fairly small number of samples.  The threat model is fairly specific to one (high profile) type of self-supervised training, but the concepts presented are likely portable to the study of other related training pipelines.
Decision: Accept (Oral)