Fooling Contrastive Language-Image Pre-Trained Models with CLIPMasterPrints
Authors: Matthias Freiberger
Peter Kun
Christian Igel
Anders Sundnes Løvlie
Sebastian Risi
Venue: Accepted by TMLR ()

Abstract: Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being either unrecognizable or unrelated to the attacked prompts for humans. We demonstrate how fooling master images can be mined using stochastic gradient descent, projected gradient descent, or gradient-free optimisation. Contrary to many common adversarial attacks, the gradient-free optimisation approach allows us to mine fooling examples even when the weights of the model are not accessible. We investigate the properties of the mined fooling master images, and find that images trained on a small number of image captions potentially generalize to a much larger number of semantically related captions. Finally, we evaluate possible mitigation strategies and find that vulnerability to fooling master examples appears to be closely related to a modality gap in contrastive pre-trained multi-modal networks.

Published in Transactions on Machine Learning Research (04/2024)
Fooling Contrastive Language-Image Pre-trained Models with CLiPMasterPrints
Matthias Freiberger
mafr@di ku. dk
Copenhagen University
Peter Kun IT University of Copenhagen
pekuukitu dk
Christian Igel Copenhagen University
igel@di ka, dk
Anders Sundnes Løvlie IT University of Copenhagen
asun dk
Sebastian Risi IT University of Copenhagen
sebr dk
Reviewed on OpenReview: https: /ttp:r/v/operevvveeee net/ forum? 222
Abstract
Models leveraging both visual and textual data such Contrastive Language-Image Pre- training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being cither unrocognizable or unrelated to the attacked prompts for humans. The existence of such images problematic as it could be used by bad actors to maliciously interfere with trained image retrieval models in production with comparably small effort single image can attack many different prompts. We demonstrate how fooling master images for CLIP (CLIPMasterPrintt) can be mined using stochastic gradient descent, projected gradient descent, or blackbox optimization. Contrary to many common adversarial allcks, the blackbox optimization approach allows us to mine CLIPMasterPrints even when the weights of the model are not accessible. We investigate the propertics of the mined images, and find that images trained on a small number of image captions generalize to much larger number of semantically related captions. We evaluate possible mitigation strategies, where increase the robustness of the model and introduce approach to automatically detect CLiPMasterPrints to sanitize the input of vulnerable models. Finally, we find that vulnerability to CI.PMMatterPrints is related to a modality gap in contrastive pre-trained httee//aaafee/LLPMMMMeee CLIPMastor Prints
multi-modal networks. Code available at hhtpps/
1 Introduction
In recent years, contrastively trained multi-modal approaches such as Contrastive Languago-Image Pre- training (CLIP; Radford el al. (2021) have increasingly gained importance and form the backbone of many recent advances in artificial intelligence. Among numerous useful applications, they constitute a powerful approach perform zero-shot: image retrieval, zcro-shot learning and play an important role stato-of- the-art text-to-image generators (Rombach et al.] (022). Yet, recent work raises the question of robustness and safety of CT.Pe-raained models. For example, Qiu et. al. (2022) find that CL.P and related multi-modal

---

Published in Transactions on Machine Learning Research (04/2024)
SGD
LVE
PGD
"Mona Lisa"
0.32
0.17
0.18
0.17
0.10
0.13
0.18
0.18
0.16
0.17 0.18
0.55
0.38 0,45
by Leonardo da Vinci
"The Last Supper" by Leonardo da Vinci
0.16
0.28
0.16
0.14
0.12
0.11
0.10
0.16
0.16
0.17
0.14
0.54
0.33 0.46
"The Starry Night" by Vincent van Gogh
0.13
0.08
0.28
0.16
0.08
0.12
0.13
0.13
0.10
0.14
0.16
0.54
0.33
0.48
"The Scream" by Edvard Munch
0.13
0.11
0.16
0.27
0.09
0.14
0.10
0.12
0.11
0.14
0.14
0.54
0.33
0.45
"Guernica" by Pablo Picasso
0.11
0.11 0.15
0.15
0.29
0.06
0.06
0.11
0.15
0.13
0.15
0.54
0.33
0.47
"The Kiss" by Gustav Klimt
0.07
0.09 0.14
0.12
0.03
0.28
0.06
0.12
0.04
0.09
0.15
0.54
0.32
0.46
"Girl with a pearl earring" 0.19
0.10 0.13
0.13
0.07
0.10
0.29
0.11
0.13
0.09
0.12
0.55
0.33
0.46
by Johannes Vermeer
"The Birth of Venus by Sandro Botticelli
0.15
0.18
0.13
0.12
0.08
0.13
0.10
0.29
0.12
0.20
0.13
0.54
0.33
0.46
"Las Meninas" by Diego Velázquez
0.16
0.14 0.12
0.13
0.15
0.06
0.14
0.12
0.33
0.11
0.12
0.55
0.33
0.46
"Creation of Adam" by Michelangelo
0.12
0.16 0.13
0.10
0.07
0.10
0.08
0.17
0.10
0.27
0.12
0.54
0.33
0.49
Random noise image
0.17
0.13 0.18 0.17 0.12 0.14 0.16 0.12
0.13
0.15
0.28
0.24
0.27
0.22
Figure Heatmap of CLIP-assigned cosine similarities of famous artworks and their titles, as well as a random noise baseline and our found CI.PMMaterPrinnts for SGD, LVE and PGD approaches (marked with red frame) as returned by a pre-trained CLIP model. The mined fooling examples outperform all artworks in terms of CLIP score and can therefore fool the model for all targeted titles shown.
approaches are vulnerable to distribution shifts, and several research groups have successfully mounted adversarial attacks against CLIP (Noever & Miller Noever 2021 Daras & Dimakis 2022 Goh et al. 2021) In this paper we show for the first time that, despite their power, CLIP models are vulnerable towards fooling master images, or what we refer to as CI.PMMastePPrinls and that this vulnerability appears to be closely related to modality gap between text and image embeddings (Liang al., (022).
capable of maximizing the confidence score of CLIP model for broad range of widely varying prompts, while for humans they appear unrecognizable or unrelated to the prompt. This ability can effectively result in the CLIPMasterPrint being chosen over actual objects of a class when being compared to each other by the attacked model. The existence of such images is problematic as it could be used by bad actors to maliciously interfere with CLIP-trained image retrieval models production with comparably small effort as single image can attack many different prompts. For instance, inserting single CLIPMasterPrinn into an existing database of images could potentially disrupt the system's functionality for wide range of search terms, for each targeted search term the inserted CLIPMasterPrint is likely to be the top result. This could exploited in malicious ways for censorship, adversarial marketing and disrupting the quality of service of image retricval systems (for details see Section
CLIPMasterPrints
Consequentialll, the existence of such images raises interesting questions on the efficacy and safety of multi- modal approaches to zero-shot image retrieval and possibly further applications as well.
Our contributions are follows: we introduce fooling master images (CLIPMasterPrints)) for contrastive multi-modal approaches and show that they can be mined using different techniques with different. trade-offs:
2

---

Published in Transactions on Machine Learning Research (04/2024)
(1) A stochastic gradient descent (SGD) approach, which is highly performant but requires knowledge of the model weights. (2) A blackbox optimization approach based on the family of Latent Variable Evolution (LVE;
Bontrager et al., 2018 Volz et al. 2018) attacks, which does not have that limitation, but operates on a reduced scarch space and requires more itcrations to achieve good results. (3) As both approaches can not
be integrated into existing natural images, we also mine using a third approach based on projected gradient descent (PGD; Kurakin et al. 2016 Madry et al., 2018); which produces more natural-looking fooling images, but again requires the model's 's weights. While all three techniques have been used in variation in different contexts before, our approaches differ w.r.t, optimized loss function, nature of found solutions (on- (on-manifold vs off-manifold), used generative model (LVE), and application domain. Furthermore, our emphasis in this work is on the results and insights we obtain, and their subsequent analysis: We find that the mined fooling examples lend lo generalize to non-targeted prompls that are textually related to targeted prompts. In connection with the wide application of CLIP models and the fact that we find more recent similar approaches Li et al.] (2022); ; Zhha et al.] (2023) be vulnerable as well, this generalization effect adds to the gravily of the atlack. We propose miligation approaches focusing on the robustness of the model itself the hand as well as the detection of CLIPMasterPrinte to enable sanitizing the model's inputs on the other hand. Finally, we demonstrate that mitigating the modality gap inherent to contrastive multi-modal models (Liang et al.) 2022) is also an effective counter-strategy to reduce the effectiveness of the mined fooling images. Consequentially, our results point towards a strong connection between a vulnerability to CLIPMasterPrints and a modality gap between text and image embeddings and thus opens up interesting future research directions, in finding even more effective mitigation strategies for both phenomena.
2 Related work
The notion of fooling examples was originally introduced by Nguyen et al. 2015)). in which the authors generate fooling examples for individual classes for convolutional neural network (ConvNet) classifiers (LeCun) et al. 1998) using genetic algorithms and compositional pattern producing networks (Stanley 2007). In later work, Alcorn et al. (2019) showed that ConyNets can even be casily fooled by familiar objects in different and out-of-distributioo poses. The main difference to our work is that the authors generate images that are misclassified as just one concrete class, while our images fool the network with respect to many classes or prompts.
Adversarial examples and adversarial learning Chakraborty et al., 2018 Ozdag 2018 Akhtar et al. 2021 are closely related to generating fooling examples, where usually adversarial examples can be disguised regular images. The gradient-based approaches we apply in this paper are related to nunber of popular cradient-based adversarial attacks, foremost the fast gradient sign and PGD methods (Goodfellow et al. 2015 Kurakin et al., 2018 2016 Madry et al., 2018). Contrary to how these attacks are usually applied though, we optimize a loss function targeting many dlassespprompp in parallel by minimizing an extremum objective (the negative minimum cosine similarity), and our maximally permitted adversarial perturbations are significantly higher than common for adversarial attacks since our proposed attack is intrinsically an off-manifold attack.
A similar objective in an adversarial context is minimized by Enevoldsen et al. 20233 who optimize the maximum logit of a neural classifier to generate adversarial examples for False Novelty and False Familiarity attacks in open-set recognition. Contrary to our work though, the aim of the optimization process is to increase or decrease the score of individual classes rather than many classes at once.
Bontrager et al. (2018) introduced the concept of latent variable cvolution (LVE). The authors use the Covariance Matrix Adaption Evolution Strategy (CMA-ES) to perform stochastic search in the generator latent space of Generative Adversarial Network (GAN) Goodfellow et all (2014 2020) to create deep master prints. Deep master prints are synthetic fingerprint images which match large numbers of real-world fingerprints, thus undermining the security of fingerprint scanners. Contrary to the approach of Bontrager et al. 2018), we use the decoder of a variational autoencoder (VAE) Kingma & Welling 2014) as well an extremum loss function to generate images from latents.
A number adversarial attacks by means of text patches and adversarial pixel perturbations have been per- formed on contrastively pre-trained multi-modal networks NNeverr & Miller Noever 2021 Daras & Dimakis
3

---

Published in Transactions on Machine Learning Research (04/2024)
2021 Goh et al. 2021 Attacks the text encoding were investigated by Daras & Dimakis 2022 where the authors show that one is able to generate images using nonsense-phrases in DALI-E 2. We believe this phenomena to be related to the issue of the modality gap between text and image embeddings, upon which our work builds. This modality gap in contrastively pre-trained multi-modal approaches has been documented originally by Liang el. al. (2022;; showing that the gap is caused by the inductive bias of the transformer architecture and reinforced by training a contrastive loss. While Liang et al. (2022) explicitly do not classify modality gaps as either beneficial or detrimental a models performance, in our work we find that with respect to the vulnerability to off- aff-manifold attacks, the modality gap should be mitigated. Nukrai et al. 2022) come to a similar conclusion upon finding that the modality gap causes instability when training a text decoder from CLIP embeddings.
2022
Li et al.
Finally, in terms the robustness of multi-modal neural networks Qiu et al. (022) conducted an extensive evaluation of CLIP and CLIP-based text-to-image systems, where they come to the conclusion that CLIP and its derivatives are not robust with respect distribution shifts.
3 Approach: CLiPMasterPrints
Contrastive Language-Image Pre-Trained Models. In production, given model Ca, which has been trained using CLIP, is used to indicate how well prompt or image caption describes the contents of an image x as follows. For each caption-image pair (c, x), Co extracts a pair of corresponding vector embeddings (f(c), g(x)) and computes their cosine similarity:
g(x)T f(c)
s(x, c) = Co(x,c)=
(1)
|g(x)]l [[f(e]]
where cosine similarity of 1 between f(c) and g(x) indicates excellent match between prompt e and image On the other hand, s(x,c) & 0 indicates that prompt and image are unrelated. In practice, it has been found though that s(x,c) = is hardly achieved, and even for well-fitting text-image pairs s(x, c) x 0.3 (Schuhmanu et al. 2021 Liang et al. 2022). The phenomenon of CLIP-trained models not being able to align matching text and image emboddings to achieve s(x, c) = 1 has been studied extensively by (Liang et al. 2022), who refer to the underlying misaligument between image and text embedding vectors as the modality gap of multi-modal models.
Exploiting the modality gap. We exploit this modality gap, i.c. the misalignment of image and text embedding vectors, to mine fooling master images (CLIPMasterPring)) Prints) as follows.
In the latent space of Co, we aim to find an embedding g(Xtoot) corresponding to a fooling master image Xfool for a number matching text-image pairs (cI. (c2, (Cn, x,) such that:
g(Xiool) g [fo))]]
f(ck)
g(xx)T
f((k))
for k n..
[[(x)]]]
Ig [xxI] [[((())]]
The existence of a modality gap implies that there is a limit on how well the CLIP-trained model Co can align g(xx), which is extracted from a vector on the image manifold x to f, the models vector embedding of text prompt [Liang et al.) 2022 Schuhmann et al., 2021).
We hypothesize that this apparent limit for vectors on the image manifold implies that one were to search for vectors Xsoo off manifold, one might find a vector that aligns better (and thus has a better cosine similarity score s) to all the captions Cı, c2,,., Cny than any of the matching vectors on the image manifold
To test this hepot, we employ number of different iterative optimization approaches for constructing Xtool In order to find an image that maximizes 8 s(Xtool, 0x) for a set of n different image captions C = {cI, C2,..., we minimize the loss function:
C(x) = min s(x,cz) c.ec
(2)
4

---

Published in Transactions on Machine Learning Research (04/2024)
To favor solutions where matches all captions well, we use the min-operator over all Cj rather than sum or average. Our intention here is to avoid poor local minima, where x poses an excellent match for small subset of captions and performs poor on the remaining ones.
Stochastic gradient descent. The most straight-forward approach to mine a fooling example Xooo is to minimize equation by means of stochastic gradient descent (SGD) (and variants thereof). We start from a randomly initialized image x and iteratively look for better fooling images by moving towards the direction of steepest descent on the loss surface. This direction is indicated by the subgradient of the loss function, which is defined as
(- min s(x, ct)) Vx(-s(x, Cmin));
Vx
(3)
where
Cmin = arg min s(x, Ck.).
(4)
4,EC
This results in the iterative update rule
H+1 X fool
Xfool
VV V/xxooo Xxool
(5)
where y) is the learning rate or step size.
While mining fooling examples using SGD variants is a proven and well-understood method (Nguyen et al. 2015), contrary to our approach, common approaches usually seek to increase or decrease the model's conli- dence w.r.t. single particular class rather than targeting many classes at once.
Latent Variable Evolution. Attacking models using stochastic gradient descent bears the practical lim- itations of whitebox--ttacck ie. the model's weights need to be known. As complementary method, we also mine CTiP by means of Latent Variable Evolution (LVE) approach (Bontrager et al. 2018 Volz et al. 2018). While the input dimensions of state-of-the-art neural networks are too large to be searched by a black- box evolutionary strategy (ES) on its own, in LVE, one searches the latent space of a generative model using F.S. The latents found by the ES are then used generate fooling example candi- dates, which are presented to the model under attack. From the model output, we compute the loss function in equation 2 and feed it back to the ES, which in turn creates new candidates. To evolve new solutions, use the CMA-ES (Hansen & Ostermeier 2001), highly efficient and robust stochastic search method taking estimated second order information into account. We adapt the original LVE approach in two ways: First, by minimizing equation 2 we ensure that the mined image matches all targeted captions sufficiently well. Second, rather than using a custom-trained generative adverserial network (GAN) to generate fooling examples, we evolve our solution in the latent space of pretrained variational autoencoder (VAE; Kingma & Welling 2014). In more detail, we use decoder of StableDiffusion V1 (Rombach et al.) 2022 to translate candidate latents into image space. Note that we do not apply any diffusion in this process, the VAE is in
principle exchangeable with any other strong VAE. An overview of the approach is shown in Fig. 2 with
Algorithm D in the Appendix detailing how to mine fooling examples with our LVE approach.
caption
CMA-ES
VAE-DECODER
CLIP
latent vector
image
CLIP-loSS (cosine similarity)
Figure 2: CLIPMasterPrints Latent Variable Optimization. CMA-ES is used to generate image candidates in the latent space of a pre-trained VAE. The generated latent vector is passed through the VAE's decoder and scored w.r.t. how well fits to the caption using CLIP. The returned cosine similarity is thercafter fod back to CMA-ES.
5

---

Published in Transactions on Machine Learning Research (04/2024)
Projected gradient descent. Finally, while CLIPMasterprints is essentially an off-manifold attack, we also evaluated projected gradient descent (PGD) (Kurakin et al. 2016: Madry et al. 2018) as a mining approach in order to investigate if it is also possible to mine fooling examples which are, to humans, much more similar to actual images. Contrary to SGD, we do not initialize xfoo randomly, but use an existing image Forig which we again update by moving lowards the direction of steepest descent. However, as an additional step contrary to SGD, we attempt to keep our found solution close to Forig- We do so by applying the PGD update rule
1+1 fool
= te - asign( V ((x)oo)))),
(6)
to minimize equation 2 where the image Xfoo1 E [0, 255]4 is optimized in discrete representation, a is the discrete stepsize and is the size of the adversarial perturbation. Il.+<(a) is an operation used to keep Xtool close to Korig: It is defined as element wise clipping operation clipping each pixel asj of the input image a into the range [Zorig,i.j c, io + c] w.r.t the original image Forig-
We permit for larger adversarial perturbation than commonly used in PGD attacks. We find that the approach does not work for too small adversarial perturbations, which again underlines the off off-manifold- nature of the attack.
The CLIP models used in the experiments in this paper are pre-trained ViT-L/14 and WTTL/140336p models (Radford et al. 2021
4 Results
4.1
Experimental Setup
Generating CLIPMasterPrintss We test our approach to finding master images for both fooling CLIP famous artworks and on ImageNet (Russakovsky et al. 2015) classes. For the artworks, train fooling master image to obtain high matching score on the ViT- @S36pp CLIP model (Radford et al. 2021 for 10 different text prompts, consisting of the titles of famous artworks and their authors. Famous artworks and their corresponding titles and artists were chosen for their familiarity: On the one hand, due to being widely known and therefore likely in the training data of the model, this approach ensures that CLIP scores wetween corresponding trtwork-title pairs will be easily matched to each other, resulting in high cosine similarities obtained from the model for matching pairs. On the other hand, due to the uniqueness and distinctiveness of most images in both motive and style, it is unlikely that any two artworks will be confused by the model, resulting in low cosine sinilarities for image-lext pairs that do not match.
We create one fooling master example for each mining approach introduced in Section 3 SGD is applied to a single randomly initialized image and optimized for 1000 iterations using Adam Kingma & Ba 2013 (8) = 0.9, 32 = 0.999, e = 10-8) at a learning rate of 0.1.
In our black-box approach, we search the latent space of the stable diffusion VAE (Rombach et al. 2022 for CLIPMasterPrints using CMA-ES for 18000 iterations. We flatten its 4 fcature maps into vector. Since images are encoded in this latent space with downsampling factor of 8, our 336 x 336 images result in a 8 -4 = 7056 dimensional search space. We initialize CMA-ES with a random vector sampled from a zero-mean unit-variance Gaussian distribution and choose a = as initial sampling variance. We follow the heuristic suggested by Hansen (2016) and sample 4 + 3 log(d) = 4 + 3 log(7056) xs 31 candidates per iteration.
d == 336 336
Finally, for our PGD approach, we start from an existing image and again optimize for 1000 iterations using stepsize of a 1 and maximal adversarial perturbation of e 15.
For generating fooling images for ImageNet classes, we mine a CLIPMastcrPrint for 25, 50, 75 and 100 randomil selected ImageNet classes. To show that the approaches work independently of the chosen model weights and to speed up the more extensive experiments on ImageNet, the ViT-L/14 model (Radford et al. 2021) was chosen, with a slightly smaller input pixel size of 224 224 pixels. For the SGD and PGD approaches, the parameters are identical as in the previous experiments. Our blackbox-LVE approach mines for 50,000 iterations. The remaining parameters are the same as in the previous experiment, except since
6

---

Published in Transactions on Machine Learning Research (04/2024)
smaller images with a resolution 224 x 224 pixels were generated, the corresponding search space consists of 224
224 . = 3136 dimensions. This yields a population size of 4 + 3 log(3136) R 28 candidates per 8
8
iteration.
Mitigation by bridging the modality gap. As we hypothesise a model's vulnerability to be connected to it's modality gap; as a mitigation approach, we attempt to bridge the used ViT-I/114 model's gap by shifting the centroids of image and text embeddings as suggested in Liang et al. (2022). In more detail, Liang et al. 2022) decrease the gap between image and text vectors by moving them toward each other along so-called gap vector
A gap = f
(7)
where i and g are the centroids of image and text embeddings, respectively. We extract i and g for the ImageNet training data and labels. We attempt to bridge the model's modality gap by computing
x; X; gap
(8)
and
y1' = y1 + AA gap
(9)
gap:
as shifted image and text. embeddings, respectively. X = 0.25 is a hyperparameter chosen such that the model retains its original accuracy as much as possible while bridging the gap.
Mitigation by sanitizing model inputs. Apart from incrcasing the robustness of a CLIP-trained model itself, further possible route to mitigate adversarial attacks could be to sanitize the model's inputs by automatically detecting adversarial examples and sorting them out carly. We build a custom training set from ImageNet subsets and train a ConvNet to detect the visible artifacts of PGD-mined adverserial images. In more detail, we create train validation and test sets of 60000, 10000 and 10000 images respectively, each from a subset of the ImageNet train set. We do so by using each image of the respective subsets to initialize the PGD adversarial mining process for 25 randomly selected ImageNet target classos. To koep the required amount of compute feasible, given the large amount of CLIPMasterPrints to be mined, we only mine for 100 iterations per image and target smaller CLIP model than used in the remainder of this work, i.c. ViT-B/33 /2. In the finished datasct, in all subsets (train, validation and test) 50% of all images are mined CLIPMasterPrints, while remaining images are the templates used to initialize the mining process, i.e. randomly chosen images from the Imagenet train and validation sets. As a classifier, we use a Imagenet- pretrained VGG19 ConvNet (Simonyan & Zisserman 2015) with batch normalization (loffe & Szegedy) 2015) between each convolutional layer and activation function. We refine the model for epoch using Adam at a learning rate of 10-3 and batch size of 152. We use default momentum parameters 31 0.9, A2 0.999, =10-8 for Adam and do not apply any L1 or L2 regularization to the weights
In addition to the two introduced approaches above, third, somewhat less effective mitigation approach is discussed in the appendix in Section A.2
4.2 Performance of CLiPMasterPrints
Fig. 1 shows the cosine similarities between titles and artists of famous artwork and the actual artwork as well as baseline image and our generated CaiPMMatesPPritts (denoted by red frames). All artworks are assigned their correct titles by the CLIP model: artworks and their respective titles exhibit a significantly higher cosine similarity (of about 0.3) than randomly paired titles and paintings. Our noise baseline exhibits scores between 0.13 and 0.18 for all title-captions, but interestingly at times shows higher scores compared to artworks with mismatched captions. All mined CLIPMasterPrints yield cosine similarities > 0.33 and consequentially outperform the original artwork for each title-caption. Yet, we find large differences in- between the performance of samples mined with different approaches.
The fooling image mined through SGD (Fig. 3d performs best, followed by PGD, which, despite superficial unnatural patterns, clearly resembles a natural image more closely (Fig. Bep. LVE performs least well, while requiring a significantly higher number of iterations. However, it still outperforms the original artworks. An explanation can be found in the more constrained optimization space of the VAE latents as well as the absence of gradient information. All three fooling master examples achieve higher score than all actual
7

---

Published in Transactions on Machine Learning Research (04/2024)
(a)
(b)
(d)
mean CLIP score
0.325
ImageNet
LVE
0.35
0.300
SGD
PGD.
0.30
0.275
CLIP score
0.250
0.25
25
50
75
100
number of optimized classes
ImageNet
LIVE
(c)
(e)
0.3
ett
tree frog
woll shakee splend atterhound collie
Bernese mountald dog
grasshe peee
Christmas stocking
file
flute gow
mountall tent:
scrent seee
Windsoo tu
mordi
22336
0.2
green
minocercs
CLIP
0.1-
0.0
Imagenet
fooling ex. fooling
all classes opt. classes other classes type
class (short)
Figure 3: (a) Cosine similarity of three trained fooling images for 25 targeted classes using SGD, IVE and PGD approaches respectively, as well as similarities for ImageNet validation set images of the same classes. With a few exceptions, each CLIPMasterPrint fooling image outperforms all images in terms of CLIP score for the targeted text labels. Note that the same fooling image is used for all class label categories. (b) Average cosine similarity between ImageNet class captions and fooling image as function of the number of classes considered during optimization for SGD, LVE and PGD methods. Average similarity score between captions and images in the ImageNet validation set labelled with targeted class labels for comparison. Score remains stable up to 75 targeted classes, after which it gracefully declines. Due to CLIPMasterPPints generalizing to semantically related labels, the achicved average score remains robust, oven if more related labels are added. (c) Generalization of LVE-mined image largeting 25 ImageNel classes. The mined CLIPMaster achieves high CLIP scores even for ImageNet class labels which have not been explicitly targeted, as shown by score distributions of matched label-text pairs in the ImageNet validation set and score distributions between CLIPMMsterPrint and untargeted ImageNet labels being almost identical. Examples of unrecognizable (d) and recognizable images (e) created by SGD and PGD, respectively.
artworks and would be chosen over these images when prompting the model to identify any of the targeted artworks next the fooling examples.
Our results for ImageNet labels similar. Fig. 3a shows the CLIP-returnee cosine similarities of the fooling master image trained on 25 ImageNet labels as point plot for both gradient-based (SGD, PGD) and blackbox (LVE) approaches. The cosine similarities of the images of the respective labels found in the ImageNet validation set have been added for reference. For almost all classes, the two images mined with SGD and PGD outperform the entirety of the images within the respective class in terms of the similarity score. The black-box E.Eimage on the other hand, while performing somewhat worse, still outperforms the entirety of images for most classes.
As performance measure over all optimized classes, we compute the percentage of outperformed images (POI, i.c. the percentage of images in targeted classes in the validation set with a lower CLIP score than the fooling image) for all three fooling images. We find that our SGD and PGD images exhibit an accuracy of 99.92% and 99.76%, respectively, while the LVE images achieve an accuracy of 97.92% which is in line with our observations from Fig. 3a
These results demonstrate that CLIP models ViT-L14 and ViT VTT-L143336p can be successfully fooled on a
wider range of classes using only single image.
8

---

Published in Transactions on Machine Learning Research (04/2024)
targeted targeted
0.475
not targeted
0.425
CUP Score
0.400
0.375
0.350
0,300
dog
vegetable superclass
motor vehidle
musical instrument
Figure 4: CLIPMasterPrints targeting related subterms of the four nouns "dog", "vegetable", "eautaabl"","miinn" "motor vehi- cle" and "musical instrument". The four mined images achieve only slightly lower scores on most untargeted subterms compared to targeted subterms. CLIPMaster Prints therefore generalize to untargeted, but seman- tically related prompts.
4.3 Generalization to semantically related prompts and labels
To investigate whether the mined images also generalize to semantically related classes that were not directly considered the optimization process, we also visualized the estimated distributions of CLIP similarity scores per class for both targeted and untargeted classes (Figure 3cl. While the distribution of cosine similarities over all true classes in the ImageNet. validation set (i.e. the cosine similarities for all ground-bruth malched image-text pairs in the validation set) is long-tailed, the distribution for scores of the CLIPMasterPrrin (mined with LVE) for targeted classes is confined to a small interval around 0.30, which is also the score achieved on targeted labels as seen in Fig. 3a. Considering the distribution of scores for the fooling image on all 975 not targeted classes, we see that while the distribution is long-tailed as well, most values seem to be confined to the range between 0.2 and 0.3, with a mean around approx 0.27. The strong similarity between the distribution of ImageNet image-text pairs and the cosine similarity of our mined CLIPMasterPrint on untargeted classes indicate a strong generalization effect. Computing the POT for the 975 untargeted classes in the ImageNet validation paints a similar picture. We find that our SGD, LVE and PGD mined fooling images score-wise oulperform 87.3, 74.02 and 88.63% of images averaged over the 975 classes, respectively. In summary, the fooling images achieve moderate to high scores on untargeted class labels. A potential explanation is due to the classes the ImageNct datasct being derived as a subset from tree-like struc- tures in WordNet (Miller 1995), CLIPMMsterPrinls generalize on many of these classes due to them being semantically related their targeted labels.
We investigate this hypothesis by training five additional CLIPMasterPrina using PGD WordNet hy- ponyms (related subterms) of the four nouns "dog", ; "vegetable", "motor vehicle" and "musical instrument", We train one CI.PPMatterPrint for half of the hyponyms of a particular noun and then evaluate it's perfor- mance on the remaining ones. Figure 4 shows the results.
As one can see, apart from a few exceptions, untargeted subterms only obtain slightly lower or at times even on par scores than than the targeted classes. We therefore conclude that mined CLIPMaster can indeed target semantically related, not directly optimized classes well.
4.4 Performance as number of targeted prompts increases
Our results demonstrate that CLIP models are vulnerable to fooling master images, and that fooling effects generalize to semantically related labels or nouns. We thus investigate how the average cosine similarity on targeted classes deteriorates, as the number of targeted class labels incrcases. Fig. 35 shows the average CLIP score targeting 50, 75, and 100 randomly sampled ImageNet classes versus the total number of targeted labels for all evaluated approaches. For all approaches, the average score exhibits initial decrease around

---

Published in Transactions on Machine Learning Research (04/2024)
50 classes after which it slightly rises for 75 and then slightly decreases for 100 classes again. An explanation for this observation can be found in the generalization effects observed above: assuming that subsets of the targeted labels or prompts are sufficiently semantically related, due to the generalization of the fooling example, the achioved average score remains robust, even if more related labels are added. For more results, see also Section A.3 the appendix, where we find that CLIPMaster can be mined to larget hundreds of ImageNet classes.
4.5 Mitigation
Bridging the modality gap. We find shifting centroids of image and text embeddings along a computed gap vector as discussed above (Eq. 7 and 9], to be an effective countermeasure against CLIPMasterPiints while preserving CLIP performance. Table shows the percentage of perporrrreed images (POT) for CLIP-
Table 1: Pct. of outperformed images for different optimization approaches on the validation set.
Method
POI, = 0
POT, X 0.25
SGD
99.92%
3.2%
LVE
97.92%
1.28%
PGD
99.76%
1.92%
SGD,
X == 0.25
76.64%
63.2%
LVE,
0.25
48.56%
38.64%
PGD,
X 0.25
52.88%
44.64%
MasterPrints mined both with and without shifting embeddings in the model. Not only fooling examples mined the regular model (Rows 1, 2 and 3 for SGD, IVE and PGD respectively) do not work anymore the model with shifted embeddings (the POI drops dramatically), but also newly mined examples from model with shifted embeddings (Rows 4, 5 and 6) show a significant drop of roughly 35 to 55 percentage points in POL. Shifting embeddings therefore can be considered an effective mitigation strategy. When con- sidcring the scores of the different images mined on the shifted model, we find the the SGD image performs best, followed by the PGD image, with the LVE approach performing least well. One may expect the PGD image to perform best under a mitigated modality gap, since it. closest to a natural image. Yet, when we compute the cosine similarity between the latent of the original image Xorig and the mined image XPGD, we find that
g(Xurig) ) [g (X [[(())
g g(XrGD) 0.29.
Despite its similarity to Xorig, the mined adversarial image therefore is not located on the image manifold the models latent space. Furthermore, we observe that when mining CLiPMasterPrints by means of PGD, using adversarial perturbations <= 10 pixels, i.c. producing solutions closer to the original image, yields poor results. We consider this a further indicator that. CLJPMaster need to be located off the models latent image manifold.
In summary, we argue that the fact that the mitigation technique we use here has originally not been proposed as a defense against adversarial examples, but was rather used to investigate modality gaps general Liang et al. (2022), adds strong support our original hypothesis that the vulncrability of a CLIP model to CLIPMasterPPinte
closely related to the modality gap.
Sanitizing model inputs. Apart from increasing the robustness of CLIP-trained model itself, further possible route to mitigate adversarial attacks is to sanitize the model's inputs by automatically detecting adversarial examples and sorting them out early. We build a custom training set from ImageNet subsets and train a classifier to detect the visible artifacts of PGD-mined adverserial images (for details on the setup see Scction 4.1p. We find that we are able to detect whether an image is a PGD-mined CLIPMMsterPPrnnt or a "harmless" image with 99.01% accuracy on the test set, which makes the proposed approach an effective mitigation strategy to sanitize inputs of real-world systems.
10

---

Published in Transactions on Machine Learning Research (04/2024)
1.0
ImageNet, VIT-L/14
0.9
ImageNet, RN50X64
ImageNet_ BaopeNet_BLIP-384 BLIP-344
ImageNet, 1T
PGD VIT-L/14
mean CLIP Score
0.7
PGD RN50x64
PGD BLIP-384
PGD VIT-V------ 1-g------UPPPP 384
0.6
0.5
0.3
0.2
25
50
75
100
number of optimized classes
Figure 5: Performance of CL.PMMsterPrinns mined using PGD CLIP-RN50664, BLIP-384 and ViT-I- 16-SigLIP-384 for 25, 50, 75 and 100 ImageNet classes respectively, CLIP- LIT- and Imagenet baselines for comparison. Models that use ResNet rather than visual transformers as well as newer models improving upon CLIP are nevertheless vulnerable to CLIPMasterPrints.
4.6 Attacking different architectures and training approaches
To demonstrate that CLiPMasterPrints are not an isolated phenomenon limited to the investigated archi- tecture or training approach, we mine additional CIIPMasterPrints on a further CLIP-trained model using an ensemble of 64 ResNet50 networks to as an image encoder ( (CLIP-RN50me) same for models trained on recently proposed improvements
Furthermore we do the CLIP, namely BLIP Li et al. (2022) and SigLIP Zhai et al. (2023). Figure 5 shows the results. All evaluated models remain vulnerable to CLIP- MasterPrints: The CLIP-model using ResNet image encoding vulnerable than the transformer-based CLIP ViT-L/14, but is still on par with the ImageNet baseline. For both newer approaches, BLIP-384 and VIT-L-16-SSgl..l form the ImageNet baselines significantly. These results clearly show that vulnerability CLIPMaster is not just limited to CLIP, but also concerns more recently proposed models trained with related contrastive approaches.
) scems to be somewhat less
the PGD-mined CLIPMasterPrints outper-
5 Potential Attack Scenarios
As emphasised by Radford et al. (2021), next to zero-shot- preeeicttt-riiiiionn, highly relevant application of CLIP is zero-shot image retrieval, which offers plenty of attack surface by means of CLIPMastenPrints. In more detail, inserting single CLIPMasterPrint into existing database of images could potentially disrupt the system's 's functionality for a wide range of search terms, for each targeted search term the inserted fooling master image is likely to be the top result. When inserting several SLiPMMattePPritss into the database, even the top n results could consist entirely of these adversarial images rather than the true results. While this also possible when inserting "regular" adversarial examples, the amount of examples needed for an attack using CIIIMMsttePPritts orders of magnitude lower than for regular adversarial examples. Practical malicious applications of this vulnerability could be 1) censorship of images related to a list of censored topics, 2) adversarial product placement: targeting variety of searched brands to advertise different product the top result, or 3) disruption of service: introducing a larger number of unrecognizable CLIPMaster for a wide range of topics, resulting in unintelligible results for many queries, reducing the quality of service of an image retrieval system.
A further interesting issue can be raised with respect to whether fooling examples, which can be recognizod by the user as such, ie. they do not resemble natural images, or show unnatural artifacts which makes them recognizeable adversarial examples to attentive user, pose real-world threat to AT systems in
11

---

Published in Transactions on Machine Learning Research (04/2024)
production. Here we argue that even if fooling images can be recognized by humans, there still remain many ways for an attacker to introduce adversarial examples where no human supervision or control is present. For instance, introducing CLIPMasterPrrint into database could be as simple as putting images online to be crawled by search engines or uploading them through webforms. Impairing the function of the attacked system; or censoring particular images in the system can in this case still be achieved using unrecognizeable fooling images. We show in Section 4.5 that sanitizing the model inputs by training an additional classifier to detect CLIPMasterPrints can be an effective way to mitigate threat surface. Of course this approach bears the drawback that an additional classifier needs to be trained and deployed in production, under which perspective further investigation into increasing the robustness of CLJP-trained models is desireable.
In cases where human supervision is present on the other hand, adversarial examples with slight artifacts may be spotted by an attentive user, but might still fool a distracted or technologically less proficient user. Furthermore, a cunning attacker might choose a template image where resulting artifacts are difficult to make out. We invite the reader to consider the PGD-mined CLIPMasterPrints in Figure bo and Oi in the appendix and form their own opinion on how prominent the resulting artifacts are, and whether they could be missed by an inattentive user or hidden by selecting appropriate image.
6 Discussion and Future Work
This paper demonstrated that CLIP models can be successfully fooled on a wide range of diverse captions by mining fooling master examples. Images mined through both gradient-basod (SGD, PGD) as well as gradient-free approaches (LVE) result in high confidence CLIP scores for a significant number of diverse prompis, image captions or labels. While the gradient-free approach performed slightly worse, it does not require access to gradient information and therefore allows for black-box attacks.
We found that the modality gap in contrastively pre-trained multimodal networks (i.e. image and text embeddings can only be aligned to a certain degree in CLIP latent space) plays a central role with respect to a model's vulnerability to the introduced attack. Low cosine similarity scores assigned to well-matching text- image pairs by a vulnerable model imply that off- manifold images, which align better with a larger number of text embeddings, can be found. PGD-mined images, while being appearing meaningful to humans, are nevertheless found to be off the latent image manifold of the attacked model. The off-manifold nature of the attack is also supported by the observation that information in fooling examples is distributed throughout the whole image for all targeted prompts, rather than locally at different places for each prompt (see Section A.4 in the Appendix), making the mined images vulnerable to occlusion and cropping.
We that a possible way to exploit the off-manifold nature of the attack for possible mitigation approaches is to train a classifier to detect the artifacts introduced by the adversarial mining process in order to sanitize model inputs. This way we are able to reliably distinguish CLiPMasterPrints from 'harmless' images with accuracy 99.01% for our test set. While this approach could be highly effective for real-world systems, it bears the drawback that an additional classifier needs to be trained and deployed in production. Under that perspective further investigation into increasing the robustness of CriPetrained models is desireable.
In terms of increased model robustness, our results demonstrate that the effects of CLIPMasterPrints on the model can be mitigated by closing the gap between centroids of image and text embeddings respectively. While Liang et al. (2022) do not explicitly classify modality gaps as cither beneficial or detrimental to a models performance, our results support the hypothesis that the modality gap leaves CLIP models vulnerable towards CLIPMasterPriing
Thus efforts to mitigate modality gaps even further, while preserving model performance, is a critical future research direction.
Finally, our mined CLIPMasterPrints seem to not only affect the prompls they target, but also generalize to semantically related prompts. In combination with the observation that recent improvements to CLIP such as BLIP and SigLIP are vulnerable as well, this generalization effect additionally increases the impact of the introduced attack. In conclusion, further research on effective mitigation strategies as well as the vulnerability of additional related models is needed.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Reproducibility Statement
We supply our code with instructions on how to reproduce our experiments supplementary material. The code is also available at htttss//githbb.coo/mattt hatpa//oommmmttee//LLLPM/arrrnn CLIPMasterPrints
Broader Impact Statement
The approaches introduced in this paper could be used to mount attacks that misdirect CLIP models in production. For instance, an attacker could manipulate the rankings of CLIP-based image retrieval system resulting in injected CLIPMasterPrints being the top result for wide range of search terms. This could be exploited in malicious ways for censorship, adversarial marketing and disrupting the quality of service of image retrieval systems (for details see Section 5p. Nevertheless, we argue that publishing this work is a necessary step towards understanding the risks of using CLaPetrained models in real-world applications. We also propose and evaluate miligation strategies and hope that our work will inspire others to build on those to make them even more effective in the future.
Acknowledgments
This work was supported by a rescarch grant (40575) from VILLUM FONDEN.
References
Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances adversarial attacks and defenses in computer vision: A survey. IEEE Access, 9:155161- 155196, 2021.
Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 4845-4854, 2019.
Philip Bontrager, Aditi Roy, Julian Togelius, Nasir Memon, and Arun Ross. Deepmasterprints: Gener- ating masterprints for dictionary attacks via latent variable evolution. In 2018 IEEE 9th International Conference Biometrics Theory, Applications and Systems (BTAS), pp. 1-9. IEEE, 2018.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Amupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: survey. arXiv preprint arXiv:1810. 10000009,, 2018.
Giannis Daras and Alexandros G Dimakis. Discovering the hidden vocabulary of DALLE-2. In NeurIPS 2022 Workshop Score-Basee Based Melhods, 2022.
Philip Enevoldsen, Christian Gundersen, Nico Lang, Serge Belongie, and Christian Igel. Familiariy- open-set recognition under adversarial attacks. In Challenges for Out-of-Distribuute Generalization in Computer Vision (OOD-CV), 2023.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Farlo, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139 144, 2020.
Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron C Courville, and Yoshua. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. Tn Proceedings of the International Conference Learning Regresentations (ICLR), 2015.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Nikolaus Hansen. The CMA evolution strategy: A tutorial. arXiv preprint arXi::1604.00772, 2016.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategics. Evolulionary Compulation, 9(2):159 195, 2001.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp. 448-456, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In Proceedings of the International Conference on Learning Representations (TCLR), 2016.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security, pp. 192.12. Chapman and Hall/CRC, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient beaeee learning applied docu- ment recognition. Proceedings of the IEFE, 86(11):2278 2222-22422, 1998.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888 12900. PMLR, 2022.
Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. Adversarial VQA: robustness of vqa models. In Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 2042 2051, 2021.
new benchmark for evaluating the
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou, Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances Neural Information Processing Systems (NewrIPS), pp. 17612 17625, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
George A Miller. Wordnet: a lexical database for english. Communicntions of the ACM, 38(1)):39-411 41, 1995.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence pre- dictions for unrecognizable images. In Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 427-436, 2015.
David A Noever and Samantha E Miller Noever. Reading isn't believing: Adversarial attacks on multi-modal neurons. arxiv preprint arXic:2108.14480 2021.
David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise- injected CLIP. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4055 4063, 2022.
Mesut Ozdag. Adversarial attacks and defenses against deep neural networks: survey. Procedia Computer Science, 140:152-161 2018.
Jielin Qin, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. Are multimodal models robust to image and text perturbations? ar preprint 211 08044, 2022.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), pp. 7748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of Compuler Vision and Pallern Recognation (CVPR), pp. 00684-10695 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 252, 2015.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaezmarczyk, Clayton Mullis, Aarush Katla, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filteed 400 million image-text pairs. In Data Centric AI NeurIPS Workshop, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramalrishha Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of International Conference on Computer Vision (ICCV), pp. 618 626, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines, 8(2):131-162, 2007.
Vanessa Volz, Jacob Schrum, Jialin Liu, Simon M Lucas, Adam Smith, and Sebastian Risi. Evolving mario levels in the latent space of deep convolutional generative adversarial network. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pp. 221 22-2288 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of European Conference on Computer Vision (ECCV), pp. 818 833. Springer, 2014.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of Computer Vision and Pattern Recognition (CVPR), 2023.
A Appendix
A.1 Mined fooling images
Figure 6 shows a selection of mined fooling images in good quality.
A.2 Other mitigation approaches
As an additional mitigation approach, we explored making the VIT-L/14 model robust by adding fooling images to the train set.
Experiment setup. First, we refine the model on the ImageNet train set, where we add for every batch presented to the network, both random noise image well as LVE fooling example. Both the noise image and the fooling example get labeled with a special naff-maaooood-tokee in order to have the model bind off- off-manifold inputs to that token rather than any valid ImageNet label. At every forward step of the model, we generate a new random noise image by leeding zero-mean unit-variance Gaussian Noise into the decoder part of our generating autoencoder. The fooling example on the other hand is generated by running CMA-ES in the loop with the training process. We start out with the best-found previous solution and run one iteration of CMA-ES for every forward step to update the fooling example the changed training weights of the model. This setup creates similar optimization process found in GANs where both models
15

---

Published in Transactions on Machine Learning Research (04/2024)
(a) SGD, optimized on artworks for ViT-L/14
(b) SGD, optimized on artworks for ViT-L L/- @336px
LVE, optimized artworks for VIT-L/14
(d) LVE, optimized artworks for VIT-L/110336pr
(e) PGD, optimized artworks for ViT-L/14
(f) PGD, optimized artworks for VIT-L/140336p
Figure 6: Examples of CLIPMasterPrint images mined through SGD (a.b), LVE (c, d) and PGD (e, f). The complementary approaches are able produce fooling images unrccognizable to humans (a-d) and images that resemble natural images but that display some artefacts perceptible to human eyes (e, f).
16

---

Published in Transactions on Machine Learning Research (04/2024)
attempt to outperform each other. We refine the model for epoch using Adam at learning rate of 10-? and a batch size of 20. We regularize the model with weight decay of 0.2 and set Adam momentum parameters as described in (Radford et al. 2021 31 = 0.9, 32 = 0.98, e = 10-6. Furthermore, we utilize mixod-precision training Micikevicius et al. (2018). Hyperparameters for CMA-ES are identical to the ones used to mine the original Tooling image. F Finally, after refining the model, we mine a new fooling example from scratch for the updated model. We do so to test the model's robustness not only to the original fooling images, but fooling images in general.
0.3
CLIP
0.2
0.1
indigo bunting
eft
green suake
wolf spider
ottehoond
Bemos mountain dog
mangogs
thinovero boctl
ant
graschpper
cassetto
Chrissmas sooking
file
Mtut
mountain tent
pelio
scewdriver
boasler
Windor the
babble
grouitta
free
ImageNet
Fooling image old
Fooling image
class (shorr)
Random noise
Figure 7: CLIP scores for fooling examples mined before and after refinement. with off-manifold token. While mapping existing fooling examples to special tokens can mitigate their impact, the model is still vulnerable to new fooling images
Results. Fig. 7 shows the CLIP scores of our refined model, which has been trained to align off-manifold vectors to special loken, in order to miligate the model's vulnerability to fooling master examples.
Shown are the average CLIP score on the ImageNet validation set, the CLIP score for the original fooling example, the score for fooling example trained after refinement, well as the score of random noise image for each largeted ImageNet label respectively. Due to the newly introduced <off-marifolb>-tokee both noise and the original fooling examples are suppressed by the model and score significantly lower as the mean label score on the ImageNet validation set.
The newly mined looling example on the other hand has not been suppressed at all by the refined model and exhibits scores similar to the ImageNet mean for all labels. The results suggest that our mitigation strategy is sufficient to mitigate existing fooling examples, yet fails to be effective as new fooling examples are mined from the updated model.
A.3 Targeting to 1000 classes
In order to explore the behavior for CLIPMasterPrints when going beyond 100 classes, we mined further fooling images targeting 25, 50, 75, 100, 250, 500, 750 and 1000 classes using PGD. To account for variations in performance, we mine 10 CLIPMasterPrints for each mumber of target classes, where we randomly vary both fooling image initialization as well as the permutation (i.e. order) of the target classes (random seeds used: 0-9). To avoid the mining algorithm to overstep, we save image with the smallest loss in equation 2 after 1000 iterations. The violin plot Figure g shows distributions of the entirety of obtained CLIP scores for all fooling examples, scores obtained by paired images of the entirety of targeted classes in the Imagenet validation set has been added. While, as expected, the overall performance of mined MLIPMasterPrinie declines as more classes are targeted, even the image targeting 1000 classes manages to achieve similar or better scores than the majority of the images the ImageNet training set. We can therefore conclude that, when targeting hundreds of classes, CLIPMasterPrints do not necessarily outperform any and all images in these classes, they still score significantly higher than a large portion of images in question.
function of the number of classes target. For comparison, the distributions of all
17

---

Published in Transactions on Machine Learning Research (04/2024)
0.40
0.35
0.30
0.25
0.20
type
ImageNet
PGD
0.15
0.10
0.05
0.00
25
50
75
100
250
500
750
1000
number optimized classes
Figure 8: Distribution of scores for CLIPMasterPrints mined using PGD for 25, 50, 75, 100, 250, 500, 750 and 1000 ImageNet classes respectively, distribution scores for Imagenet images in the targeted classes for comparison. While the score declines as the number of targeted classes increases, even CLIPMasterPrinss targeting all 1000 Imagenet classes achieves largely on par better scores than the majority of images in the TmageNet training set..
A.4 Analysis of information distribution in CLIPMasterPrints
To understand how information is distributed the found fooling master examples, we create occlusion maps (Zeiler & Fergus 2014 Selvaraju et al.] 2017)) of the fooling master example trained on the titles of famous artworks (Fig, 9 As we blur 75 75 rectangles of the fooling master image in a sliding-window manner with 2 pixel stride and large (o 75) Gaussian blur kernel, measure the change in cosine similarity returned by the ViT- @336pr model. As reference, the same procedure is performed number of artworks the fooling image is intended to mimic. Blurring any part of CLIPMaster results in a significant decrease (between 0.1 and 0.2) of the resulting similarity score of the model. It os noteworthy that the image optimized sing LVE seems to be more robust to occlusions than images obtained by SGD and PGD methods. This can be explained by the more grainy and contrasted patterns in the LVE. image. The individual increases and decrcases for actual artworks on the other hand are more moderate and vary based the location in the image.
For the Random noise image prompt, which has been excluded from optimization, bluring parts of the image results in significantly smaller changes in model output score. Interestingly, the mined CLIPMaster react differenty to occlusion based on the used optimization approach. For the SGD image, blurring different regions of the image affects decreases the score in some regions, while it increases it others. As the SGD image closely resembles a pure noise image, scems intuitive that blurring certain parts of the image decreases model similarity, yet it is unclear why blurring parts of the image increases the similarity, For the L.VE image the other hand, blurring does not result in improving scores, but again, different regions
18

---

Published in Transactions on Machine Learning Research (04/2024)
Prompt: "The Scream" by Edvard Munch Random noise
Munch
Vermeer
SGD
LVE
PGD
Prompt: "Girl with a pearl earring" by Johannes Vermeer Random noise
Munch
Vermeer
SGD
LVE
PGD
-0.05
Prompt: Random noise image
Munch
Vermeer
Random noise
SGD
LVE
PGD
0.025
0.000
-0.02
Figure Occlusion maps for famous artworks, random noise baseline, and mined CLIPMasterPrints for different prompts. Note that, while each row shows the same CLIPMasterPrint, occlusion maps vary for certain part of the image denoted
different prompis. Increases in cosine similarities when blurring in red, decreases shown in blue. Information in the CLIPMasterPrrinn is distributed over the whole image; individual regions in the image that can be mapped to particular prompt.
respond differently the noise prompt. Finally, for large parts of the PGD image, the similarity score improves as information in the image erased through blurring.
While these results demonstrate that the way that CLIP assigns similarity is often far from intuitive to humans, can conclude that information from CLIPMasterPrints resulting in high CLIP confidence scores is spread throughout the image for all captions, and is quite sensitive to occlusions and cropping.
While CLIP has learned to deal with blurring and occlusions in natural images due to the large amount of sufficiently varying images presented during training, this robustness does not translate to other patterns such as noise and adversarial images. We have shown above that blurring the latter two results in significant misalignment of the resulting vector in relation to the text vectors it has been targeting.
19

---

Published in Transactions on Machine Learning Research (04/2024)
A.5 Pseudocode for black-box mining of CLiPMasterPrints
Algorithm illustrates our black-box approach to mining CLIPMosterPrind as pseudocode listing.
Algorithm Black-box approach to find CLIPMasterPrintss
Input: imitial vector hg ~ (0 1),
list of objective prompts c1, C2....,Cos number of to-be-run iterations imar
pre-trained CLIP model Co pre-trained image decoder D& Initialize CMA-FI with ho for =- to iana do Generate candidates h1, hn using CMA-ES mutation Decode images X1, X2.... Ny from h1, bg,..., hn using De for all x, in X1, X2,,..; Xn do
Set 8j.min
for all C in (2,02
Cn do
Set 8j,k Co, (x,, (k)
if 8j,k < 8j.uin then
Set 85.inn 8j.k end if
end for
end for
Update CMA-ES statistics with 81,min, 82,min, ..n 8n,min end for
20


Reviews:
[{"review_id": "VkKUVBuXY2", "reviewer": "Reviewer_udbx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Very well written.\n2. Clear evidence that supports the authors claims.\n3. Nice figures and formatting throughout. \n4. Interesting findings with practical applications.\n\nWeaknesses:\n1. The authors take way too long to explain why one should care about these adversarial attacks.\n2. The term modality gap is mentioned a lot, and explained very little, and a bit too late. Having a succinct, but robust description of what this is would be very useful, as well as the practical details of computing it. Does one need the full ImageNet to do this? How do you backprop through all those images? etc.\n3. In general there seems to be a very strong bias in the writing that the readers are very much aware of all the latest and greatest in adversarial attack land, and while that may be the case for some, rewriting bits of the paper with an assumption that most folks know how CLIP works, and they have previously studied neural networks, backpropagation and deep learning would make this paper more accessible and thus increase its impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper the authors do the following:\n\n1. Introduce a novel set of methods for acquiring adversarial \"master key\" images that tend to show up with very high similarity to a number of key prompts and their classes on the CLIP models.\n2. Demonstrate that a concept previously dubbed the \"modality gap\" seems to be related to the effectiveness of these adversarial attacks, and that optimizing a model to directly reduce this gap, improves a models robustness to such attacks.\n3. Show various pieces of evidence that support their claims and offer further information for exploration.\n4. Demonstrate that while they learned these images on CLIP, these tend to generalize to models that have various degrees of transfer from CLIP, such as BLIP and SIGLIP.\n5. Offer some practical insights into how such methods may be used to cause various degrees of problems in image retrieval systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Narrative Related: \n1. Make it more clear as to why someone should care about such adversarial attacks on multi modal models. Ideally as a paragraph in the introduction that links things to a practical application domain.\n2. Clearly explain what a modality gap is in this context early on.\n3. Consider rewriting bits of the paper with an assumption that most folks know how CLIP works, and they have previously studied neural networks, backpropagation and deep learning to make this paper more accessible and thus increase its impact. \n\nTypos/Odd Phrasing Related:\n\nIn the section titled \"Potential Attack Scenarios,\" the word \"exploited\" is likely meant to be \"be exploited.\" The sentence should read, \"This could be exploited in malicious ways for censorship, adversarial marketing, and disrupting the quality of service of image retrieval systems.\"\n\nIn the section \"4.2 Generalization to semantically related prompts and labels,\" the term \"LVE-mined image\" may cause confusion. It might be clearer to specify that it refers to an image mined using the Latent Variable Evolution (LVE) technique.\n\nIn the \"Broader Impact Statement,\" the term \"adverserial\" should be corrected to \"adversarial.\" The correct sentence should read, \"This could be exploited in malicious ways for censorship, adversarial marketing, and disrupting the quality of service of image retrieval systems.\"\n\nThroughout the document, there's inconsistent use of \"CLIPMasterPrints\" and \"fooling master images.\" For clarity and consistency, it may be beneficial to choose one term and stick with it throughout the document.\n\nIn several places, the document refers to \"gradient-free approach\" and \"black-box approach\" interchangeably when discussing the mining of fooling images without access to the model's gradients. For clarity, it would be helpful to stick to one terminology or clearly define these terms if they are intended to convey different nuances.\n\nThe document uses both \"SGD\" (Stochastic Gradient Descent) and \"PGD\" (Projected Gradient Descent) without a clear introductory explanation of these acronyms on their first use. While these are common terms in the field, a brief explanation or expansion of the acronyms at their first mention would make the text more accessible to readers not deeply familiar with these techniques.\n\nIn the appendix section titled \"A.3 Analysis of information distribution in fooling master images,\" the sentence structure and clarity could be improved. Specifically, the last paragraph could be rewritten for clarity to better explain the significance of the findings related to the distribution of information in fooling images and their sensitivity to occlusions and cropping.\n\nThe document frequently switches between British and American spelling conventions (e.g., \"optimisation\" vs. \"optimization\"). It would be more coherent to stick to one spelling convention throughout the document.", "broader_impact_concerns": "The authors cover their bases well in the paper. I would argue that this work is something that bad actors would already be thinking or doing, and such work can at least make it more clear when it happens, and stimulate more work towards defending against such attacks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vUge8md0wO", "reviewer": "Reviewer_m6KN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper effectively demonstrates the existence of CLIPMasterPrints.\n- The study investigates several optimization procedures to identify CLIPMasterPrints.\n- The research explores various CLIP methodologies (CLIP, BLIP, SigLIP).\n- The paper suggests a straightforward and promising method to decrease the sensitivity to CLIPMasterPrints.\n\nWeaknesses:\n- The range of visual diversity examined in the empirical protocol is somewhat limited. The paper only considers painting images and a subset of ImageNet. It would be beneficial to confirm the existence of CLIPMasterPrints using random image/caption pairs from internet data, similar to the data used for CLIP pretraining.\n- The paper demonstrates that a CLIPMasterPrint can maximize similarity with images extracted from up to 100 classes. However, it is unclear what would happen if an image is randomly selected from the full ImageNet or ImageNet-22k dataset. It remains to be seen whether a CLIPMasterPrint exists in this scenario, or if their existence is limited to specific data subsets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study illustrates the vulnerability of contrastive vision-language pretraining methods to adversarial master images (CLIPMasterPrint), which are designed to maximize their similarity with any given caption. The existence of these master images is confirmed across various methodologies, including CLIP, BLIP, and SigLIP, and can be identified through different means such as SGD, LVE, or PGD.\nThe paper provides empirical evidence of the impact of these master images on a subset of ImageNet and a selection of painting image collections.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be nice to clarify the behavior of CLIPMasterPrints as you increase the number classes to 1000 and beyond.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LOuMXXzHDU", "reviewer": "Reviewer_FGxL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nStudying attacks/defenses on CLIP systems is of interest to the adversarial ML community.\n\nThe results in Figure 3a are more-or-less convincing that the attacks have the intended effect.\n\n**Weaknesses**\n\nBesides image retrieval, zero-shot classification is perhaps the main other application of CLIP models. This paper does not examine whether their \"fooling images\" can successfully attack such classifiers. If these images can be detected, then I am unsure what risk they really are to real-world systems.\n\nGeneralization results in Figure 3 appear marginal: the difference in means between \"Imagenet all classes\" and \"fooling ex. other classes\" appears small (~0.01 or 0.02). Thus I am unsure how much generalization to untargeted classes is taking place.\n\nRegarding \"potential explanation\" of the results in Section 4.2 that generalization occurs to \"semantically related results\", this could be explicitly tested by using the WordNet tree to define exactly which held-out classes are \"semantically related\" or not.\n\n**Minor comments**\n\nSome of the writing comes off as speculative and unfounded:\n* \"A likely explanation can be found\" (Section 4.1)\n* \"A possible explanation could be\" (Caption of Figure 3)\n* \"presumably due to a few mislabeled images or very hard examples\" (Section 4.2)\n* \"which seems to imply that\" (Section 4.4)\n\nI recommended that the authors either substantiate these claims or make it crystal clear that it is just speculation.\n\nSome of the writing is not specific enough\n* \"We found that... plays a central role with respect to\" (Section 6).\n\nBe more specific about what your findings are rather than use generic language like \"plays a central role\" .", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work follows the usual template of the many other works in adversarial ML: propose new attacks, study defenses, draw a possible connection to real-world systems. This work empirically studies adversarial attacks and mitigations on CLIP models. The proposed attacks involve finding a \"fooling image\" (or CLIPMasterPrint), an image that achieves high confidence scores across several classes (prompts) despite not resembling any of said classes/prompts.\nThe optimization proposed to find such fooling images is the minimum cosine similarly between the embedded image and embedded prompts over a collection of given prompts. Three methods of solving the optimization problem are studied: SGD, PGD, and an evolutionary strategy in latent space called LVE.\n\nAn experiment on the generalization of fooling images to classes/prompts unseen at attack time is made, finding that fooling images can sometimes transfer to \"semantically related\" classes. An investigation of a mitigation strategy based on the \"modality gap\" is made, where it is found that reducing this gap leads to less effective attacks. The potential practical relevance of such attacks is envisioned to involve manipulation of image retrieval systems and others.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**\n\nCan the proposed fooling images be reliably detected? If so, does this not eliminate the envisioned risk to real-world systems?\n\nRevise the experiment in Section 4.2 to explicitly test whether generalization to \"semantically related\" or not classes using the Wordnet hierarchy.\n\n**Strengthen**\n\nFix legend in Figure 4.\n\nPlease clean up writing as described above.", "broader_impact_concerns": "A Broader Impact Statement is given. No further concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper identified a new limitation of CLIP models by finding fooling master images
for CLIP (CLIPMasterPrints) that are of high confidence but non-meaning to human users. The authors demonstrated the crafting of CLIPMasterPrints in the black-box setting and proposed some mitigation strategies.

This paper provides new findings to understand the limitations of CLIP models and their implications on robustness. The authors' rebuttal has addressed the main concerns of reviewers. All reviewers are in favor of accepting this submission, and I concur.
Decision: Accept (poster)