What do we learn from inverting CLIP models?

## Abstract

Abstract We employ an inversion-based approach to examine CLIP models. Our examination reveals that inverting CLIP models results in the generation of images that exhibit semantic alignment with the specified target prompts. We leverage these inverted images to gain insights into various aspects of CLIP models, such as their ability to blend concepts and inclusion of gender biases. We notably observe instances of NSFW (Not Safe For Work) images during model inversion. This phenomenon occurs even for semantically innocuous prompts, like “a beautiful landscape,” as well as for prompts involving the names of celebrities.

## 1 Introduction

CLIP (Contrastive Language-Image Pre-training) models (Radford et al., ) have gained significant attention in the field of artificial intelligence. Serving as a link between textual and visual data, these models have found application in numerous deep learning contexts (Nichol et al., ), (Rombach et al., ), (Patashnik et al., ), (Mokady et al., ), (Chegini & Feizi, ), (Parelli et al., ), (Lüddecke & Ecker, )). They not only demonstrate zero-shot performance comparable to fully supervised classification models but also exhibit resilience to distribution shifts. A key factor contributing to this resilience is their training on extensive web-scale datasets, which exposes them to a diverse array of signals within the input data.

While large-scale training offers numerous advantages, little is known about the content of the proprietary dataset used to train the original CLIP model, or the biases this data may impart on the model.
Despite prior exploration into the knowledge acquired by CLIP models (Ghiasi et al., ), (Goh et al., ), our work is the first attempt to analyze them through the lens of model inversion.

Most of our knowledge about model biases comes from generative models for which we can explicitly observe and interpret their outputs. But how do we study the knowledge of a non-generative model like CLIP?
Model inversion is the process of generating content, either images or text, that minimizes some function of a neural network’s activations. When applied to classification tasks, model inversion is used to find inputs that are assigned a chosen class label with high confidence. In this study, we put a different twist on model inversion, using it to invert the CLIP model by finding images whose embeddings closely align with a given textual prompt. Unlike inverting image classification models that have a limited number of classes, the inversion of CLIP models provides us the freedom to invert a wide range of prompts and gain insights into the knowledge embedded within these models.

By utilizing the extensive set of prompts available for inverting CLIP models, we delve into analyzing various aspects of this family of models.
Our contributions are summarized as follows:

I. In recent years, generative models like DALLE (Ramesh et al., ) and IMAGEN (Saharia et al., ) have shown the capability to blend concepts. We demonstrate that the same holds true for CLIP models, and the knowledge embedded inside CLIP models is capable of blending concepts.

II. We demonstrate that through inversion, seemingly harmless prompts, such as celebrity names, can produce NSFW images. This is particularly true for women celebrities, who the CLIP model seems to strongly associate with sexual content. Certain identities, like “Dakota Johnson”, are close to many NSFW words in the embedding space. This may be problematic since the embeddings of CLIP models are being used in many text-to-image generative models. Addressing this issue requires more meticulous curation of data during the training of large-scale models.

III. We demonstrate that CLIP models display gender bias in their knowledge through inversions applied to prompts related to professions and status.

IV We investigate the scale of the training data on the quality of the inversions, and we show that more training data leads to better inversions.

It should be noted that we study the caveats and biases of CLIP when used as a generative model, and these caveats do not necessarily manifest themselves when CLIP is used in a non-generative way. Still, these studies give us insights into the training data used by CLIP, and the kinds of biases that model developers should be aware of when red teaming a CLIP-dependent model (of which there are many).

## 2 Related Work

### 2.1 Class Inversion

Class inversion is the procedure of finding images that activate a target class maximally. The process starts by initializing input x randomly and using gradient descent to optimize the expression

$$ $\max_{x}L(f(x),y)+R(x),$ $$

where $f$ denotes a trained classification neural network, $L$ is the classification loss function (typically cross-entropy), and $y$ is the target label. Regularization term $R$ aims to prevent the optimized image from devolving into meaningless noise by incorporating priors associated with natural images. DeepDream (Mordvintsev et al., ) uses two regularization terms: $\mathcal{R}_{\ell_{2}}(\mathbf{x})=\|\mathbf{x}\|_{2}^{2}$ which penalizes the magnitude of the optimized image, and $\mathcal{R}_{tv}(\mathbf{x})$ which penalizes Total Variation forcing adjacent pixels to have similar values. DeepInversion (Yin et al., ) uses an additional regularization term

$$ $\mathcal{R}_{feat}(\mathbf{x})=\sum_{k}\left(\|\mu_{k}(\mathbf{x})-\hat{\mu}_{ k}\|_{2}+\|\sigma_{k}^{2}(\mathbf{x})-\hat{\sigma}_{k}^{2}\|_{2}\right)$ $$

where $\mu_{k},\sigma_{k}^{2}$ are the batch mean and variance statistics of the $k$-th convolutional layer, and $\hat{\mu}_{k},\hat{\sigma}_{k}^{2}$ are the running mean and running variance of the $k$-th convolutional layer. The $\mathcal{R}_{feat}$ is only applicable to architectures using batch normalization (Ioffe & Szegedy, ), restricting its application for other networks, such as ViTs (Dosovitskiy & Brox, ) and MLPs (Tolstikhin et al., ). In this study, we explore the inversion of CLIP models. Unlike traditional models with predefined classes during training, CLIP models undergo training with language supervision, wherein specific classes are not explicitly specified.

Figure: Figure 2: Progression of Inverted Images for prompts “A peaceful sunset,” “Professor Albus Dumbledore,” and “A loving couple”. We start with resolution 64 and increase the resolution to 128, and 224 at iterations 900, and 1800 respectively.
Refer to caption: extracted/5446351/figures/progression/sunset/s_0.png

### 2.2 CLIP Visualization

Exploring CLIP models from a visualization standpoint has been previously undertaken, and we present a brief summary of the insights derived from such examinations. A study conducted by (Ghiasi et al., ) revealed that CLIP features exhibit activation based on semantic features rather than visual characteristics. For instance, they identified features activated by concepts such as death and music despite the absence of visual similarity among the images that triggered these features. Additionally, (Goh et al., ) found that akin to the human brain, CLIP models possess multi-modal neurons that respond to the same concept in photographs, drawings, and images of their name. However, our investigation in this work focuses on unraveling the knowledge embedded in CLIP models through the lens of model inversion.

### 2.3 Bias and NSFW content

Recent research in deep learning has aimed at tackling biases and NSFW content in large multimodal datasets like LAION-400M and text-to-image generative models. Concerns raised by (Birhane et al., ) highlight explicit and problematic content in LAION-400M, with (Birhane et al., ) indicating a $12\%$ increase in hateful content with the growth of the LAION dataset. This underscores the crucial need for dataset curation practices to minimize harmful biases.

In the realm of Text-to-Image generative models, (Perera & Patel, ) delves into bias within diffusion-based face generation models, particularly regarding gender, race, and age attributes. Their findings reveal that diffusion models exacerbate bias in training data, especially with smaller datasets. Conversely, GAN models trained on balanced datasets exhibit less bias across attributes, emphasizing the necessity to address biases in diffusion models for fair outcomes in real-world applications. A promising solution introduced by (Gandikota et al., ) is the Erased Stable Diffusion (ESD) method, designed to permanently remove unwanted visual concepts from pre-trained text-to-image models. ESD fine-tunes model parameters using only text descriptions, effectively erasing concepts such as nudity and artistic styles. This approach surpasses existing methods and includes a user study, providing code and data for exploration.

Additionally, (Luccioni et al., ) proposes an assessment method focusing on gender and ethnicity biases, revealing the under-representation of marginalized identities in popular systems like Stable Diffusion and Dall·E 2. Furthermore, the “Safe Latent Diffusion (SLD)” method presented in (Schramowski et al., ) actively suppresses NSFW content in text-conditioned image models, addressing challenges posed by NSFW image prompts.

## 3 Method

A CLIP model consists of two key networks. The first is the visual encoder network, denoted as $V$, responsible for creating image embeddings. The second is the text encoder network, marked as $T$, which generates embeddings for textual content. The training process of a CLIP model is guided by a contrastive loss function designed to both increase the similarity between an image and its associated caption and reduce the similarity between that image and all other captions in the same batch. To invert a CLIP model for a prompt $p$, we solve the following optimization problem starting from a random noise:

$$ $\max_{x}cos(V(A(x)),T(p))+Reg(x)$ $$

which $cos(.)$ is the cosine similarity, $A$ is a random augmentation chosen at each iteration step, and $Reg$ are regularization terms used.

We adopt using augmentations from (Ghiasi et al., ) into our methodology. These augmentations are employed to invert classification models and serve as image priors. Specifically, if an image is classified as a bird, its augmentation is also expected to be classified as a bird. Similarly, in CLIP inversion, if an image aligns with a given prompt, its augmentations must align with that prompt as well. The main augmentation used in (Ghiasi et al., ) is ColorShift; however, we incorporate random affine, color jitter, and Gaussian noise as augmentations in our experiments. Details can be found in Section 5.
We also integrate the ensembling technique outlined in (Ghiasi et al., ), where we concurrently optimize $b$ augmented versions of the input to align with the prompt, with $b$ representing the batch size.
We use Total Variation (TV) and L1 loss as regularization terms as also been used in (Mordvintsev et al., ).

$$ $Reg(x))=\alpha TV(x)+\beta||x||_{1}.$ $$

The sequence of images, evolving from random noise, is illustrated in Figure 2. We begin at a resolution of 64 and gradually increase to 128 and then to 224.

Figure: Figure 3: Inverted images for prompt “An astronaut exploring an alien planet, discovering a mysterious ancient artifact” for different models.
Refer to caption: extracted/5446351/figures/different_clips/rn50.png

Figure: Figure 4: Inverting the prompt “A person jumping in a park”
Refer to caption: extracted/5446351/figures/jump/jump1.png

## 4 Analysis

In this section, we investigate the varied insights enabled by model inversion for CLIP models. We begin by exploring the capacity of model inversion to generate novel concepts. Following this, we provide an analysis of NSFW content detected within these inversions. We then delve into the gender biases inherent in CLIP models, followed by an investigation into the impact of the scale of training data. Lastly, we examine the limitations of CLIP models in making accurate associations.

Figure: Figure 5: Inverted images of certain celebrity names lead to NSFW imagery.
Refer to caption: extracted/5446351/figures/celeb/cencored/zendaya_1.png

### 4.1 Blending Concepts

The initial observation we make regarding CLIP model inversions is their capacity to merge concepts. As highlighted in (Ramesh et al., ), text-to-image generative models possess the notable ability to blend different concepts convincingly. Interestingly, we notice this phenomenon in the inverted images generated by CLIP models, even though these models aren’t primarily intended for generation. Instances of these combinations can be seen in Figure 1. Take the prompt “panda mad scientist mixing sparkling chemicals” as an example; the resulting inverted image perfectly captures its intended meaning. The majority of the visualizations presented throughout the paper originate from the ViT-B16 model (Dosovitskiy et al., ). However, as depicted in Figure 3, the blending concept capability is also observable in other model variants.

It is important to highlight the refined nature of CLIP model inversions beyond their capability to blend concepts. For instance, when inverting prompts related to celebrity names, as depicted in Figure 12, the resulting images are completely recognizable. For example, consider the prompt “Hugh Jackman”; we can readily identify this actor from the inverted image, which also portrays him as a fit individual.

In another instance, we employ model inversion to explore prompts associated with emotions, as illustrated in Figures 10 and 11. These inverted images provide fascinating insights into how the model perceives emotions. For instance, when given the prompt “an interested person,” the resulting image emphasizes enlarged ears, implying attentiveness and careful listening. Additionally, our examinations yield further notable observations. For instance, as shown in Figure 4, the model effectively portrays the concept of jumping by deliberately blurring the image of the jumper. These examples represent only a fraction of the investigations that can be made with the help of model inversion, illustrating its potential to understand various aspects of CLIP models.

### 4.2 NSFW Content Analysis

Recently, researchers discovered instances of child abuse material within the LAION dataset, leading to its public removal. This underscores the urgent need for improved detection methods for sensitive content and better NSFW (Not Safe For Work) filters. When we apply model inversion on a CLIP model, specific prompts generate NSFW imagery, even those seemingly innocuous, such as using celebrity names, “A beautiful landscape,” “The map of the African continent,” and “A scientist conducting groundbreaking research.” In Figure 6, examples of these images and their associated prompts are depicted. This emphasizes the critical necessity for robust content filtering during CLIP model training.

Figure: Figure 6: Inverting prompts “A beautiful landscape”, “The map of the African continent”, and “A scientist conducting groundbreaking research” results in NSFW imagery. All these images with red squares were flagged as NSFW when processed through a stable diffusion safety checker.
Refer to caption: extracted/5446351/figures/sensitive2/1_1.png

As depicted in Figure 6, when we invert the prompt “A beautiful landscape,” it produces NSFW visuals. Our verification through the Stable Diffusion safety checker confirms NSFW detection in three separate inversion attempts, each initialized with different random noise.

Table: Table 1: In the initial word series, we see words closely associated with ’A beautiful landscape’ within the embedding space. In the second word series, we see words that are proximate to the embedding of the inverted image.

We speculated that this could stem from the prompt’s nearness to NSFW language. Similar to (Rando et al., ), we utilize a word list including 10,000 most common English words(^1^11[Most common English Words](https://github.com/first20hours/google-10000-english)), Naughty, Obscene, and Otherwise Bad Words(^2^22[List of Dirty Naughty Obscene and Otherwise Bad Words](https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words)), Names for body parts (^3^33[List of Body Parts](https://github.com/janester/mad_libs/blob/master/)), Offensive/Profane Word List (^4^44[Offensive/Profane Word List](https://www.cs.cmu.edu/%C2%A0biglou/resources/bad-words.txt)), 11913 words in total, to identify the 20 words most closely associated with the prompt in the embedding space. However, upon reviewing the list of words as shown in Table 1, none of them seemed NSFW upon examination.
Yet, when we examined words whose embeddings closely matched those of the inverted image, several NSFW words emerged, see Table 1.

**Table 2: The words closest to the names of the celebrities in the embedding space.**
| Prompts |  |
| --- | --- |
| Dakota Johnson | dakota, emma, lisa, sexy, maria, fit, petite, hot, latina, ana, melissa, mia, eva, busty, cute, shakira, joy, dana, brunette, lauren, mariah, xx, victoria, dylan, d, seo, boobs, julia, mm, slut, bon, nsfw, jap, dog, to, elegant, j, sarah, barbara, me, rebecca, ooo, bikini, booty, k, titty, yea, jessica, honk, yes, ero, dat, yo, liberal, erotic, nicole, oh, ye, wow, eh, l, pamela, xxx, bmw, jo, tits, big tits, z, aw, dammit, clara, abs, ya, tb, cocktease, h, cia, je, nastyslut, jj, oo, new, linda, ah, f**kable, ha, hi, dm, deluxe, qt, t, ecchi, di, amanda, b, um, jesus, katrina, o |
| Timothée Chalamet | petite, dylan, eminem, to, hot, harry, samuel, ye, xx, he, yo, boy, aye, oscar, eh, sam, man, me, ya, yea, um, mm, oo, yes, lit, lauren, fit, his, oh, emma, jesus, ooo, sexy, o, cute, matt, lil, ian, tom, of, tb, ah, h, aw, uh, i, liberal, adam, ha, osama, hi, peterson, fw, dm, new, wow, hh, n–ga, ch, rob, mac, im, on, es, hey, shit, model, k, max, og, men, jon, rl, jim, rt, fr, xxx, que, af, www, y, avi, santorum, yet, le, cho, shitty, t, cw, ok, pamela, f**k, x, b, oc, f**kin, je, tf, ho |
| Leonardo Dicaprio | leo, marco, ye, oscar, jesus, carlo, yea, dylan, yo, ben, oh, oo, sean, le, eminem, rl, ha, to, jim, eh, lol, lo, yet, ok, um, uh, l, ooo, tom, ya, yes, man, og, louis, hi, liberal, wow, so, dan, osama, but, ah, mm, me, lit, aw, ian, cia, mem, dat, rob, fr, apollo, o, aye, my, ob, xi, meta, latino, mac, ol, diego, kinda, hey, how, k, relevant, title, jpeg, bet, political, america, paul, oc, he, f**kin, rp, on, tremendous, mariah, who, d, hh, carlos, and, apt, af, i, bc, h, usa, op, ou, ryan, fa, lou, b, shit |
| Shakira | shakira, mariah, britney, melissa, pamela, dylan, barbara, latina, sarah, emma, maria, mia, sara, madonna, dakota, lauren, linda, sh, dat, sandra, hot, mm, lisa, que, michelle, ia, ya, shited, , rica, she, shitty, to, diego, sexy, yea, da, si, ali, es, yes, shit, stephanie, wow, i, shitola, clara, o, eh, ah, fit, amanda, shitf**k, oh, oo, pam, sierra, ooo, ha, nicole, las, aka, carlos, pocha, af, suckme, k, my, marco, sg, sd, solar, d, suckmyass, yo, y, jesus, ok, persian, jo, jim, dale, hi, yet, shitdick, marilyn, me, f**k, re, liz, s, ye, karen, hey, f**ked, por, rat, allah, laura, so |

Furthermore, using celebrity names as prompts can lead to the generation of NSFW images through inversion. We can see examples of these images in Figure 5. We count the NSFW-flagged images out of 100 inverted images using the stable diffusion safety checker for each of these prompts to quantify the extent of potentially NSFW content generated through inversion. As depicted in table 3, there is a notable prevalence of NSFW-flagged images for female celebrities. For example, for the prompt “Dakota Johnson” 94 images out of 100 images are flagged as NSFW.

Providing analysis on this prompt, we find the closest words in the embedding space to the embedding of “Dakota Johnson”. Surprisingly, as shown in Table 2, we can find many NSFW words present in the list of words. This situation can present challenges, particularly since CLIP models serve as text encoders in numerous text-to-image generative models.

**Table 3: The number of NSFW-flagged images determined from 100 images identified by a stable diffusion safety checker for ViT-B/16 OpenAI CLIP and ViT-B/16 OpenCLIP trained on Laion2b, and ViT-B/16 OpenCLIP trained on Laion400B.**
| Prompt | CLIP | OpenC2B | OpenC400M |
| --- | --- | --- | --- |
| Jennifer Anniston | 9 | 6 | 50 |
| Dakota Johnson | 94 | 43 | 53 |
| Demi Lovato | 80 | 11 | 29 |
| Zendaya | 60 | 7 | 20 |
| Jennifer Lopez | 88 | 19 | 32 |
| Johnny Depp | 18 | 14 | 18 |
| Leonardo DiCaprio | 22 | 1 | 4 |
| Brad Pitt | 9 | 25 | 19 |
| George Clooney | 7 | 2 | 3 |

The proximity of a celebrity name’s embedding to NSFW words can be undesirable. In a separate experiment, as illustrated in Table 5, we identify the words closest to the embedding of an image featuring “Dakota Johnson” on the internet. Once more, among the first 200 closest words, there are several instances of NSFW words. This underscores the existence of NSFW content during the training of CLIP models, emphasizing the necessity for enhanced curation of training data, especially when involving authentic human images.

Initial experiments counting the number of NSFW images for celebrity names utilized a ViT-B16 OpenAI CLIP model trained on a web-scale dataset not known to the public. Upon conducting the same experiment with a ViT-B16 OpenCLIP model (Ilharco et al., ) trained on Laion2b (Schuhmann et al., 2022), the incidence of inappropriate NSFW-flagged images notably decreases. However, when utilizing models trained on Laion400M (Schuhmann et al., 2021), the number of NSFW flagged images rises once more. The presence of troublesome explicit images in Laion400M is investigated by Birhane et al. (2021). Once again, this underscores the critical importance of meticulously curating training data for CLIP models. The results are shown in Table 3.

Figure: Figure 7: Top row: Inverting the prompt “A successful student in university” yields 100 images, all classified as depicting a man. Bottom row: Inverting the prompt “A successful female student in university” for 100 trials results in all images being classified as depicting a woman. Interestingly, for the latter prompt, as demonstrated in the second row, some of these inversions exhibit partial nudity despite no mention of it in the prompt.
Refer to caption: extracted/5446351/figures/university/1.png

**Table 4: For each prompt, we generate 100 inverted images and conduct classification to determine whether these inverted images are associated with a man or a woman. The classification is performed using a separate CLIP model. The “Neutral” column indicates prompts as shown in the table. The “FEMALE” and “MALE” columns represent scenarios where gender specification is added to the prompt. For instance, using “A male nurse in the pediatric ward.”**
| Prompt | Man | Woman | Man | Woman |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| STATUS RELATED | NEUTRAL | FEMALE | MALE |  |  |  |
| A thief, dangerous person, criminal | 96 | 4 | 0 | 100 | 99 | 1 |
| A person with a low-paying job | 97 | 3 | 0 | 100 | 98 | 2 |
| A person with a high-paying job | 96 | 4 | 0 | 100 | 99 | 1 |
| A successful student in university | 98 | 2 | 0 | 100 | 100 | 0 |
| PROFESSION RELATED |  |  |  |  |  |  |
| A nurse in the pediatric ward | 0 | 100 | 0 | 100 | 100 | 0 |
| A kindergarten teacher in the colorful classroom | 0 | 100 | 0 | 100 | 100 | 0 |
| A midwife in the delivery room | 0 | 100 | 0 | 100 | 100 | 0 |
| A firefighter at the scene of a blazing fire | 99 | 1 | 0 | 100 | 100 | 0 |
| A construction worker at a bustling construction site | 99 | 1 | 0 | 100 | 100 | 0 |
| A mechanic in the busy auto repair shop | 97 | 3 | 0 | 100 | 99 | 1 |

### 4.3 Gender Biases

Works like (Perera & Patel, ) have analyzed biases and stereotypes in generative models. This analysis is possible with generative models because we can see the generations. However, in non-generative models like CLIP, this is not possible. (Agarwal et al., ) investigated biases and stereotypes in CLIP models.
In this work, we use model inversion to conduct bias and stereotype analyses on CLIP models. We focus on examining gender bias. Inverting 100 images from a ViT-B16 model with various initializations for the prompt “A successful student in university,” we then employ a different CLIP model (ViT-B32) to classify the inverted images into “man” and “woman” categories. The outcome reveals that 98% of the examples are classified as “man.” However, when specifying a prompt where gender is indicated, such as “a successful male/female student in university,” the inversions are nearly entirely (more than 99%) classified according to the prompt’s specification. This suggests that when the prompt is neutral, the inversions tend to exhibit bias toward a specific gender, reflecting the bias present in the model. Examples of these inversions are visible in Figure 7. The top row displays images inverted from a neutral prompt, all depicting a male student. In contrast, the bottom row showcases inversions where the prompt specifies the gender as female. Remarkably, upon closer inspection, numerous images in the latter category feature bras and partial nudity. We can see more examples of the second row in Figure 13 in the Appendix.

We conducted a similar experiment for two categories of prompts: one related to status and another related to profession, as illustrated in Table 4. Professions such as a nurse, kindergarten teacher, and midwife are predominantly categorized as female, while professions like firefighter, construction worker, and mechanic are mainly categorized as male.

### 4.4 Effect of Training Data Scale

Figure: Figure 8: The effect of training data scale on the quality of inversions.
Refer to caption: extracted/5446351/figures/scale/astronaut_openai.png

The impact of the training dataset on the quality of inverted images is significant. Comparing to inversions performed on classification models like in papers (Ghiasi et al., ), the inversions done on CLIP models are much better. We speculate that this might be because of the scale of the training dataset. For example ImageNet (Deng et al., ) only contains 1M images, and Imagenet22k only contains 14M images. This also holds true for CLIP models. When a CLIP model is trained on a limited dataset, the resulting image quality is poor. We observe instances of inverted images from RestNet50 CLIP models that were trained on three different datasets: OpenAI CLIP training data with 400 million image-caption pairs, CC12M (Changpinyo et al., ) with 12M images, and yfcc15M (Thomee et al., ) with 15M images. We hypothesize that the success of inversions is closely tied to the scale of the training data. We can see examples of these inversions in Figure 8.

Figure: Figure 9: The examples where CLIP fails to make correct associations. Prompts from left to right: “A big dog chasing a small kitten.,” “a female mannequin dressed in a black leather jacket and gold pleated skirt”.
Refer to caption: extracted/5446351/figures/bow/cat.png

### 4.5 Bag of Words

(Yamada et al., ) demonstrates that CLIP models perceive prompts as aggregations of concepts. For instance, when presented with an image containing both a yellow lemon and a purple eggplant, along with the prompt “In this picture, the color of the lemon is [mask]”, with choices “yellow” and “purple”, the model selects “purple” over “yellow”. This choice reflects the model’s attempt to encompass as many concepts as possible from the given image. Due to the strong association between “eggplant” and “purple”, the model opts for “purple” to account for the presence of the “eggplant” concept in the image. In a separate experiment, they demonstrate that shuffling the words within a sentence has minimal impact on the CLIP score. This phenomenon is also evident in our inversions. Illustrated in Figure 9 is an example where the prompt “A big dog chasing a small kitten” results in an inverted image depicting a “big kitten chasing a small dog.” This suggests that the CLIP model forms inaccurate associations, treating the prompt more like a set of individual words rather than a coherent sentence.

## 5 Experimental Details

We utilize Adam as our optimizer with a learning rate set to 0.1. To implement various random augmentations for different inputs within the batch, we employ the Kornia library. Unlike PyTorch’s default augmentations, which use the same augmentation for all images in a batch, we require different augmentations for each element in the batch due to identical inputs. In our experiments, we employ random affine, color jitter, and Gaussian noise augmentations. We apply random affine and color jitter with a probability of 1, while Gaussian noise is applied with a probability of 0.5. For random affine, we configure degrees, translate, and scale parameters to 30, [0.1, 0.1], and [0.7, 1.2], respectively. Regarding color jitter, we set the parameters for brightness, contrast, and saturation to 0.4 each, and hue to 0.1. We complete a total of 3400 optimization steps. Initially, we begin with a resolution of 64, then increase it to 128 at iteration 900, and finally to 224 at iteration 1800.

## 6 Reproducibility

Our code has been made accessible via [https://github.com/hamidkazemi22/CLIPInversion](https://github.com/hamidkazemi22/CLIPInversion).

## 7 Discussion and Limitations

We present a method for studying biases and knowledge inherent in CLIP models using qualitative methods that are typically only available for generative models. While the dataset used to train the original CLIP model is proprietary, visualization methods give us a glimpse into its construction. The strong tendency of the CLIP model to produce NSFW imagery across a wide range of contexts suggests that the dataset is not carefully curated, and it likely contains a considerable amount of NSFW content.

Furthermore, the close proximity of specific prompts, such as celebrity names, to NSFW (Not Safe For Work) words in the embedding space raises notable concerns. This is particularly significant given the widespread use of these embeddings across various applications, including text-to-image generation models like Stable Diffusion (Rombach et al., ). Despite efforts to mitigate the generation of NSFW images in diffusion models like Stable Diffusion (Rombach et al., ), none of these endeavors have explored the possibility that the issue might stem from the text encoder employed by these models. Addressing this concern earlier in the diffusion model pipeline may be necessary.

A notable limitation of this study is that we use generative strategies to extract conclusions from a model that is not typically operated in a generative way. While model inversion gives us a powerful window into CLIP’s behaviors, these behaviors do not have to be represented in other operational modes.

## 8 Impact Statement

We want to clarify that we have not intentionally sought to create any NSFW images during the inversion process. The emergence of such behavior is inherent to CLIP models. Despite not using any NSFW prompts, we have observed that specific prompts can still result in NSFW imagery. This raises a significant concern that warrants attention within the community. It underscores the importance of employing improved data filtering and curation techniques for training models on web-scale datasets.

## 9 Acknowledgments

This work was made possible by the ONR MURI program, the AFOSR MURI program, and DARPA GARD. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IIS-2212182), and by the NSF TRAILS Institute (2229885).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Chris Olah's work on feature visualizaion (https://distill.pub/2017/feature-visualization/) shows that applying image inversion on fourier-transformed images can yield better results; would be interesting to see if the results are different if this fourier-based regularization technique is used.
- Is there a way to leverage the proposed image inversion approach to mitigate identified biases in CLIP models?
soundness: 2
strengths: Strengths:
- The analysis shows that CLIP models trained on more data are more amenable to image inversion (better quality of inverted images
summary: The paper investigates CLIP models using image inversion techniques to analyze their learned representations and biases. Their analysis reveals three findings: (a) CLIP models can effectively blend different concepts in inverted images, (b) models contain embedded NSFW associations that emerge even from innocent prompts, and (c) strong gender biases exist, particularly in occupational and status-related concepts.
weaknesses: Weaknesses:
- No novelty in methodology or findings. The problem of image inversion has been studied in the context of discriminative (deepdream and papers cited in the submission) and generative models (e.g., https://arxiv.org/abs/2405.15012, https://dl.acm.org/doi/abs/10.1145/3372297.3417270). The problem of identifying CLIP biases has also been studied in the past (e.g., https://arxiv.org/abs/2311.05746, https://ojs.aaai.org/index.php/AIES/article/view/31657 and references therein).
- Fairly indirect approach to study model biases. The image inevrsion approach proposed in this paper requires a second human-in-the-loop inspection of inverted images to identify biases. This can be tedious, error-prone, and is not scalable. Also, it is unclear if there is a principled way to quantify the extent to which a given model is biased w.r.t. a given concept, which is essential for making progress in this direction of forgetting / unlearning undesirable concepts.
- This paper specifically focuses on CLIP models, even though the image inversion approach can be applied to more recent (and better performing) image-text models. Studying biases across models (e.g., do inverted images w.r.t. CLIP models tranfer to other models) would be interesting.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: I would love to hear from the authors regarding the following questions:
1. From the above section on weaknesses, how can the research community today concretely leverage these insights to inform data, modelling or evaluations? For example, how do any of these insights translate into useful modelling pointers for generative diffusion models? Do any of the learnings from CLIP-like models transfer to the generative context?
2. Tangentially, this work also demonstrates the ability to inspect training data ~ is it accurate to say that this can imply training data leakage (and inspection via model inversion) and thus there could be evidence of memorization? If yes, how can we ensure downstream models are robust to such inversion analysis?
3. How can we improve text rendering based on the analysis presented? What is the intuition behind the choice of regularization being correlated with the text rendering quality? 
4. On the statement of "Similarly, in CLIP inversion, if an image aligns with a given prompt, its augmentations must align with that prompt as well. " - why was a different set of transformations chosen? What is the motivation? It is suprising that this is a big discriminator of image quality.
soundness: 3
strengths: This work presents an interesting analysis on interpreting embedding-based image features from the widely popular CLIP model (where the information about training data is proprietary). It flags important drawbacks elicited via model inversion which point out potential flaws in the training data, flagging an important issue given that CLIP image embeddings are widely used. Further, the authors have aptly presented the different implications in an organized and coherent fashion making it easy to follow the paper.
summary: This work presents an insight into embeddings learnt by CLIP via the lens of model inversion. Since the data used for training CLIP is proprietary, the authors essentially leverage model inversion to generate images whose embeddings align closely with the embeddings of certain text prompts. An analysis leads to the following conclusions on both safety and model capabilities: a) Seemingly normal prompts elicit NSFW images unexpectedly, specifically for many female celebrities b) There exists prevalent gender bias based on providing prompts on factors like profession etc c) the images represent blending in the semantic space as expressed by concept blending in the prompt, implying the model’s understanding of semantic relationships (as opposed to more visual, pixel level features). The paper is accompanied by appropriate visualizations and statistics where apt to support the presented claims.
weaknesses: While the work presents an interesting analysis, it is unclear how these insights can be concretely leveraged to improve image generation pipelines as of today. Can we inform any of the following? a) Modelling strategies and making models more robust to such potentially bad data points? Any kind of safety finetuning? b) data curation strategies if any? Further, some modelling choices (e.g. choice of transformations) are not well motivated. These questions have been outlined in the next section.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Does using random affine transformations instead of ColorShift affect the conclusions? I believe an analysis like this would be beneficial as well. You mention in line 198 that it has a significant impact on the quality of images, but it is not clear whether the underlying conclusions change and whether the generated images semantically change or remain the same.

What classification model is used for experiments in Table 4? Could you provide details about the architecture, training data and some performance metrics of the model used please?
soundness: 3
strengths: The paper presents a solid analysis of CLIP models through a novel approach using model inversion. The paper is well written and the motivation is clear. The problematic studied in the paper is timely and will be of good use to the community. To my knowledge, the idea is indeed novel.

Specifically, the authors clearly demonstrate that CLIP models possess the capability to blend concepts, akin to generative models like DALLE and IMAGEN. Their study reveals associations between seemingly harmless prompts and NSFW content, particularly concerning female celebrities, highlighting the necessity for better data curation. Additionally, the paper uncovers gender biases related to professions and roles.  Furthermore, the investigation into textual components within inverted images, especially the effects of TV regularization, enriches the understanding of model behavior, as well as unwanted artefacts that might arise from usage of CLIP models. Overall, the paper offers good contributions to the understanding of CLIP models, emphasizing both their capabilities and areas for improvement.
summary: The paper examines the biases present in the CLIP model and proposes using model inversion to generate content that is assigned to a chosen class label with high confidence. The authors claim to demonstrate that CLIP can blend concepts, produce NSFW images from seemingly harmless prompts, and exhibit inherent gender bias. They also show that more training data can lead to better model inversions and that textual components are present within the inverted images, which they link to TV regularization not being used in the loss function.
weaknesses: Overall, the paper appears to be poorly formatted, giving the impression of being rushed without proper attention to formatting guidelines. For example, Table 4 is misaligned and requires reformatting, and there are 10 unnecessary empty lines between Figure 2 and the text. The same issue occurs with Table 6 in the appendix. This is disappointing, as the text within the paper is well-written and addresses an important topic.

Additionally, while the experiments in Table 4 seem convincing, I believe additional experiments would be beneficial, such as examining potential racial biases (for example you can do the same for Table 4 but for black/white people). The paper generally lacks comprehensive experimental analyses - for example, for "a beautiful landscape, "the map of the african continent" or "a scientist conducting groundbreaking research", how many times did you see NSFW images? Do all of them not pass through the safety checker because the residuals (as marked with red squares in your images) are always the same? Is there anything else that appears in these images (for the same prompts) that is flagged as NSFW? The findings are very interesting and find the reader wanting for more but even after reading through the appendix, the paper falls short. Also, the authors should provide details regarding the model which they perform classification with (this model is vaguely mentioned in line 392).

Finally, regarding the fourth point which you raise (line 106), I am not sure what contribution does it bring to the paper. Although it is visible that scale has quality on the inversions, why is this not obvious? Furthermore, since the paper does not improve the techniques for inversions, what does this insight contribute to the paper, in particular in examining whether CLIP contains biases or not? In my opinion it would be better to remove this point from the paper as it does not really relate to the problem that you are trying to investigate, but I would love to hear why you think it is important if you decide to keep it.

Since there are no space constraints and an additional page is available, I would be inclined to raise my score if the authors reformat the paper, improve its overall structure, and provide further experiments similar to those in Table 4.

Other minor details:
Line 107: The abbreviation "TV" is used without introduction.
Line 112: Please clarify what "activating a target class maximally" means.
Line 146: Reference error.
Line 150: Change "delves" to "delve."
Line 183: Typo ("which").
Tables 1 and 2: More words should have an asterisk, e.g., "sh*thead," and the filtering should be stronger.
Line 321: Reference the word lists by providing links (e.g., English words, Naughty Obscene and Otherwise bad words).
Lines 395 and 396 overlap.
Line 448: Typo with a double period.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please address the points raised in the Weaknesses section. Here are additional relevant questions:
- How do the findings affect typical downstream tasks of CLIP (embeddings)?
- The generated images of Fig. 4 seem to be most surprising because the prompts are harmless. Is there a reasonable explanation for this behavior? The paper does not include a quantitative evaluation such as the ratio of flagged images by the stable diffusion safety checker, e.g., when generating 100 images of each of the three prompts, how many are flagged?
- Please explain the hypothesis of line 320 that the prompts of Fig. 4 are near to NSFW language when Fig. 1 (top) only shows safe words as close in the embedding space.
- What is the similarity value of each word in Tab. 1 (top) and in Tab. 2? Do the unsafe words have a high similarity with celebrity names or other harmless words?
- Regarding Sec. 4.3, what if the prompt is "a successful male or female student in university"? Is the bias still one-sided?
- In this study cosine similarity is optimized in isolation. In practice, CLIP is often used contrastively (the way it was trained), e.g., in zero-shot classification. What happens if the Softmax of the cosine similarity is optimized, e.g., for the three prompts in Fig. 4, or for the different celebrities?

Verdict:  
The paper applies model inversion to CLIP models, yielding interesting insights about their training data, as well as some expected results regarding model behavior. However, it is unclear how these findings would translate to downstream applications of CLIP. Unfortunately, the experimental evaluation falls short of ICLR publication standards and does not adequately support the claims made. To strengthen the study, quantifying the frequency or severity of each insight wherever possible and seeking explanations for the observed behaviors could provide a more robust analysis. This could involve inspecting open CLIP models where both model and data are available.
soundness: 1
strengths: - Applying model inversion to CLIP models is a suitable way to obtain insights into its proprietary and unavailable training data.
- The paper reveals that the CLIP training data was not cleaned from potentially harmful content.
- It is interesting to see that CLIP model inversion can produce (somewhat) coherent objects and text
summary: The paper employs model inversion on CLIP models to visualize images that induce high similarity with a variety of text prompts. By studying the generated images, the paper concludes that:
1) CLIP can blend multiple semantic concepts,
2) the prompt of (primarily female) celebrities is close to sexually explicit words and can produce sexually explicit images,
3) the inverted images reflect gender biases learned from data,
4) CLIP models trained on a larger dataset produce higher quality inversions, and
5) CLIP inversions include written text in the images.
weaknesses: - While CLIP model inversion is interesting for the sake of scientific curiosity, the paper does not discuss any practical implications on downstream tasks, such as retrieval, classification, segmentation, text-to-image modeling. Sec. 7 (l. 470) even states that "these behaviors do not have to be represented in other operational modes".)
- The paper makes rather strong claims which are mostly supported by few qualitative examples. Furthermore, the experimental analysis is not thorough enough. The following elaborates on each contribution.
  1) Very few images show blending of concepts, and it is not clear if such combined concepts existed in the original training data (e.g., fictional art). If the study was performed on CLIP models trained on open data, it is relatively easy to verify through retrieval if the given concept combination is novel (or at least to provide stronger support).
  2)
    1) For the prompt analysis, a word pool of the 10,000 most common English words and 1,913 additional unsafe words are used. Such overrepresentation of unsafe words can skew and bias the results and their interpretation. For example, it is likely the that unsafe words are much closer to each other in the embedding space than most of the top 10,000 words which have greater variety. Hence, if one unsafe word is retrieved, multiple similar words will follow, potentially exaggerating the qualitative results of Tab. 2. Extending the most common word list naturally would be a better way to incorporate more words, especially because the top 10k words already include unsafe words.
    2) Top-k closest word lists (Tab. 1, 2) can be misleading because they do not reveal the similarity of the words. If the retrieval word list does not contain relevant words, similarity can be low and decrease quickly. The closest words in Tab. 2 contain stop-words, characters, or other non-words such as d, j, to, ok, lol, mm, yea, yo, ha, um, ia, da, si, which suggests that the surrounding words have rather low similarity with the celebrity names.
  3) Tab. 4 suggests strong gender biases. However, model inversion is not a faithful generative, but an optimization process. If CLIP was gender biased 55% to 45%, i.e., slightly higher similarity for a prompt when depicting one gender over the other, an optimization process will always seek to generate that one gender. As a result, model inversion likely exacerbates model biases. This could explain the very one-sided values in Tab. 4. which do not allow to draw conclusions about the severity of the CLIP model's biases.
  4) \+ 5. The findings are not particularly surprising and not quantitatively supported, e.g. through prompt similarity or OCR. As stated in line 434, it is well known that CLIP recognizes text in images.
- While the paper presents some insights from model inversion, it does not try to explain them. E.g., could a confounding latent concept such as "(female) celebrity" cause these findings? An ablation could test for regular male and female names to support this hypothesis. Similarly, more general concepts/words such as "woman" or "man" could be tested.
- The paper does not try to make a connection to real images. Inverted images are out-of-distribution for CLIP. Are the image-text similarities also out-of-distribution, e.g., much higher than what real images typically achieve? Do the conclusions change if the inversion process is restricted/regularized to similarity magnitudes of real images?

Minor:
- L. 107: The abbreviation 'TV' is used without prior introduction.
- L. 146: Missing citation
- L. 181: Equation should be numbered; x and p are not introduced
- Sec. 5 seems to better fit as Sec. 4.5
- Tab. 4 extends beyond page boundaries and there is not enough whitespace between caption and main text
- While the text mentions ViT-B16 (OpenAI?) is used unless otherwise specified, it would be clearer if the captions of each image specify the model