Is What You Ask For What You Get? Investigating Concept Associations in Text-to-Image Models

## Abstract

Abstract Text-to-image (T2I) models are increasingly used in impactful real-life applications. As such, there is a growing need to audit these models to ensure that they generate desirable, task-appropriate images.
However, systematically inspecting the associations between prompts and generated content in a human-understandable way remains challenging. To address this, we propose Concept2Concept , a framework where we characterize conditional distributions of vision language models using interpretable concepts and metrics that can be defined in terms of these concepts. This characterization allows us to use our framework to audit models and prompt-datasets. To demonstrate, we investigate several case studies of conditional distributions of prompts, such as user defined distributions or empirical, real world distributions. Lastly, we implement Concept2Concept as an open-source interactive visualization tool facilitating use by non-technical end-users. Warning: This paper contains discussions of harmful content, including CSAM and NSFW material, which may be disturbing to some readers.

## 1 Introduction

Text-to-image (T2I) models have become central to many real-world AI-driven applications. However, the complexity of these models makes it difficult to understand how they associate concepts in images with textual prompts. Existing works have shown that T2I models can resolve prompts in unexpected ways (Bianchi et al. ()). Furthermore, the training datasets for T2I models are often large, uncurated, and may contain undesirable prompt to image associations that models can learn to internalize (Birhane et al. ()).
Thus, without robust auditing frameworks that help us detect these undesirable associations, we risk deploying T2I models that generate unexpected and inappropriate content for a given task.

However, auditing T2I models is challenging because it is difficult to systematically, efficiently, and intuitively explore the vast space of prompts and possible outputs. Because raw pixel values alone are difficult to semantically reason about, previous works learn mappings from raw inputs to high-level concepts. This can be achieved post-hoc (Kim et al. (); Zhou et al. (); Ghorbani et al. ())
or as an intervention during training (Koh et al. ();Chen et al. (). Although these methods were designed for classification networks, it is this general intuition which motivates our work.

In this paper, we propose a framework for producing interpretable characterizations of the conditional distribution of generated images given a prompt, $p(\mathtt{image}|\mathtt{prompt})$. We do so by extracting high-level concepts from each image and summarizing $p(\mathtt{image}|\mathtt{prompt})$ in terms of such concepts. Here, we define concepts as a class of objects/nouns, ideas, open vocabulary detected classes or labels.

Our contributions are as follows:

(1) We propose an interpretable framework for concept-association based auditing of conditional distributions.
Specifically, in our framework: we sample images from a T2I model under audit, given a prior distribution over prompts–either a user defined distribution or an empirical real-world distribution. Then, using a fast, scalable visual grounding model, we extract concepts from generated images. We characterize the conditional distribution of the generated images by analyzing the distribution of concepts.
This framework allows users to systemically investigate associations of conditional distributions at varying levels of granularity, from broad concept trends, co-occurrences, to detailed visual features.
By design, our framework utilizes visual grounding models that localize concepts in images, enabling a deeper analysis of visual representations. Simple association mining metrics help uncover non-obvious concept relationships.

(2) We demonstrate a wide range of concrete use-cases for our framework, by applying it to audit models and prompt datasets. In addition to demonstrating the effectiveness of the framework, our analysis unearthed new findings that are independently interesting.
In particular, *we discovered child-sexual abuse material (CSAM)* in a human-preferences prompt dataset and misaligned classes in a synthetically generated ImageNet dataset. These findings not only demonstrate the utility of our framework but also contribute to the broader discourse on the safety, fairness, and alignment of T2I models.

(3) We introduce an interactive visualization tool, based on our framework, for human-in-the-loop auditing of T2I models. Our tool allows users to explore and inspect the identified concept associations. To facilitate widespread use, we provide our framework as an open-source package, enabling researchers and practitioners to easily audit their own models and datasets.

## 2 Related Work

Biases in T2I models. There is a body of works that have qualitatively investigated biases in T2I models, focusing on social biases related to gender, race, and other identity attributes. For example, Bianchi et al. () qualitatively demonstrated a range of social biases in T2I models, including biases related to basic traits, social roles, and everyday objects. Similarly, Ungless et al. () manually analyzed images generated by T2I models and found that certain non-cisgender identities were misrepresented, often depicted in stereotyped or sexualized ways. Through several focus groups, Mack et al. () found that T2I models repeatedly presented “reductive archetypes for different disabilities”. Qualitative evaluations play a critical role in exposing instances where generative models can be biased. However, given the large space of possible prompts and images, instance-based bias probing alone cannot paint a systematic picture of how T2I models may (mis)behave in application.

A number of works have focused on automating bias detection at scale. For example, in Cho et al. (), the authors measured visual reasoning skills and social biases in T2I models by using a combination of automated detectors and human evaluations to assess the representation of different genders, skin tones, and professions. Likewise, Luccioni et al. () employed Visual Question Answering (VQA) models and clustering-based evaluations to measure correlations between social attributes and identity characteristics. TIBET (Chinchure et al. ()) dynamically generated axes of bias and counterfactual prompts based on a single input prompt. However, these works either do not operate on the general concept-level (e.g. only specifically probe for concepts related to social attributes) and/or do not leverage the rich information in the concept co-occurrences, the stability of concepts, nor do they pinpoint and extract specific concepts. We found these key elements to being integral to uncovering deeper insights relating to T2I models. Most closely related to our work is Try Before You Bias (TBYB) (Vice et al. ()), which proposes an object-centered evaluation methodology to quantify biases in T2I models using image captioning and a set of proposed metrics. Also like us, CUPID Zhao et al. () presents a visualization framework that enables users to discover salient styles of objects and object relationships by leveraging low-dimensional density-based embeddings. Our approach generalizes and builds upon these previous works. While existing methods focus primarily on social bias and style relationships, our framework enables a more nuanced audit of model behavior, capturing not only social biases but also the underlying patterns in how models represent and associate visual concepts.

Important use cases of synthetic data. One important use case of synthetic data is for training backbone or foundation models. Works have demonstrated that training backbone models using synthetic ImageNet (Deng et al. ()) clones can achieve similar performance on specific evaluation benchmarks as compared to real ImageNet dataset (Azizi et al. ();He et al. ();Sarıyıldız et al. ()). They can also be used to realign or mitigate bias in foundation models Abdel Magid et al. () Howard et al. () or evaluate vision-language models Fraser & Kiritchenko () Smith et al. (). In addition to training foundation models, synthetic images and their corresponding prompts are used in reinforcement learning human feedback (RLHF). Many datasets of real user prompts and preferences have been collected. Examples include RichHF-18K Liang et al. (), ImageReward Xu et al. (), and Pick-a-Pic Kirstain et al. (). In this work, we demonstrate how to use our framework to audit synthetic datasets as well as prompts datasets for RLHF alignment of T2I models. For auditing prompt datasets, we focus on StableImageNet (Kinakh ()) and Pick-a-Pic. The latter is used to train PickScore which is then used as an evaluation metric and to better align T2I models with human preferences.

## 3 Concept2Concept: An Intuitive Framework for Characterizing the Conditional Distribution of T2I Models

We propose *Concept2Concept*, a novel framework to provide systematic and interpretable characterizations of the conditional distribution of images generated by a T2I model given a prompt, $p(\mathtt{image}|\mathtt{prompt})$. We do so by first extracting high-level concepts from generated images, then characterizing the conditional distribution of these concepts given prompts, $p(\mathtt{concept}|\mathtt{prompt})$.

Obtaining Concept Distributions from T2I Models. We assume a distribution of text prompts $p(t)$, defined by the user or the auditing task. We empirically represent $p(t)$ with $N$ sampled prompts $\{t_{i}\}_{i=1}^{N}$ from $p(t)$:

$$ $\displaystyle t_{i}\sim p(t),\quad\text{for }i=1,2,\dotsc,N.$ (1) $$

For each sampled prompt $t_{i}$, we approximate the conditional distribution of images given prompt $t_{i}$ by generating $K$ images $\{x_{i,k}\}_{k=1}^{K}$ from the T2I model $G$:

$$ $\displaystyle x_{i,k}\sim p_{G}(x_{i,k}|t_{i}),\quad\text{for }k=1,2,\dotsc,K.$ (2) $$

As image distributions are difficult for humans to work with at a global level, we focus on studying the distribution of concepts in the generated images. Specifically, for each image, we are interested in $C(x)$, the set of concepts in image $x$. In practice, we compute $C(x_{i,k})$ for each generated image $x_{i,k}$ by applying an object detector $D$ to label and localize (e.g., bounding box) the concepts in the image $C_{i,k}=D(x_{i,k})$. The choice of object detector $D$ is not fundamental to our framework and can be application-specific. For instance, in our experiments, we utilize two distinct detectors—Florence 2 (Xiao et al. ()) and BLIP VQA (Li et al. ())—each offering different levels of detection capabilities. The flexibility to choose $D$ allows us to adapt the framework to various tasks, depending on what is important to detect and at which level of granularity. Recent large vision-language models like Florence 2 offer multiple modes including visual grounding. We note that our use of an object detector $D$ can introduce uncertainty in the extracted concepts, $C_{i,k}$ (e.g., due to detection confidence levels or the probabilistic nature of the model).
Thus, we consider $C_{i,k}$ as samples from a distribution $C_{i,k}\sim p(C|x_{i,k})$. In the case that concepts are extracted deterministically from a given image $x_{i,k}$, $p(C|x_{i,k})$ is a delta distribution.

Finally, we empirically approximate two distributions of concepts – the marginal distribution of concepts over the prompt distribution, $p(C)$; and the conditional distribution of concepts given a prompt, $p(C|t)$:

$$ $\displaystyle p(C)=\int_{t}p(C|t)p(t)\,dt,\;\;\;\;$ $\displaystyle p(C|t)=\int_{x}p(C|x)p_{G}(x|t)\,dx.$ (3) $$

Summarizing Concept Distributions.
We further summarize the concepts distributions $p(C)$ and $p(C|t)$ we obtain from the T2I model to enable end-users in exploring and discovering associations between concepts in the prompt and concets in the generated images.
Towards this end, we use a number of metrics to aid in our analysis of concept associations.

*Concept Frequency $P(c)$.* We calculate the empirical frequency of each concept $c$ across all generated images. The probability $P(c)$ is estimated by:

$$ $\displaystyle P(c)=\frac{\sum\limits_{i=1}^{N}\sum\limits_{k=1}^{K}\mathbb{I}[ c\in C_{i,k}]}{N\times K},$ (4) $$

where $\mathbb{I}[c\in C_{i,k}]$ is the indicator function that equals 1 if concept $c$ is present in $C_{i,k}$, and 0 otherwise. This identifies the dominant concepts associated with the prompt distribution $T$.

*Concept Stability.*
To assess the variability of concept $c$ across prompts, we compute its coefficient of variation (CV) as:

$$ $\displaystyle CV(c)=\frac{\sigma_{c}}{P(c)},\quad\sigma_{c}=\sqrt{\frac{1}{N} \sum\limits_{i=1}^{N}\left(P(c\mid t_{i})-P(c)\right)^{2}}.$ (5) $$

We set a threshold $\tau$ to focus on concepts that occur with sufficient frequency:
$\mathcal{C}_{\tau}=\{c\in\mathcal{C}\mid P(c)>\tau\}$.
Persistent concepts are those that consistently appear regardless of the prompt (small CV) , while triggered concepts are more sensitive to specific concepts within the prompts (large CV) .

*Concept Co-Occurrence.*
To uncover rich associations between concepts in the generated images, we analyze concept co-occurrences. For each pair of concepts $(c,c^{\prime})$, we compute the co-occurrence probability:

$$ $\displaystyle P(c,c^{\prime})=\frac{\sum\limits_{i=1}^{N}\sum\limits_{k=1}^{K} \mathbb{I}[c,c^{\prime}\in C_{i,k}]}{N\times K}.$ (6) $$

This analysis helps us map the relationships between concepts present in the images. However, since the number of detected concepts can be large and co-occurrences grow quadratically, we employ simple association mining metrics to identify significant and relevant co-occurrences: support, confidence, and lift. We refer the reader to the appendix for additional details.

Choosing Task-Relevant Prompt Distributions $p(t)$.
The concept distribution $p(C)$ depends on the choice of the prompt distribution $p(t)$. Generally, the choice of $p(t)$ should be informed by the task, e.g. auditing models for social biases. In this paper, we consider two primary scenarios for auditing: *model auditing* and *prompt dataset auditing*.

*Model Auditing.* In this scenario, the prompt distribution $p(t)$ should be user-defined and should capture realistic ways users may interact with the model in order to understand its behaviors. Here, users may generate controlled sets of prompts, possibly including counterfactual examples, to audit how the T2I model $G$ represents specific concepts. By carefully designing $p(t)$, users can manipulate the input conditions and study the resulting concept distribution $p(C)$ marginalized over prompts $p(t)$. This allows for targeted analysis of the model’s behavior with respect to particular concepts or biases. We provide several experiments in section 4.

*Prompt Dataset Auditing.* When we are trying to understand the images generated from a set of prompts, $p(t)$ should be an empirical distribution derived from real-world prompt datasets, such as those used in reinforcement learning from human feedback (RLHF). By examining the concept distribution $p(C$ marginalized over prompts $p(t)$, we can surface potential issues like harmful or inappropriate content in training datasets. We provide several experiments in section 5.

## 4 Application 1: Auditing the Model

### 4.1 Case Study 1: Toy Examples

Figure: Figure 1: Top concepts detected by our framework. Concepts are curated to highlight the effectiveness of the framework for user-defined prompt distributions in Section 4.1
Refer to caption: x1.png

Figure: Figure 2: Examples of concepts as they are extracted from our framework along with sample co-occurrences.
Refer to caption: x2.png

On a small, pedagogically designed prompt set, we demonstrate how to use Concept2Concept to probe for unexpected generation behaviors. We design a prompt set which varies along a social attribute of interest, age, and a second prompt set which adds an axis of variation along semantically similar words (e.g. jogging vs running). Concretely, our prompt distribution is a uniform distribution over the set $\{\text{``A photo of a {[age]} person }\texttt{[action]}\text{''}\}$, where [age] takes value in $\{\text{young},\text{middle-aged},\text{old}\}$, and [action] takes value in $\{\text{jogging},\text{sprinting},\text{running}\}$.

In Concept2Concept, comparing conditional concept distributions helps us identify concept associations. Figure 1 shows the conditional concept distributions $p(C|t)$ we obtain through Concept2Concept. By contrasting these distributions, we find that the concept jogging is largely associated with the concept woman (the concept of “woman” occurs in roughly 60% of the generations). Conversely, running is associated with man in about 80% of the generations.
We are also able to discover that different attires are associated to the concept of jogging and running, respectively (see Figure 2).

In Concept2Concept, visually grounding concepts helps us verify that concepts are resolved as we desire. Figure 2) provides a small example of concept co-occurrences. Even seemingly concrete concepts can be visually resolved in diverse ways, in Concept2Concept, we visually ground each concept (see Figure 2). The localization of our framework is highly precise, even for small objects like glasses, which occupy only a small fraction of the entire image. Using Concept2Concept, we can identify, compare, and contrast the conceptual representations in generated images resulting from different prompts. This enables us to uncover unexpected concept associations (e.g. boy and sprinting vs. woman and jogging). Additional results, including concept stability are included in 2.

### 4.2 Case Study 2: Replicating Bias Probing Results from Literature

**Table 1: The average percentage of detections of woman and man generated by a concept detector in our framework for the StableBias (Luccioni et al. ()) and TBYB (Vice et al. ()) case studies. Note that these are two different case studies with different experimental settings. U.S. Bureau of Labor Statistics.**
| Model | Concept Detected | U.S. Labor Bureau |  |  |
| --- | --- | --- | --- | --- |
|  | % woman | % man | % woman | % man |
| StableBias (Luccioni et al. ()) | 31.10 % | 68.90% | 47.03% | 52.97% |
| Ours | 28.41 % | 71.59% |  |  |
| TBYB (Vice et al. ()) | 31.64% | 68.36% |  |  |
| Ours | 19.56 % | 80.44 % |  |  |

With Concept2Concept, we demonstrate that we can replicate experiments from existing works on gender-based bias probing. We consider two studies, each using a different probing framework: StableBias (Luccioni et al. ()) and Try Before You Bias (TBYB) (Vice et al. ()).
In both works, the authors prompt a T2I model with names of professions and report the distribution of gender representation (in percentages) amongst the generated images. Our findings are summarized in Table 1. Consistent with the two existing studies, we found that the concept woman is underrepresented across most professions, with only about 30% of the images depicting the concept woman, while approximately 70% of the images depicted the concept man. While we were able to reproduce similar gender distributions as StableBias, our distributions are notably different from those reported for TBYB. We provide a discussion for this discrepancy in 2.

### 4.3 Case Study 3: Scaling Up Qualitative Studies on Disability Representation

Figure: Figure 3: Concept distributions for two examples on disability representation. We can leverage the concepts to determine how to alter the images in a human-understandable way through simple negative prompting. We can also visualize unexpected co-occurrences of specific concepts: shorts and sticks.
Refer to caption: x3.png

We replicated and extended findings from a qualitative study on disability representation in T2I models, which involved a focus group to evaluate the generated outputs (Mack et al. ()). By automating this process with our framework, we conceptually quantify how the model represents disabilities across various prompts. Concretely,
$\mathcal{T}_{\text{disability}}=\{t_{i}=\text{``A person with {[value]} }\}$ where $\texttt{[value]}\in\{\text{a disability},\text{bipolar disorder},\text{a
chronic illness},\text{cerebral palsy},\text{a limb difference},\text{hearing
loss}\}$. Figure 3 (top left) shows for the prompt “a person with a disability,” nearly 100% of the generated images depicted wheelchair, despite not being explicitly stated in the prompt. When analyzing specific disability-related prompts, the model produced similarly stereotypical associations. For instance, the prompt “cerebral palsy” primarily generated images of young and boy, while “a limb difference” Figure 3 (bottom left) resulted in images with the concepts shorts and foot; individuals in the images are typically dressed in shorts to emphasize the disability. Unexpectedly,stick co-occurred with shorts. We visualize this in Figure 3 (bottom right) and find that the model produces branch-like sticks, perhaps to represent crutches. In the case of “chronic illness,” the model often depicted people in hospital, beds, with their faces covered. Additional results and a detailed experimental setup can be found in the appendix (A.6).

We demonstrate how the framework’s conceptual characterization of the conditional distribution can be useful for adjusting the T2I outputs. Figure 3 (top right) shows how we can use the concepts and apply negative prompting with concepts we wish to attenuate and/or amplify. Suppose we want to exclude the concept wheelchair and emphasize face and person in the images. Our framework coupled with simple prompt revision enables users to directly alter conceptual output distributions. This case study illustrates the framework’s ability to identify harmful and unexpected biases.

## 5 Application 2: Auditing Prompt Datasets

Figure: Figure 4: Co-occurrences of concepts with the detected concepts girl and woman in 10 random samples of the Pick-A-Pic DatasetKirstain et al. (). Most frequent concepts overall can be found in A.7.
Refer to caption: x4.png

### 5.1 Case Study 4: Detecting Unexpected Issues in Pick-A-Pic

Warning: This section contains discussions of harmful content, including CSAM and NSFW material, which may be disturbing to some readers.

Figure: Figure 5: Prompts in the Pick-a-Pic dataset that trigger the naked, underwear, and thong concept associations. None of the prompts explicitly call for nudity or hyper-sexualization (HS).
Refer to caption: x5.png

The Pick-a-Pic dataset (Kirstain et al. ()) is one of many human preferences datasets consisting of prompt-image pairs. Authors reason that these “human preferences datasets” are useful for realigning T2I models so that they produce output users actually want to see. Kirstain et al. () train the PickScore on the Pick-a-Pic training set to learn their collected human preferences. The PickScore is then used (1) as a standalone evaluation metric to measure the quality of any given T2I model and (2) to improve T2I generations by providing a ranking of a sample of images given a prompt. It is clear that these two use cases are incredibly safety-critical.
We used Concept2Concept to explore concept associations in Pick-a-Pic and audit the dataset for unexpected and undesirable associations. Notably, our analysis of concept associations in Pick-a-Pic revealed child sexual abuse material (CSAM), pornography, and hyper-sexualization of women, girls, and children.

We draw 10 random samples of size 1K each from the training split of the Pick-a-Pic dataset (^1^11https://huggingface.co/datasets/yuvalkirstain/pickapic_v1). In addition to the prompts and images, each row indicates which image the user ranked higher. When sampling, we save the images that the user ranked as better for a corresponding prompt. (^2^22In the case of a tie, we randomly choose one of the two images.) Our analysis is thus conducted on the prompts and images that would reward a model for choosing similar content when trained on this dataset.

Figure 4 shows concept co-occurrences for the concepts girl and woman. In addition to the stereotypical and non-diverse concept co-occurrence distribution, we highlight in red concepts that may warrant additional investigation or probing. We find that the concept girl co-occurs with the concepts young, naked, nude, thong, underwear, and lingerie among others. Similarly, woman co-occurs with naked, breast and lingerie. We investigate, through localizing each concept, and determined the input concepts (prompts) associated with the detected concepts. In Figure 5, we show examples of this process. Notably, none of these prompts explicitly call for harmful material, yet the models output–and the users chose–nudity, hyper-sexualization, CSAM, and pornographic material. For example, the top row of figure 5 shows that the prompt “Japanese redhead woman” produced a naked individual. Similarly, the prompt “An asian woman” and “Afrikan woman wet clothes” produced hyper-sexualized (thong) and naked content. We note that hyper-sexualization when not necessarily desired or explicitly stated in the prompt is not limited to woman or girl but is also exhibited for man and boy. Additional results are shown in the appendix, along with the overall top detected concepts, with confidence intervals.

Where a user may not elicit pornographic material, a T2I model will enforce it. Moreover, due to the design of the web-app used to collect the dataset, users are presented with two images at a time and a new image is presented only when the user ranks one of the existing images. The user can only break out of the ranking if they change the prompt. The authors of the Pick-a-Pic dataset conducted an automatic filtering of NSFW prompts by using a list of keywords. This list was not released. Using our framework, we showed that these problematic concepts do not necessarily occur in the prompt, but the association still occurred, and thus their filtering scheme may not be the most effective way of auditing the dataset. We emphasize that this is of high importance for several reasons. First, T2I models have been shown to memorize the original training data Carlini et al. (), so there is a possibility of replicating real CSAM and pornography. Second, the fact that this harmful material is also in a dataset that is used to realign and evaluate T2I models should not go understated. The human has to be in the loop; using our framework simplifies that by characterizing the distribution in terms of human understandable concepts.

### 5.2 Case Study 5: Detecting Misalignment in Synthetic ImageNet

Figure: Figure 6: Sample of misaligned synthetic ImageNet images detected through the conceptual characterization of conditional distributions through our framework. The first 9 images from each class. Clear misalignment. All 100 images for these classes as well as other detected misaligned classes can be found in the appendix.
Refer to caption: x6.png

In this section, we demonstrate another example of auditing the prompts (and the T2I model) used to generate a synthetic ImageNet1k Russakovsky et al. () dataset. Many works demonstrate that using synthetic ImageNet, either to augment real ImageNet or entirely replace it, boosts performance, as discussed in the related works section 2. Moreover, works also use synthetic versions of ImageNet to evaluate other models Bansal & Grover (). These are two important and safety-critical use cases of T2I model outputs and we use our framework to investigate their concept associations.

Following TBYB Vice et al. (), we audit the synthetic StableImageNet dataset Kinakh (). Concretely,
$\mathcal{T}_{\text{StableImageNet}}=\{t_{i}=\text{``a photo of {[value]},
realistic, high quality }\}$ where $\texttt{[value]}\in\{\text{ImageNet1K Classes}\}$. Several existing works have experimented with a similar setup to this generation procedure; see Bansal & Grover () and Sarıyıldız et al. ()). Using our framework, we identified misaligned concept associations in Figure 6. Through the localization and co-occurrence of concept dreadlocks with beanie, mask, ski we found several classes had completely misaligned images. For example, the class turnstile in real ImageNet is intended to be “A narrow, mechanical gate, with rotating arms of wood or metal…”(^3^33Oxford Dictionary), however, the T2I model generated photos of a musical band called Turnstile.
Similarly, for the class redbone, the intended ImageNet class refers to
“A variety or breed of American hound with a predominantly red coat…”(^†^†footnotemark:) However, the model instead generated images of human individuals. Another set of issues arises in the two closely related classes: ski and ski mask. First, the model did not produce ski content and second, the model replaced it with individuals with certain skin tones and hairstyles. The issue is thus two fold; one of prompt adherence and one of fairness. One can attribute the failure to either a vague prompt or a poor T2I model. In any case, it raises concerns regarding both the dataset and the model’s accuracy and bias.
It is also important to note that while this exact dataset was not published in a specific paper, the recipe for generation is replicated in other works as a comparison point (Bansal & Grover ();Sarıyıldız et al. ()) demonstrating that the model (1) actually learns good representations with this recipe and (2) presents an approach practitioners actively use and investigate.

## 6 Interactive Tool

Given the ubiquity of T2I models and, as demonstrated in the case studies, the problematic concept associations and underlying prompts they may contain, there is a broad need for further analysis of these models and their corresponding datasets. To lower the technical barrier for such auditing, we propose an interactive visualization tool. This tool embeds into a user’s Jupyter notebook and accepts a broad array of data sources. A user can investigate specific concepts, their stability, and co-occurrence with other concepts (Figure 23). Additionally, users may search for specific concepts to identify the prompts used to generate the concept, the distribution of these prompts, and, localize how the concept is depicted in different images (Figure 24).

## 7 Conclusion

In this work, we proposed an interpretability framework designed to characterize the conditional distribution of T2I models in terms of high-level concepts. The purpose of this framework is to provide users with an in depth understanding of how T2I models interpret prompts and associate concepts in generated images. By providing in depth analysis through metrics such as concept frequency, stability, and co-occurrence, we reveal biases, stereotypes, and harmful associations that other frameworks may overlook. We also note that our findings of misaligned classes in StableImageNet and child sexual abuse material and pornographic material in the Pick-a-Pic dataset are independently significant.

## Appendix A Appendix

An overview of our method can be found in Figure 7.

Figure: Figure 7: Concept2Concept enables users to systematically analyze the conditional distributions by investigating concept frequency and stability, co-occurrences, and detailed visual features using simple association mining metrics. This approach enables comprehensive insights into underlying concept associations.
Refer to caption: x7.png

### A.1 Ethics Statement

We recognize that the detection of a concept does not imply an absolute truth. The definition of a concept is subjective and can vary across different contexts, shaped by societal, cultural, and historical influences. Concepts are not neutral; they carry power dynamics that affect how they are understood and applied, often reflecting dominant ideologies and reinforcing existing inequalities while marginalizing alternative perspectives. For instance, when the framework detects woman or Asian it is important to recognize that these labels are not necessarily true, as such attributes cannot be reliably inferred from visual cues alone—especially in the context of synthetic images. Since these images are artificially generated by models, the concept of identity tied to real-world characteristics, such as gender or ethnicity, becomes even more ambiguous. In this sense, the labels applied to synthetic images are inherently inaccurate, as they refer to constructs rather than real individuals. However, these detections are still valuable because they help expose biases within the models and datasets. By surfacing such issues, our tool provides insight into how certain concepts are (mis)represented or (over)simplified, allowing for critical evaluation and improvement of text-to-image models.

Similarly, other design choices in our work reflect inherent biases. For example, our use of U.S. labor statistics as a comparison point introduces bias by privileging a specific cultural and national framework, which may not be representative of broader, global contexts. This comparison inherently reflects the dominant perspective from which the data was sourced, potentially excluding or misrepresenting other groups.

These biases and design choices, whether in concept detection or the data we use, shape the outcomes of our work and how it is interpreted. We acknowledge that the definitions we apply and the concepts we choose to highlight are not neutral; they actively influence the narrative and meaning of our results. Therefore, we strive to remain aware of the ethical implications of our decisions and aim for transparency in acknowledging the limitations and biases inherent in our work.

### A.2 Limitations

While our framework provides valuable insights into the concept associations learned by text-to-image (T2I) models, it has several limitations that are important to acknowledge. First, the interpretability of the results depends heavily on the quality of the object detection model. If this model fails to accurately detect objects or introduces its own biases, the subsequent analysis can be skewed. Second, the computational complexity of analyzing co-occurrences can grow significantly with the number of detected concepts, especially in large-scale datasets or highly complex prompts. Another important direction is to explore active mitigation strategies that go beyond prompt revision. This could include integrating the framework with model training pipelines to intervene during the training process, helping to guide the model toward learning more equitable and unbiased representations.

### A.3 Experimental Setup

The experimental setup for each case study is detailed below.

### A.4 Additional Results and Details: Toy Example

Table 2 details the experimental setup for this case study. Figure 8 shows additional visual examples of detected concepts. We note how the concept jacket can clearly manifest in different styles and colors. This visualization supports our argument for this component of our framework. Figure 9 shows the concept stability for each of the actions. We can clearly see which concepts generally persist for a single action (holding age constant) and across actions. Moreover, we can also determine which output concepts are triggered by one or more of the input concepts.

**Table 2: Case study toy example: hyper-parameters and their corresponding values**
| Hyperparameter | Value |
| --- | --- |
| Object Detector | Florence 2 |
| Object Detector Mode | caption+grounding |
| Text-to-Image Model | ByteDance/SDXL-Lightning 4 step model |
| T2I Model Hyperparameters | inference steps= 4; guidance scale= 0 |
| Number of Images | 300 |
| Prompt Distribution | a uniform distribution over the set $\{\text{``A photo of a {[age]} person }\texttt{[action]}\text{''}\}$, where [age] takes value in $\{\text{young},\text{middle-aged},\text{old}\}$, and [action] takes value in $\{\text{jogging},\text{sprinting},\text{running}\}$. |

Figure: Figure 8: Additional examples of concepts localized by our framework.
Refer to caption: x8.png

Figure: Figure 9: Concept stability across actions. This is an effective way of capturing how a concept persists or is triggered by a specific input concept.
Refer to caption: x9.png

### A.5 Additional Results and Details: StableBias and TBYB

Tables 3 and 4 detail the experimental setting for StableBias and TBYB, respectively. For the StableBias case study, the list of adjectives and professions can
found in Table 4 of the original work’s supplementary material Luccioni et al. (). The professions for TBYB are listed in Table 1 of the original work’s supplementary material Vice et al. (). We note that the list of professions for the two experiments (TBYB and StableBias) are different. Second, the TBYB reports their results on a much larger set of prompts that do not include professions. We omit these as that is not the focus of this case study. Moreover, we observed that due to the prompt template, many images did not render with a detectable person. This is likely the reason for the discrepancy in results between our detections and theirs.

**Table 3: Case study Stable Bias: hyper-parameters and their corresponding values**
| Hyperparameter | Value |
| --- | --- |
| Object Detector | BLIP VQA |
| Object Detector Mode | direct VQA: “What is this person’s gender?” |
| Text-to-Image Model | Stable Diffusion v2.1 |
| T2I Model Hyperparameters | inference steps= 50; guidance scale= 7.5 |
| Number of Images | 30 |
| Prompt Distribution | a uniform distribution over the set $\{\text{``Portrait photo of a {[adjective]} {[profession]}}\}$ |
| Image Size | $768\times 768$ |

**Table 4: Case study TBYB: hyperparameters and their corresponding values**
| Hyperparameter | Value |
| --- | --- |
| Object Detector | BLIP VQA |
| Object Detector Mode | direct VQA: “What is this person’s gender?” |
| Text-to-Image Model | Stable Diffusion v2.1 |
| T2I Model Hyperparameters | inference steps= 50; guidance scale= 7.5 |
| Number of Images | 30 |
| Prompt Distribution | a uniform distribution over the set $\{\text{``A photo of a {[desc]} }\texttt{[profession]}\text{''}\}$, where [desc] takes value in $\{\text{who is a good},\text{who is a bad},\text{that looks like a}\}$ |
| Image Size | $768\times 768$ |

### A.6 Additional Results and Details: Disability Representation

Table 5 details the experimental setup for this case study. Figure 10 shows the concept stability over all disabilities and Figure 11 shows the top concepts by disability.

**Table 5: Case study disability representation: hyper-parameters and their corresponding values**
| Hyperparameter | Value |
| --- | --- |
| Object Detector | Florence 2 |
| Object Detector Mode | caption+grounding |
| Text-to-Image Model | Stable Diffusion v2.1 |
| T2I Model Hyperparameters | inference steps= 50; guidance scale= 7.5 |
| Number of Images Per Prompt | 100 |
| Prompt Distribution | {A person with [value] } where [value] $\in$ { a disability, bipolar disorder, a chronic illness, cerebral palsy, a limb difference, hearing loss} |
| Image Size | $768\times 768$ |

Figure: Figure 10: Concept stability for the disability representation case study.
Refer to caption: x10.png

Figure: Figure 11: Top concepts for all disabilities.
Refer to caption: x11.png

### A.7 Additional Results and Details Case Study: Pick-a-Pic

Table 6 shows the experimental setting for this case study.
We drew 10 random samples of size 1k each from the train split of the Pick-a-Pic dataset. Each image can be generated by a different text-to-image model. We refer the reader to the dataset for the details of each image’s generation hyper-parameters. Figure 13 shows the top detected concepts, along with confidence intervals. Figure 12 shows prompts within the Pick-a-Pic dataset that request child nudity, violence, slurs, and sexually explicit material. We have hidden most of them so as to not overwhelm the reader. The dataset can be accessed at [https://huggingface.co/datasets/yuvalkirstain/pickapic_v1](https://huggingface.co/datasets/yuvalkirstain/pickapic_v1).

**Table 6: Case study Pick-a-Pic: hyperparameters and their corresponding values**
| Hyperparameter | Value |
| --- | --- |
| Object Detector | Florence 2 |
| Object Detector Mode | dense region caption and detection |
| Text-to-Image Model | variable |
| T2I Model Hyperparameters | variable |
| Number of Images Per Prompt | variable (typically 1) |
| Prompt Distribution | $p(t)$ drawn from train split of Pick-a-Pic |

Warning: This section contains discussions of harmful content, including CSAM and NSFW material, which may be disturbing to some readers.

Figure: Figure 12: Harmful prompts in the Pick-a-Pic dataset.
Refer to caption: extracted/5905615/appendix_figures/RLHFPROMPTSPNG.png

Figure: Figure 13: The top detected concepts over 10 random samples of size 1k drawn from the Pick-a-Pic dataset.
Refer to caption: x12.png

### A.8 Additional Results and Details Case Study: StableImageNet

Table 7 details the experimental setting of this dataset. The dataset can be accessed at [https://www.kaggle.com/datasets/vitaliykinakh/stable-imagenet1k.](https://www.kaggle.com/datasets/vitaliykinakh/stable-imagenet1k.) We show all 100 images for the misaligned classes in StableImageNet in Figures 14-22.

**Table 7: Case study StableImageNet: hyperparameters and their corresponding values**
| Hyperparameter | Value |
| --- | --- |
| Object Detector | Florence 2 |
| Object Detector Mode | caption + grounding |
| Text-to-Image Model | Stable Diffusion v1.4 |
| T2I Model Hyperparameters | inference steps= 50; guidance scale= 7.5 |
| Number of Images Per Prompt | 100 |
| Prompt Distribution | a photo of class, realistic, high quality |
| Image Size | $512\times 512$ |

Figure: Figure 14:
Refer to caption: x13.png

Figure: Figure 15:
Refer to caption: x14.png

Figure: Figure 16:
Refer to caption: x15.png

Figure: Figure 17:
Refer to caption: x16.png

Figure: Figure 18:
Refer to caption: x17.png

Figure: Figure 19:
Refer to caption: x18.png

Figure: Figure 20:
Refer to caption: x19.png

Figure: Figure 21:
Refer to caption: x20.png

Figure: Figure 22:
Refer to caption: x21.png

### A.9 Interactive Tool

Figures 24 and 23 show screenshots of the Concept2Concept interactive tool.

Figure: Figure 23: Users can interactively inspect concept metrics, such as the persistence and fickleness scores, their stability, and co-occurrence with other concepts. The interactive tool also includes features for users to sort by certain metrics and filter by keywords.
Refer to caption: extracted/5905615/appendix_figures/conceptmetrics.png

Figure: Figure 24: The interactive tool also includes a concept inspection tab that allows users to search for concepts of interest. For each concept, the tool uses visual grounding models to localize how the concept is represented in different images. Localized concepts are displayed as thumbnails. The prompts that were used to generate the concept are also displayed in a table below. Users can search for and compare two concepts in this panel.
Refer to caption: extracted/5905615/appendix_figures/conceptinspection.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could you clarify the conceptual and mathematical connection between the marginal and conditional distributions in Equation 3 and the summarization metrics in Equations 4-6? An explanation of how these are linked would help in understanding the core framework.

- Since the marginal and conditional distributions are defined for continuous distributions, while the summarization metrics are based on discrete cases, could you provide a derivation or rationale that bridges these two? How do you address this foundational difference?

- You mention handling uncertainty from object detectors by sampling from a distribution of concepts, but the practical details of this approach are unclear. Could you elaborate on how this sampling is implemented and how effective it is in managing detection uncertainty?

- Given the similarities between concept frequency, concept stability, and concept co-occurrence and the metrics used in counterfactual explanations (e.g., validity, proximity, and diversity), could you discuss any connections or differences between your proposed metrics and those commonly used in counterfactual work?
soundness: 2
strengths: - This work addresses the important challenge of auditing text-to-image (T2I) models to assess their reliability, fairness, and bias.
- The authors introduce an interpretation of concept distributions, which forms the basis for their marginal and conditional distribution notations.
- Through various case studies—including bias analysis, disability representation, and model misalignment—the authors explore essential aspects of T2I model auditing.
summary: The authors propose Concept2Concept, a framework that characterizes the conditional distributions of vision-language models using interpretable concepts and metrics. This enables systematic auditing of both models and prompt datasets. Through case studies, they analyze various prompt distributions, including user-defined and real-world examples. Concept2Concept is also an open-source interactive visualization tool, making it accessible to non-technical users.
weaknesses: - The primary innovation of the paper lies in interpreting distributions over concepts, leading to the marginal and conditional distributions defined in Equation 3 and summarization metrics in Equations 4-6. However, the connection between these two sets of equations is not well-explained, making it difficult to understand how they are conceptually or mathematically linked.

- Although marginal and conditional distributions are defined for continuous distributions, the summarization metrics—concept frequency, concept stability, and concept co-occurrence—are framed in discrete terms. The authors do not provide a derivation or proof to clarify the connection between continuous and discrete cases, leaving this foundational aspect unclear.

- The authors mention addressing uncertainty from off-the-shelf object detectors by sampling from a distribution of concepts. However, they provide little information on the practical implementation of this approach, making it challenging to interpret how this sampling is achieved or how effective it is in managing uncertainty.

- To address the uncertainty introduced by the object detector, the authors need a more comprehensive analysis, particularly in handling cases where the detector may be over-confident or under-confident. A systematic empirical study to quantify and validate this uncertainty would greatly improve clarity and demonstrate how well the framework manages these corner cases.

- The metrics introduced by the authors—concept frequency, concept stability, and concept co-occurrence—resemble the validity, proximity, and diversity metrics used for counterfactual explanations as defined in [1]. However, there appears to be no discussion connecting these proposed metrics to previous work on counterfactual explanations.

[1] Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: -
soundness: 4
strengths: - The paper is well motivated and well written
- The proposed methods and metrics are simple and intuitive
- It is nice that the paper reproduces the findings of prior works using a different evaluation framework
- The paper has some important and worrying finding such as NSFW data in a human preferences dataset
- Open sourcing such a framework would be very useful for practitioners
summary: This work introduces a framework for auditing T2I models and datasets. The framework uses discriminative models to find objects or concepts in generated images. Using the proposed method, the findings of several works that explore the biases of T2I models can be reproduced. Furthermore,
weaknesses: - My main concern is that there is an existing work [1], that has not been acknowledged. It introduces a similar method that uses discriminative models to find co-occurences and biases in the generations T2I models, somewhat limiting the contributions of this work. Nonetheless, I think the other contributions and alaysis of this paper still have merit. 
- The method section has too much fluff and introduces too many concepts that are not used later on. For example, the concept co-occurence formula is never being used, and the concept stability is never explored in the main part of the paper.
- Figure 3: Methodologically, it is not clear what the prompt revision means. Are some concepts used as negative prompts?

[1] OpenBias: Open-set Bias Detection in Text-to-Image Generative Models, CVPR 2024

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the Weakness section.
soundness: 2
strengths: The paper provides insightful and valuable findings concerning harmful associations present in popular datasets, offering critical observations that can guide future research and model development. The topic itself is highly relevant, and the authors’ motivation is clearly articulated, underscoring the importance of addressing these issues.
summary: This paper systematically examines the associations between text prompts and generated image content in a human-understandable way. Its goal is to audit text-to-image models to ensure they produce desirable and task-appropriate images. The authors propose a framework called Concept2Concept that 1) extracts high-level concepts from generated images and 2) calculates concept distribution to uncover associations between prompts and generated images. Using this framework, the authors have identified potentially harmful associations in popular datasets like Pick-a-Pic.
weaknesses: One limitation of this paper is that the overall framework still relies on human examination and investigation, which may impact its scalability. 

The technical and theoretical contributions are fair but could be strengthened. Further elaboration on the differences from existing work would help to clarify the novelty of this framework. As it stands, the paper resembles more of a technical application report than a traditional academic paper. To demonstrate the framework’s utility, the authors present five case studies that effectively showcase its application; however, they lack cross-model analysis, which would add depth to the evaluation. Using concepts as tools to analyze bias in text-to-image (T2I) models holds strong potential, and it would be beneficial for the analysis to extend into other domains, such as ethnicity, offering a more comprehensive evaluation across multiple models and datasets. The current five case studies, though useful, may fall short of meeting the quality criteria expected in a top conference.

Besides, why there is no information about the used T2I model in the main paper? and in the appendix, there is no discussion about the choice of the model and no discussion about the different comparisons of different models 

Additionally, there are minor typos (e.g., in Line 236, Figure 2) that could benefit from correction.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: See weaknesses W1 to W3.
soundness: 3
strengths: - The paper is very well written and easy to follow. The authors handle the sensitive topics with the appropriate sense of responsibility.
- The selected applications of the method as well as the results are very interesting and I hope will spark a discussion in the communities using the respective datasets. 
- The presented framework increases evaluation robustness due to the three presented metrics, evaluating the relationship between prompts and generated images from different perspectives.
summary: The paper introduces Concept2Concept, a framework for auditing text-to-image models by analyzing the associations between generated images and prompts using interpretable concepts. It helps uncover biases, harmful content, and unexpected associations in models and datasets, demonstrated through various case studies.
weaknesses: **W1:** To my knowledge, there is no anonymous code repository provided with the paper, neither for the experiments nor the tool. As a result, I am unable to comment on the tool's usefulness. It would be beneficial if the experiments could be replicated either within the tool or through a dedicated repository to also validate the correctness of the results. 
The selection and robustness of the results regarding the T2I and VLM detector models are in my opinion just weakly addressed: 

**W2:** Only in the Appendix, it is revealed that the audited T2I model is Stable Diffusion 2.1. This information should be part of the main manuscript as the results of Study 4 only apply to this model architecture. It would be interesting how results would change for other model architectures, as especially closed-source models are strongly safety fine-tuned. If I understand correctly your framework is model-agnostic and could also be applied to closed-source models accessed via API calls. Did you perform any experiments with other model architectures? And if not please argue in the manuscript why you restrict Application 1 to this specific model architecture.

**W3:** While the authors acknowledge that the detection model introduces uncertainty in the extracted concepts (Line 132), they do not address how sensitive the application results are to the choice of the detector model. Could specific concepts be overlooked if a different grounding model is used? Additionally, how does the safety fine-tuning of the detection model potentially conflict with the task of identifying sensitive concepts, such as in CSAM?

I am open to raising my score if the identified weaknesses are either adequately addressed through revisions to the manuscript or convincingly argued to be non-issues.

Comments:
- There is a closed bracket missing in line 187 „P(c“
- There is a closed bracket too much in line 236 „Figure 2)“.

Meta-review: This paper introduces a framework to audit T2I models and prompt datasets, which analyzes association between text prompts and generated images with interpretable tools. Four knowledgeable reviewers went over this submission. The reviewers recognized that the paper tackles an important topic (8F6o, bRZR), that the paper is well written and well motivated (3MPH, eKz3), and that the proposed methods and metrics are simple and intuitive (3MPH).

The reviewers also raised concerns about:
1. The connection between marginal and conditional distributions, and connection between continuous and discrete cases, which was not well established (8F6o)
2. The little discussion on the models used to address uncertainty: no empirical study to validate the captured uncertainty, nor about the sensitivity of results to the choice of detector model (8F6o, eKz3)
3. Missing discussion on related work: counterfactual explanations literature (8F6o) and OpenBias (3MPH), questioning the technical and theoretical novelty of the paper (bRZR)
4. Experiments not fully convincing (bRZR, eKz3): Cross-model analysis would be beneficial; extension to other domains such as ethnicity would be beneficial; validation across multiple models and datasets would solidify the experimental validation. The selection of T2I model should be justified.
5. The scalability of the method relying on human examination (bRZR)
6. The novelty being unclear and the utility of the proposed framework difficult to assess (bRZR, eKz3)

The rebuttal and discussion partially addressed the reviewers concerns. For example, the rebuttal established the missing connections pointed out by the reviewers, discussed the differences with the missing related work, showcased with a few qualitative examples the output of OpenBias, analyzed the output of additional models, and argued the importance of having humans in-the-loop. During the reviewers' discussion period, one reviewer tended towards acceptance, one reviewer towards rejection, and the two remaining reviewers positioned themselves as borderline. The AC went over the paper, and all the discussion materials. After careful considerations, the AC agrees with some of the concerns raised by the reviewers. In particular, the technical or experimental novelty remains unclear. Although the topic covered is very important and relevant to the community, it remains unclear why one should choose the proposed framework over prior work. To address this concern, it might be worth doing an in-depth benchmarking of the proposed approach vs. previous works (e.g. OpenBias and other cited works in the paper). This benchmarking should ideally go beyond a few prompt qualitative analysis and arguments highlighting differences (a user study could be considered). Alternatively, the authors could focus on better highlighting what are the novel findings enabled by the framework. The current case studies replicate known findings in the community about Internet crawled datasets (such as CSAM and NSFW content) or synthetic data (biases, misalignment, and overall problematic content similar to real datasets). For these reasons, the AC recommends to reject and encourages the authors to consider the feedback received to improve future iterations of their work.

Decision: Reject