OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes

Authors: Sepehr Dehdashtian, Gautam Sreekumar, Vishnu Boddeti
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: stereotype measurement, responsible ai, trustworthy ai, interpretability, generative ai, text-to-image models, multimodal models

Abstract:
Images generated by text-to-image (T2I) models often exhibit visual biases and stereotypes of concepts such as culture and profession. Existing quantitative measures of stereotypes are based on statistical parity that does not align with the sociological definition of stereotypes and, therefore, incorrectly categorizes biases as stereotypes. Instead of oversimplifying stereotypes as biases, we propose a quantitative measure of stereotypes that aligns with its sociological definition. We then propose OASIS to measure the stereotypes in a generated dataset and understand their origins within the T2I model. OASIS includes two scores to measure stereotypes from a generated image dataset: **(M1)** Stereotype Score to measure the distributional violation of stereotypical attributes, and **(M2)** WALS to measure spectral variance in the images along a stereotypical attribute. OASIS
also includes two methods to understand the origins of stereotypes in T2I models: **(U1)** StOP to discover attributes that the T2I model internally associates with a given concept, and **(U2)** SPI to quantify the emergence of stereotypical attributes in the latent space of the T2I model during image generation. Despite the considerable progress in image fidelity, using OASIS, we conclude that newer T2I models such as FLUX.1 and SDv3 contain strong stereotypical predispositions about concepts and still generate images with widespread stereotypical attributes. Additionally, the quantity of stereotypes worsens for nationalities with lower Internet footprints.

Decision: accept (spotlight)
Reviewer scores: 6, 10, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730166947776

thread_type:
official_review

<thread_object>
creator: reviewer_qudx

<content>
summary:
This paper aims to measure and understand stereotypes in Text-to-Image models. 

* Measurement: it introduces metrics to quantify the presence of stereotypes in generated images by comparing them to real-world distributions, and also assess diversity along stereotypical attributes. 

* Track Origins:  The author attempts to identify hidden associations that T2I model makes with certain concepts and trace how stereotypical attributes emerge over the generation process

strengths:
1. The paper tackles the significant problem of stereotypes in T2I models, which has important implications for ethical AI use and social impact.
2. The paper is clearly written and organized. The visual presentations are good. 
3. It proposes StOP and SPI that try to understand the origin of the stereotypes, which may offer further insights.

weaknesses:
1. The authors emphasize that their method is innovative in focusing specifically on stereotypes rather than just bias in the abstract. However, this distinction is not fully explained in the main content, and the methodology still seems quite similar to bias detection to me.

2. The way they identify stereotypes is by prompting LLMs to generate stereotypical attributes for a given concept, but this method is already common in other papers. For example, [1] prompts LLMs to generate stereotypical / anti-stereotypical sentences for certain concepts and then identifies the stereotypical / anti-stereotypical attributes from those sentences.

3. Although the paper includes many formulas, I found that the core idea is actually quite simple, and the formulas only make it harder to follow. If I understand correctly, the main process is: identify a concept, use LLMs to generate stereotypical attributes, generate images, measure the appearance of these attributes, and compare to real-world distributions & similarities across groups. This could be communicated in just a few sentences, which would make the approach clearer.

4. I am concerned with the idea of comparing to real-world distributions. For example, men are dominant in many professions even today. In society, an educated person can be aware of stereotypes to some extent, but telling the model that it’s okay to learn patterns such as “men are more likely to be doctors” because it reflects real-world distributions doesn’t seem fully convincing to me.

[1] chatgpt based data augmentation for improved parameter-efficient debiasing of llms

questions:
I would appreciate it if the authors addressed the questions mentioned in the weaknesses section, and I will raise the score if their responses are convincing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730762376329

thread_type:
official_review

<thread_object>
creator: reviewer_amaa

<content>
summary:
The authors present OASIS, a set of 4 metrics/methods to analyze the presence of stereotypes in T2I models, how those stereotypes form, and the connections between them. The authors use 3 popular models for this analysis. In the paper, OASIS shows the performance of these models on various prompts (e.g. "Show me a picture of a Mexican person") and how certain stereotypes are strongly associated with the prompt, at rates greater than those stereotypes in the real world. The paper shows weaknesses in models in their attempts to prevent stereotypes such as revealing stereotypes when prompts or combined or in how they produce low-variance along the stereotype's dimension. The paper also shows how certain stereotypes are clustered together and at what stage in the image generation process these models form (which is almost always in the early stages). 

In all, OASIS provides a useful toolkit for understanding stereotypes in T2I models and highlights the importance for further work in this very important area.

strengths:
The paper was very well-written, and it was quite easy to follow. I greatly appreciated how the terminology was constantly linked to concrete examples. The general formatting was also intuitive. 

The paper did a good job of thinking about the broader context of this work, thinking a bit more critically about what phrases like biases actually mean and what we should think about when studying these models. Also incorporating real-world census data was a neat idea. 

I liked the results about how the biases are being generated early in the image-generation process.

The formulation for the stereotype score was quite intuitive and seems easy enough to compute at scale when provided with accurate descriptors from these LLMs.

weaknesses:
I feel that overall, there was a lack of bench-marking for the methods presented in this paper beyond providing a few visual examples (which are still useful!). There were a few parts of the paper where I felt this significantly 

1. Computing the stereotype score: How accurate is the probability computation? The intuition of the metrics' formulation is clear, but there is no validation that the score provides accurate results in practice. As the paper mentions, CLIP is a biased VLM on its own, so its not fair to assume that this score will work perfectly. A smaller dataset demonstrating the accuracy of the score could be quite useful. 

2. Spectral variety along a stereotype. There was a nice description of each method for measuring $\delta A$ in the appendix, but its not clear which one is used in the paper and which one is more accurate in practice. 

3. The biases of models like CLIP are well-documented (e.g. Birhana, Prabhu, & Kahembwe, 2021). If CLIP is being used extensively to identify additional biases, I think it makes sense to add more validation about how said biases may or may not be affecting the performance of OASIS. 

4. I understand the SPI claims about how certain stereotypes are intrinsically associated with a concept. However, the paper only provides a handful of examples of where this is occurring. I would appreciate a larger-scale analysis that includes something like 100 contexts generated in some way (i.e. top 100 most common ethnicities), and the average step at which each stereotype attribute occurs. This would be much more convincing than what could be cherry-picked examples. 

I think that the breadth of the experiments/validation were also a bit limited. I understand that gaining access to real-world data for a ground truth distribution is not always feasible, but consistently focusing on broad nationalities feels like it lacks nuance. 

Another broad critique is that certain figures were unclear. I'll bring up those missing details in the following sections

questions:
Is this definition of stereotypes correct? Does it not implicitly push the stereotypes in society that cause imbalances to exist? Relying on census data to define an expected rate of a given attribute is perpetuating the stereotypes that exist in today's society. For example, using the example provided in the paper, if there are less women than male doctors in the world, it doesn't necessarily mean that the model is creating a 'stereotype' by generating an equal number of woman and male doctor images. 

For experimental details, what is each point in Figure 3? What does each point correspond to in terms of attribute? 

In Figure 5, there is a claim that "We observe that some models have lower stereotypes at the cost of lower attribute variance". This suggests a trade-off can exist between these two, but there is not much correlation observed for any model across nationalities. I agree that both metrics are important to measure, but it would be great to get more clarification on if there indeed is a trade-off that occurs often. 

I would appreciate any additional discussion about the accuracy of the metrics as well and if they work as intended.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730766735487

thread_type:
official_review

<thread_object>
creator: reviewer_hz7r

<content>
summary:
This paper introduces OASIS, a toolbox for auditing visual stereotypes in text-to-image (T2I) models. OASIS offers novel methods for measuring stereotypes and analyzing their origins within T2I models. OASIS stereotype scores reveal that newer T2I models produce significant stereotypical biases, potentially limiting their applicability.

strengths:
- The paper is well-written, clear, and easy to follow.
- The paper addresses a complex and socially significant problem by evaluating biases and stereotypes in T2I models. Additionally, it provides new methods to trace the origins of these stereotypes in T2I models, offering valuable insights to help mitigate these biases.
- The findings of this work highlight the need for greater inclusion and representation of underrepresented communities, particularly those from regions with limited internet presence, in the development of T2I models.

weaknesses:
- This work focuses exclusively on generated images based on nationality, which raises several concerns. For instance, in racially diverse countries like the United States, how can an image indicate that someone is a U.S. citizen without relying on stereotypes and social biases? Additionally, in regions where people from neighboring countries may share similar facial features or cultures (such as North and South Korea), how useful can OASIS be in distinguishing these cases?
- How can OASIS be useful beyond nationality domains? It is not clear in the paper how OASIS can be extended to evaluate other biases in T2I models, such as gender biases, racism, etc.
- Overall, this is a solid piece of work; however, a deeper discussion of its limitations and the specific types of stereotypes for which OASIS may or may not be effective would be beneficial.

questions:
Please, refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730768549374

thread_type:
official_review

<thread_object>
creator: reviewer_n4np

<content>
summary:
The proposed work has a novel contribution in an existing research gap. Defining, separating, and quantifying a measure for stereotypes and biases in Image generation models (from text prompt) is a strong case of Ethical Machine Learning. The authors built a foundation of definitions that allowed them to measure estimates of data and image stereotypes. Then they apply that on well-known Text to Image models to uncover the stereotypes risks of applying those models according to the proposed measures/estimates.

strengths:
Addressing a research gap
Clear mythology
Application to existing models

weaknesses:
It would have been nice if the writeup included a detailed case study of qualitative analysis of cases of stereotypes

questions:
- In the abstract the authors said "incorrectly categorizes biases as stereotypes", do you mean the opposite (i.e. "incorrectly categorizes stereotypes as biases")?
- How about using other measures of visual variations of the generated images (e.g. a similarity measure based on feature extraction (e.g. HoG))?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730863700488

thread_type:
official_review

<thread_object>
creator: reviewer_efbv

<content>
summary:
In this paper, the authors propose multiple new metrics to understand stereotyping in T2I models.  In particular, they propose a metric for stereotyping that compares the prevalence of a given attribute to a societally-anchored prior probability.  They go on to present numerous other analyses for diversity of images and measuring how differnt stereotypical concepts relate to demographics in latent space.  They show that generally newer models exhibit less stereotyping and that stereotypes arise mostly in the earlier stages of optimization.

strengths:
1. Measuring sterotyping is an important and challenging problem for generative models.

2. The diversity of metrics seem quite interesting to get a more complete picture of how bias and stereotypes arise in these models.

3. The paper is fairly thorough in its testing.

weaknesses:
1. The biggest weakness in my opinion is that the paper is quite hard to follow for many of the metrics. I believe this is due to making the description of the metrics unnecessarily complex, and I would encourage the authors to try to describe all of the concepts more simply.  In addition to making the paper harder to understand, it also obfuscates the insights and contributions.

2. One of the papers claimed main contributions is measuring stereotyping relative to societal priors on the distribution.  However, as the authors state in the appendix, the prior is often unknown and thus falling back to 50% is a reasonable choice.  That said, this seems to limit the contribution over prior work.

3. The paper does not validate their work with human judgements

4. More minor: the paper only presents the evaluations but do not work on mitigations.

5. The intro could be more clear and sensitive in how it characterizes what are stereotypes, why they matter, and how they define which stereotypes are important.

questions:
S3.2 - WALS is quite confusingly described.  In particular (6) doesn't really make sense as an equation.  If E(D) is the SVD that is just estimating the dataset? Do you mean the rank or something else?  Even the motivation doesn't make much sense to me of why this is necessary.

"Existing bias definitions are not applicable for some attributes studied in Tab. 1. For example, in the case of wearing turban, a T2I model needs to depict 50% of the images of Iranian with turban to be unbiased according to Eq. (1), which incorrectly represents Iranian people." - This seems like an odd remark in that this is largely what is done here just conditioned on gender.  What do you see as the primary difference here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731880216833

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response (Part 1)

comment:
We thank all the reviewers for their valuable feedback. Our primary goal in this work was to evaluate stereotypes in T2I models so that they may achieve wider use in society, and we are happy that [**EfBv, QudX**] found the problem that we are addressing socially significant and important. As pointed out by [**Hz7R**], we hope that our work leads to more engagement from the research community to build an inclusive AI.

We are also encouraged that the reviewers found our paper well-written [**Hz7R, AMAa**], clear [**Hz7R, QudX**], easy to follow [**Hz7R, AMAa**], interesting and thorough [**EfBv**], with scalable formulation [**AMAa**] that provides valuable insights [**Hz7R, QudX**].

We have incorporated the feedback in the paper and updated it with our changes, which are highlighted in blue. We first summarize the primary concerns of the reviewers and how we addressed them:

- Reviewers **AMAa** and **QudX** asked for more clarity on the definition of the stereotype used in the paper. To address that, we compared bias definitions and our stereotype definition in common response part 2 (”Difference between existing bias definitions and our stereotype definition”), providing mathematical definitions and numerical examples. Moreover, we also justified the use of true distribution of the data in the definition of the Stereotype Score in common response part 2 (”On the ethics of using census data to measure stereotype”).
- Reviewers **EfBv** and **AMAa** asked for further accuracy analysis of the CLIP model used in the paper and validation of its results with human judgments. To address this concern, we added an analysis of the CLIP model's performance on a small manually annotated generated dataset and the CelebA dataset, a large dataset with annotations. This analysis is added to Sec. A.5 of the paper.
- Reviewer **AMAa** requested more samples on SPI. To address this, we generated 100 samples and averaged their SPI for a more robust result. This result further confirmed our observation that stereotypical attributes arose early in the image-generation process. This result is added to Sec. A.3. However, as we mention in our individual response to **AMAa**, the purpose of SPI is to provide a sample-wise stereotype attribute measurement that can be used by the mitigation methods in contrast to other metrics in OASIS that are applied to the distribution of generated data.
- Reviewers **EfBv** and **n4np** requested a qualitative analysis of the generated data. To address that we added a qualitative human analysis of the three T2I models on the generated images of four nationalities in Sec. A.7.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732375546847

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-Up on Rebuttal Discussion

comment:
Dear Reviewers,

Thank you for your valuable feedback on our submission. We have carefully addressed your comments and concerns in our rebuttal and the main paper, which were submitted six days ago.

As the end of the discussion period approaches, we kindly encourage you to share any remaining questions or thoughts to further clarify our responses. If our rebuttal has already addressed your concerns, we would greatly appreciate it if you could consider revisiting or updating your evaluation scores.

We deeply value your time and effort in reviewing our work and look forward to your thoughts.

Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
Text-to-image (T2I) models are widespread and tend to exhibit behavior that is negative towards certain communities. This is well documented in the T2I literature. This paper does a nice job separating "stereotyping" from other forms of bias and focusing on measuring it and identifying its origin in T2I models. I agree with the reviewers' assessment of this work and recommend that it be highlighted in the conference with the hope this contributes towards the design of more inclusive T2I models.