OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes
Authors: Sepehr Dehdashtian
Gautam Sreekumar
Vishnu Boddeti
Venue: ICLR 2025 Spotlight (2025)
Keywords: Stereotype Measurement
Responsible AI
Trustworthy AI
Interpretability
Generative AI
Text-to-Image Models
Multimodal Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a toolbox to quantify stereotypes in Text-to-Image models

Abstract: Images generated by text-to-image (T2I) models often exhibit visual biases and stereotypes of concepts such as culture and profession. Existing quantitative measures of stereotypes are based on statistical parity that does not align with the sociological definition of stereotypes and, therefore, incorrectly categorizes biases as stereotypes. Instead of oversimplifying stereotypes as biases, we propose a quantitative measure of stereotypes that aligns with its sociological definition. We then propose OASIS to measure the stereotypes in a generated dataset and understand their origins within the T2I model. OASIS includes two scores to measure stereotypes from a generated image dataset: **(M1)** Stereotype Score to measure the distributional violation of stereotypical attributes, and **(M2)** WALS to measure spectral variance in the images along a stereotypical attribute. OASIS
also includes two methods to understand the origins of stereotypes in T2I models: **(U1)** StOP to discover attributes that the T2I model internally associates with a given concept, and **(U2)** SPI to quantify the emergence of stereotypical attributes in the latent space of the T2I model during image generation. Despite the considerable progress in image fidelity, using OASIS, we conclude that newer T2I models such as FLUX.1 and SDv3 contain strong stereotypical predispositions about concepts and still generate images with widespread stereotypical attributes. Additionally, the quantity of stereotypes worsens for nationalities with lower Internet footprints.

Published as a conference paper at ICLR 2025
T OASIS UNCOVERS: HIGH-QUALITY T2I MODELS, SAME OLD STEREOTYPES
Sepehr Dehdashtian Gautam Sreekumar Michigan State University (sepehr, sreekuml, vishnu) @msu.
Vishnu Naresh Boddeti
ABSTRACT
Images generated by text-to-image (T2I) models often exhibit visual biases and stereotypes of concepts such as culture and profession. Existing quantitative measures of stereotypes are based on statistical parity that does not align with the sociological defiition of stereotypes and, therefore, incorrectly categorizes biases as stereotypes. Instead of oversimplifying stereotypes as biases, we propose quantitative measure of stereotypes that aligns with its sociological definition. We then propose OASIS to measure the stereotypes in a generated dataset and understand their origins within T2I models. OASIS includes two scores measure stereotypes from a generated image dataset: (MI) Stereotype Score to measure the distributional violation of stereotypical attributes, and (M2) WALS to measure spectral variance in the images along a stereotypical attribute. OASIS also includes methods understand the origins of stereotypes in T21 models: (U1) StOP discover attributes that the T2I model internally associates with given concept, and (U2) SPI to quantify the emergence of stereotypical attributes in the latent space of the T2I model during image generation. Despite the considerable progress in image fidelity, using OASIS, we uncover that newer T2I models such as FLUX.1 and SDv3 contain strong stereotypical predispositions about concepts and still generate images with widespread stereotypical attributes. Additionally, the severity of stereotypes worsens for nationalities with lower Internet footprints. Content warning: This paper contains images with potentially offensive stereotypes.
FLUX SDv3
SDv2 Man Turban
Arriibtt -Grader Agy Baunl
FLUX
Old Cloibs Reard
0.15
SDv3
0.05
-2.05
SDv2
10
-0.5
22
2.3
2.4
2.5
2.6
2.7
15
WALS
Time
(a)
(b)
(c)
Figure 1: Measuring Stereotypes in Text-to-Image Models. (a) The images generated by T2I mod- els corresponding to the prompt "A photo of an Iranian person" overwhelmingly contain stereotypical tropes such as beard, turban, and religious attire although the prompt devoid of this information. (b) The proposed toolbox OASIS includes complementary methods for quantifying stereotypes. Stereotype Score measures the rereeetprsnnntnnnnn of stereotypical attributes while WALS measures the variance of images along these attributes. (c) SPI quantifies the emergence of stereotypes from the latent space of these models and helps understand the origin of stereotypes within T2I model.
1 INTRODUCTION
In a sociological context, stereotypes are generalized beliefs or assumptions about a particular group of people, things, or categories (Bordalo et al. 2016). These stereotypes widespread

---

Published as a conference paper at ICLR 2025
in the images generated by text-to-image (T2I) models when the input textual prompts contain concepts such as culture and profession. For instance, consider the images in Fig. [ generated by FLUX.1 (BlackForestLabs) 2024), SDv3 (Esser et al. 2024), and SDv2 (Rombach al.. 2022) for the prompt "A photo of a/an <nationality> person" There are clear portrayals of ethnic stereotypes in attributes such clothing, skin tone, and facial features across different nationalities, despite no references to such attributes in the prompt. For example, the model consistently depicts an Iranian person as middle-aged or senior with a long beard, wearing a turban, and dressed in religious attire, reinforcing harmful stereotypical representations about people with Iranian nationality. Besides being demographically incorrect, stereotypical biases in these models can lead to broader harm. For instance, when the biased outputs of these models are shared online, they can perpetuate damaging stereotypes about marginalized groups, further exacerbating societal polarization on issues such as beauty standards, ethnicity, and disability representation (Zhang et al. 2023 D'Incà et al. 2024 Vázquez& & Garrido-Merchán 2024).
Existing methods detect stereotypes primarily rely on feedback from human annotators, which both subjective and resource-intensive. It also becomes impractical in the era of the fast-paced development of generative models and changing regulations. Additionally, the feedback from human annotators may be affected by their personal and political leanings (Sap et al.) [022] Geva et al. 2019), e,g., annotation of continuous-valued attributes such nose size and skin tone. Human annotation can also affect the users' privacy by exposing the generated images to external evaluators.
In contrast, automated methods classifiers to detect stereotypes (Cho et al. 2023 Friedrich et al.) 2023 Zhang et al. 2023), overcoming several drawbacks of human annotators. However, these methods incorrectly rely on a general bias metric, i.e.. statistical parity, as a stereotype measure that fails to account for the directionality in the sociological definition of stereotypes. For example, consider a biased T2I model that generates images of predominantly female doctors. Existing works categorize this bias as a stereotype, although the generally known gender stereotype associated with the concept of doctor is that all doctors are male (Vogel 2019).
This paper presents new mathematical definition of stereotypes that aligns with the sociological definition. Building upon this formulation, we propose Open-set Assessment of Stereotypes in Image generative models (OASIS), novel toolbox for quantifying stereotypes and understanding their origins T2I models, addressing the limitations of prior studies. OASIS provides two metrics for measuring stereotypes based on the distribution and spectrum of the generated data in a feature space. OASIS comprises two additional methods to (1) discover the stereotypical attributes that a T2I model internally associates with concept and (2) quantify the emergence of stereotypical attributes in the latent space of T2I models. Our work is an important step toward automated auditing and mitigating stereotypical content in T2I models during development and deployment.
Contributions.
1. We formulate the quantitative measurement of stereotypes in text-to-image (T2I) models as the over-representation of attributes that are generally associated with concept, aligning with the sociological definition of stereotypes (Def. D
2. Using the new formulation, we propose OASIS, a toolbox for auditing visual stereotypes in T21 models. OASIS has two methods measure stereotypes: (M1) Stereotype Score (V) quan- tify the presence of a stereotype ($ 1. and (M2) Weighted Alignment Score (WALS) to mea- sure the spectral variety of the generated images along a given stereotypical attribute ($ (.2).
3. In addition to methods to measure stereotypes, OASIS includes two methods for understanding the origins of stereotypes in T21 models: (U1) Stereotypes from Optimized Prompts (StOP) that discover stereotypical attributes that a T2I model internally associates with a given concept ($ 3.3). and (U2) Stereotype Propagation Index (SPI) that quantifies the emergence of stereotypical attributes during the generation steps of T2I model ($ 3.4).
4. Analyzing the scores from OASIS, we conclude that newer T2I models such as FLUX.1 and SDv3 contain strong stereotypical predispositions regarding the concepts and generate significant stereotypical biases that can limit their potential applications. Moreover, these stereotypes worsen for under-represented groups with lower Internet footprints. When these models demonstrated lower stereotypes, they did so at the cost of lower attribute variance.
2

---

Published as a conference paper at ICLR 2025
2 STEREOTYPES IN T21 MODELS: BACKGROUND AND PROBLEM DEFINITION
Despite the growing awareness and efforts from the research community, stereotype measurement in T2I models faces challenges on two major fronts: Defining candidate set of stereotypes for each concept is challenging due to stereotypes' subjective and evolving nature. This difficulty is further compounded by the personal biases of the candidate set designer, particularly for under-represented and desser-known communities. Challenges regarding candidate set definition are sometimes ad- dressed with the help of large language models (LLMM)) to generate a set of potential stereotypical
attributes for given concept (Jha et al. 2023 D'Incà et al. 2024). 2) Developing analytical meth- ods to measure known stereotypes requires mathematical definitions and numerical quantification of visual stereotypes. Existing mathematical definitions of stereotypes do not align with the sociological definition of stereotypes. This work focuses on the analytical challenges in developing methods to measure stereotypes and understand their origins in T2I models.
Definitions and Notations. We use the term concept to refer to groups of people, things, or categories related to which stereotypes may exist, e.g., culture and profession. We denote con- cepts using& a random variable C. If C denotes the concept of nationality, then it takes values from {Iranian, American, ...). For given concept C C, define the set of potential stereo- typical attributes as A. C A. where A is the set of all possible attributes. Every attribute A; Ae is binary random variable that assumes values from {ar 0, }. where of and oj in- dicate the presence and the absence of A,, respectively. For example, if с = Iranian, then A.- {beard, religious symbols, hijab, ... }. We denote concepts and stereotypes different colors.
Problem Setting, The objective is to measure stereotypes in T2I model M from the set Ac. that purportedly exists, related to a concept c. For example, in Fig. I could correspond to Mexican nationality and A. could include sombrero and serape. The distribution of images / generated by M conditioned on text prompt T(c) PT T(c)). The notation ((c) indicates that the text prompt contains information about only the concept and not of any stereotype. To detect the presence of E Ac, we are provided with a dataset D of N samples generated by A from text prompts T(c) where D = {4 I ~ PM[[] | T(c)), 1,..,, NN.
3 OASIS: A STEREOTYPE MEASUREMENT AND UNDERSTANDING TOOLBOX
Motivation. The measurement of a stereotype related to concept is subjective without a formally defined metric. Prior works have not considered the differences between stereotypes and biases and have employed bias definitions stereotype metrics. The dataset D is considered unbiased w.r.t. an attribute A E A. if
D ~ U
where U is uniform distribution. However, not all biases necessarily stereotypes.
Quantitative Measure of Stereotype. Stereotypes are generalized beliefs or assumptions about particular group of people, things, or categories (Bordalo et al., 2016). "Generalization" in this definition can be translated to statistical terms as exceeding the true distribution of the data for a concept e in the real world. As an example, if D contains generated images of doctors in the US and male, the distribution of male in D must match with its true distribution in the real world PM (*AA i... P(A = male DD Doctor) Moreover, stereotypes are directional, which means male having a smaller likelihood of doctors in the US compared to the real-world distribution is not considered a stereotype. although it bias. Accounting for this directionality, we say a dataset D contains stereotype A w.r.t. cif
the stereotype of interest
P (A- male C Doctor).
Definition Stereotype
max(0, P(A D, - PP(
where Ç is margin for the violation from the real-world distribution. Note that this definition of stereotype differs from the definition of bias in Eq. (T). Our definition (i) compares the distribution of the generated dataset against its true societal distribution, and (ii) concerns the violation only along the direction of the attribute prone to be stereotype.
'' (A C') be obtained from census and online sources. For details, refer
A.1.3
3

---

Published as a conference paper at ICLR 2025
OASIS
A toolhox for stereotype measurement and understanding
Prompt T(C Meticur) "A photo of a Mexicon person"
41Z, A.) 3T
SiOP(Z, () ( 3.3
Male
LLM
Sombrero
Measures stereotypes A. in dataset D from VLM features 2.
tributes that M internally associates with concept C.
Discovers stereotypical at-
Sterestype
Latent Space (P)
Candidates (A.)
WALSIZ. A.)
M2
SPKFF A.)
12
Mcasures spectral variety along stereotypes pes A. in the dataset D using features Z.
Quantifies the emergence
stereatypes A. during image generation.
Generated Images (D)
A
T21
CLIP
Model (.M)
D
Feature Space (2) Clusters (C)
Figure An overview OASIS. Given a text prompt, a set of images is generated using T21 model M. Simultaneously, a stereotype candidate set is created using an LLM. OASIS then performs four quantitative analyses: (MI) Stereotype Score V to measure stereotypes based on Def. [ (M2) WALS to assess the spectral variance of D w.r.t. a stereotypical attribute, (UI) StOP to discover the stereotypical attributes that M associates with given concept, and (U2) SPI to quantify the emergence of stereotypical attributes in the latent space of M during image generation.
Finding Stereotype Candidates. To find open-set stereotype candidates for a concept c, we follow the approach by II'Inca al. 2024). Let MILM be large language model (LLM). By providing prompt (c) and a template instruction 7 we have
MILM (T(c), { (A di d 1. .nA.
where d and d, are the descriptions for the presence and the absence of A,, respectively. Subse- quently, Ar = {AI,...: ) Ana.)......)) For example, let (c)) be "A photo of a doctor" and A, be male Here, di is "A photo of a man" and d. is "A photo of woman".
Based these definitions, we propose OASIS, toolbox to measure stereotypes in M from distri- butional and spectral perspectives and to understand the origin of these stereotypical attributes in the T2I model. Given concept c, OASIS takes in as input the dataset 'D corresponding T(c), the latent space F' from M at every time step of image generation, and the candidate set of stereotypes Ag. OASIS first extracts features Z from the images using pre-trained vision-language model (VLM) such as CLIP (Radford et al., 202T). Using these inputs, OASIS calculates the metrics we define below. Fig. illustrates an overview of the proposed toolbox OASIS.
prompt
3.1 STEREOTYPE SCORE: MEASURING STEREOTYPES T2I MODELS
Following Def. stereotype (1) of A E A. for given dataset D and concept is defined V (A D,C) = max(0, P(A D, D,C) (A
(3)
where P*(A C) is the real-world density of A in concept c. Using Bayes' rule, P(A D. D,C) is,
(,A) P(A 1,C)
I P(A= 4,C)
P(A=at (D,)
(4)
I. P(A 4, C)
We obtain P(A 4. C) by means of attribute classifiers. Instead of training attribute-specific classifiers, zero-shot predictor such CLIP (Radford al. (021) can be used, where P(A 4, C) is obtained using softmax over cosine similarity scores of image features and text descriptions for a and a However, these cosine similarity scores are often numerically close (Liang et al. 2022
*Refer to A.1. for more details on the template instruction.
"The number of categories for gender restricted by the annotations of the existing datasets.
4

---

Published as a conference paper at ICLR 2025
requiring additional temperature parameter to obtain accurate probability measures. Therefore, in such cases, we estimate P(A 4, C)) as
P(A at 0) - (
> (Z1, Za -coo))
(5)
where (I, cos is the cosine similarity between and y. is the indicator function, and Z₁. 2 and Z are features of image, d't, and d from Eq. (2) respectively.
3.2 WALS: MEASURING SPECTRAL VARIETY ALONG STEREOTYPE
Motivation. Since v/ measures stereotypes from distributional perspective, it is possible for a dataset D to appear free of stereotypes at the cost of reduced variance along the stereotypical attribute. For example, in the case of measuring male stereotype among images of doctors in the US, a T2I model may repeatedly generate images of the same male and female doctors and yet satisfy Def. T Moreover, it is challenging measure variety through human inspection due to its subjective nature, and therefore, a quantitative method to inspect variance is beneficial. To encapsulate these requirements, we propose a metric named Weighted Alignment Score (WALS) that measures the spectral alignment of the data D with a given attribute A.
Method. To quantify the changes in a given stereotypical attribute across images generated by T2I model, WALS involves two steps: 1) Estimating the structure of data D through the singular value decomposition of the CLIP image features & (D) i... (D) = UEVT where & is the image encoder of the CLIP model, 2) Finding the direction of change in A, denoted by 5A, using one of the following two approaches: (i) estimating 8A as the difference between the text embeddings of pair of positive and negative descriptions, d+ and d
8A ET (d+) ET ET (d),
(6)
where E9 is the text encoder the CLIP model, or (ii) estimating 8A as the direction of maximum change along A in the image embedding space of a set of A-aware images corresponding to positive (a and negative (a categories of A, using supervised principal component analysis (Barshan all 2011). Detailed descriptions and derivations can be found in $ A.2 These approaches make different assumptions, and one of these can be chosen based on the available information and the availability of computational resources. The first approach assumes alignment between text and image embeddings in the CLIP model, and 5.A is more accurate when the embeddings of these modalities are more aligned. The second approach estimates öA accurately at the cost of increased computation due to generating two A-aware image sets and calculating the kernel matrices. Moreover, the first approach captures linear dependency. while the second one can be adopted for both linear and non-linear dependencies. We use the former approach in our experiments. Using the two components explained above, WAL.S measures the data variance along 8A in the feature space, as
WALS(A) == Σ 0 -8A
Σ/
where o; is tth singular value of DD and u; is the associated singular vector.
3.3 STOP: DISCOVERING INTERNALLY ASSOCIATED STEREOTYPICAL ATTRIBUTES
Motivation. Stereotypes might occur due to T2I models internally associating concept with stereotypical attributes. This means that the prompts with these attributes can equivalently generate images corresponding to c. However, these attributes may not be present in A.. Therefore, qualitative methods are devised to discover these open-set attributes, which we refer to as M-attributes.
Method. Since the distribution of stereotypical attributes is not uniform within D, we have to find M-attributes for individual clusters of images that share common stereotypes. Given an image spectral clustering (Von Luxburg 2007) on CLIP features
dataset corresponding to concept c, extracted from these images and visually identify clusters that share stereotypes. To discover M- attributes for given cluster with prominent stereotypes, following ZeroCap (Tewel et al. 2022). The solution to this optimization problem is a sequence that maximizes its mean CLIP score with the images in the chosen cluster. Formally, with cluster
design sequence optimization problem,
5

---

Published as a conference paper at ICLR 2025
D' =
In siisi n) containing images, the objective is (I,.... =I
π Σ (2)))
= arg max
(8)
where Er and Er are image and text encoders from CLIP. Following ZeroCap, is produced by an LLM that is conditioned on the starting sequence "This is photo of of The subsequent optimization problem reduces iteratively finding 2-token sequence that maximizes the mean CLIP score in Eq. ( using beam search. Since a single prompt s* may not contain diverse stereotypical attributes, we output the top-K K prompts in the final iterative step of the optimization in Eq.
3.4 SPI: UNDERSTANDING THE EMERGENCE OF STEREOTYPES IN T21 MODELS
Motivation. In addition to measuring stereotypes from generated images, it is important to quantify the aggregation of stereotypical attributes during image generation to design successful mitigation strategies. To that end, propose stereotype propagation index (SPI) to quantify the addition of stereotypical attributes in the latent space of M at each time step of image generation.
Method. In the flow-based models such SDv3, the latent in each inference step updated as Tt-1 = xy + (1te 4, (t) where Tú and T++1 are the latent representation in the current and next step, respectively, ve(.) ) the velocity of Ie for time step f. and 6 = c t, Gp) is the noise predicted in time step & for latent by the noise predictor ée, where cp is the conditioning text prompt. The velocity decides the attributes of the generated image based on the provided text prompt. Our goal to measure the amount of a stereotypical attribute added during each step of image generation, which requires knowing the direction of change in the attribute (6.A) in the latent space of the T2I model.
To find 5A in the latent space of the T2I model, we first predict two A-aware noises that correspond to positive d+ and negative d' descriptions of A as
= (x
E = e (xr,t,
Using these predicted noises, we find the velocities that model could have in this step the text prompt A-aware, i.e., ve 1, e,, f and ve (x,,,,,, Here, the direction of change in the attribute can be calculated as
8A uy (z,,t, e t,
(10)
We define SPI as the cosine similarity between the velocity time step and the direction of change the given attribute
SPI = (ve(x), t, t,), e.),
(11)
positive SPI means the stereotypical attribute is being added to the image in time step f, and a negative SPI means that the image losing the stereotypical attribute A.
4 WHAT DOES OASIS UNCOVER ABOUT STEREOTYPES IN T2I MODELS?
We apply OASIS three open- weight T2I models SDv2 (Rombach al. 2022), SDv3 (Esse et all 2024), and FLUX.. 1 (BleskForestl abs) 2024). In the first step, as illustrate in Fig. 2 we generate dataset of 2000 images of people from each of the nationalities and with each T2I model. In the next step. for each nationality, a candidate set for stereotypes and their descriptions is generated according Eq. (2). We used ChatGPT ol l--rreview (OpenAl 2024b) and ChatGPT 4o (OpenAI
2024a) as MLLM in Eq. (2). Implementation details are mentioned in
4.1 LOWER, YET SIGNIFICANT STEREOTYPES IN NEWER T21 MODELS
We use CLIP VIT-G-14 14 from OpenCLIP (Iharcoo al.) 202T) trained on LATON2B (Schuhmann et al.) 2022) to estimate P(A] D, C). Tab. compares the T2I models in terms of their stereotype scores defined in $ 3.1 from the images generated by these models corresponding to three nationalities Iranian, Indian, and Mexican. Although the fidelity of the generated images has improved dramatically
We use Laama 3.1 Dubey al.
2024

---

Published as a conference paper at ICLR 2025
Table 1: Stereotype Score. Comparison of three T2I models, SDv2, SDv3, and FLUX.1 I on stereotype
score in three nationalities. P* (A C) is the true density of the attribute obtained from real-world
statisties (details provided in $ A.1.3 P(A | D, C) is the density of the attribute the generated dataset, and (A D, C) is the stereotype score. All values are in %.
Stereotype Candidate
(*(A (C)
P(A
D,C)
4(A (2.)
P[4 ID.))
9(4| D.C)
P(A
D.C)
((4
D. C)
48
99.8
49.8
83.6
Tronion
Wearing Turban
0.2
27.3
27.1
69
68.8
38.2
38
Old
93.2
53.2
S7
47
66.5
26.5
Traditional Claths
6.1
Beard
96.6
62.6
99.7
65.7
83.5
49.5
Mon
78.5
27.5
78.1
27.1
31.6
Indian
Tarbun
2,2
0.2
0.9
0.1
Mustache
25
17.7
0
12.4
0
25.9
0.9
TilkhRind
61.7
11.7
59.3
9.3
86.7
36.7
VirronColorCluta
41.5
58.3
8.3
53.8
3.8
Mon
48
95.1
47.1
85
37
50.1
2.1
Meaicon
94.4
Sombrero
56.6
6.6
17.5
0
58.6
8.6
Mustuche
25
77.8
52.8
34.1
9.1
84.7
59.7
Embroidered Clothing
32.6
94.2
from SDv2 to SDv3 and FLUX.I, our results demonstrate that stereotype scores of newer models are generally lower than those of the older ones. However, some cases, there are exceptions. As an example, when generating images of Mexican person, FLUX.1 depicts 84.7% of the faces with
mustache while SDv2 and SDv3 generate 77.8% and 34. 4.1%% faces with mustache, respectively. In high-level attributes such as gender, FLUX.1 has lower stereotype scores than other models. For example, in the case of Iranian, FLUX.1 depicts 83.6% of images as man. But in comparison, 98% and 99.8% of the images generated by SDv2 and SDv3, respectively, depict man.
Existing bias definitions are not applicable for some attributes studied in Tab. I E.g., a T2I model needs to depict 50% of the images of Iranian with turban to be unbiased according to Eq. (T. which incorrectly represents Iranian people among whom only 0.2% wear turban.
Remark.
Previous works have noted the gender imbalance in the generated images for certain professions such as doctors and teachers (D' O'Incà et al., 2024). We observe similar trend in the newer T2I models as shown in Tab. How- ever, SDv3 has lower gender imbalance compared to SDv2 for doctor. We hypothesize that this is due to the data balaneing methods taken ensure unbiased gender representation in the images of doctor following the scrutiny it has faced. However, the imbal- ance worsens when a nationality is added to the profession (e.g., Iranian doctor). This example demonstrates that stereotype mitigation through data balancing is insufficient against intersectional stereotypes as it is infeasible collect data samples corresponding to every possible combination.
Table 2: P(A= man C, D) for C doe- for, C Iranian doctor, and Indian Doctor.
Model
Doctor
Indian Doctor
Iraiinn Doctor
SDv2
93
97 (44)
98 (+5)
SD+3
FLUX.I
93
(+7)
100 (+7)
4.2 IMAGES OF UNDER-REPRESSE REPRESENTED NATIONALITIES CONTAIN MORE STEREOTYPES
T2I models are often trained on image-caption pairs that are scraped from the Internet. Therefore, their training data may be biased by the Internet footprint of various nationalities. To investigate the impact of a nationality's 's Internet footprint on stereotypes in T2I models, we compare the stereotype scores of generated images from various nationalities against their corresponding number of Internet users. We consider generated images corresponding to Indian, Mexican, and Iranian nationalities, which have populations of 881.3 million, 96.8 million, and 78.1 million, respectively (WorldP Popllation| 2024). Fig. pressenss the stereotype scores across different attributes for each country and model. The results indicate that the maximum and the average stereotype scores for a nationality decrease as the number of Internet users increases. These findings suggest that the stereotypes in T2I models may be exacerbated for represeedd nationalities when trained on image-caption pairs from the Internet.
FLEX spoe
bardebitatandd
Stereotype Soore
7B. (Tuan) 96.M (Musico) 881.3M (Indaa
Figure 3: type scores for nationalities against the number of Inter- net users shows that stereo- types are higher for under- represented nationalities.
Comparing stereo-
7

---

Published as a conference paper at ICLR 2025
4.3
EFFECTIVE T21 MODEL COMPARISON REQUIRES BOTH STEREOTYPE SCORE AND WALS
Model SD22 SDx3
(de
Model
Shv2 SDx3
(dev
Model
SDw2
SD13
28
28
26
Amihunes
Amilunas
Amihmnss
(a) Iranian
(b) Mexican
(c) Indian
Comparison of SDv2, SDv3, and FLUX.1 on spectral variance in the generated images across different attributes, calculated for Iranian, Mexican, and Indian nationalities.
Figure 4: WALS:
Fig. 4 compares the WALS for T2I models on three nationalities. higher WALS(A indicates more variance in the images along the attribute A. We observe that FLUX.1 1 generates images that show higher variety in clothing items such as hats and turbans, but have a lower variance regarding facial attributes such as beard and mustache across all three nationalities. contrast, images generated by attributes.
SDv2 show a higher variance on beard and mustache than on
SD33
Boan
Man
Mhatate
Stereceype Score
Stereotppe Score
Sterootype Score
-0
50
20
[LAXX
25
25
Sbvd
22
2.2
2.4
2.6
WALS
WALS
WALS
(a) Iranion
(b) Indian
(c) Mexican
Figure 5: Comparison T2I models based on stereotype scores and WALS on three nationalities.
Different colors show different T2I models and the shapes of the markers denote the altributes.
As mentioned in $ 3.2, stereotype score and WALS are complementary measures of stereotypes. We can compare the models jointly on these scores to verify if models demonstrate lower stereotypes the cost lower variety. Fig- S plots stereotype score and WALS for various T2I models and attributes for each nationality. An ideal T21 model must have low stereotype score and high WALSS and therefore must appear towards the bottom-right corner of these plots. We observe that some models have lower stereotypes while having lower attribute variance. For instance, images from SDv3 tend to have lower WALS across all nationalities, although they succeed in reducing stereotypes in some attributes. These observations highlight the importance of employing both distributional (Stereotype Score) and spectral (WALS) metrics together to compare the T2I models.
4.4 T2I MODELS INTERNALLY ASSOCIATE CONCEPTS WITH STEREOTYPES
We use StOP to discover the internal associations that the T2I model M makes with given concept c. In Tab. 3 we show M-attributes FLUX.1 discovered using StOP for three concepts: Iranian, Mexican, and American. We obtain clusters of images using spectral clustering of aligned faces and manually identify those with shared stereotypes. The average of the faces in the clusters are shown in the second column. The attributes that expect StOP discover can be visually identified from these averaged images. For example, the average of the cluster corresponding Iranian shows an old man wearing a turban and sporting long beard, characteristic the Islamic religious leaders in Iran. Therefore, the expected M-attributes include religious terminology. In the third column, we show some of the optimized prompts that StOP produces. The optimized prompts contain Unicode characters in vernacular languages. The optimized prompts corresponding to Iranian images include religious terms such as "Imam" and "Sheikh". Similarly, the optimized prompts for Mexican images include "brero" (short for sombrero). In the last column, input one of these prompts to FLUX.1 to visually inspect the resulting images. Unsurprisingly, the images generated from these optimized prompts nationality. For example, the US national flag can be seen as a blurred background the cluster average and is also present in the samples generated from optimized prompts for American person.
the CLIP features
visually similar to those generated from prompts containing only
8

---

Published as a conference paper at ICLR 2025
Table StOP first identifies image clusters for each concept using spectral clustering. The averages of the images from these clusters are shown in the second column. StOP finds the captions shown in the third column by solving the optimization problem in Eq. (8) These captions contain stereotypical attributes such "/mam" and "brero". The fourth column shows the images generated using these optimized captions. Unsurprisingly, these images contain insignia the corresponding culture.
Culture
Cluster average
Optimized prompts
Samples from highlighted prompt
"This is a photo of (u093f (002c Imam"
Iranian
"This is photo of ( reb"
"This photo of (u093f u0035 Sheikh"
"This is a photo of brero mayor"
Mexican
"This a photo bero Garcia"
"This a photo of brero pastor"
"This is a photo of EO Democrat"
American
"This a photo of :border counsclor"
"This photo of :border ambussador"
4.5
STEREOTYPICAL ATTRIBUTES EMERGE IN THE EARLY STEPS OF IMAGE GENERATION
Juu
Atrilet
Atriliat
Time Step
Time
Time Soy
Time Sazy
(a) Iramian, 1
(b) Iranian, SDv3
(c) Indian, FLUX.1
(d) Indian, SDv3
Time Step
Figure 6: SPI tracks the change in attributes in the image generation processes of FLUX.1 1 and SDv3. We observe that these attributes are affected during the early time steps of image generation.
We quantify the emergence of stereotypical attributes during image generation in FLUX.1 and SDv3 for image prompts of the form "A photo of an <nationality> person" using SPI. For a given stereotype, we first obtain positive and negative descriptions corresponding it. For example, for the attribute age, old and young were used in d+ and d' in Eq. (2). respectively. SPI is then calculated as the cosine similarity between 8.A and velocity of the latent x, at time step as shown in Eq. (T]. We plot SPI(A. t) for all time steps during the generation of four images from Iranian and Indian nationalities in Fig. 6 We observe that a relatively high amount of information on attributes such as beard, traditional cloths, and sombrero is added to the images at the first step of generation in FLUXX11 indicating that the model readily associates these attributes with the concept. Specifically, we observe that the stereotypical attributes arise during the earlier time steps of generation in both Iranian and Indian images. the example of Iranian person in Fig. we observe that age and beard attributes form in the image within the first 3 time steps and gender attribute emerges at time step 7. After time step 20, the changes in these attributes are negligible. Likewise, for Indian nationality, stereotypical attributes undergo little change after time step 20. Additional results for other nationalities and the average SPI over 100 samples are provided in & A.3 and Fig. 9 respectively.
4.6 T21 MODELS HAVE STEREOTYPICAL PREDISPOSITIONS ABOUT CONCEPTS
In
we noted that stereotypical attributes aggregate in the early steps of image gener- ation. A question that naturally follows this observation is: T21 models predisposed to

---

Published as a conference paper at ICLR 2025
generate stereotypical images for given concept? This can be answered by considering the velocity th ( t, 6)) of the early time steps since they guide towards the mean of the data distribution (Kynkäänniemi et al.) 2024). This enables us to identify the stereotypical predispositions qualita- tively. For each time step i, we estimate the final time step image îT based on velocity ue (I t, Et) as TT It + ( &, (T) t), as illustrated in Fig. 7a Fig. shows these images for three samples corresponding to Iranian person. The images generated using the velocity at time / 0 appear to be of person with lurban and beard, even when the final generated images lack these attributes. Conflating with our observations from & a s $ we conclude that T2I models associate stereotypical attributes with scemingly innocuous prompts. Additional results are provided in S A.4
Time Step
Random Stedd
Time Skep
(a) Estimation
(b) Iranian person
Figure Stereotypical predisposition in
T21 models for Iranian person.
RELATED WORK
Many studies have shown that deep learning models tend to learn and, at times, amplify the biases
present in their datasets (Bolukbasi al., 2016, Buolamwini & Gebru 2018 Luccioni & Viviano
2021
Blodgett et al. 2021 H Agarwal et al. 2021 Aka et al. [2021] 1 Sadeghi ct al. 2022] Birhane et al.
2023
2024 Chuang et al. 2023 Dehdashtian et al., 2024b a Phan et al. 2024). and T2I models
are no exception. Most of the existing work about stereotypes in T2I models has focused on gender and ethnic biases in the generated images. Some studies have shown that prompts play significant role in the bias generated by T2I models (Bansal et al. 2022 Zhang et al. [223]] Seshadri et al.
2024). Seemingly neutral prompts lead to geographical biases favoring Western nations such as the
US and Germany. leading to lighter skin tones and Western norms in the images (Bianchi et al. 2023
Naik & Nushi 2023). while prompts containing certain cultural and gender terms sometimes generate
NSFW images, reflecting the biases in the training datasets (Birhane et al. 2021 Ungless et al., 2023
Schramowski et al.] 2023). [uccioni et al. 2024) measured distributional biases in professions W.r.t
closed set of genders and ethnicities.
Unlike these works, we use an open set of stereotypes obtained from an LLM, following D Inca all 2024). Although these studies have achieved breadth in terms of sources for stereotypes, they have primarily used statistical parity as the definition of stereotype. For example, (Jha et al.) 2024) uses "stereotype tendeney" defined as the ratio of the likelihood of a stereotype appearing in group to that of it appearing in the general population, ignoring the directionality of stereotypes. In contrast, we measure stereotypes following their true sociological definition. We additionally provide insights into the origins of the stereotypical attributes in T2I models.
6 CONCLUDING REMARKS
This paper proposed OASIS to measure and understand the origin of stereotypes in T21 models based quantitative measure that aligns with the sociological definition of stereotype. OASIS includes: (MI) Stercotype Score ($ 3.T to measure the directional violation of the true stereotypical attribute distribution in the T21 model, (M2) WALS ($ 3.2) 2 to measure the spectral variety of the generated images along the stereotypical attributes, (U1) StOP (.3) to discover the stereotypical attributes that the T2I model internally associates with a concept, and (U2) SPI ($ 3.4p measure the emergence of stereotypical attributes during image generation from the latent space. Despite the considerable progress in the image fidelity of T2I models, using OASIS, conclude that the newer models such as FLUX.1 and SDv3 have strong stereotypical predispositions about concepts and still struggle to avoid stereotypical attributes in the generated images.
Recommendations. OASIS unveils the extent of stereotypes in T2I models. However, commonly pursued solutions for correcting biases in generative models such as data balancing are not suitable for resolving stereotypes due to the sheer number of concepts that could potentially have stereotypes. Additionally, concepts such as nationalities worsen stereotypes in unrelated concepts such as doctors as observed in Tab. 2 It is infeasible to collect data samples at the intersection of multiple concepts. Therefore, training- time mitigation and post hoc correction techniques that are tailored to remove stereotypes in T2I models must be developed (Phan et al.) 22224] Zhang et al. 2023) Friedrich et al.) 2023 Parihar et al. 2024). Our observations also underscore the need for increased participation of under-represennted communities in the development of large generative models.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements: This work was supported in part by the National Science Foundation (award #21471 116 and the Office of Naval Research (award #N00014-23-1-24117, Any opinions, findings, and conclusions recommendations expressed in this material those of the authors and do not necessarily reflect the views of NSF or ONR.
REFERENCES
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2 02818, 18, 2021.
Osman Aka, Ken Burke, Alex Bauerle, Christina Greer, and Margaret Mitchell. Measuring model biases the absence of ground truth. In AAAI/ACM Conference AI, Ethics, and Society, 2021.
Ananya. AI image generators often give racist and sexist results: can they be fixed?, 2024. URL https:// www www.natuueee..m.. com awt.iaaetileeeeeeeeeeeeeeeeeeere d41 0011:/2401000000055055000000000-
Hritik Bansal, Da Yin, Masoud Monajatipoor, and Kai- Wei Chang. How well can Text-to-Image Gen- erative Models understand Ethical Natural Language Interventions? In Conference on Empirical Methods in Natural Language Processing, 2022.
Elnaz Barshan, Ali Ghodsi, Zohreh Azimifar, and Mansoor Zolghadri Jahromi. Supervised principal component analysis: Visualization, classification and regression subspaces and submanifolds. Pattern Recognition, 2011.
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes large scale. In ACM Conference Fairness, Accoumabilitt, and Transparency, 2023.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021.
Abeba Birhane, Vinay Uday Prabhu, Sanghyun Han, Vishnu Naresh Boddeti, and Sasha Luccioni. Into the LAION's Den: Investigating Hate in Multimodal Datasets. Advances in Neural Information Processing Systems, 2023.
Abeba Birhane, Sepehr Dehdashtian, Vinay Uday Prabhu, and Vishnu Naresh Boddeti. The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models. In ACM Conference Fairness, Accountability, and Transparency, 2024.
BlackForestl Inrounciing-becc--foree---------
FLUX.
https://blackforetti.ds ai/
2024.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping Norwegian salmon: An inventory of pitfalls faimess benchmark datasets. In Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 2021.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman to homemaker? debiasing word embeddings. Advances in Neural Information Processing Systems, 2016.
Pedro Bordalo, Katherine Coffman, Nicola Gennaioli, and Andrei Shleifer. Stereotypes. The Quarterly Journal of Economics, 2016.
Sarah Brotherton and Sylvia Etzel. Graduate medical education. 2022-2023. The Journal of the American Medical Association, 330(10)9888-0111 2023.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In ACM Conference on Fairness, Accountability, and Transparency, 2018.

---

Published as a conference paper at ICLR 2025
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In /EEE/CVF International Conference on Computer Vision, 2023.
Ching- Yao Chuang. Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.000700 2023.
Sepehr Dehdashtian, Ruozhen He, Li, Guha Balakrishnan, Nuno Vasconcelos, Vicente Ordonez, and Vishnu Naresh Boddeti. Fairness and Bias Mitigation in Computer Vision: A Survey, arXiv preprint arXiv:2408.02444 2024a.
Sepehr Dehdashtian, Bashir Sadeghi, and Vishnu Boddeti. Utility-Farness Trade-Offs and How to Find Them. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Sepehr Dehdashtian, Lan Wang, and Vishnu Naresh Boddeti. FairerCLIP: Debiasing CLIP's Zero- Shot Predictions using Functions in RKHSs. International Conference on Learning Representa- fions, 2024c.
Moreno D' D'Incà, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang, Humphrey Shi, and Nicu Sebe. OpenBias: Open-set Bias Detection in Text-to-Image Generative Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama herd of models. arXiv preprint arXhv:2407.21785 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024.
Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv preprint arXiv:2302.10893, 2023.
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, 2019.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical Schmidt norms. In International conference on algorithmic learning
dependence with theory. Springer, 2005.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, July 2021. htthhttss://ddt /ttp://000..00/1.221111111 Zenoooo 31
Akshita Jha, Aida Mostafazadeh Davani, Chandan K Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. SeeGULL; A Stereotype Benchmark with Broad Geo-Cultural Coverage Leverag- ing Generative Models. In Annual Meeting The Association For Computational Linguistics, 2023.
Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan Reddy, and Sunipa Dev. ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation. In Annual Meeting of the Association for Computational Linguistics, 2024.
Tuomas Kynkäääniemii, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724 2024.
Marie Lamensch. Generative AI tools are perpetuating harmful gender stereotypes. Centre for International Governance Innovation, 14, 2023.
12

---

Published as a conference paper at ICLR 2025
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In IEEF/CVF International Conference on Computer Vision 2015.
Alexandra Sasha Luccioni and Joseph D Viviano. What's in the box? a preliminary analysis of undesirable content in the common crawl corpus. arXiv preprint arXiv:2105.02732, 2021.
Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jemnite. Stable bias: Evaluating societal representations diffusion models. Advances in Neural Information Processing Systems, 2024.
Helmut Lütkepohl. Handbook of matrices. John Wiley & Sons, 1997.
Ranjita Naik and Besmira Nushi. Social biases through the text-to-image generation lens. In AAAT/ACM Conference on AI, Ethics, and Society, 2023.
Leonardo Nicoletti and Dina Bass. "Humans are Biased. Generative Al Even Worse", 2023, URL https://wwes http:///www.p./oooooccccc/coo///
OpenAI. Chatgpt 4o.
2024a.
OpenAl. Chatgpt ol -preview.
2024b.
Rishubh Parihar, Abhijnya Bhat, Abhipsa Basu, Saswat Mallick, Jogendra Nath Kundu, and R Venkatesh Babu. Balancing Act: Distribution Debiasing in Diffusion Models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Hoang Phan, Andrew Gordon Wilson, and Qi Lei. Controllable Prompt Tuning For Balancing Group Distributional Robustness. arXiv preprint arXiv:2403.02695, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In International Conference on Machine Learning. 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In JEEE/CVF conference on computer vision and patlern recognition, 2022.
Bashir Sadeghi, Sepehr Dehdashtian, and Vishnu Boddeti. On Characterizing the Trade-off in Invariant Representation Learning. Transactions Machine Learning Research, 2022. ISSN 2835-8856. Featured Certification.
Maarten Sap. Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022.
Patrick Schramowski, Manuel Brack, Bjöm Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration diffusion models. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, 2022.
Preethi Seshadri, Sameer Singh. and Yanai Elazar. The Bias Amplification Paradox in Text-to-Iman to-Ima Generation. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024.
13

---

Published as a conference paper at ICLR 2025
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-ter genera- tion for visual-semantic arithmetic. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Eddie Ungless, Björ Ross, and Anne Lauscher. Stereotypes and Smut: The (Mis) representa- tion of Non-cisgender Identities by Text-to-lmage Models. In Findings of the Association for Computational Linguisties, 2023.
Adriana Fernández de Caleya Vázquez and Eduardo Garrido-Merchán Taxonomy of the Biases of the Images created by Generative Artificial Intelligence. arXiv preprint arXiv:2407.01550 2024.
Lauren Vogel. When people hear "doctor", most still picture a man. CMAJ: Canadian Medical Association joumall=journde de I'Association medicale canadienne, (110)):2955-229 2019.
Ulrike Von Luxburg. tutorial on spectral clustering. Statistics and computing, 2007.
WorldPopulationR: "Internet Users by Country", 2024. [Online; accessed 1-Oct-2024).
Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wu, Dmitry Lagun, Thabo Beeler, and Fernando De la Torre Iti-gen: Inclusive text-to-image generation. In JEEE/CVF International Conference on Computer Vision, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
In our main paper, we proposed OASIS for quantifying stereotypes and understanding their origins in T2I models. Here, we provide some additional analysis to support our main results, The appendix section structured follows:
1. Implementation Details in A.1
2. Finding 8A Using A-Aware Generated Images in
A.2
3. More Results on SPI and average SPI
A.3
4. More Results on T2I models' Stereotypical Predispositions in A.4
5. Limitations in A.5
6. Ethical Considerations in A.6
7. Importance of Detecting Stereotypes in A.7
8. Qualitative Descriptions the Generated Datasets in $ A.8
A.1 IMPLEMENTATION DETAILS
In this section we provide all the details that are employed the experiment section in order to obtain the mentioned results.
A.1.1 LLM TEMPLATE INSTRUCTION
To generate open-set candidate for stereotypes corresponding given concept, follow the
approach provided in D'Incà et al. 2024). The template instruction Z that was used in Eq. 2 is as follows:
I am using a text-to-image model to generate facial images of individuals from various nationalities. These images depict faces along with surrounding margin that may include headwear or clothing visible above the chest. I need a candidate set of at least 15 potential stereotypes monitor for in the generated images. will provide the prompts used to generate the images, and I would like you to identify this candidate set. Additionally. I plan to check the presence or absence of these attributes the images using a CLIP model. For each attribute, need at least two sentences: one that positively describes the existence of the attribute and another that describes its absence.
An example of the output produced by ChatGPT ol ol-preview is,
15

---

Published as a conference paper at ICLR 2025
Certainly! Based on your prompt, "A photo of Tunisian person," " here is list of potential attributes to look for in the generated images. These attributes are associated with cultural cle- ments and common depictions of Tunisian people. For each attribute, I've provided a positive and negative sentence that can be used with CLIP model for zero-shot classification.
1. Wearing Traditional Chechia (Hat)
Positive: "The person is wearing a traditional Tunisian chechia."
Negative: "The person is not wearing any headwear."
2. Traditional Clothing
Positive: "The person is wearing traditional Tunisian clothing."
Negative: "The person is wearing modern Western clothing."
3. Wearing a Djebba (for men)
Positive: "The man is wearing a traditional djebba."
Negative: "The man is not wearing a djebba."
15. Wearing a Sefsari (for women)
Positive: "The woman is wearing a traditional sefsari."
Negative:
"The woman is not wearing a sefsari."
A.1.2 IMAGE GENERATION PROMPTS
To generate images from FLUXX.1 and SDv3, we used the prompt template "A photo of <nationality> person", where <nationality> could take values like American, Indian, etc. For SDv2, we observed that the images could be lower quality or contain the torso of the person. Therefore, for each images, sampled prompt randomly from the following array of prompts: "A photo of <nationality> person", "A picture of <nationality> person", "A portrait photo of <nationality> person", "A front profile photo of <nationality> person".
A.1.3 OBTAINING TRUE DISTRIBUTIONS
Stereotype score measured the violation of the underlying distribution of attribute given a concept, denoted by [* C). in the generated images. One could obtain P*(A C) from official census data and online statistes. For example, [Brotherton Etzel (2023) provides various demographic details about doctors in the US such as ethnicity and gender in various specializations. For attributes where is difficult obtain precise statistics, e-g., traditional clothing, we consider their presence a choice and assign a 50% chance for their presence. For example, for mustache for people from Mexican nationality, we calculate p* ("mustache MMxxica) = 0.5 Pa (male Mexican) R 0.255.
A.2 FINDING 8A USING A-AWARE GENERATED IMAGES
As mentioned in 3.2 to find the direction of change in A, we propose two approaches: (i) using text embeddings of a pair of positive and negative descriptions, dt and d and (ii) using A-aware generated images. The first approach is explained in 3.22 and in this section, we explain how to find A using A-aware generated images both linear and non-linear cases.
A.2.1 LINEAR 8A
As mentioned in $.22 using A-aware generated images to find A can be more precise than using text embeddings. As example, for finding the direction of change in male for images corresponding to "A photo of an Iranian person", two sets of images using prompts "A photo of a man" and "A photo of a woman" created. The of CLIP features of these images
denoted by ZA )m and their corresponding labels of as YA = {y: (m Then we find an orthogonal transformation matrix
that maps ZA subspace that maximizes the variance of the labeled data using supervised principal
16

---

More samples for SPI on Mexican person and Tunisian person are Published as a conference paper at ICLR 2025
component analysis (Barshan et al., 2011) that maximizes the dependency between the mapped data IT A and Ya. Hilbert-Schmide Independence Criterion (HSIC) (Gretton et al. 2005) is employed as the dependence metric where its empirical version is defined as HSIC where H is the centering matrix, Ky is a kernel matrix of Y, and Kzz is a kernel matrix of the mapped data. When using linear kernel, it becomes Kzz Z rT Z. Therefore, can be calculated by solving the following optimization
Iz Ky J.
arg max Tr(T" r
ZHKyy }
(12)
subject to IT==I
(13)
This optimization has closed-form solution, and the columns of the optimal I' are the eigenvectors of M :- H corresponding to the d largest eigenvalues where d is the dimensionality of the subspace (Lütkepohl 1997). Here, since we only need a direction vector, we choose the eigenvector 81 associated with the largest eigenvalue of M.
6A
Ū1.
(14)
To capture the non-linear relations of the attribute, a non-linear kernel can be used to calculate Ky and Kzz. The closed-form solution for the non-linear case is provided in $ $/.22.2
A.2.2 NON-LINEAR 8A
If we are interested in finding non-linear relations between the images in order to find direction of change in attribute, non-linear version of the formulation mentioned in the previous subsection can be used. In this approach, similar to the linear case, we generate two sets of images associated with positive (a ) and negative (a categories of A. The set of CLIP features of these images are denoted by ZA- = {= and their corresponding labels of A as YA } Then we find an orthogonal transformation matrix I' that maps kernelized Z
subspace that maximizes the
variance of the labeled data using supervised principal component analysis (Barshan et al. 2011 that maximizes the dependency between the mapped data I' Kzz and YA.
Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al. metric where its empirical version defined as HST centering matrix, Ky is a kernel of Y, and Kzz is the kernelized Zu using similarity measure of the mapped data. I' can be calculated by solving the following optimization
2005) is employed as the dependence Tr TrH H Ky), J. where is the
arg max Tr(TT
Kzz H Hy l},
subject to -T-=I
(15)
This optimization has a closed-form solution and the optimal solution for I are the eigenvectors of M : == H HKT2 corresponding to the d largest eigenvalues where is the dimensionality of the subspace (Lütkepohl] 1997). Here, since we only need a direction vector, we choose the eigenvector & associated with the largest eigenvalue of M.
8A --= -
(16)
A.3 MORE RESULTS ON SPI
More Sample- Wise Results. illustrated Fig. X
The average SPI in 100 images generated by SDv3 corresponding to "A photo of an Iranian person" is demonstrated in Fig. [ As illustrated, the T2I model adds a high amount of information attributes such as turban in the earlier steps. This confirms our earlier conclusions from sample-wise SPI in Fig. 6b Additionally, we note that the variance for SPI is small in the time step T O, suggesting the stereotypical predispositions noted in 4.6 However, in the next few time steps, we see a slightly larger variance that indicates that these models tend to correct the stereotypical attributes added in the former time steps.
Average SPI.
17

---

Published as a conference paper at ICLR 2025
Arriheae Gendo Seebaoo Hanate
Atrdaa Geade Sortees Sunde
Anrhere GGnder Age
Trackee Satton
Antisute Goode
AAg TTdienal Kallok
4.2
Tine
flae
Time
(a) Mexican, FLUX.1
(b) Mexican, SDv3
(c) Tumisian, FLUX.1
(d) Tunisian, SDv3
Figure 8: Additional samples of SPI plots of generated images by FLUUX.1 1 and SDv3 for Mexican and Tunisian nationalities. A positive value for SPI(A, t) means that at is added to the image at time step Similarly, negative SPI means that the image is moving toward a
Antribute
Time Step
Figure 9: Average SPI in 100 samples for Iranian person generated by SDv3.
A.4 MORE RESULTS ON T21 MODELS' STEREOTYPICAL PREDISPOSITIONS
In this section, we provide additional results that show that T2I models are predisposed create stereotypical images for various nationalities. In Fig. TO we show additional results for FLUXX.1 on Iranian, Indian, and Mexican nationalities. Similar to our observations $ $4.66 we note that the images generated from the velocity at t 0 for Iranian person contain stereotypical attributes such as beard and turban. Likewise, for images of Indian personality, we observe vibrant clothing (e.g.. orange veil). In the images of Mexican person, we can see faded sombrero in the early images. Moreover, the attributes that appear in the early stages of image generation are absent the final generated image, indicating that these stereotypical attributes arise due to their intrinsic association with the concept.
A.5 LIMITATIONS
Obtaining (A C). Access to (* (A|C) C) is a crucial component for any stereotype measuring method. As mentioned in A.1.3 Pt (*(A C) is obtained from census data and online sources when they are available. We note that reliable sources may not be available for every attribute and changes in survey methods can affect the results. However, for most stereotype evaluation and mitigation applications, reliable data can be found from government and survey agencies.
Use of CLIP. As mentioned in $ 3.1 we obtain P(A) 1,, C) using attribute classifiers. Instead of training attribute-specific classifiers, zero-shot predictor like CLIP (Radford et al. (02T can be utilized. However, some attributes may be unfamiliar the model, resulting in lower accuracy in detecting them within the images. Additionally, these models may be biased in terms of concepts such as ethnicity. With advancements zero-shot prediction models and the introduction of more accurate versions, newer models can seamlessly replace the existing ones in OASIS, thanks to its modular design.
For small dataset of doctors that is used in Tab. 2. we evaluate the performance of the CLIP model in predicting the gender. As mentioned earlier, for each T2I model, we generated 100 images of doctors and manually labeled their genders. The accuracy of the CLIP model in predicting gender is demonstrated in Tab. The results suggest that on this small dataset, the CLIP model can predict gender almost as accurately human annotators.
18

---

Published as a conference paper at ICLR 2025
Time Step
(a) Iranian person
Time Step
(b) Indian person Time Step
(c) Mexican person
Figure 10: First 9 steps of image generation in FLUX.1 model for three nationalities: (a) Iranian
person, (b) Indian person, and (c) Mexican person
Additionally, we evaluate the performance of the employed CLIP model on CelebA (Liu al. 2013) that contains more than 200,000 face images of celebrities annotated with 40 binary attributes. Since primarily used CLIP to predict attributes such beard and hat, evaluate the model on similar attributes (i.e., having beard, man, wearing a hat, having a mustache). The accuracy in predicting each attribute is reported in Tab. 5 The results demonstrate that the CLIP model can
19

---

Published as a conference paper at ICLR 2025
Table 4: Performance of the CLIP model on predicting gender in the generated doctors dataset.
Model
Accuracy
SDv2
100%
SDv3
99%
FLUX.1
99%
predict the attribute with an acceptable accuracy. As we noted earlier, although the CLIP model may not accurately predict certain general attributes, our results indicate that the CLIP model is suitable for predicting the attributes that we considered this work.
Table Performance the CLIP model on predicting four attributes in CelebA dataset.
Attribute
Accuracy
having beard gender
83.06%
99.38%
wearing a hat
96. 14%
having a mustache
94.77%
The thooee-mentioned experiments, show the effectiveness of using CLIP model in automating the classification of the images. However, the accuracy of the model is not 100% which indicates that by newer usion-language model with higher accuracy compared to the CLIP model that is employed in this paper, should be replaced in the OASIS.
A.6 ETHICAL CONSIDERATIONS
In this work, we measure stereotypes and explore their origins in images generated by publicly avail- able T2I models. We find that these models pose ethical concerns by reinforeing social stereotypes and potentially offending some users. By introducing OASIS, we aim to highlight these issues and move the community one step closer mitigating stereotypes in generative models.
A.7 IMPORTANCE OF DETECTING STEREOTYPES IN GENERATIVE MODEL.
Visual content produced by generative models inadvertently perpetuates stereotypes about various ethnicities, cultures, nationalities, and professions (Ananya 2024). Such images and videos are shared online social media accounts such as X and Reddit, and this reinforce stereotypical notions about certain social groups. This content could also influence public perception of marginalized communities and could undermine ongoing efforts to integrate them into mainstream society. For example, it has been noted that generated images of women from certain ethnicities tend to be sexualized (Lamensch) 2023). Additionally, the adoption of these generative models by various companies and institutions may have unforeseen consequences. For example, Nicoletti & (2023 states that using generative Al to develop suspect sketches could lead wrongful convictions.
A.8 QUALITATIVE DESCRIPTIONS OF THE GENERATED DATASETS
We evaluated OASIS on the images corresponding various nationalities generated by different T2I models. In this section, give qualitative description of the generated images. A few randomly selected representative samples from each culture and T21 model are shown in Fig. [T]
FLUX.1. The images produced by FLUX.1 are of high quality and look realistic. However, some stereotypes can be qualitatively observed from the images Fig. IIa For example, some images of American people contain the American flag, Images of Indian people tend to show vibrant colored clothing. Most the generated images of Iranian people are of men and most of them wear turban. Among the images of Mexican people, sombrero is the most common stereotypical element. We additionally note general superficial diversity among the samples. For example, the images each were generated with the same random seeds. We observe that the backgrounds in these photos
20

---

Published as a conference paper at ICLR 2025
sometimes repeated. For instance, compare the first columns of Indian and Mexican samples. Additionally, there is a clear disparity in the backgrounds across nationalities. The backgrounds for American and Iranian images are more often indoors than for Indian and Mexican. Images of American and Iranian people also more often contain images of officials compared to Indian and Mexican images. Interestingly, Donald Trump's image appeared when prompted generate images of American person.
SDv3. Fig. Ilb shows the samples generated by SDv3. Although the generated images are of high fidelity, unlike FLUX.1, they lack variety in background and poses. Surprisingly, images of American people are relatively free of stereotypes and show ethnic diversity. However, the face images of Indian people look very similar and contain elements such as tilak/bindi. The diversity drops further the images of Iranian people. All the randomly selected samples contained images of men wearing turban and religious aftire. Among the images of Mexican people, sombreros were present but in fewer proportions compared to the images generated by FLUX.1.
SDv2. Some representative samples generated by SDv2 are shown in Fig. Il Among the consid- ered T21 models, SDv2 produced images with the least photorealism, with some displaying distorted facial expressions. However, these images generally contain diverse facial attributes such as hairstyle. Images of American people are of higher quality compared to other nationalities, although they include black & white portraits. We note the lack of ethnic diversity among these images compared to those in SDv3 and FLUXX.1. Although identity diversity is lower for images of Indian people compared FLUX.11 and SDv3, also observe fewer stereotypical attributes. Similar to SDv3, the images of Iranian people generated by SDv2 are primarily of men, mostly donning turban. Stereotypes such as sombrero and colorful clothing are present in the images of Mexican people. Among all the T2I models that we considered, SDv2 seems to have the least gender diversity across all nationalities.
21

---

Published as a conference paper at ICLR 2025
American
Indian
Iranian
Mexican
(a) FLUX.1
American
Indian
Iranian
Mexican
SDv3
American
Indian
Iranian
Mexican
(c) SDv2
Figure 11: A few randomly selected representative samples from each culture and T2I model.
22


Reviews:
[{"review_id": "3PYA0hUasj", "reviewer": "Reviewer_EfBv", "summary": "In this paper, the authors propose multiple new metrics to understand stereotyping in T2I models.  In particular, they propose a metric for stereotyping that compares the prevalence of a given attribute to a societally-anchored prior probability.  They go on to present numerous other analyses for diversity of images and measuring how differnt stereotypical concepts relate to demographics in latent space.  They show that generally newer models exhibit less stereotyping and that stereotypes arise mostly in the earlier stages of optimization.", "questions": "S3.2 - WALS is quite confusingly described.  In particular (6) doesn't really make sense as an equation.  If E(D) is the SVD that is just estimating the dataset? Do you mean the rank or something else?  Even the motivation doesn't make much sense to me of why this is necessary.\n\n\"Existing bias definitions are not applicable for some attributes studied in Tab. 1. For example, in the case of wearing turban, a T2I model needs to depict 50% of the images of Iranian with turban to be unbiased according to Eq. (1), which incorrectly represents Iranian people.\" - This seems like an odd remark in that this is largely what is done here just conditioned on gender.  What do you see as the primary difference here?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. Measuring sterotyping is an important and challenging problem for generative models.\n\n2. The diversity of metrics seem quite interesting to get a more complete picture of how bias and stereotypes arise in these models.\n\n3. The paper is fairly thorough in its testing.", "weaknesses": "1. The biggest weakness in my opinion is that the paper is quite hard to follow for many of the metrics. I believe this is due to making the description of the metrics unnecessarily complex, and I would encourage the authors to try to describe all of the concepts more simply.  In addition to making the paper harder to understand, it also obfuscates the insights and contributions.\n\n2. One of the papers claimed main contributions is measuring stereotyping relative to societal priors on the distribution.  However, as the authors state in the appendix, the prior is often unknown and thus falling back to 50% is a reasonable choice.  That said, this seems to limit the contribution over prior work.\n\n3. The paper does not validate their work with human judgements\n\n4. More minor: the paper only presents the evaluations but do not work on mitigations.\n\n5. The intro could be more clear and sensitive in how it characterizes what are stereotypes, why they matter, and how they define which stereotypes are important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gdO6i9O5mA", "reviewer": "Reviewer_n4np", "summary": "The proposed work has a novel contribution in an existing research gap. Defining, separating, and quantifying a measure for stereotypes and biases in Image generation models (from text prompt) is a strong case of Ethical Machine Learning. The authors built a foundation of definitions that allowed them to measure estimates of data and image stereotypes. Then they apply that on well-known Text to Image models to uncover the stereotypes risks of applying those models according to the proposed measures/estimates.", "questions": "- In the abstract the authors said \"incorrectly categorizes biases as stereotypes\", do you mean the opposite (i.e. \"incorrectly categorizes stereotypes as biases\")?\n- How about using other measures of visual variations of the generated images (e.g. a similarity measure based on feature extraction (e.g. HoG))?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Addressing a research gap\nClear mythology\nApplication to existing models", "weaknesses": "It would have been nice if the writeup included a detailed case study of qualitative analysis of cases of stereotypes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOP2YUzOqn", "reviewer": "Reviewer_Hz7R", "summary": "This paper introduces OASIS, a toolbox for auditing visual stereotypes in text-to-image (T2I) models. OASIS offers novel methods for measuring stereotypes and analyzing their origins within T2I models. OASIS stereotype scores reveal that newer T2I models produce significant stereotypical biases, potentially limiting their applicability.", "questions": "Please, refer to the weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, clear, and easy to follow.\n- The paper addresses a complex and socially significant problem by evaluating biases and stereotypes in T2I models. Additionally, it provides new methods to trace the origins of these stereotypes in T2I models, offering valuable insights to help mitigate these biases.\n- The findings of this work highlight the need for greater inclusion and representation of underrepresented communities, particularly those from regions with limited internet presence, in the development of T2I models.", "weaknesses": "- This work focuses exclusively on generated images based on nationality, which raises several concerns. For instance, in racially diverse countries like the United States, how can an image indicate that someone is a U.S. citizen without relying on stereotypes and social biases? Additionally, in regions where people from neighboring countries may share similar facial features or cultures (such as North and South Korea), how useful can OASIS be in distinguishing these cases?\n- How can OASIS be useful beyond nationality domains? It is not clear in the paper how OASIS can be extended to evaluate other biases in T2I models, such as gender biases, racism, etc.\n- Overall, this is a solid piece of work; however, a deeper discussion of its limitations and the specific types of stereotypes for which OASIS may or may not be effective would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5oJsg2Z6Y", "reviewer": "Reviewer_AMAa", "summary": "The authors present OASIS, a set of 4 metrics/methods to analyze the presence of stereotypes in T2I models, how those stereotypes form, and the connections between them. The authors use 3 popular models for this analysis. In the paper, OASIS shows the performance of these models on various prompts (e.g. \"Show me a picture of a Mexican person\") and how certain stereotypes are strongly associated with the prompt, at rates greater than those stereotypes in the real world. The paper shows weaknesses in models in their attempts to prevent stereotypes such as revealing stereotypes when prompts or combined or in how they produce low-variance along the stereotype's dimension. The paper also shows how certain stereotypes are clustered together and at what stage in the image generation process these models form (which is almost always in the early stages). \n\nIn all, OASIS provides a useful toolkit for understanding stereotypes in T2I models and highlights the importance for further work in this very important area.", "questions": "Is this definition of stereotypes correct? Does it not implicitly push the stereotypes in society that cause imbalances to exist? Relying on census data to define an expected rate of a given attribute is perpetuating the stereotypes that exist in today's society. For example, using the example provided in the paper, if there are less women than male doctors in the world, it doesn't necessarily mean that the model is creating a 'stereotype' by generating an equal number of woman and male doctor images. \n\nFor experimental details, what is each point in Figure 3? What does each point correspond to in terms of attribute? \n\nIn Figure 5, there is a claim that \"We observe that some models have lower stereotypes at the cost of lower attribute variance\". This suggests a trade-off can exist between these two, but there is not much correlation observed for any model across nationalities. I agree that both metrics are important to measure, but it would be great to get more clarification on if there indeed is a trade-off that occurs often. \n\nI would appreciate any additional discussion about the accuracy of the metrics as well and if they work as intended.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper was very well-written, and it was quite easy to follow. I greatly appreciated how the terminology was constantly linked to concrete examples. The general formatting was also intuitive. \n\nThe paper did a good job of thinking about the broader context of this work, thinking a bit more critically about what phrases like biases actually mean and what we should think about when studying these models. Also incorporating real-world census data was a neat idea. \n\nI liked the results about how the biases are being generated early in the image-generation process.\n\nThe formulation for the stereotype score was quite intuitive and seems easy enough to compute at scale when provided with accurate descriptors from these LLMs.", "weaknesses": "I feel that overall, there was a lack of bench-marking for the methods presented in this paper beyond providing a few visual examples (which are still useful!). There were a few parts of the paper where I felt this significantly \n\n1. Computing the stereotype score: How accurate is the probability computation? The intuition of the metrics' formulation is clear, but there is no validation that the score provides accurate results in practice. As the paper mentions, CLIP is a biased VLM on its own, so its not fair to assume that this score will work perfectly. A smaller dataset demonstrating the accuracy of the score could be quite useful. \n\n2. Spectral variety along a stereotype. There was a nice description of each method for measuring $\\delta A$ in the appendix, but its not clear which one is used in the paper and which one is more accurate in practice. \n\n3. The biases of models like CLIP are well-documented (e.g. Birhana, Prabhu, & Kahembwe, 2021). If CLIP is being used extensively to identify additional biases, I think it makes sense to add more validation about how said biases may or may not be affecting the performance of OASIS. \n\n4. I understand the SPI claims about how certain stereotypes are intrinsically associated with a concept. However, the paper only provides a handful of examples of where this is occurring. I would appreciate a larger-scale analysis that includes something like 100 contexts generated in some way (i.e. top 100 most common ethnicities), and the average step at which each stereotype attribute occurs. This would be much more convincing than what could be cherry-picked examples. \n\nI think that the breadth of the experiments/validation were also a bit limited. I understand that gaining access to real-world data for a ground truth distribution is not always feasible, but consistently focusing on broad nationalities feels like it lacks nuance. \n\nAnother broad critique is that certain figures were unclear. I'll bring up those missing details in the following sections", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZlOXnF5bg1", "reviewer": "Reviewer_QudX", "summary": "This paper aims to measure and understand stereotypes in Text-to-Image models. \n\n* Measurement: it introduces metrics to quantify the presence of stereotypes in generated images by comparing them to real-world distributions, and also assess diversity along stereotypical attributes. \n\n* Track Origins:  The author attempts to identify hidden associations that T2I model makes with certain concepts and trace how stereotypical attributes emerge over the generation process", "questions": "I would appreciate it if the authors addressed the questions mentioned in the weaknesses section, and I will raise the score if their responses are convincing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles the significant problem of stereotypes in T2I models, which has important implications for ethical AI use and social impact.\n2. The paper is clearly written and organized. The visual presentations are good. \n3. It proposes StOP and SPI that try to understand the origin of the stereotypes, which may offer further insights.", "weaknesses": "1. The authors emphasize that their method is innovative in focusing specifically on stereotypes rather than just bias in the abstract. However, this distinction is not fully explained in the main content, and the methodology still seems quite similar to bias detection to me.\n\n2. The way they identify stereotypes is by prompting LLMs to generate stereotypical attributes for a given concept, but this method is already common in other papers. For example, [1] prompts LLMs to generate stereotypical / anti-stereotypical sentences for certain concepts and then identifies the stereotypical / anti-stereotypical attributes from those sentences.\n\n3. Although the paper includes many formulas, I found that the core idea is actually quite simple, and the formulas only make it harder to follow. If I understand correctly, the main process is: identify a concept, use LLMs to generate stereotypical attributes, generate images, measure the appearance of these attributes, and compare to real-world distributions & similarities across groups. This could be communicated in just a few sentences, which would make the approach clearer.\n\n4. I am concerned with the idea of comparing to real-world distributions. For example, men are dominant in many professions even today. In society, an educated person can be aware of stereotypes to some extent, but telling the model that it’s okay to learn patterns such as “men are more likely to be doctors” because it reflects real-world distributions doesn’t seem fully convincing to me.\n\n[1] chatgpt based data augmentation for improved parameter-efficient debiasing of llms", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)