Association of Objects May Engender Stereotypes: Mitigating Association-Engendered Stereotypes in Text-to-Image Generation

Authors: junlei  zhou
jiashi  gao
xiangyu  zhao
xin  yao
xuetao  wei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:50:22

Abstract: Text-to-Image (T2I) has witnessed significant advancements, demonstrating superior performance for various generative tasks. However, the presence of stereotypes in T2I introduces harmful biases that require urgent attention as the T2I 
 technology becomes more prominent.
Previous work for stereotype mitigation mainly concentrated on mitigating stereotypes engendered with individual objects within images, which failed to address stereotypes engendered by the association of multiple objects, referred to as *Association-Engendered Stereotypes*. For example, mentioning  ''black people'' and ''houses''  separately in prompts may not exhibit stereotypes. Nevertheless, when these two objects are associated in prompts, the association of ''black people'' with ''poorer houses'' becomes more pronounced. To tackle this issue, we propose a novel framework, MAS, to Mitigate Association-engendered Stereotypes. This framework models the stereotype problem as a probability distribution alignment problem, aiming to align the stereotype probability distribution of the generated image with the stereotype-free distribution. The MAS framework primarily consists of the *Prompt-Image-Stereotype CLIP* (*PIS CLIP*) and *Sensitive Transformer*. The *PIS CLIP* learns the association between prompts, images, and stereotypes, which can establish the mapping of prompts to stereotypes. The *Sensitive Transformer* produces the sensitive constraints, which guide the stereotyped image distribution to align with the stereotype-free probability distribution. Moreover, recognizing that existing metrics are insufficient for accurately evaluating association-engendered stereotypes, we propose a novel metric, *Stereotype-Distribution-Total-Variation*(*SDTV*), to evaluate stereotypes in T2I. Comprehensive experiments demonstrate that our framework effectively mitigates association-engendered stereotypes.

Main paper: 
 introduction: 
 Text-to-Image (T2I) (Nichol et al. [2021] , Rombach et al. [2022] , Ramesh et al. [2022] , Saharia et al. [2022] ), based on the diffusion model, has achieved significant breakthroughs in image generation, showing potential for various downstream tasks such as image creation, editing (Brooks et al. [2023] ), etc. In addition, the scale of T2I applications has expanded impressively. For example, the opensource stable diffusion project (Rombach et al. [2022] ) has gained the favor of more than 10 million users. As of August 2023, the number of images created through T2I has surpassed 15 billion † .Figure 1 : Original T2I models can generate stereotypes when given prompts involving multiple objects; when the prompts are "a photo of black/white people" or "a photo of a house", they do not engender stereotypes. However, if the prompt is "a photo of black/white people and a house", the model may engender stereotypes that the white people's house is better than the black people's house. It is noted that previous mitigation approaches have yet to mitigate these association-engendered stereotypes. Our approach, MAS, demonstrates the capability to effectively mitigate these stereotypes.As images generated by diffusion models become more prevalent, mitigating the stereotypes they engender is increasingly critical (Seshadri et al. [2023] , Luccioni et al. [2024] ).The stereotype in social psychology is a mental construct of a particular type of person, object, or thing, often based on observing individuals who fit that type. Previous work explored sensitive attributes (such as gender, race, region, religion, and age) and corresponding stereotypes present in T2I diffusion models (Wang et al. [2023a] , Esposito et al. [2023] , Bianchi et al. [2023] ). For instance, when the prompt is "a photo of a politician", the object (politician) in the image generated by the T2I diffusion model always appears male in the sensitive attribute of gender. Previous works (Shen et al. [2024] , Kim et al. [2023] , Esposito et al. [2023] ) on stereotype mitigation have been limited to a single object, referred to as Non-Association-Engendered stereotypes (e.g., only mitigating the stereotype problem in the occupation or gender dimension), and cannot effectively address stereotypes involving the association of multiple objects, referred to as Association-Engendered Stereotypes. For example, as shown in Figure 1 , when the prompt is "a photo of a white people and a house" versus "a photo of a black people and a house", T2I diffusion models often generate images where the houses associated with white people appear better than those associated with black people, reflecting an association-engendered stereotype. Therefore, mitigating such association-engendered stereotypes is highly desired for T2I models.In this paper, our contributions are as follows: ❶ To the best of our knowledge, we take the first step towards addressing association-engendered stereotypes in T2I models. We define the stereotype problem in the T2I diffusion model as a probability distribution alignment problem, aiming to align the stereotype probability distribution of the generated image with the stereotype-free distribution. ❷ We present a novel framework MAS to mitigate the association-engendered stereotypes, which primarily consists of the Prompt-Image-Stereotype CLIP (PIS CLIP) and Sensitive Transformer. The PIS CLIP learns the association between prompts, images, and stereotypes, which can construct the mapping of prompts to stereotypes. The Sensitive Transformer produces the sensitive constraints, which guide the stereotyped image distribution to align with the stereotype-free probability distribution. ❸ Given the insufficiency of existing metrics for accurately evaluating association-engendered stereotypes, we propose a novel metric for evaluating stereotypes in T2I: Stereotype-Distribution-Total-Variation (SDTV). ❹ We conduct a comparative evaluation of our stereotype mitigation approach on five popular T2I diffusion models and six advanced stereotype mitigation approaches in T2I. Extensive experiments demonstrate the superiority of our approach in effectively mitigating both non-association-engendered and association-engendered stereotypes. Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our abstract and introduction accurately reflect the contributions and scope of the paper. In Sections 3 and 4, we provide detailed theoretical explanations and comprehensive experimental validation of the contributions proposed in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Stereotypes in T2I models. Although the current image quality and capabilities generated by T2I are gradually approaching maturity, it is worth noting that various biases and stereotypes may still arise from neutral prompt words (Zhou et al. [2024] ). Fraser et al. [2023] and Wan and Chang [2024] found that text-generated image models often engender stereotypes of gender, race, and demographics in sociology. These biases can manifest in multiple aspects, such as occupations (Bianchi et al. [2023] , Cho et al. [2023] , Luccioni et al. [2023] ), objects (Mannering [2023] ) and adjectives (Luccioni et al. [2023] , Naik and Nushi [2023] ). Additionally, T2I diffusion models may exhibit gender bias when depicting interactions between two or more people. Based on the above findings, previous researchers proposed stereotype detection frameworks such as Language Agent (Wang et al. [2023a] ) and T2IAT detection (Wang et al. [2023b] ). These research results provide practical tools for identifying stereotypes and lay a solid foundation for further mitigating stereotypes.Stereotypes mitigation in T2I models. Existing approaches to mitigate stereotypes in T2I diffusion models have effectively mitigated some of the stereotypes but still have some limitations. Friedrich et al. [2023] randomly introduced additional text clues by identifying known occupations in the prompts to ensure a fair distribution of the generated images. However, this approach may ineffectively address biases related to occupations that are not predefined. Kim et al. [2023] developed a destereotyping loss function and adjusted specific parameters of the soft prompts to balance the sensitive attributes of the generated images but only achieved de-stereotyping in the T2I model containing occupational prompts. Chuang et al. [2023] proposed a technique that involves projecting biased directions in text embeddings and debiasing by utilizing text data with a calibrated projection matrix. Fine-tuning the diffusion model by modifying the capabilities of the pre-trained model has also proven to be an effective strategy. Inspired by Gal et al. [2023] , Zhang et al. [2023] , Brooks et al. [2023] , Dai et al. [2023] , Shen et al. [2024] fine-tuned the sampling process of the diffusion model and used an optimized loss function to align the T2I model with fairness principles directly. Moreover, He et al. [2024] utilized an iterative distribution alignment approach to align biased distributions with a unified distribution. Schramowski et al. [2023] built a Safe Latent Diffusion (SLD) capable of suppressing sensitive image portions during diffusion. Previous researchers adjusted the original T2I model and had to consider the image quality and mitigate stereotypes in the original T2I model. They mainly focused on mitigating non-association-engendered stereotypes in the T2I model (such as those engendered by occupation/gender), so it is unable to mitigate the stereotypes generated when multiple objects are associated effectively.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of the paper in section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
our framework: mitigate association-engendered stereotypes (mas): 
 In this section, we construct a mathematical model of stereotypes in T2I. Based on this model, we propose a novel metric, Stereotype Distribution Total Variation (SDTV), for evaluating stereotypes. Then, we model the stereotype mitigation problem as a probability distribution alignment problem and build a new stereotype mitigation framework, Mitigate Association-engendered Stereotypes (MAS). Finally, we train PIS CLIP and Sensitive Transformer models to construct sensitive constraints, which guide the stereotype probability distribution to align with the stereotype-free probability distribution.Inspired by Shen et al. [2024] , we denote the object of the image as x and the object's sensitive attributes as s (such as gender, race, and adjective). We introduce the probability distribution function (PDF) P (s = v(s)|x) to quantify the likelihood of the image's object exhibiting specific sensitive attributes, where v(s) represents the value of the sensitive attribute. For instance, when the generated image's object x is person and s is gender, v(s) could be male or female. The PDF represents the probability of a person appearing as male or female in the sensitive attribute of gender. Furthermore, to distinguish the difference between association-engendered stereotypes and stereotypes explored by previous work (Kim et al. [2023] , He et al. [2024] , Shen et al. [2024] ), we categorize stereotypes into four distinct categories (Table 1 ) based on the association between objects and sensitive attributes: Here, we utilize a joint PDF to define categories with multiple objects with multiple sensitive attributes.EQUATIONwhere x 1 and x 2 represent the two objects in the image. For instance, in Figure 1 , x 1 corresponds to a people, and x 2 corresponds to a house. The variables s x1 and s x2 denote the race and house's description. See Appendix A for more PDF definitions of other stereotype categories.Stereotypes engender when a sensitive attribute in an image consistently presents one or a few states in most instances (Naik and Nushi [2023] ). Therefore, Saravanan et al. [2023] and Wang et al. [2023a] have utilized various metrics to evaluate the stereotype from the perspective of the overall proportion of different subgroups. However, when evaluating association-engendered stereotypes, the proportion of subgroups in the population cannot accurately evaluate association-engendered stereotypes. To address this shortcoming, we propose a new stereotype evaluation metric using total variation distance based on the probability distribution model of stereotypes in T2I.Consider a T2I model, denoted as G, which generates an image, and x is an object of the image. We define a set S x denotes that object x contains multiple sensitive attributes such that S x = {gneder, race, region, . . . }. We assign the sensitive attributes as v(s x ) 1 , v(s x ) 2 , . . . , v(s x ) Y for each sensitive attribute s x , ∀ s x ∈ S x (e.g. v(s x ) = female, v(s x ) = black, . . . ). For an object (i.e., x ="doctor" or x ="nurse"), the stereotype-free T2I model should generate images with equal probability across different sensitive attribute values and should avoid significant probability distribution disparities caused by the dependence of sensitive attributes on the object generating.To evaluate the extent of stereotypes, we calculate the difference in probability distributions for each sensitive attribute value using the Total Variation distance Chung et al. [1989] :D T V (p θ (s x = v(s x ) i |x), p θ (s x = v(s x ) j |x)). Therefore, based on the PDF definition of stereotype types in Section 3.1, the SDTV value of model G under single-object x with single-sensitive attribute v(s x ) is:EQUATIONIn practice, for a given prompt text τ in T2I models, the sensitive attribute's probability of the generated images under this prompt τ is represented as p θ (s x = v(s x ) i )|x, τ ). For instance, consider the sensitive attribute of gender (s x ="gender") when the input prompt τ is set to "a rich person", the gender-sensitive attributes' probability of the corresponding generated images are considered and set to p θ (s x = v(s x ) i )|x, τ ) (where v(s x ) i represents "male") and p θ (s x = v(s x ) j )|x, τ ) (where v(s x ) j represents "female") respectively. We could incorporate the prompt and sensitive attribute and obtain the following:EQUATION)where θ is the parameter of model G. Given an input prompt τ , p θ (s x = v(s x ) i |x, τ ), and p θ (s x = v(s x ) j |x, τ ) represent the probabilities that the sensitive attribute appears as v(s x ) i and v(s x ) j in the output, where {i, j} ⊆ Y and i ̸ = j. If the value of SDTV(G) is considerable, it suggests that the image exhibits very severe stereotypes. Conversely, suppose its value is close to 0. In that case, it implies that, under the given prompt conditions, the sensitive attributes displayed in the generated image are relatively evenly distributed, indicating no apparent stereotypes in this sensitive attribute dimension. When calculating the SDTV of association-engendered stereotypes, we deal with multiple objects and sensitive attributes simultaneously, so we need to adjust Equation (3). Let's define a set of objects X = {X h , X ′ h }, where X h and X ′ h represent the sets of human and non-human objects, respectively. For a human object x h ∈ X h with sensitive attributes S h , and a non-human object x ′ h ∈ X ′ h with descriptions S ′ h , we can adjust Equation (3) accordingly to obtain:EQUATIONThe sensitive attribute s h and describe s ′ h have Y and Z corresponding values, respectively. See Appendix B for the detailed proof process of the SDTV.Our framework aims to alter the stereotype probability distribution in T2I to align it with a stereotype-free probability distribution. As shown in Figure 2 , we construct MAS to implement this alignment process, which consists of two main network structures, PIS CLIP and Sensitive Transformer. PIS CLIP learns prompts, images, and stereotypes in T2I and constructs mapping of prompts to stereotypes; Sensitive Transformer builds sensitive constraints based on prompts and mapping of prompts to stereotypes. Finally, we embed sensitive constraints into the T2I model by utilizing the Sensitive Transformer, which guides the probability distribution associated with stereotypes toward alignment with the stereotype-free distribution. We approach the solution by answering two questions: ❶ How do we learn the association between prompts, images, and stereotypes? ❷ How do we construct constraints?PIS CLIP. In T2I, stereotypes are typically not in prompts but in images. Therefore, we must learn the stereotype in the generated image corresponding to the prompt, which involves mapping the prompts, images, and stereotype descriptions. Inspired by CLIP (Radford et al. [2021] ), we construct a novel three-dimensional mapping approach to learn the features and relationships of prompt-image stereotypes. We use a T2I-generated image, which contains a stereotype, as a bridge to create three pairs: <prompt, image>, <image, stereotype>, and <prompt, stereotype> for training. PIS CLIP learns the potential mapping relationship between the prompt and the stereotype by maximizing the cosine similarity of these three pairs. We optimize a symmetric cross-entropy loss based on these similarity scores. Algorithm 1 presents the process of the PIS CLIP (See Appendix C.1 for more detailed experimental settings).Sensitive Transformer. In PIS CLIP, we learn and establish a mapping of prompts to stereotypes. Then, based on this learned mapping, we can create sensitive constraints for prompts. When given a prompt, we aim to provide specific sensitive constraints tailored to that prompt. To achieve this, we approach this as a query problem: the aim is to construct the value (V: sensitive constraints) by matching a query (Q: prompts) with keys (K: the mapping of prompts to stereotypes). We implement this task based on the transformer architecture (Vaswani et al. [2017] ).EQUATIONIn Equation ( 5), the input consists of queries and keys of dimension d k . We compute the dot products of the query with all keys, divide each by √ d k , and apply a softmax function to obtain the weights on the values.Sensitive Constraints Guide Distribution Alignment. Consider a T2I diffusion model G, which generates images x with a sensitive attribute s. The attribute s can have Y different categories and needs to align with a target distribution D that is free of stereotypes. We use a prompt to generate a batch of images, denoted asI = {x v(s) i } i∈N . For the PDF of the sensitive attribute of image x, h(x v(s) i ), let h(x v(s) i ) = p v(s) i x = [p v(s) 1 x , p v(s) 2 x , • • • , p v(s) Y x ], i ∈ |Y|, where p v(s) xrepresents the estimated probability of generating images with stereotypes using model G. Assume another set of images, Ĩ = {x u(s) i } i∈N , generated with the same prompt P but without stereotypes. The probability distribution of its sensitive attribute is h(x u(s) i ) = p ui(s) i x = [p u(s) 1 x , p u(s) 2 x , • • • , p u(s) Y x], i ∈ |Y|, and the corresponding probability distribution without stereotypes is denoted as p u(s) x. We compute the distribution distance between p v(s) x and pEQUATIONwhereS Y is the constraint space, S Y = {V (s 1 ), V (s 2 ), . . . , V (s n )} n∈N , V (s) = [v(s) 1 , v(s) 2 , . . . , v(s) Y ],and σ is a set of constraints within this space. The optimal constraint for minimizing the distance between two probability distributions in the constraint space is σ * . The σ contains all possible sensitive attributes S = {s 1 , s 2 , . . . , s n } n∈N and their value corresponding to the prompts. When a sensitive constraint σ is added to the input of the T2I diffusion model, it is combined with the original prompt embedding to form a new input for the T2I sampling process. Since the new input includes the sensitive constraint information, it guides the T2I process to produce a new outputσ(p vi(s) x ) = [p v(s) 1 σ x , p v(s) 2 σ x , • • • , p v(s) Y σ x ]. σ(p vi(s) x) is a distribution that closely approximates the stereotype-free distribution p u(s) x .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In this paper, we discuss the potential societal impacts of our work. Our research effectively mitigates association-engendered stereotypes in T2I, thereby reducing the generation of biased content by generative AI. This provides a strong assurance for the safe development of generative AI. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiment: 
 We conduct experiments on popular T2I diffusion pipelines to evaluate the generalizability of our approach across various T2I diffusion pipelines, which include ❶ runwayml/stable-diffusion-v1-5(SD-1.5) (Rombach et al. [2022] ), ❷ stabilityai/stable-diffusion-xl-base-1.0 (SDXL) (Meng et al. [2021] ), ❸ ByteDance/SDXL-Lightning (Lightning) (Lin et al. [2024] ), ❹ stabilityai/sdxl-turbo (Turbo) (Sauer et al. [2023] ), and ❺ stabilityai/stable-cascade (Cascade) (Pernias et al. [2024] ). All these models are openly accessible from Hugging Face * . As mentioned above, we apply our stereotype mitigation approach to the five mainstream T2I diffusion pipelines. We use the optimal sampler and keep the same hyperparameters setting to generate images for all T2I diffusion pipelines (See Appendix C.2 for detail experiment setting). We create ten prompts for each of the four categories * https://huggingface.co of stereotypes discussed in Section 3.1, generate 100 images per prompt, and calculate attribute probability distributions by utilizing 100 × 10 = 1000 images. In addition, for comparison, we test each pipeline using two sets of data, one from the original pipeline and the other from the pipeline with MAS. We utilize the SDTV value to evaluate the severity of stereotypes in T2I models. For four types of stereotypes, we apply the corresponding SDTV to evaluate the severity of each type (See Appendix B for detailed calculation methods for the SDTVs corresponding to different types of stereotypes).Evaluation of stereotype mitigation effectiveness across different T2I diffusion models. Table 2 reports the performance of our stereotype mitigation approach. Our approach effectively mitigates the association-engendered stereotypes in T2I and maintains excellent performance in mitigating the non-association-engendered stereotypes. Besides, Table 2 shows that the latest T2I models (Lightning, Turbo, and Cascade) display more pronounced stereotypes than their predecessors, SD-1.5 and SD XL. Nevertheless, our approach performs robust generalizability, demonstrating superior performance in mitigating stereotypes across various scenarios, including traditional and the latest T2I models involving non-association-engendered/association-engendered stereotypes (Figure 3 ).Comparative evaluation with different stereotype mitigation approaches. To validate the effectiveness of our approach, we conduct a comparative analysis with state-of-the-art methods. Table 3 shows that our approach outperforms six other solutions, demonstrating fewer stereotypes for all eight single-object and multi-object scenarios. Our approach offers a distinct advantage compared to the prompt fine-tuning approach. We directly learn the relationship between prompts, images, and stereotypes, which enables us to construct more targeted constraints for the stereotypes in T2I rather than relying on broadly sensitive attributes as supplements. As a result, our approach can maintain an effective mitigation strategy even when faced with more complex and concealed stereotypes.Compared to the model fine-tuning approach (Shen et al. [2024] ), our approach only requires integra- tion within the T2I workflow to perform mitigation, eliminating the need to retrain the original T2I model. During the training stage, the fine-tuning mitigation approach must ensure that it cannot affect the original T2I diffusion model's alignment of text to image. Therefore, not all resources can be utilized to learn stereotypes, significantly limiting the learning of more types of stereotypes. However, our model can fully use computing resources to learn more types of stereotypes. This enables our approach to demonstrate a practical mitigation effect in covert stereotype scenarios involving multiple objects. To explore generalization to non-template prompts, we randomly select 100 prompt instances from the diffusionDB (Wang et al. [2022] ) dataset-data that might inadvertently perpetuate stereotypes. Table 5 shows the evaluation results and demonstrates the effectiveness of our approach in mitigating stereotypes. Although we only implement stereotype mitigation for template prompts, the stereotype-mitigation effect also generalizes to more complex nontemplated prompts. We list some images generated with non-template prompts in Appendix F. Impact on T2I diffusion models. Since our approach integrates a Sensitive Transformer model into the original T2I workflow, it could introduce additional overhead to the T2I model or impact the quality of the generated images. To evaluate these potential effects, we conduct a series of experiments focusing on the quality of image generation and the efficiency of the generation process. We set four different batch sizes (10, 20, 50, 100) for the five different T2I diffusion models. We evaluate the impact of using MAS on the quality and efficiency of generated images of original T2I by measuring the Fréchet Inception Distance (FID) (Heusel et al. [2017] ) value of generated images and the time taken to generate images in different batch sizes. Table 6 demonstrates the evaluation results. Our mitigation approach maintains comparable image quality and generation efficiency to the original T2I diffusion models. Compared to other approaches, although our solution adds sensitive constraints to the original T2I diffusion model, it only introduces low additional overhead. Stereotype mitigation experiments in more T2I scenarios. This section aims to evaluate the effectiveness of MAS in mitigating stereotypes in more complex T2I scenarios.In practical T2I generation processes, it is necessary to incorporate models such as LoRA and ControlNet to control the image's style and structure. Consequently, our mitigation approach must be capable of reducing stereotypes in multi-model collaboration T2I scenarios. We conduct stereotype mitigation experiments by combining several mainstream models, including the diffusion-based retraining model Realistic † (R-SD), LoRA models, and ControlNet models (Con). Table 7 demonstrates that our stereotype mitigation approach maintains outstanding performance even in T2I generation scenarios involving multiple model combinations. See Appendix G.1 and Appendix G.2 for detailed descriptions of the experiment and examples of images. 
conclusion: 
 In this paper, we took the first step toward mitigating association-engendered stereotypes in Text-to-Image (T2I) diffusion models. We innovatively modeled stereotypes as a probability distribution alignment problem and constructed a probability distribution model for both non-association-engendered and association-engendered stereotypes. Then, we proposed the Mitigate Association-engendered Stereotypes (MAS) framework for the first time. MAS learned the mapping of prompts, images, and stereotypes and constructed sensitive constraints to guide the T2I diffusion model in generating stereotype-free images by embedding these sensitive constraints into the T2I diffusion process. Additionally, we proposed a novel metric to evaluate stereotypes, Stereotype Distribution Total Variation (SDTV). Finally, comprehensive experiments demonstrated that we contribute an effective mitigation approach for association-engendered stereotypes in T2I, establishing a more ethical and reliable foundation for future text-to-image generation development. 
limitations: 
 This research effectively mitigated both non-association-engendered and association-engendered stereotypes. While it successfully addressed the identified stereotypes, exploring other subtle stereotypes will be an interesting direction for future research. To our knowledge, modeling the problem as aligning existing distributions with target distributions holds significant potential for further extension into research areas such as debiasing and detoxification in large models. Investigating these avenues could provide deeper insights and more comprehensive solutions to ethical issues in generative models. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: To reduce the potential risks of T2I diffusion models generating unsafe images, we report the models, parameter settings, prompts, and other relevant configurations used in our experiments in Appendix C, D, E, F and G. These configurations help ensure the generation of safe images and reduce the risk of producing unsafe content. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All data, models, and baselines used in this paper are open source. We provide citations or footnotes for all sources throughout the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification:We provide the open-source code in the supplemental material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: A.1 Probabilistic model of four types of stereotypesIn our research, we define stereotype as the representation that links objects to sensitive attributes. Specifically, for generated images, we denote the object as x, the subject's sensitive attribute as S x , and the attribute value of s as v(s x ). Based on this foundation, we introduce the PDF P (s x = v(s x )) to describe the likelihood of an image subject displaying specific sensitive attributes. In traditional psychology, a stereotype refers to people's overall view or concept of a particular person, thing, or object. This understanding often arises from individual observations and perceptions within that category. However, defining their stereotypes in artificial intelligence, especially for unconscious systems like AI models, becomes complex and challenging. The lack of clear definitions poses a significant challenge for our study. Therefore, our work interprets stereotypes within the T2I model from a probabilistic perspective. By carefully examining different stereotypes, we categorize them into four distinct types:• Single Object with a Single Sensitive Attribute• Single Object with Multiple Sensitive Attributes• Multiple Objects with a Single Sensitive AttributeWe utilize random conditional probability distributions for single object with single sensitive attribute to describe their behavior.EQUATIONThe formula 7 represents the probability density distribution of the attribute value v(s x ) of the object x concerning the sensitive attribute s x . For instance, consider the prompt phrase "a photo of a nurse".In this case, the object depicted in the image is a nurse. If we pay attention to the sensitive attribute of gender, the object in the picture always displays "female" gender-sensitive attribute values (v(s x )).We utilize multiple dimension-sensitive attributes of a single object by using probability distributions under various conditional constraints:EQUATIONi.e. still taking the prompt as "a photo of a nurse" as an example, but this time, in formula 8, we focus on sensitive attributes such as "s 1 x = gender", "s 2 x = race", then the image always shows "v(s 1 x ) = female", " v(s 2x ) = white race" sensitive attribute value. For generated images of multiple objects (x 1 , x 2 ), we use a joint probability density distribution to describe the stereotypes of multiple objects.EQUATIONThe formula 9 describes how multiple objects within the image collectively exhibit the same sensitive attributes. However, these attributes consistently manifest distinct stereotypes due to the presence of different objects. For instance, consider the prompt "a photo of a doctor and a nurse." When the object is a "x 1 = doctor," it invariably appears as "v(s) = male". Conversely, when the object is a "x 2 = nurse", it consistently portrays a "v(s) = female", thus reflecting gender stereotypes.Likewise, we utilize the joint probability density function:EQUATIONThe formula 10 represents the stereotype probability distribution across multiple objects and multiple sensitive attributes. i.e., when the prompt is "a photo of a black man/white man and his house", a common stereotype emerges: the "white man's" house tends to be portrayed as more better than the "black man's" house. This stereotype frequently arises when multiple objects and sensitive attributes intersect. Prompts: "a photo of a engineer." "a photo of a driver."For the engineer and driver objects in the figures, there are stereotypes that gender is always male and race is always white people.Only one sensitive attribute can be mitigated at once.Prompts: "a photo of a farmer." "a photo of a CEO."For the farmer and CEO objects in the figures, there are stereotypes that gender is always male and race is always white people.Multiple sensitive attributes can be mitigated simultaneously.Nonassociationengendered stereotypesPrompts: "a photo of a boss and a employee." "a photo of a professor and a teacher."For these images, there are stereotypes that white people always have a higher status than other races.Prompts: "a photo of a nurse and a doctor." "a photo of a manager and a secretary."For these images, there is always a stereotype that men have a higher status than women and are always present as white people. B SDTV Main Proof.In section 3.2, we have explained the variables in the SDTV formula. The single-object with singlesensitive attribute proof process is as follows:SDT V (G) = max {i,j}⊆Y D T V p θ (s x = v(s x ) i |x), p θ (s x = v(s x ) j |x) = max {i,j}⊆Y D T V p θ (s x = v(s x ) i |x, τ )dτ -p θ (s x = v(s x ) j |x, τ )dτ = max {i,j}⊆Y D T V E p(sx=v(sx) i |τ ) p θ (s x = v(s x ) i |x, τ ) -E p(sx=v(sx) j |τ ) p θ (s x = v(s x ) j |x, τ ) = max {i,j}⊆Y E p(sx=v(sx) i |τ ) p θ (s x = v(s x ) i |x, τ ) -E p(sx=v(sx) j |τ ) p θ (s x = v(s x ) j |x, τ ) = max {i,j}⊆Y p θ (τ v(sx) i |x) -p θ (x|τ v(sx) j |x)Based on the above proof, we consider a sensitive attribute set S x = {gneder, race, region, . . . }, which contains different attributes. We evaluate the extent of single-object with multi-sensitive attribute stereotypes by calculating the average SDTV values of all attributes.SDT V (G) = E |Sx| sx∈Sx max {i,j}⊆Y D T V p θ (s x = v(s x ) i |x, τ ), p θ (s x = v(s x ) j |x, τ ) = E |Sx| sx∈Sx max {i,j}⊆Y p θ (s x = v(s x ) i |x, τ ) -p θ (s x = v(s x ) j |x, τ )Similarly, we expand the object set X = {x 1 , x 2 , ..., x n } n∈N , where the set X contains multiple different objects, and calculate the average SDTV of all objects to evaluate the multi-objects with single-sensitive attribute stereotype extent of the image:SDT V (G) = E |X| X x max {i,j}⊆Y D T V p θ (s x = v(s x ) i |x), p θ (s x = v(s x ) j |x) = E |X| X x max {i,j}⊆Y p θ (s x = v(s x ) i |x, τ ) -p θ (s x = v(s x ) j |x, τ )Based on the above process, we simultaneously expand the object and attribute dimensions. Let's define a set of objects X = {X h , X ′ h }, where X h and X ′ h represent the sets of human and non-human objects, respectively. For a human object x h ∈ X h with sensitive attributes S h , and a non-human object x ′ h ∈ X h with descriptions S ′ h , we can adjust Equation (3) accordingly to obtain:SDT V (G) = E s h ∈S h s ′ h ∈S ′ h max {i,j}⊆Y, m∈Z p θ (s h = v(s h ) i |s ′ h = v(s ′ h ) m , τ ) -p θ (s h = v(s h ) j |s ′ h = v(s ′ h ) m , τ ) , B.2 Why is it the maximum and not the average?We compute the distances between the two distributions using the total variation distance (D T V ). The evaluation typically utilizes the average in the study by Dwork et al. [2012] and Yang et al. [2022] . However, our research has revealed that the pair of average values do not accurately depict stereotypes in T2I. Therefore, we will conduct further discussions and analyses on this issue. Ultimately, we demonstrate that utilizing the maximum total variation distance can effectively characterize the extent of stereotypes in T2I. Moreover, we can extend the maximum total variation distance to other studies. , where the D T V value for the white and brown categories is assumed to be the highest at 0.9. In contrast, it falls between 0.2 and 0.4 for other racial categories.When we calculate the average, the SDTV value of model G is roughly 0.3. Model G has significantly reduced its extent of stereotypes in the attribute dimension of race by almost three times. However, the SDTV value, calculated using the maximum value, is 0.9. This value indicates that the model still has a clear preference for one of the sensitive attribute values related to the sensitive attribute of race, thereby displaying severe stereotypes.Another situation is that the high D T V of some sensitive attribute values maintains a relatively balanced state. As shown in Figure 5 (b), at this time, using the mean SDTV value to describe the stereotype extent of model G will still lower the extent of sensitive attributes of the model, and at this time, SDTV will remain at a low level. Thus, we mistakenly believe that model G does not exhibit stereotypes.Example. We analyze how model G stereotypes are based on the attribute of race (s = race). Among them, the possible values of the sensitive attribute s are v(s) = {white, black, yellow, brown,...}. We consider a balanced case, as shown in Figure 5 (b), to calculate several races. When looking at the D T V , it can be seen that the high D T V value of model G remains in the range of 0.5-0.6, while other D T V are around 0.2. When using the average, the SDTV value of model G equals approximately 0.3. The model does not show a very obvious stereotype. On the contrary, using the maximum value can show that the model still has stereotypes.Some other thoughts from the perspective of mitigation: when we use the maximum value D T V as a description, at the exact moment, we only need to pay attention to the attribute value corresponding to the most severe stereotype under the current SDTV performance. At this time, when facing the extreme situation in Figure 5 (a), we focus on mitigating the stereotype under this attribute value, which can significantly alleviate the T2I model stereotype; when facing the situation in Figure 5 (b), When we mitigate the maximum D T V , the SDTV will still show the maximum D T V among the remaining attribute values, and then continue to mitigate the maximum to achieve the step-by-step mitigation of stereotypes in the T2I model. On the contrary, when we use mean SDTV as a description, we will disperse the focus of mitigation to all possible attribute values, and the effect of dispersed mitigation will not be able to solve the stereotype problem effectively.association-engendered stereotypes and generated 100 images for each prompt. We calculate the SDTV value of each T2I model based on 10 × 100 = 1000 images. In Tables 2 ∼ 7, the ".XX/.XX" represents the average value of 1000 images, and "±.XX/±.XX" denotes the error bounds. Additionally, Tables 2∼7 illustrate that the greater the stability of the T2I diffusion model, the smaller the error bounds. All evaluation experiments are performed on a single NVIDIA A100-80GB GPU. Prompts: best quality,masterpiece,realistic,absurdres,front view,concert, violinist, (female:1.2), solo, standing, <lora:more_details:0.67> city, street, downtown, alleyway, lawn, bench, skyscraper, park, main street, magnificent architecture, cathedral, castle, turret, bridge, ruins, remains, rubble ruins, pasture, plateau, garden, fields, flower ocean, cafe, castle, concert, stage, harbor, casino, dungeon, grave, auditorium] In this paper, we use the following prompt template for image generation to evaluate the effectiveness of the MAS method in mitigating stereotypes: NeurIPS Paper Checklist† The most downloaded retraining stable diffusion model on the Civitai: https://civitai.com/models/4201 


Decision: The paper studies the problem of association biases in text to image generation that occurs when the text prompt includes both people and objects. All of the reviewers found the paper proposed a novel method for an important problem and the evaluation was solid.

Reviews: review: paper_summary: paper_summary: The paper proposes a framework to detect and mitigate stereotype association in Text-to-Image models. They conduct extensive experiments to demonstrate the usability of the framework.
strength_weakness: strength_weakness: - I strongly recommend renaming the acronym for the Text-Image-Text CLIP model. The current acronym 'TIT-CLIP' is inappropriate, particularly given the project's focus on detecting stereotypes in Text-to-Image models. A more suitable name that reflects the project's serious and sensitive nature would be advisable. 

- The claim that the default representation of different identity groups (e.g., 'black' and 'white' people) is not stereotypical is false (Lines 8-10, Figure 1). Research has demonstrated that the default representations of identities are indeed stereotypical, and default representations of objects can also exhibit stereotypical characteristics [1, 2, 3, 4].

[1] Easily Accessible Text-to Image Generation Amplifies Demographic Stereotypes at Large Scale. Bianchi et al. 2023 \
[2] ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation, Jha et al., 2024 \
[3] Stable bias: Evaluating Societal Representations in Diffusion Models, Luccioni et al. 2023 \
[4] AI’s Regimes of Representation: A Community-Centered Study of Text-to-Image Models in South Asia. Qadri et al., 2023 \
questions: questions: - Since the default representation of the images can also be biased, did the authors study how much of the bias stems from the isolated representations of identity groups and objects vs the two combined? 
- Can authors provide more details on the dataset used for training and evaluation and the grounding used for evaluating the presence of stereotypes before and after mitigation?
limitations: limitations: Missing limitations and discussion section.
paper_summary: paper_summary: This paper presented the first step to mitigate association-engendered stereotypes in Text-to-Image (T2I) diffusion models. A probability distribution alignment problem was first formulated, and then a probability distribution model was constructed for non-association-engendered and association-engendered stereotypes. This paper further presented a MAS framework, which consists of the Text-Image-Text CLIP (TIT CLIP) and Sensitive Transformer. Comprehensive experiments demonstrated that the MAS framework is an effective mitigation approach for association-engendered stereotypes in T2I.
strength_weakness: strength_weakness: - In line 167, for the Algorithm 1, what is the output?  Please clarify. 

- More output examples of the proposed framework MAS and baseline methods should be shown to illustrate the mitigation effects.

- While the paper provides an Appendix for detailed experimental settings, some implementation details may be needed in Section 4.
questions: questions: Q1: In line 167, for the Algorithm 1, what is the output?  Please clarify. 

Q2: More output examples of the proposed framework MAS and baseline methods should be shown to illustrate the mitigation effects.

Q3: In Appendix B2 Figure 5(a), what does "extreme" mean in this context?
limitations: limitations: - Please clarify the "stereotype-free distribution", which is not very clear in this paper. 

- While the paper provides an Appendix for detailed experimental settings, some implementation details may be needed in Section 4.
paper_summary: paper_summary: This paper presents a novel framework (MAS) to address biases in text-to-image (T2I) models. Traditional methods focus on individual object stereotypes but fail to tackle stereotypes arising from object associations. MAS models the stereotype issue as a probability distribution alignment problem, utilizing a Text-Image-Text CLIP (TIT CLIP) and a Sensitive Transformer to align generated image distributions with stereotype-free distributions. The paper also introduces the Stereotype-Distribution-Total-Variation (SDTV) metric for better stereotype evaluation. Experiments show MAS effectively mitigates both single and association-engendered stereotypes.
strength_weakness: strength_weakness: 1. The proposed framework adds considerable complexity to the existing T2I pipeline, which might pose challenges for practical implementation and integration into existing systems, especially for developers with limited resources or technical expertise.
2.  It is advisable to provide more details on the computational overhead introduced by the Sensitive Transformer within the MAS framework, especially in terms of real-world application performance and scalability for large-scale T2I generation tasks.
3. I wonder if the MAS framework can be adapted to other generative models beyond text-to-image, such as text-to-video or text-to-audio models.
questions: questions: Please see the weaknesses.
limitations: limitations: Please see the weaknesses.
paper_summary: paper_summary: The authors observe that some stereotypes only appear when an association of objects is required in T2I and propose to address association-engendered stereotypes for the first time. They use a pre-trained Text-Image-Text CLIP to map a prompt to its potential stereotypes and generate sensitive constraints using a Sensitive Transformer. The sensitive constraints are then used with the input prompt to affect T2I generation to be fairer. Besides, a new metric to evaluate association-engendered stereotypes in T2I results is proposed.
strength_weakness: strength_weakness: - I am wondering why the TIT CLIP can learn the mapping from prompts to stereotypes by maximizing the cosine similarity of three pairs. It is intuitively understandable that the original 2-dimensional CLIP uses contrastive learning to associate visual appearance and textual description of a single image. However, in this case, the stereotypes are something unrevealed in a single image but spotted by a statistical distribution over a set of images. The validity of pulling <prompt, stereotype> pair closer should be further explained. Besides, it would help the reader to understand the concept better if the authors could provide some examples of the stereotype descriptions used for training the TIT CLIP.
- Technical details of training the Sensitive Transformer is missing. Is it trained together with the diffusion model with the distribution alignment guidance or what is the training objective of it?
- How does the Sensitive Transformer handle engendered stereotypes across multiple attributes? Is it achieved implicitly or explicitly? Please provide further explanation.
- How to get the probability distribution for evaluation is unclear. Does it involve using pre-trained sensitive attribute classifiers like in previous works? If so, I am wondering how accurate the classifiers are when there are multiple objects in a given image.
- It would be beneficial to provide ablation studies to showcase the effectiveness of each proposed component. For example, I am curious about how a simple MLP would work in mapping the prompts to their stereotypes, compared to the TIT CLIP.
- It is visually hard to tell that the race bias is reduced in Figure 3(b), leading to concerns that the method cannot effectively mitigate biases that are strongly bonded to certain races. 
- Examples after stereotype mitigation in G.2 show violation to prompt requirement "female", leading to concerns that the method might overlook the prompt input. 
- One can hardly tell which person is "poor" from some examples in Figure 12. It raises concerns about how to distinguish harmful stereotypes (like relating poverty to certain races) from desirable visual concepts (like relating poverty to affordable clothes) during generations. 
- The authors do not provide a discussion on the limitation of the method and its societal impact although claim to do so in the checklist. 
- The stereotype mitigation target is even distribution. How can the model adapt to certain circumstances where not all attribute classes are correct? For example, the model should not generate Asian figures when asked for German soldiers in WW2.
questions: questions: See Weaknesses for major questions.

Other minor issues include:
- L41: I think the explanation in brackets is unrelated to its previous statement. Stereotype mitigation on a single object is different from stereotype mitigation on a single sensitive attribute.
- Why there are [SA] indications in some prompt templates? If the SA is given in the prompt, the evaluation of stereotype mitigation across different SA classes is rather trivial.
- How are the sensitive constraints incorporated into the prompt input? Maybe the authors could consider redrawing Figure 2 (3) for a clearer illustration.
limitations: limitations: The authors fail to provide a discussion on limitations and potential societal impact.

Comments: title: Response to Rebuttal
comment: Thank you for your detailed rebuttal and the effort you have put into addressing the concerns raised. Your responses are thoughtful and show a clear understanding of the issues at hand. Based on your rebuttal, I am pleased to inform you that I am increasing my score for your submission.
Thank the authors for their carefully prepared rebuttal. It addresses most of my concerns. I have raised my rating to 5.
Thank you for addressing my concerns. I decide to increase the Rating to 6.