Title: Association of Objects May Engender Stereotypes: Mitigating Association-Engendered Stereotypes in Text-to-Image Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Stereotypes

Diffusion Model

Text-to-Image
Abstract: Text-to-Image (T2I) has witnessed significant advancements, demonstrating superior performance for various generative tasks. However, the presence of stereotypes in T2I introduces harmful biases that require urgent attention as the T2I 
 technology becomes more prominent.
Previous work for stereotype mitigation mainly concentrated on mitigating stereotypes engendered with individual objects within images, which failed to address stereotypes engendered by the association of multiple objects, referred to as *Association-Engendered Stereotypes*. For example, mentioning  ''black people'' and ''houses''  separately in prompts may not exhibit stereotypes. Nevertheless, when these two objects are associated in prompts, the association of ''black people'' with ''poorer houses'' becomes more pronounced. To tackle this issue, we propose a novel framework, MAS, to Mitigate Association-engendered Stereotypes. This framework models the stereotype problem as a probability distribution alignment problem, aiming to align the stereotype probability distribution of the generated image with the stereotype-free distribution. The MAS framework primarily consists of the *Prompt-Image-Stereotype CLIP* (*PIS CLIP*) and *Sensitive Transformer*. The *PIS CLIP* learns the association between prompts, images, and stereotypes, which can establish the mapping of prompts to stereotypes. The *Sensitive Transformer* produces the sensitive constraints, which guide the stereotyped image distribution to align with the stereotype-free probability distribution. Moreover, recognizing that existing metrics are insufficient for accurately evaluating association-engendered stereotypes, we propose a novel metric, *Stereotype-Distribution-Total-Variation*(*SDTV*), to evaluate stereotypes in T2I. Comprehensive experiments demonstrate that our framework effectively mitigates association-engendered stereotypes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors observe that some stereotypes only appear when an association of objects is required in T2I and propose to address association-engendered stereotypes for the first time. They use a pre-trained Text-Image-Text CLIP to map a prompt to its potential stereotypes and generate sensitive constraints using a Sensitive Transformer. The sensitive constraints are then used with the input prompt to affect T2I generation to be fairer. Besides, a new metric to evaluate association-engendered stereotypes in T2I results is proposed.\n\nSTRENGTHS:\n- The authors make direct observations of association-engendered stereotypes in T2I results. From these observations, the motivation to address association-engendered stereotypes is validated.\n- The idea of learning mappings from the prompts to their corresponding stereotypes is interesting and the overall framework is a novel design.\n- The novelly-proposed evaluation metric for association-engendered stereotypes with extensive proof and discussion is valuable.\n- Extensive experiments are conducted with structural evaluations, proving the effectiveness of the method.\n\nWEAKNESSES:\n- I am wondering why the TIT CLIP can learn the mapping from prompts to stereotypes by maximizing the cosine similarity of three pairs. It is intuitively understandable that the original 2-dimensional CLIP uses contrastive learning to associate visual appearance and textual description of a single image. However, in this case, the stereotypes are something unrevealed in a single image but spotted by a statistical distribution over a set of images. The validity of pulling <prompt, stereotype> pair closer should be further explained. Besides, it would help the reader to understand the concept better if the authors could provide some examples of the stereotype descriptions used for training the TIT CLIP.\n- Technical details of training the Sensitive Transformer is missing. Is it trained together with the diffusion model with the distribution alignment guidance or what is the training objective of it?\n- How does the Sensitive Transformer handle engendered stereotypes across multiple attributes? Is it achieved implicitly or explicitly? Please provide further explanation.\n- How to get the probability distribution for evaluation is unclear. Does it involve using pre-trained sensitive attribute classifiers like in previous works? If so, I am wondering how accurate the classifiers are when there are multiple objects in a given image.\n- It would be beneficial to provide ablation studies to showcase the effectiveness of each proposed component. For example, I am curious about how a simple MLP would work in mapping the prompts to their stereotypes, compared to the TIT CLIP.\n- It is visually hard to tell that the race bias is reduced in Figure 3(b), leading to concerns that the method cannot effectively mitigate biases that are strongly bonded to certain races. \n- Examples after stereotype mitigation in G.2 show violation to prompt requirement \"female\", leading to concerns that the method might overlook the prompt input. \n- One can hardly tell which person is \"poor\" from some examples in Figure 12. It raises concerns about how to distinguish harmful stereotypes (like relating poverty to certain races) from desirable visual concepts (like relating poverty to affordable clothes) during generations. \n- The authors do not provide a discussion on the limitation of the method and its societal impact although claim to do so in the checklist. \n- The stereotype mitigation target is even distribution. How can the model adapt to certain circumstances where not all attribute classes are correct? For example, the model should not generate Asian figures when asked for German soldiers in WW2.\n\nQUESTIONS:\nSee Weaknesses for major questions.\n\nOther minor issues include:\n- L41: I think the explanation in brackets is unrelated to its previous statement. Stereotype mitigation on a single object is different from stereotype mitigation on a single sensitive attribute.\n- Why there are [SA] indications in some prompt templates? If the SA is given in the prompt, the evaluation of stereotype mitigation across different SA classes is rather trivial.\n- How are the sensitive constraints incorporated into the prompt input? Maybe the authors could consider redrawing Figure 2 (3) for a clearer illustration.\n\nLIMITATIONS:\nThe authors fail to provide a discussion on limitations and potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework (MAS) to address biases in text-to-image (T2I) models. Traditional methods focus on individual object stereotypes but fail to tackle stereotypes arising from object associations. MAS models the stereotype issue as a probability distribution alignment problem, utilizing a Text-Image-Text CLIP (TIT CLIP) and a Sensitive Transformer to align generated image distributions with stereotype-free distributions. The paper also introduces the Stereotype-Distribution-Total-Variation (SDTV) metric for better stereotype evaluation. Experiments show MAS effectively mitigates both single and association-engendered stereotypes.\n\nSTRENGTHS:\n1. The paper addresses a relatively unexplored area in stereotype mitigation for T2I models, focusing on the association of multiple objects rather than individual objects.\n2. The paper is well-structured, with a clear presentation of the problem, methodology, and results. The introduction provides a solid context for the need to address association-engendered stereotypes, and the explanation of the MAS framework is easy to follow.\n3. The paper extends to improving the societal and ethical aspects of AI-generated content, making it a valuable contribution to the field.\n\nWEAKNESSES:\n1. The proposed framework adds considerable complexity to the existing T2I pipeline, which might pose challenges for practical implementation and integration into existing systems, especially for developers with limited resources or technical expertise.\n2.  It is advisable to provide more details on the computational overhead introduced by the Sensitive Transformer within the MAS framework, especially in terms of real-world application performance and scalability for large-scale T2I generation tasks.\n3. I wonder if the MAS framework can be adapted to other generative models beyond text-to-image, such as text-to-video or text-to-audio models.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presented the first step to mitigate association-engendered stereotypes in Text-to-Image (T2I) diffusion models. A probability distribution alignment problem was first formulated, and then a probability distribution model was constructed for non-association-engendered and association-engendered stereotypes. This paper further presented a MAS framework, which consists of the Text-Image-Text CLIP (TIT CLIP) and Sensitive Transformer. Comprehensive experiments demonstrated that the MAS framework is an effective mitigation approach for association-engendered stereotypes in T2I.\n\nSTRENGTHS:\n+  A novel and important research problem that tackles the stereotypes engendered by the association of multiple objects, which was ignored by previous work. This research adds critical insights into the stereotype mitigation area.  \n\n+ A neat framework MAS was proposed to mitigate such association-engendered stereotypes. MAS innovatively models the stereotype problem as a probability distribution alignment problem, which means aligning the stereotype probability distribution of the generated image with the stereotype-free distribution.  \n\n+ Two effective components, TIT CLIP and Sensitive Transformer, were proposed to enable MAS.  The framework MAS learns the mapping of prompts, images, and stereotypes via the TIT CLIP and constructs sensitive constraints via the Sensitive Transformer to guide the T2I diffusion model in generating stereotype-free images by embedding these sensitive constraints into the T2I diffusion process.\n\n+ A novel metric, Stereotype-Distribution-Total-Variation (SDTV), was introduced to evaluate association-engendered stereotypes accurately due to the insufficiency of existing metrics. \n\n+ Extensive experiments were conducted, supported by the 13 Appendix pages.\n\nWEAKNESSES:\n- In line 167, for the Algorithm 1, what is the output?  Please clarify. \n\n- More output examples of the proposed framework MAS and baseline methods should be shown to illustrate the mitigation effects.\n\n- While the paper provides an Appendix for detailed experimental settings, some implementation details may be needed in Section 4.\n\nQUESTIONS:\nQ1: In line 167, for the Algorithm 1, what is the output?  Please clarify. \n\nQ2: More output examples of the proposed framework MAS and baseline methods should be shown to illustrate the mitigation effects.\n\nQ3: In Appendix B2 Figure 5(a), what does \"extreme\" mean in this context?\n\nLIMITATIONS:\n- Please clarify the \"stereotype-free distribution\", which is not very clear in this paper. \n\n- While the paper provides an Appendix for detailed experimental settings, some implementation details may be needed in Section 4.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework to detect and mitigate stereotype association in Text-to-Image models. They conduct extensive experiments to demonstrate the usability of the framework.\n\nSTRENGTHS:\n+ The authors aim to assess \"association-engendered\" stereotypes in T2I models. They model the stereotype mitigation problem as a probability distribution alignment problem and propose MAS framework to mitigate the association-engendered stereotypes.\n+ The authors conduct extensive experiments to demonstrate the effectiveness of the framework.\n\nWEAKNESSES:\n- I strongly recommend renaming the acronym for the Text-Image-Text CLIP model. The current acronym 'TIT-CLIP' is inappropriate, particularly given the project's focus on detecting stereotypes in Text-to-Image models. A more suitable name that reflects the project's serious and sensitive nature would be advisable. \n\n- The claim that the default representation of different identity groups (e.g., 'black' and 'white' people) is not stereotypical is false (Lines 8-10, Figure 1). Research has demonstrated that the default representations of identities are indeed stereotypical, and default representations of objects can also exhibit stereotypical characteristics [1, 2, 3, 4].\n\n[1] Easily Accessible Text-to Image Generation Amplifies Demographic Stereotypes at Large Scale. Bianchi et al. 2023 \\\n[2] ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation, Jha et al., 2024 \\\n[3] Stable bias: Evaluating Societal Representations in Diffusion Models, Luccioni et al. 2023 \\\n[4] AI’s Regimes of Representation: A Community-Centered Study of Text-to-Image Models in South Asia. Qadri et al., 2023 \\\n\nQUESTIONS:\n- Since the default representation of the images can also be biased, did the authors study how much of the bias stems from the isolated representations of identity groups and objects vs the two combined? \n- Can authors provide more details on the dataset used for training and evaluation and the grounding used for evaluating the presence of stereotypes before and after mitigation?\n\nLIMITATIONS:\nMissing limitations and discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}