Title: A Large Scale Analysis of Gender Biases in Text-to-Image Generative Models
Venue: NeurIPS 2025
Accepted: no
Keywords: gender bias

social bias

text-to-image generation
Abstract: With the increasing use of image generation technology, understanding its social biases, including gender bias, is essential. This paper presents a large-scale study on gender bias in text-to-image (T2I) models, focusing on everyday situations. While previous research has examined biases in occupations, we extend this analysis to gender associations in daily activities, objects, and contexts. We create a dataset of 3,217 gender-neutral prompts and generate 200 images per prompt from five leading T2I models. We automatically detect the perceived gender of people in the generated images and filter out images with no person or multiple people of different genders, leaving 2,293,295 images. To enable a broad analysis of gender bias in T2I models, we group prompts into semantically similar concepts and calculate the proportion of male- and female-gendered images for each prompt. Our analysis shows that T2I models reinforce traditional gender roles, and reflect common gender stereotypes in household roles. Women are predominantly portrayed in care- and human-centered scenarios, and men in technical or physical labor scenarios. Code and prompts to evaluate models will be released upon acceptance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of the paper perform a large-scale analysis of gender biases in T2I generation. They focus on 5 state-of-the-art models and they perform such analysis by considering more than 3000 prompts about activities, contexts, objects and occupations. Overall, they consider a set of more than 2 million images. Through their analyses, they show that T2I reinforce gender stereotypes, and particularly the association between women and household roles.\n\nSTRENGTHS AND WEAKNESSES:\nThe presented experiments are extensive and technically sound. The claims of the authors are well supported by the obtained results. In addition, the authors provide a profound analysis of the limitation of this work, and it is particularly positive how they mention, multiple times, the utilization of binary genders as a limitation, as well as the issues related to inferring gender on images. The work is written clearly and the structure is easy to follow. The extensive literature review shows that the authors have a profound knowledge of the related work and have done a notable effort in showing (in the last part of the appendix) how they compare to major existing publications in this field. The obtained results are significant and interesting. However, while the effort of providing a larger-scale evaluation, as well as the focus on more dimensions (rather than just occupations) is notable, the obtained results are not groundbreaking. In this sense, the paper, while being strong, sound and interesting, is not particularly innovative and does not present unique or creative ideas. That being said, this weakness does not constitute a reason for rejecting the paper. The field of T2I generation advances very fast and a large-scale evaluation as the one provided in this paper, while not being groundbreaking, still provides interesting and more nuanced insights on the different dimensions (beyond occupation) where gender biases can be evaluated.\n\nQUESTIONS:\nMy current evaluation is 5 (accept). I would not go for a 6 (strong accept) even after the rebuttal because I do appreciate this paper and the effort put by the authors, but I do not believe that this is a groundbreaking piece of work. Nevertheless, here are some minor suggestions:\n\n1. Lines 224-225 are a bit confusing, since it is not clear why the parent-son prompts are not gender-neutralized (e.g., parent-child).\n2. For clarity reasons, it would be great if the authors could briefly explain the difference between \"bias analysis\" and \"open bias detection\" at the beginning of Section 2.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations of their work extensively. In the part in which they mention the utilization of generated images for training other models, they might consider adding some reflections on the impact, considering the results shared in the following paper:\n\nChen, T., Hirota, Y., Otani, M., Garcia, N., & Nakashima, Y. (2024). Would Deep Generative Models Amplify Bias in Future Models?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10833-10843).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an in-depth and comprehensive analysis of gender bias in text-to-image (TTI) models. While prior studies have explored gender bias in highly limited scenarios using small sets of prompts, this work investigates biases across a wide range of real-world contexts, including everyday activities, occupations, objects, and broader situational prompts.\n\nIn their experimental setup, the authors generate images from each prompt using five widely adopted TTI models. The generated images are first filtered based on whether they contain a person, after which an automated binary gender classification is performed using an object detector and a multimodal large language model (MLLM).\n\nThe results reveal the presence of gender bias in specific categories, with further evidence showing that male-oriented prompts in the dataset not only exhibit bias but that these biases are amplified by the TTI models.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n**1. The first large-scale, in-depth exploration of gender bias in recent TTI models**\n\nThis paper presents the first large-scale, systematic analysis of gender bias in TTI models, using a diverse set of prompts derived from real-world scenarios. Compared to previous studies that focused on narrow prompt sets, this work's comprehensive design—covering four distinct categories—provides a significant advantage for investigating bias in TTI models. The large-scale nature of the study adds credibility to the findings, providing stronger empirical support for the existence of gender bias in pre-trained TTI models.\n\n**2. Fully automated framework for large-scale gender bias evaluation**\n\nThe authors propose a fully automated framework to evaluate gender bias in TTI models at scale. This framework eliminates the need for human intervention, making it a highly practical and reproducible tool for bias assessment.\n\n### **Weaknesses**\n\n**1. Impact of Classifier-Free Guidance (CFG) on gender bias**\n\nClassifier-Free Guidance (CFG) is known to enhance the quality of generated images by sacrificing diversity, effectively narrowing the sampling distribution toward high-quality, representative images aligned with the text prompt. As noted in the appendix, all models except Flux-Schnell employ CFG. This raises concerns that CFG may unintentionally amplify gender bias by favoring stereotypical or dominant gender representations within specific categories. The potential role of CFG in reinforcing or amplifying gender bias warrants further investigation, ideally through controlled ablation studies.\n\n**2. Ambiguity in the source of observed gender biases — prompt design or model bias?**\n\nIn the Bias Amplification in Activities experiment (Table 3), it is unclear why male-majority and female-majority groups exhibit opposite patterns. The current explanation, attributing these results to inherent male-gender bias in the TTI models, seems insufficient. One possible alternative is that the seemingly neutral prefixes (e.g., “a person”) used in the prompts may themselves introduce unintended gender bias. While such prefixes appear gender-neutral from a human perspective, they may encode bias when processed by TTI models. It is recommended that the authors explicitly investigate potential gender bias in prompt prefixes through controlled experiments. This would strengthen the fairness and reliability of the overall bias evaluation framework.\n\nMoreover, if the model itself exhibits an inherent male-gender bias, some of the results shown in Figure 2 may primarily reflect model-level bias rather than category-specific biases. This possibility casts doubt on the interpretation of male-dominated outcomes reported in Sections 4.1 and 4.2. A clear justification distinguishing between model-inherent bias and category-specific biases is necessary to ensure a fair and accurate analysis.\n\nQUESTIONS:\nIt would be more informative if this figure were separated by model and provided in the supplementary material. For instance, in the Object Categories analysis, it is difficult to accurately discern the effects specific to Flux-Schnell, making model-wise comparisons less clear. Presenting model-specific results would facilitate a more precise understanding of the bias patterns across different TTI models.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates social biases in image generation models by moving beyond prior analyses that primarily focused on limited prompt templates and narrow gender-occupation associations. Instead, the authors conduct a large-scale analysis across a broader range of bias dimensions—including activity, context, object, and occupation—using a diverse set of prompts. The study covers several recent state-of-the-art image generation models and examines which wordings exhibit a majority bias toward male or female representations.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The authors rightly identify a key limitation in prior work: many earlier studies rely on narrow prompt sets when analyzing gender-occupation bias. In contrast, this work conducts a broader, large-scale investigation.\n- The paper provides a comparative analysis across multiple state-of-the-art image generation models, offering valuable insight into how widespread the issue is.\n- The authors improve the trustworthiness of their findings by introducing filtering techniques to ensure cleaner and more meaningful prompt-image pairings.\n\n## Weaknesses\n- The paper primarily presents an empirical analysis but falls short of delivering a strong message beyond confirming the existence of known biases. Since the presence of gender bias in generative models is already well recognized, the paper would benefit from articulating what new insights are derived from the analysis. Furthermore, it would improve the completeness of the work to include recommendations or implications for future bias mitigation strategies based on the findings.\n- Given the analysis spans multiple generative models, a discussion on model-wise differences would be valuable. Are some models more biased than others? What factors (e.g., model size, training objective) contribute to the differences, if any? These aspects are not sufficiently addressed in the current version.\n- While the authors justify analyzing LAION-400M (as a proxy for training data) to study majority class reduction and amplification, most of the evaluated models are trained on datasets much larger and more diverse than LAION-400M. This raises concerns about how much trust can be placed in the LAION-based conclusions when extrapolating to those models.\n- The analysis is limited to binary gender bias, while recent studies have also explored broader social dimensions such as race and age.\n\nQUESTIONS:\n- What novel findings can we extract from this analysis beyond reaffirming that bias exists?\n- Given that many of the evaluated models are trained on larger, more diverse datasets than LAION-400M, how reliable are the comparisons made using LAION-400M?\n- A discussion or acknowledgment of the additional axes of bias would have made the work more comprehensive and aligned with current trends in fairness research.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a large-scale analysis of gender bias in images generated using T2I models. This analysis considers not only occupations, but different activities, contexts and objects that could influence the gender in the generated images. Furthermore, it studies how these models amplify biases w.r.t. the internet data that they are trained on. Detailed results indicate how these models follow traditional gender roles.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The diversity of prompts is better than previous works that only consider occupation prompts. Additionally, the large number of images is certainly useful for the analysis. \n- Positive correlations between models shows that most models learn similar biased representations. All analyses are reported with model-specific statistics, enabling comparisons between models.\n- I liked the decision to consider bias amplification with respect to LAION-400m. Results in Table 3 highlight some interesting findings.\n- The paper is well written and very easy to read, and considers ethical issues related to their analysis.\n\n**Weaknesses**\n\n1. *[Major] Novelty:* The authors highlight two areas of novelty - (1) the total number of images they used for the analysis, and (2) the diversity of prompts (going beyond occupations) on which they do this analysis. However, I find there is limited novel technical contribution or analysis methodology in this paper. The pipeline consists of VQA (used previously by StableBias and several other works cited in the paper) for gender detection and a fairly simple metric (Rf) that measures the ratio of females in the image sets, and is limited to the select open-source models (SD & Flux families) that the authors have chosen. It is also unclear in the paper whether the ~2.3M images will be released to the public (abstract only mentions code and prompts) - which could be a useful contribution for further research on biases in T2I models.\n\n2. *[Major] Justification for the need for large image sets:* While I agree that using larger image sets is generally a good idea for bias detection, there needs to be some justification for spending the additional GPU hours and time. One way to evaluate this would be to sample smaller subsets of images (10, 25, 40, 100, 200 images per set - as an example) and evaluate how much Rf changes if smaller subsets of images were used. Does increasing the number of images change your analysis? What should be the ideal number of images per prompt? How much variance is present from prompt cluster to cluster?\n\n3. *[Minor] Quality of VQA model:* I see that VisoGender is used to justify the use of InternVL for gender detection. However, VisoGender only contains real world images rather than synthetically generated images. It is possible that images containing artifacts from generation (esp. from older diffusion models) may make the VQA model fail to accurately detect gender. It would be valuable to conduct a small user study to test VQA performance on synthetic images, or at least identify any failure cases. \n4. *[Minor] Qualitative examples:* The paper (or appendix) does not contain any qualitative examples of prompt clusters and their images, or any examples of VQA. Given how many images that were generated, it would be valuable to observe how different models represent certain clusters & what kinds of prompts show the most variation. For example, in Fig 5, the laundry cluster seems to show that all models except SD3.5-Large are female learning. This would be an interesting example to show qualitative results on.\n\n**Other comments:**\n- Lines 306-308: I agree that LAION-400m may be representative, but is there any citation to justify this?\n- Lines 5-7 (abstract): the wording used here seems to suggest 200 images per prompt per model, but it is 40 images per prompt over 5 models or over 5 prompt variations. Please make this clear. Also, some prior works do consider more images per prompt per model (e.g. TIBET used 48 images per prompt, on any given model), so it would be important to clarify this.\n\nQUESTIONS:\nMy rating is primarily based on the aforementioned weaknesses and the questions associated with those, which I encourage the authors to address in their rebuttal. To summarize the main weaknesses: \n\n1. Novelty: Is there any novel technical contribution I missed, that could be highlighted? Will all images be released? How will this work enable future works in bias analysis?\n\n2. Image set size: How much do analysis results change when image set size changes? Is 200 a sufficient number of images, or do we need less or more for drawing similar conclusions?\n\n3. VQA: can you evaluate how well the VQA model does in detecting genders in synthetic images, even in cases where there may be artifacts?\n\nLIMITATIONS:\nYes - a detailed limitations section is provided in the appendix, in addition to discussions about ethical considerations on gender classification. I find this sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-16"}