Title: More of the Same: Persistent Representational Harms Under Increased Representation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Natural Language Processing (NLP)

evaluation

bias

generative AI

gender bias
Abstract: To recognize and mitigate the harms of generative AI systems, it is crucial to consider whether and how different societal groups are represented by these systems. A critical gap emerges when naively measuring or improving *who* is represented, as this does not consider *how* people are represented. In this work, we develop GAS(P), an evaluation methodology for surfacing distribution-level group representational biases in generated text, tackling the setting where groups are unprompted (i.e., groups are not specified in the input to generative systems). We apply this novel methodology to investigate gendered representations in occupations across state-of-the-art large language models. We show that, even though the gender distribution when models are prompted to generate biographies leads to a large representation of women, even representational biases persist in how different genders are represented. Our evaluation methodology reveals that there are statistically significant distribution-level differences in the word choice used to describe biographies and personas of different genders across occupations, and we show that many of these differences are associated with representational harms and stereotypes. Our empirical findings caution that naively increasing (unprompted) representation may inadvertently proliferate representational biases, and our proposed evaluation methodology enables systematic and rigorous measurement of the problem.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper asks 2 questions related to gender bias in LLMs: (1) has there been a change over time in how LLMs associate gender with occupations (2) do models generate text with gender-associated stereotypes even if the prompt doesn't contain an explicit gender marker. The paper compares one old model (GPT 3.5) with two newer models (GPT 4o mini and Llama 3.1) by prompting the model to generate biographies of given occupations and finds that models have increased in their tendency to generate biographies of women, across occupations. The authors then test whether the texts (generated from prompts without mentioning gender) that describe women/men tend to be more similar to texts describing women/men from prompts with explicit gender, and finds they are, and that this similarity increased from GPT 3.5 to GPT 4omini.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n(1) Knowing whether models have changed over time in gender stereotypes in their output is an important question\n\n(2) The paper looks at two ways to answer this question, both at the change in the percentage of biographies generated for different genders for different occupations, and for the actual stereotypes generated.\n\n(3) The finding that OpenAI models have increased in their tendency to generate biographies of women (at least between 3.5 and 4o-mini)  is an interesting finding.\n\nWeaknesses:\n\nThe main weakness is that overall the method and findings are insufficiently novel and insufficiently well-motivated. Some examples:\n\n(1) One of the two main findings, that biographies about women contain more women-associated terms (even if the prompt doesn't explicitly ask for a biography of a particular gender) is quite unsurprising.  Dozens of papers have shown that these models generate biased gender associations, and many of them use prompts that don't have gender in the prompt (e.g., recent papers like https://aclanthology.org/2025.findings-naacl.398/ or https://doi.org/10.1007/s10579-024-09780-6).  But more worrisome, given that these models have amassed so much evidence that models generate these biased associations, it's difficult to defend the possibility that model might write a biography that describes a woman but fail to generate gender-associated language simply because the prompt didn't mention the gender.\n\n(2) Methods are insufficiently motivated and insufficiently compared to alternative methods.  The calibrated marked words method is not motivated (why remove frequent words? the Fightin' Words method specifically argues the method doesn't need frequent word removal). If this was necessary because of a failing of Fightin' Words to handle this use case, that should be discussed and motivated. If Calibrated Marked Words is to be a contribution, it has to be carefully compared to alternative methods (perhaps even just simple raw log-likelihood). The Gender Association Method (count pronouns and honorifics) is simplistic and not different from what other papers do in this situation (e.g. https://aclanthology.org/2025.findings-naacl.398/ or https://doi.org/10.1007/s10579-024-09780-6).  The Subset Representational Bias score  and the use of Chamfer distance is not motivated or compared to other cosine-based methods for quantifying gender bias, like GenBit (Sengupta,  B.;   Maher,  R.;   Groves,  D.;   Olieman,  C.  (2021).  GenBiT:  measure and  mitigate  gender  bias  in  language  datasets, Microsoft  Journal  of  Applied  Research,   16,   63–71,   2021.)  \n\n(3) The finding that there is a historical shift in the percentage of women biographies generated across occupations is interesting but both unclear and insufficiently supported. The main paper simply claims it based on visual evidence in Figure 1. But without any statistical tests, it's hard to know how robust this is. This claim should be tested statistically. In addition, it's hard to know what the exact claim is about this change. Is the claim that GPT4o is generally more likely to generate biographies of women, irrespective of occupation or other attributes?  Or that specifically there is a change in something about the occupation-gender association? These are not distinguished. The reader also can't tell if the claim is simple that models generate more women biographies than BLS, or that there is a historical change and prior models did not generate more women biographies and current models do. (the relationship between GPT3.5 and BLS is not clear in the paper or the appendix). All of the historical claims mainly rest on changes from a single model GPT3.5 to another single model, GPT4o-mini, since differences between GPT3.5 and Llama may be due to architectures or randomly different training data rather than explicit changes in gender associations over time.\n\n(4) The paper is generally weak on statistical tests of any kind, making many of the comparisons feel anecdotal\n\n(5) The stereotypes in seciton 5.3 seem quite similar to stereotypes already described in the very extensive gender bias literature. Are some of these new? if so which ones and what do they contribute to the literature?\n\n(5) The paper is poorly written and poorly checked.  Many terms (associated gender, specified gender) are used well before they are defined.  The equations are not numbered, making it difficult to point out that first equation is wrong ( \"min dx(A,B)\" should be  \"min dx(v,u)\") and the second equation has a typo (a period inside the second Chamfer distance). The latex for equations should be cleaned up (just as one example CH is currently not be typeset as two variables C and H).\n\nQUESTIONS:\nMy questions are mainly highlighted in the weaknesses, but are repeated here for clarity:\n\n1) Why is it unexpected that models that generate biographies of women should use women-associated terms, even if the prompt doesn't mention a woman?\n\n2) For each method, how is it better than prior methods:\na) Calibrated Marked Words\nb) SRBS including Chamfer Distance\n\n\n3) What exactly is the historical claim about  generation: that models generate more biographies of women, or that models generate more association of women with different occupations. Can you distinguish these?\n\n4) What exactly is the historical claim about the relationship of occupations to BLS stats, and are your results for GPT3.5 consistent with earlier papers using GPT3.5 that have suggested underrepresentations of women?\n\n5) How does the Gender Asssociation method differ from prior methods?\n\n6) which of your findings hold with statistical significance and which do not?\n\n7) How are the sterotypes described in section 5.3 different than similar stereotypes already described in the literature?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates **persistent representational harms** in large language models (LLMs) despite increased demographic representation. Focusing on gender bias in occupational portrayals, the authors:\n (1) Analyze \"who is represented: Generate biographies/personas for 63 occupations *without* specifying gender, finding women are significantly overrepresented (vs. U.S. BLS data) in GPT-3.5, GPT-4o-mini, and Llama-3.1-70b. Non-binary representation remains near zero.  (2) Develop novel methods:  \n Gender Association Algorithm infers implicit gender in outputs via pronoun/honorific frequency (99.6% validation accuracy), \n  Calibrated Marked Words identifies statistically significant words differentiating gendered descriptions (z-score ≥1.96), filtering common English terms and Subset Representational Bias Score quantifies bias by comparing word embeddings of *implicitly* vs. *explicitly* gendered outputs using Chamfer Distance.  (3) Examine \"how people are represented\": Reveal persistent stereotypes—e.g., women associated with *empathy*, *resilience*, and *passion*; men with *excellence* and *academia*—even in gender-neutral prompts. Biases intensify in newer models (e.g., GPT-4o-mini shows 100% increased bias for \"software engineer\").  (4) Critique bias mitigation: Increased female representation likely stems from interventions but fails to address *descriptive biases*, amplifying harmful narratives.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n1. **Novel methodological contributions**:  \n   - The **Gender Association Method** enables large-scale analysis of *implicit* gender biases in unmarked prompts, addressing a critical gap in bias evaluation.  \n   - **Subset Representational Bias Score** provides a quantifiable framework to compare biases across *associated* (implicit) vs. *specified* (explicit) gender contexts.  \n2. **Rigorous empirical design**:  \n   - Large-scale experiment: 63 occupations × 100+ generations/model (≈6K+ samples), validated across multiple SOTA LLMs.  \n   - Statistical robustness: *p*<0.05 for bias scores; careful calibration of marked words avoids noise from common terms (e.g., \"the\").  \n3. **Critical societal implications**:  \n   - Exposes limitations of current \"diversity-focused\" interventions (e.g., overrepresenting women without addressing stereotypes).  \n   - Highlights proliferation of neoliberal tropes (e.g., framing success as individual \"resilience\" vs. systemic change).  \n4. **Transparency**: Code/data released; detailed supplement (hyperparameters, validation, cluster analysis).  \n\n **Weaknesses**  \n1. **Limited generalizability**:  \n   - U.S.-centric bias: Occupational gender distributions sourced from U.S. BLS, ignoring cultural variability.  \n   - Non-binary exclusion: <10% representation prevents meaningful analysis (acknowledged but unaddressed).  \n2. **Methodological constraints**:  \n   - **Pronoun dependency**: Gender Association Method may miss cultural/linguistic nuances (e.g., gender-neutral language).  \n   - **Subjective cluster interpretation**: Stereotype labels (e.g., \"passion,\" \"excellence\") applied post hoc to word clusters lack inter-rater reliability metrics.  \n3. **Incomplete causal analysis**:  \n   - Speculates about \"bias mitigation interventions\" but lacks access to model training data/RLAIF details to verify causality.  \n   - No ablation on prompt phrasing impact (e.g., \"biography\" vs. \"persona\").  \n4. **Underdeveloped harm taxonomy**:  \n   - Groups all stereotypes under \"representational harm\" without severity gradation (e.g., \"empathy\" vs. \"subservience\").  \n   - Neglects intersectional biases (e.g., race/class + gender).\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a nuanced and critical dimension of representational harm in LLMs. The authors argue that merely increasing the representation of a demographic group (who is represented) does not automatically mitigate, and can even proliferate, stereotypical and harmful portrayals (how they are represented). To demonstrate this, the paper analyzes gender representation in occupational personas generated by state-of-the-art LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper addresses a crucial and nuanced issue in AI safety. \n* The proposed bias measurement pipeline is technically sound, particularly the enhancements to “Marked Personas.”\n\nWeaknesses:\n* Limited originality in framing: The underlying insight—that improving demographic parity (e.g., 50/50 gender distribution) doesn't eliminate harmful stereotypes—is widely discussed in fairness and safety literature.\n* The study is narrowly focused on occupational prompts. It remains unclear how the findings extend to other real-world tasks (e.g., summarization, Q&A, dialogue).\n* Details of the main methodology are pushed to the Appendix, making readers difficult to follow and understand the rigor of the proposed method. \n* There is little discussion about how findings might guide practical LLM development beyond “do better.”\n* Misc: clarify what A and B stand for in the Chamfer Distance formula.\n\nQUESTIONS:\n* What do your results imply for practitioners? Would interventions at the RLHF or fine-tuning stages be more effective in targeting “how” people are represented?\n* How is your Subset Representational Bias Score substantially different or more effective than other existing bias quantification tools, such as WEAT, StereoSet, or Marked Personas alone?\n* Can your methods be applied to domains outside occupation modeling? For example, do you observe similar representational harms in education, family, or politics contexts?\n* What about dealing with intersectional identities (e.g., race + gender)?\nMisc:\n* What does \"English language\" in line 153 refer to?\n* What does \"topic prior\" in line 663 refer to? What dataset is it?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes that previous studies have failed to account for who and how gender biases were studied when it comes to representational harms. The authors propose a methodology that shows how word differences in describing occupations when it comes to gender distributions makes a difference in showing how the real representational harms are perpetuated

STRENGTHS AND WEAKNESSES:
It is an interesting point of research to understand word associations in gender based representional harms. This approach is much needed. To look into how words describing occupations vs how the words explicitly calling out genders in occupations is indeed very clever and very well done. 

But overall the analysis and approach fails to dive deep into the question of generated image which in fact uses text to generate an image vs text to text generation. To quote the part here "although previous work has investigated gender occupational bias in generated94
images without specifying gender [ 36 ], to our knowledge, no previous work has investigated gender
occupational bias in generated text without specifying gender." Specifying or not specifying gender, image generating research in this area has indeed pointed this out many times.

QUESTIONS:
Would be nice if the authors add more context in how such representational harms harm more women even when the occupations described here have more representation. That part was briefly mentioned in the conclusion but not given more context. Given the authors continued enagagement with this topic and how it adds an layer of insight to their input, it would be helpful to have this opinion or added references to this

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-08