Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution
Authors: Falaah Arif Khan
Nivedha Sivakumar
Yinong Oliver Wang
Katherine Metcalf
Cezanne Camacho
Barry-John Theobald
Luca Zappella
Nicholas Apostoloff
Venue: COLM 2025 (2025)
Keywords: fairness
uncertainty
intersectionality
TL;DR: We propose a fairness benchmark that evaluates intersectional biases in LLMs based on disparities in model confidence while performing coreference resolution on different intersectional identities

Abstract: Large language models (LLMs) have achieved impressive performance, leading to their widespread adoption as decision-support tools in resource-constrained contexts like hiring and admissions. There is, however, scientific consensus that AI systems can reflect and exacerbate societal biases, raising concerns about identity-based harm when used in critical social contexts. Prior work has laid a solid foundation for assessing bias in LLMs by evaluating demographic disparities in different language reasoning tasks. In this work, we extend single-axis fairness evaluations to examine intersectional bias, recognizing that when multiple axes of discrimination intersect, they create distinct patterns of disadvantage. We create a new benchmark called WinoIdentity by augmenting the WinoBias dataset with 25 demographic markers across 10 attributes, including age, nationality, and race, intersected with binary gender, yielding 245,700 prompts to evaluate 50 distinct bias patterns. Focusing on harms of omission due to underrepresentation, we investigate bias through the lens of uncertainty and propose a group (un)fairness metric called \emph{Coreference Confidence Disparity} which measures whether models are more or less confident for some intersectional identities than others. We evaluate five recently published LLMs and find confidence disparities as high as 40\% along various demographic attributes including body type, sexual orientation and socio-economic status, with models being most uncertain about doubly-disadvantaged identities in anti-stereotypical settings, such as when assigning transgender women to historically male-dominated occupations. Surprisingly, coreference confidence decreases even for hegemonic or privileged markers (e.g., 'White' or 'cisgender'), indicating that the recent impressive performance of LLMs is more likely due to memorization than logical reasoning. Notably, these are two independent failures in value alignment and validity that can compound to cause social harm.

Published as a conference paper at COLM 2025
Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution
Falaah Arif Khan', Nivedha Sivakumar", Yinong Oliver Wang', Katherine Metcalf, Cezanne Camacho², Barry-John Theobald?, Luca Zappella', Nicholas Apostoloff Work done while Apple, AApple,
Abstract
Large language models (LLMs) have achieved impressive performance, leading to their widespread adoption as decision-support tools in resource- constrained contexts like hiring and admissions. There is, however, scien- tific consensus that Al systems can reflect and exacerbate societal biases, raising concerns about identity-based harm when used in critical social con- texts. Prior work has laid a solid foundation for assessing bias in LLMs by evaluating demographic disparities in different language reasoning tasks. this work, we extend single-axis fairness evaluations to examine intersec- tional bias, recognizing that when multiple axes of discrimination intersect, they create distinct patterns of disadvantage. We create a new benchmark called Winoldentity by augmenting the WinoBias dataset with 25 demo- graphic markers across 10 attributes, including age, nationality, and race, intersected with binary gender, yielding 245,700 prompts to evaluate 50 distinct bias patterns. Focusing on harms of omission due to underrepre- sentation, we investigate bias through the lens of uncertainty and propose group (un)fairness metric called Coreference Confidence Disparity which measures whether models are more or less confident for some intersectional identities than others. We evaluate five recently published LLMs and find confidence disparities as high as 40% along various demographic attributes including body type, sexual orientation and socio-economic status, with models being most uncertain about doubly-disadvantage mnttiisteeottppcal settings, such as when assigning transgender women to historically hale-dominated occupations. Surprisingly, coreference confi- dence decreases even for hegemonic or privileged markers (e.g., 'White' or 'cisgender') ) indicating that the recent impressive performance of LLMs is more likely due to memorization than logical reasoning. Notably, these are two independent failures in value alignment and validity that can compound to cause social harm.
identities in
Introduction
Social decision-making, such as hiring, lending and admissions, is often highly resource- constrained, with a large number of applicants vying for a limited number of positions. Artificial intelligence (Al) has emerged an efficient and scalable solution to alleviate this burden, with large language models (LLMs) being particularly well-suited to process textual inputs such as resumes and cover letters. A major hurdle to their widespread adoption, however, is the potential to reproduce and exacerbate pre-existing societal biases (Bender et al., 2021; Katzman et al., 2023; Wang et al., 2022), leading to representational harm, where certain groups are underrepresented misrepresented; allocational harm, where resources or opportunities are unfairly distributed; and stereotyping harm, where algorithms perpetuate harmful stereotypes.
A plethora of textual fairness benchmarks have been proposed, focusing on demographic disparities in different Natural Language Processing (NLP) tasks including text classifi- cation (Garg et al., 2019), natural language inference (Dev et al., 2020), question answer- ing (Parrish et al., 2022), and coreference resolution (Zhao et al., 2018). These benchmarks
1

---

Published as a conference paper at COLM 2025
are predominantly focused on single-axis evaluations, and have generally overlooked inter- sectional bias. More recently, LLMs have achieved impressive performance on a variety of NLP tasks, leading to their rapid adoption into society, and so fairness evaluations have become more contextual and application-specific, for example, directly evaluating LLMs for bias when used for resume screening (Armstrong et al., 2024). Importantly, this involves composition of several language reasoning capabilities including named entity recognition, coreference resolution and text classification. LLM fairness evaluations thereby often take for granted the validity of LLMs as decision-support tools, and instead focus only on value alignment, i.e., that LLMs do not discriminate against any social group. Drawing from Coston et al. (2023), we challenge this assumption; asserting that an invalid instrument is unfit for use critical contexts, regardless of its fairness. Conversely, a valid instrument is not auto- matically fair, highlighting the need to evaluate both criteria separately. For example, recent evaluations show that LLMs struggle with simple reasoning tasks (Mirzadeh et al., 2024; Williams & Huckle, 2024), indicating that their impressive performance is more suitably attributed to memorization than to reasoning abilities - emergent or otherwise (Schaeffer et al., 2023). Memorization poses a threat to validity because tool employed for lan- guage reasoning tasks should genuinely perform reasoning, rather than merely repeating memorized training data labels. Put differently, models that rely on memorization over reasoning fail to satisfy external validity desiderata (Coston et al., 2023).
In this work, we focus on the task of intersectionally fair coreference resolution, which has been overlooked so far. We propose a flexible framework that extends the WinoBias dataset ddeiigned to probe gender stereotypes using a coreference resolution task-using three augmentations cinnnnttaiionssioo that prefix demographic markers to the referent occupation only (R-Aug), the non-referent occupation only (ÑR-Aug) and to both occupations in the sentence (C- Aug). Using set of 25 markers from 10 demographic attributes intersected with binary gendered pronouns, we create Winoldentity; a benchmark of 245,700 distinct probes to evaluate intersectional biases in LLMs. We pair this corpus with new group (un)fairness metric, coreference confidence disparity, defined as the difference in model performing coreference resolution different identities. Uncertainty is relevant here because representations harm towards coubly-disadvantand certainty that these identities will perform poorly in given occupation, but rather due to high uncertainty about whether they will perform well in the given position due to historic underrepresentation and lack of access opportunities (Suresh & Guttag, 2021). Hence, in order to probe for representational harms from omission alongside stereotyping harms, we design an uncertainty-based fairness evaluation. An additional technical reason for adopting an incertainty-based approach is that recent work has shown that uncertainty provides a more fine-grained analysis of model performance than accuracy and existing influential error-based fairness measures (Kuzucu et al., 2024).
model confidence when
groups is often not due to
In this work, we make two contributions: First, the Winoldentity' benchmark for evaluating intersectional bias through the lens of uncertainty, and the flexible empirical framework used to generate it, described in $ 3. Second, the findings of our empirical evaluation of five recently published LLMss, discussed in S 4, raising the following concerns:
1. Validity concerns: LLMs perform poorly on the augmented coreference resolution task, even for hegemonic markers which are usually unmarked, indicating that they rely more on memorization than reasoning and are thereby unable to reason well about intersectional identities;
2. Value misalignment concerns: We find coreference confidence disparities higher than 20% in 7 out of 10 demographic attributes we evaluated, with exacerbated bias towards identities in anti-stereotypical contexts, such as when assigning feminine subgroups to historically male-dominated occupations.
httpps//giihhub http:///tttuuu.ccoopppel///////e wttpa////wwwwwwiieeetniiee winoidentity
2

---

Published as a conference paper at COLM 2025
2
Related Work
The literature abounds with LLM fairness benchmarks that evaluate bias along gender and sexual identity, including WinoBias (Zhao et al., 2018), WinoGender (Rudinger al., 2018), BUG (Levy et al., 2021), BEC-Pro (Bartl et al., 2020), GAP (Webster et al., 2018), WinoBias+ (Vanmassenhove et al., 2021), SOWinoBias (Dawkins, 2021), WinoQueer (Felkner et al., 2023), GICOREF (Cao & Daumé III, 2021) and SoWinoBias (Dawkins, 2021). Benchmarks such as StereoSet (Nadeem et al., 2020), BBQ (Parrish et al., 2022), Bias-NLI (Dev et al., 2020), CrowS- Pairs (Nangia et al., 2020), RedditBias (Barikeri et al., 2021), Equity Evaluation Corpus (Kiritchenko & Mohammad, 2018) and PANDA (Qian et al., 2022) evaluate discrimination across several attributes including ethnicity, nationality, physical appearance, religion, socio- economic status and sexual orientation, but only across a single axis at a time (not using intersectional or multi-attribute group definitions). Other bias evaluation studies include Curto et al. (2024) who uncover socio-economic bias in word embeddings, and Sheng et al. (2019) who introduce the notion of regard towards demographic as way quantify bias along gender, race and sexual orientation in text generation.
Hossain et al. (2023) evaluate LLMs' ability to reason with gender-neutral and neo-pronouns, and attribute poor performance to lack of representation of non-binary pronouns in training data and memorized associations. Ju et al. (2024) study the typicality of genders in different occupations and find that comaleedominated occupations are 'more gendered than male dominated ones. We see these behaviors reproduced in our study. Lastly, Kotek et al. (2023) is recent work contemporaneous with ours that extends the WinoBias corpus to evaluate different reasoning strategies that models might employ to perform coreference resolution on ambiguous sentences only, including syntactic, stereotypical, random guessing and biased strategies. By contrast, we evaluate the model's uncertainty under syntactically ambiguous and unambiguous contexts, with and without demographic augmentations. Further, their evaluation is limited to binary gender, whereas we study 50 bias patterns at the intersection of gender and 10 demographic attributes.
Intersectionality (Crenshaw, 1989; Collins et al., 2021) has been centered in several re- cent evaluations, mostly to assess stereotypical biases in hiring contexts. Howard et al. (2024) study stereotypes at the intersection of gender and race in vision language models using counterfactuals; Charlesworth et al. (2024) propose flexible procedure to extract intersectional stereotypes from word embeddings and report significant disparities along gender, race and class, and their intersections; and Ma et al. (2023) construct dataset for intersectional stereotype research using 6 demographic attributes (namely race, age, religion, gender, political leaning and disability) and all their possible combinations. Howard et al. (2024) find that covert raccoliiguistic bias against speakers of African American English (AAE) can be masked by overtly positive stereotypes about African Americans, whereas Hada et al. (2024); Yu et al. (2024); Liu et al. (2024) uncover bias at the intersection of gender and language. We defer to Li et al. (2023) and Blodgett et al. (2020; 2021) for a more extensive survey of fairness evaluations and to Ovalle et al. (2023); Gohar & Cheng (2023) and Kong (2022) for more comprehensive review of intersectionality in fair-ML.
Our experimental design is inspired by the seminal field study by Bertrand & Mullainathan (2004) which uncovered discriminatory hiring practices by comparing call-back rates for identical resumes with distinct racial and gender-identifying names. Several recent studies have reproduced these findings with LLMs; Wilson & Caliskan (2024) found systematic racial bias on a resume assessment and matching task, while An et al. (2024) find that L.M-ggenerted hiring decision emails are most likely to favor White-associated names. Additional works study bias in job recommendation tasks (Salinas et al., 2023; Kirk et al., 2024), and resume assessment and generation tasks (Armstrong et al., 2024). All of these studies are tailored to specific stages of the hiring pipeline and evaluate downstream allocational harms. Instead, we focus developing a flexible framework to measure representational harms towards intersectional identities.
Uncertainty, a long recognized factor in model performance (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Hüllermeier & Waegeman, 2021; Depeweg et al., 2018; Kendall & Gal, 2017; Mukhoti et al., 2023), has recently become of interest to the fair-ML.
3

---

Published as a conference paper at COLM 2025
The p0yace Ph Nos collcc
Dopur
Nit OIT portoe fon
phephpptiin the dusa calos
10mm
Num
05
0.1
0.00
Type
Langagg
yanng
Mh-lap
Wit to the DVn portor for oeleer
PMD
PM2
N0w
Nmm
30.35
087
068
8.0
Lou Levoola the anto The velro
Denopoogh
Daratty
Vi to the 6N6 The pirnce fun orforit
harto do DVS pinnur han wenaeeeiii
Figure 1: Dataset Construction: Winoldentity is comprised of 245,700 bias probes (1575 unique sentences pronouns x (25+1) demographic markers x augmentations)
community. Wang et al. (2024) formalize and contrast 'epistemic discrimination' due to modeling choices under lack of knowledge about the optimal 'fir'' model, with 'aleatoric discrimination due to inherent biases in the data. Ali et al. (2021) and Tahir et al. (2023) propose methods to improve fairness by identifying and mitigating ambiguous regions where errors are likely to occur due to epistemic and aleatoric uncertainty respectively. Recently, Kuzucu et al. (2024) show formally how uncertainty-bast provides more fine- fine-grained yet complementary analysis to existing error-based measures like Equalized Odds (Hardt et al., 2016) and Statistical Parity (Zafar et al., 2017; Dwork et al., 2012). Creel & Hellman (2022) and Gomez et al. (2024) argue that arbitrariness - closely related to uncertainty - can be ethically and legally problematic when is systematic (or disparate) because it violates principles of non-discrimination including procedural fairness. We adopt this perspective in our bias evaluation, asserting that models with large coreference confidence disparities are unfair because they apply more confident procedure to members of one social group and a more uncertain one for others.
fairness evaluation
3
The Winoldentity Dataset
3.1 Background on WinoBias
WinoBias (Zhao et al., 2018) is an evaluation corpus of 3,150 sentences, with equal number of sentences containing male and female pronouns i.e., 1575 unique base prompts. This corpus was proposed to detect gender bias in coreference resolution, where the model selects which occupation (of two in a sentence) particular pronoun refers to, which we will call the 'referent occupation'. Sentences are further categorized into Type-1 and Type-2, designed to evaluate semantic and syntactic reasoning separately.
Type-1: Follow schema [occ1] [interacts with] [occ2] [conjuction] [pronoun]ccirumstaanc]]
where syntactically the pronoun can refer to either occu- pation, and additional semantic reasoning is required to make the correct coreference prediction. Example: The librarian helped the CEO to find the book because [pronoun] was asked to. The pronoun "[pronoun]" refers to the. There are 783 unique Type-1 sentences.
Type-2: Follow schema [occ1] [interacts with] [occ2] and then [pronoun] for [circumstances], where the pronoun always refers to the second occupation in the sentence, and therefore
be resolved with syntactic reasoning only. Example: The physician saw the nurse and called [pronoun] to the CPR. The pronoun "[pronoun]" refers to the. There are 792 unique Type-2 sentences.
4

---

Published as a conference paper at COLM 2025
3.2 Winoldentity Construction
Demographic Attributes We draw from The Wheel of Power and Privilege2 to identify 10 attributes that impact social dynamics, from which we select 25 demographic groups that are well represented in the US context. In Tab. 1, identities are divided into hegemon- ic/pne ac/privileged that historically hold power and advantage, and disadvantaged that have been historically marginalized or oppressed.
Attribute
Heggemonicprivilleed
Disadvantaged
age
young
old fat neurodivergent (ND), disabled
bady type
thin
disability
neurotypical (NT), able-bodied
gender identity
cisgender
transgender
language
English speaking
non speakiing
nationality iity
American
immigrant
secual orientation
helensexual
gay poor
socio meci--cenooiic status
rich
White
Black, Asian, Hispanic
religion
Christian
Muslim, Jewish
Table Demographic markers used in augmentations, drawn from The Wheel of Power and Privilege. The above 25 markers combined with binary gender categories produce 50 demographic groups on which we evaluate intersectional bias in LLMs. *Privilege and disadvantage along the lines of age is highly context-specific. For example, old is disadvantaged hiring contexts, while young is disadvantaged in lending contexts.
Augmenting WinoBias We design three augmentations:
Referent augmentation (R-Aug): We prepend the demographic marker to the referent occu- pation. We aim to investigate whether providing additional demographic information about the referent occupation influences the model's confidence in assigning gender (pronoun) to that occupation. For the example in Fig. 1, this augmentation assesses: 'how likely is the [pronoun] the nurse, given that the nurse is young?
Non-referent augmentation (NR-Aug): We prepend the demographic marker to the non- referent occupation. In contrast to R-Aug, NR-Aug examines whether the model's confidence in assigning a gender (pronoun) to a referent occupation changes when additional information provided about a non-referent occupation. For the example in Fig. 1, this augmentation assesses: 'how likely is the [pronoun] the nurse, given that the physician young?'
Contrastive augmentatiin (C-Aug): We prepend markers to both occupations: we prepend the referent occupation with the marker being evaluated (e.g,, young), and prepend the non-referent occupation with contrastive marker from Tab. 1 (e.g., old). Some markers such as White and Christian have several contrastive demographic markers, and we marginalize over all of them. For the example in Fig. 1, this augmentation assesses: 'how likely is the [pronoun] the nurse, given that the nurse is young and the physician is old?'
Our prompt construction procedure is shown in Fig. 1. For each base prompt from WinoBias, a chosen augmentation type (R-Aug, NR-Aug, C-Aug) and demographic attribute from Column 1 of Tab. 1, we generate augmented set of size 2(| +1), where |G] is the cardinality of the set of intersectional demographic markers for that attribute. For example, G=age has only two unique values (young, old) whereas when G=race has four (White, Black, Asian, Hispanic). Each augmented set includes two bbseline' WinoBias prompts to evaluate single-axis gender bias, using feminine (fem) and masculine (masc) pronouns. The other 2|G| prompts will combine markers from Column and 3 of Tab. 1 with binary feminine (fem) and masculine (masc) pronouns to evaluate intersectional biases. Overall, Winoldentity comprises of 245,700 distinct sentences.
4 Bias Evaluation via Coreference Confidence Disparities
We assess intersectional bias by evaluating how the addition of demographic markers affects the model's coreference confidence. For LLMs to be reliable decision-support tools, they
htttps: //tttps://kb..o wisc. wis.c..eh/instruutiiooa---eoos wet.aaaeeeeeeeeeeeeeeeeeeeeeeeee phtp::/eaaapppppppipi1pppaa
5

---

Published as a conference paper at COLM 2025
must demonstrate language reasoning, which includes the ability to distinguish between relevant and irrelevant information. In the coreference resolution task, demographic infor- mation is irrelevant and should be treated as such. Additionally, from faimess perspective, any impact of demographic information should be consistent across all markers.
Next-token Occupation Probability To assess the probability of candidate occupation token w being the next token given an input prompt sequence L, we query the causal language model fy output probability distribution over its vocabulary P(w L) softmax(fa(L) ar- We calculate the log probability of each occupation candidate in a sen- tence, 3 such as 'physician' and 'nurse' in the example in Fig. 1. For multi-token occupations, we sum the log probabílities of individual tokens obtain the overall noxt probability. We use greedy decoding as this ensures deterministic predictions for reproducibility.
Coreference Confidence We define the coreference confidence on a sentence L, as the difference between next-token probabilities of the two occupations:
CC(L,) = P(referent L;) P(non-reeerenn L;).
(1)
A performant model will assign higher probability to the referent occupation than to the non-referent occupation. A confident and performant model will have a coreference con- fidence score close to 1. We expect that LLMs will exhibit higher confidence on Type-2 sentences, which are syntactically unambiguous, compared to Type-1 sentences, which require additional world knowledge. A model without gender bias will exhibit similar coref- erence confidence scores for the same prompt across different pronouns. An intersectionally fair model will demonstrate consistent coreference confidence scores across all combinations of demographic markers (e.g., age, ethnicity) and pronouns, given the same prompt.
Coreference Confidence Disparity We adopt group-fairness perspective, and define the coreference confidence disparity along demographic attribute G, on a set of base prompts (L,, 12....,,nn, Ln), as the maximum disparity in the average coreference confidence over all subgroups:
ACC maxge ( xYg)
Σ CC(L)) min ( xYg)
CC(LS).
(2)
For each base prompt Li, we create an augmented prompt L associated with the inter- sectional subgroup g. Here, g is combination of gender (masculine or feminine) and demographic marker YG (unique markers for attribute G). For example, when G represents age and Yg = {old, young), then {old men, old women, young men, young women]. In an ideally egalitarian society, no disparities would exist among any subgroup. This metric measures the deviation from the strictest notion of fairness, highlighting the extent to which current disparities fall short of true equality.
The coreference confidence disparity metric captures two types of harm arising from dis- parate uncertainty: (i) when the model consistently declines to provide a response for certain demographic groups-suu sucp when using reject option-which of information for those groups, not because the information doesn't t exist, but because LLMs unable to confidently retrieve in the given context; and (ii) when predictions are systematically more inconsistent for some demographics than for others, making outcomes for certain groups effectively lotteries, while for others they remain rule-based.
result in the omission
4.1 We are still far from achieving intersectionally fair coreference resolution
Using the Winoldentity dataset, we evaluate five recently published causal language mod- mixtral-8x7B-inntruee,
els, namely mist mistral-7B-insttuut-we pythia-128, and falcon-40B-instruuc..
Recall that each prompt ends with the phrase 'The pronoun [pronoun] refers the'

---

Published as a conference paper at COLM 2025
We report the coreference confidence (averaged over all base prompts) for each identity (combination of pronoun and demographic marker) using referent augmentation (R-Aug) in Fig. 2 (Type-1) and Fig. 3 (Type-2). Results for other augmentations
deferred to App. A.1.
An ideal model would have a coreference confidence of for identities (brown poly- gon in Fig. 3), while an intersectionally fair model would have comparable coreference confidence across identities. Recall that Type-1 sentences are syntactically ambiguous and evaluate semantic reasoning, whereas Type-2 sentences only assess syntactic reasoning. We therefore anticipate models to be more uncertain (with lower coreference confidence) on Type-1 sentences than Type-2. Comparing the radar plots in Figs. and 3, we observe that this holds true before augmentation: the horizontal axes with fem and masc have maximum values near 0.8 for Type-2 sentences, but only 0.4 for Type-1 sentences.
Jama3 Instruct
Mitrai 8:78
Nistral Instruct
Pytha
Falcon 40b
223
Figure 2: Average coreference confidence on Type-1 sentences with referent augmentation (R-Aug). In each radar plot,fen and masc represent single-axis gender attributes, whereas all other attributes are intersectional. The ideal coreference confidence is 1; we omit the ideal polygon in this figure so as to not compress the plot too much.
ESS
RUR
Figure 3: Average coreference confidence on Type-2 sentences with referent augmentation (R-Aug). In each radar plot, and masc represent single-axis gender attributes, whereas all other attributes are intersectional. The ideal coreference confidence is 1, and is shown in brown.
The most striking observation is that we are far from achieving intersectionally fair corefer- ence resolution, as none of the models approach a fair polygon. Trends vary by demographic attribute and model. Notably, the radar plots for age, body type, nationality, religion and
7

---

Published as a conference paper at COLM 2025
race are stretched along the horizontal axis for both Type-1 and Type-2 sentences. This indicates that the sharpest drop in confidence is achieved by adding a demographic at- tribute to the gender attribute. When comparing different gender-demograppic attributes the differences are milder. For instance, for all models in Fig. 2, the performance disparity between fem and fem:young is larger than the disparity between fem:young and fem:old, and fem:young and masc:young.
Llama3 generally outperforms other models on Type-1 sentences; with its blue polygon being the outermost on nearly all identities. The only exception is fem-immigrant, where Mistral performs slightly better. In contrast, there is no clear top-performing model on Type-2 sentences, with Llama3, Mistral and Mixtral being competitive. Pythia, however, clearly under-performs, with baseline average coreference confidence scores (without any augmentations) close to 0.04 and 0.3 on Type-1 and Type-2 sentences, respectively. Un- surprisingly, despite its poor performance, Pythia is the fairest model according to our coreference confidence disparity criteria. This is visually evident in Figs. 2 and 3, and is further corroborated by Tab. 2. Conversely, Mistral (shown in green) is the least fair model, exhibiting skewed performance parity between fem and fem:fat is larger than that between fem and fem:thin and between fem:fat and masc:fat. We can see similar trends for socio-economic status, language and gender identity.
Type-1 sentences. For instance, the performance dis-
Hegemonic identities such as cisgender, White and heterosexual are considered 'unmarked' because they are deeply ingrained and rarely questioned in Western society (Bucholtz & Hall, 2004; Blodgett et al., 2021); therefore, we expected that adding hegemonic markers would have little impact on model performance. Instead, we observe that coreference confidence scores consistently decrease after referent augmentation, even for hegemonic markers, albeit not as much as for non-hegemonic identities, shown in Fig. 9 (Type-1) and Fig. 10 (Type-2) in App. A.2. This suggests that LLMs struggle to reason about intersectional identities in general (eg., White men) compared to single-axis gendered identities (egg, men), indicating that LLMs rely heavily on memorization, perhaps more than language reasoning.
Additionally, models are generally stable to non-referent augmentation (NR-Aug) on Type-2 sentences, where syntactically unambiguous answer exists, but less so on Type-1 sentences that rely on semantic information. This finding confirms that the change in confidence after augmentation is not merely the result of adding an extra word. Indeed, it is the specific positional context of the augmentation that triggers and reveals bias. This is corroborated by looking at accuracy alongside uncertainty in Tab. 3, discussed in App. A.3.1
Lastly, in order to disentangle the effects of memorization (or lack of reasoning) from biased reasoning, we use Chain of Thought (CoT) prompting with Mistral. We find that while CoT prompting improves parity in coreference confidence for certain attributes, also leads to decrease in overall confidence. See App. A.5 for further details.
Ilama3
mixtral
mistral
pythia
falcon
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
no augmeniation
0.097
0.065
0.001
0.068
0.056
0.178
0.007
0.055
0.023
0.019
nun-denoographic
0.061
0.052
0.016
0.052
0.049
0.165
0.002
0.039
0.006
0.023
age
0.192
0.026
hody type
0.259
0.251
0.153
0.157
0.243
0.392
0.072
0.053
0.152
0.118
disability
0.145
0.192
0.115
0.125
0,222
0.382
0.089
0.114
0.203
0,251
gender identity
0.172
0.151
0.349
0.269
language
0.145
0.113
0.136
0.141
0.255
0.358
0.111
0.102
0.075
0.064
0.185
0.036
0.112
0.109
0.094
0.119
0.014
0.028
0.09
0.041
sexual orienlation
0.25
0.346
0.346
0.194
0.23
0.204
0.11
0.087
0.227
0.18
socio-economic status
0.342
0.382
0.213
0.271
0.389
0.4
0.1
0.134
0.292
0,246
0.127
0.16
religion
0.034
0.025
0.09
0.105
0.108
0.092
0.03
0.054
0.042
0.045
Table Maximum co-reference confidence disparity over all subgroups with referent augmentation (R-Aug). The lower disparity, the fairer the model. Differences larger than 0.20 are marked in bold. In the no augmentation baseline the disparity is between masculine and feminine pronouns. In the non-demographie baseline we use markers such confused, unfocused, and relaxed, and the disparity is averaged over each pair of masculine and feminine augmentations. See App. A.6 for more details on non-demographic augmentations.
8

---

Published as a conference paper at COLM 2025
19toi 24 2r
Natrul entruddo0.2 on sortenees
047
EX0
032
0:1
1.01
0.71
1004
0330
1:44
1.-
1.6
6/27
332
26
84.m
0:21
0438
0111
06.
01
0.22
01:1
1100
04:4
2010
0224
031
00011
18:08
0.11
0.34
02-
0411
5.5
3.3
0.35
0.00%
0,800
0.30
11.01
8.6
1.86
08:04
0.311
811
0.391
0.35
0.14
8.01
0.6
0.001
0.34
04:30
5:01
6.00
0.11
0.51
81:34
1.005
1:20
0.61
-977
kim
6:21
HOnoy
(a) Gender identity
(b) Sexual orientation
Figure 4: Mistral's average subgroup coreference confidence on Type-2 sentences with contrastive augmentation (C-Aug), broken down by referent occupation. Values close to 1 indicate that the model is correct and confident, values around 0 indicate the model is highly uncertain, negative values indicate the model is wrong.
Coreference Confidence Disparities Tab. 2 reports the maximum subgroup disparity de- fined in Eq. (2) for referent augmentations (R-Aug), with the results for other augmentations deferred to Tab. 4 in App. A.3.2. As noted earlier, Pythia shows disparities below 20% across all 10 attributes. In contrast, Mistral shows disparities between 20 to 40% on 7 out of 10 demographic attributes. This presents an interesting albeit common trade-off: more perfor- mant models tend to exhibit larger performance disparities, achieving high performance across all subgroups is more challenging. Conversely, less performant models are more likely to be fair because the bar to equalize subgroup performance is significantly lower.
Worryingly, we observe substantial confidence disparities, ranging from 20 to 40% for body type, disability, gender identity, sexual orientation, socio-economic status, and race. These findings support our previous discussion on performance disparities along different axes, and are further corroborated by significant accuracy disparities along the same dimensions, reported in Tab. 5 in App. A.3.2. Notably, we find the smallest disparities along age, nationality and religion.
4.2 LLMs reinforce gender stereotypes on intersectional subgroups
We now examine uncertainty trends by occupation, focusing on Mistral's results for con- trastive augmentations (C-Aug) related to gender identity and sexual orientation. Additional results are in App. A.4. Fig. 4 displays the average coreference confidence across all prompts for each referent occupation (columns) and identity (rows). The correctness of the prediction is determined by the sign. A value of 1, shown in red, indicates confident and correct coref- erence predictions, whereas a value of -1, shown in blue, indicates confident and incorrect predictions, with grey values indicating under-confident predictions.
Firstly, on male-dominated occupations' such as mechanic and construction worker, we see strong gender bias with the model being more confident and correct on masc pronoun (dark red), and uncertain and wrong on fem pronouns (light blue). Conversely, on female- dominated occupations, the model is less certain (but still correct) on masc pronouns than
4 according to US Bureau of Labor Statistics data

---

Published as a conference paper at COLM 2025
fem ones. This underscores the need for an uncertainty-based evaluation, error-based evaluations would overlook this nuance.
Lastly, drawing from the social psychology literature (Fiske et al., 2002; Klysing et al., 2021), we identify intersectional bias through the lens of donble-disaddonntae,, where female intersectional subgroups are worse off male-dominated occupations, and vice versa. For example, the average coreference confidence for fem on mechanic is -0.065, compared to -0.11 for transgender fem and -0.24 for fem.
5 Conclusion
In this paper, we evaluated five recently-published LLMs for intersectional bias. Recogniz- ing that human annotation is expensive and that existing fairness benchmarks are likely to be included in LLM training data and potentially memorized, we proposed a flexible data augmentation procedure to extend single-axis fairness evaluations to capture inter- sectional biases. Applying this framework to the WinoBias dataset with a diverse set of 25 markers across 10 demographic attributes, we constructed a new evaluation corpus called Winoldentity. We paired this corpus with new (un)fairness metric called Coreference Confidence Disparity, which is attentive to whether model predictions are systematically underconfident for some social groups than others.
We found that models perform poorly on the augmented coreference resolution task, in- dicating that LLMs don't reason well about intersectional identities. Further, we found confidence disparities as high as 40%, with exacerbated bias towards doubly-disadvvantgeed identities in anti-stereotypical settings, such as when assigning feminine subgroups to historically male-dominated occupations.
Coston et al. (2023) translate concepts from validity theory to examine when it is appro- priate to use predictive systems in critical social contexts. Applying their framework, we distinguish threats to validity, i.e., brittleness to augmentations and indications of memoriza- tion, from value misalignment i.e., procedural unfairness and representational harms from omission and stereotyping. Worryingly, we found empirical evidence of two independent failure modes that, together, can amplify identity-related social harm.
Implications These systematic errors could lead to discrimination in real-world contexts such as hiring and admissions, by down-ranking application materials that contain phrases such as "Black Feminist Scholars", "Society of Hispanic Scientists" and "Neurodivergent in Al Affinity Group".
Our findings also highlight the limitations of current bias mitigation strategies. For instance, the method proposed by Zhao et al. (2018), which involves augmenting the training dataset with anti-stereotypica examples, has been shown to improve quantitative fairness metrics. A similar strategy could be applied our work; by adding the augmented sentences to the training corpus. However, we view this approach as temporary fix that leverages memorization rather than genuinely solving the underlying issue of poor model performance. Put differently, while such mitigations can improve value alignment, they do not solve the problem of invalidity.
Limitations and Future Work This study's limitations include its focus on 50 identities and 1575 sentences from WinoBias. Future research can expand this study with other evaluation corpuses like StereoSet and WinoQueer, incorporating additional demographic markers, and systematizing insights across datasets to provide a comprehensive understanding of socially-salient brittleness.
Another limitation of our work is the combinatorial explosion that results from our inter- sectional analysis. Future work could address this by using subsampling techniques (over questions or demographic markers) to make the evaluation more efficient, while preserving statistical guarantees for all groups
10

---

Published as a conference paper at COLM 2025
6 Ethical Considerations
Our research is specifically focused on the US context, and therefore, generalizability of our findings other cultural and sociolinguistic settings is uncertain and may be limited.
Further, in this work, we proposed a fairness metric called coreference confidence disparity defined the difference in model confidence when performing coreference resolution different identities. We used this metric to study intersectional bias in LLMs based on the assertion that models which are more confident on one social group than another violate procedural fairness doctrines. However, our research, like all work on algorithmic fairness, faces a limitation: fairness is complex, nonhtechnical concept that cannot be fully captured by mathematical criteria. While metrics can identify undesirable model behavior that may cause social harm, it's unclear how to translate mathematical unfairness into practical social harm. The Al fairness community acknowledges this challenge and has adopted pragmatic approach, recognizing that exact equality is often impossible. Instead, researchers aim for approximate fairness, but this raises critical question: how much disparity is unacceptable? This context-dependent question requires input from stakeholders, not just machine learning researchers. Nonetheless, the path towards answering this question will necessarily go through the process of quantifying how much disparity we observe in practice, laying the groundwork for stakeholders to determine what threshold of disparity constitutes unfair model behavior.
References
Junaid Ali, Preethi Lahoti, and Krishna P. Gummadi. Accounting for model uncertainty in algorithmic discrimination. In Proceedings of the 2021 AAAÎ/ACM Conference on Al, Ethics, and Society, AIES '21, pp. 336-345, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 97814503. doi: 1011145/4461702.344266.. URL https://doi.own org/1/...g..... 11 144/3461702.3346233
Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. Do large language models discriminate hiring decisions on the basis of race, ethnicity, and gender? Xrv preprint ar (tt:22006110886 2024.
Lena Armstrong, Abbey Liu, Stephen MacNeil, and Danaë Metaxa. The silicone ceiling: Auditing gpt's race and gender biases in hiring. arxiv preprint arX:::2405.0 04412,, 2024.
Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. Redditbias: A real-world resource for bias evaluation and debiasing of conversational language models. arviv preprint arXiv:2106.0352 03521,, 2021.
Marion Bartl, Malvina Nissim, and Albert Gatt. Unmasking contextual stereotypes: Measur- ing and mitigating bert's 's gender bias. ar preprint ar Xrx::22000 145344 2020.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
Marianne Bertrand and Sendhil Mullainathan. Are emily and greg more employable than lakisha and jamal? field experiment on labor market discrimination. American Economic Review, 94(4):991-1013, September 2004. doi: 10.1257 /00022280442005561. URL https://www. aeaweb. wwgg/attcclesidd110 12w. 1257/0002828842202556
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in NLP. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), ACI,, 2020.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015, 2021.
11

---

Published as a conference paper at COLM 2025
Mary Bucholtz and Kira Hall. Language and identity. A companion to linguistic anthropology, 1:369-394, 2004.
Yang Trista Cao and Hal Daumé III. Toward gender-inclusive coreference resolution: An analysis of gender and bias throughout the machine learning lifecycle. Computational Linguistics, 47(3))61--661, 2021.
Tessa ES Charlesworth, Kshitish Ghate, Aylin Caliskan, and Mahzarin R Banaji. Extracting intersectional stereotypes from embeddings: Developing and validating the flexible intersectional stereotype extraction procedure. PNAS nexus, 33))pg8,, 2024.
Patricia Hill Collins, Elaini Cristina Gonzaga da Silva, Emek Ergun, Inger Furseth, Kanisha D Bond, and Jone Martinez-Palacios Intersectionality as critical social theory: Intersectional- ity as critical social theory, patricia hill collins, duke university press, 2019. Contemporary Political Theory, 20(3):690, 2021.
Amanda Coston, Anna Kawakami, Haiyi Zhu, Ken Holstein, and Hoda Heidari. A validity perspective on evaluating the justified use of data-driven decision-making algorithms. In 2023 IEEE conference on secure and trustworthy machine learning (SaTML), pp. 690-704. IEEE, 2023.
Kathleen Creel and Deborah Hellman. The algorithmic leviathan: Arbitrariness, fairness, and opportunity in algorithmic decision-making systems. Canadian Journal of Philosophy, 52(1):26 2022.
Kimberlé Crenshaw. Demarginalizing the intersection of race and sex: A black feminist cri- tique of antidiscrimination doctrine, feminist theory and antiracist politics. The University of Chicago Legal Forum, 140:139-167, 1989.
Georgina Curto, Mario Fernando Jojoa Acosta, Flavio Comim, and Begoña Garcia ZapiiaZa Are ai systems biased against the poor? machine learning analysis using word2vec and glove embeddings. Al & society, (972)6117-632, 2024.
Hillary Dawkins. Second order winobias (sowinobias) test set for latent gender bias detection in coreference resolution. arv preprint ar arXi::2109.14047, 2021.
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi- Velez, and Steffen Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International conference on machine learning, pp. 1184-1193. PMLR, 2018
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 34, pp. 7659-7666, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Faimess through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226, 2012.
Virginia K Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. Winoqueer: A community-in-thh -iee----i--tt-ooooe loop benchmark for anti-Igbtq+ bias in large language models. Xiv preprint ar Xiv:233. 15087, 2023.
Susan T Fiske, Amy JC Cuddy, Peter Glick, and Jun Xu. A model of (often mixed) stereo- type content: Competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology, 82(6):878--00 2002.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger
(eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL hwwph:ttttss/sseeeeee http:///eiooocoo.////oo/ mlr, poress/448/ge /g/l 16. html.
12

---

Published as a conference paper at COLM 2025
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. Coun- terfactual faimess in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pp. 219-226, 2019.
Usman Gohar and Lu Cheng. A survey on intersectional fairness in machine learning: Notions, mitigation, and challenges. ariiv preprint ar.:::2305 069.9, 2023.
Juan Felipe Gomez, Caio Machado, Lucas Monteiro Paes, and Flavio Calmon. Algorithmic arbitrariness content moderation. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 7234-2253, 2024.
Rishav Hada, Safiya Husain, Varun Gumma, Harshita Diddee, Aditya Yadavalli, Agrima Seth, Nidhi Kulkarni, Ujwal Gadiraju, Aditya Vashistha, Vivek Seshadri, et al. Akal badi ya bias: An exploratory study of gender bias in hindi language technology. In The 2024 ACM Conference Fairness, Accountability, and Transparency, pp. 1926-1939, 2024.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Tamanna Hossain, Sunipa Dev, and Sameer Singh. Misgendered: Limits of large language models in understanding pronouns. arviv preprint ar Xii::2306003950 2023.
Phillip Howard, Avinash Madasu, Tiep Le, Gustavo Lujan Moreno, Anahita Bhiwandiwalla, and Vasudev Lal. Socialcounterfactualss Probing and mitigating intersectional social biases in vision-languaae models with counterfactual examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11975-11985, 2024.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3)457-506, 2021.
Da Ju, Karen Ullrich, and Adina Williams. Are female carpenters like blue bananas? corpus investigation of occupation gender typicality. In Findings of the Association for Computational Linguistics ACL 2024, pp. 4254-4274, 2024.
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, and Solon Barocas. Taxonomizing and measuring representational harms: look at image tagging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, Pp. 14277-14285, 2023.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL https: http///poeeeennc.eeee///o heapeddinnn.euuiiissp neurips. cc/paper/
Svetlana Kiritchenko and Saif M Mohammad. Examining gender and race bias in two hundred sentiment analysis systems. arXiv preprint arXiv:1805.04500 2018
Hannah Rose Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frederic A. Dreyer, Aleksandar Shtedritski, and Yuki M. Asano. Bias out-of-the-box: an empirical analysis of intersectional occupational biases in popular generative language models. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713845393.
Amanda Klysing, Anna Lindqvist, and Fredrik Björklund. Stereotype content at the inter- section of gender and sexual orientation. Frontiers in Psychology, 12:713839, 2021.
Youjin Kong. Are "intersectionally fair" ai algorithms really fair to women of color? philosophical analysis. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 494, 2022.
13

---

Published as a conference paper at COLM 2025
Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pp. 12-24, 2023.
Selim Kuzucu, Jiaee Cheong, Hatice Gunes, and Sinan Kalkan. Uncertainty as fairness measure. Journal of Artificial Intelligence Research, 81:307-335, 2024.
Balaji Lakshminarayanan Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Shahar Levy, Koren Lazar, and Gabriel Stanovsky. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. ar arXiv preprint Xitv:2009003858, 2021.
Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arviv preprint ar Xiv:2308.10119) 2023.
Geng Liu, Carlo Alberto Bono, and Francesco Pierri. Comparing diversity, negativity, and stereotypes in chinese-language ai technologies: a case study on baidu, ernie and qwen. arxiv preprint arXiv:2408.15686 2024.
Weicheng Ma, Brian Chiang, Tong Wu, Lili Wang, and Soroush Vosoughi. Intersectional stereotypes in large language models: Dataset and analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8589-8597, 2023.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. Xiv preprint ar ftt:2410.052299 2024.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H.S. Torr, and Yarin Gal.
Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Patlern Recognition (CVPR), pp. 24384-24394, June 2023.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arviv preprint ar XrXi:22004.09456, 2020.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. Crows-pairs: A challenge dataset for measuring social biases in masked language models. arXio preprint 1rXiv:2010.0013: 2020.
Anaelia Ovalle, Arjun Subramonian, Vagrant Gautam, Gilbert Gee, and Kai-Wei Chang. Fac- toring the matrix of domination: A critical review and reimagination of intersectionality in ai fairness. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp.
496-511, 2023.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086-21055 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/20 findings-acl.165. URL https: / http://caaaaaatolooooooooo org/2222. findings-acl. 165.
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer nlp. arXiv preprint ar arXv::205..12586 2022.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arviv preprint ar Xri::1804.09301, 2018
Abel Salinas, Parth Shah, Yuzhong Huang, Robert McCormack, and Fred Morstatter. The unequal opportunities of large language models: Examining demographic biases in job recommendations by chatgpt and llama. In Proceedings of the 3rd ACM Conference Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO '23, New York, NÝ, USA, 2023. Association for Computing Machinery. ISBN 9798400703812 doi: 00.1145/3617694.62320 URL https://dol org/10 0/5//0.111550 1 76222766177744.33277
14

---

Published as a conference paper at COLM 2025
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36:55565- 55581, 2023.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. arv preprint Xit::900900332 2019.
Harini Suresh and John Guttag. A framework for understanding sources of harm throughout the machine learning life cycle. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms,, and Optimization, pp. 1-9, 2021.
Anique Tahir, Lu Cheng, and Huan Liu. Fairness through aleatoric uncertainty. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM '23, Pp. 2372-2381, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701245. doi: (0.1145/3583780.600 3583780 3614875.
URL httpp:///ol/ http://ooooorg.....///// 1145/
Eva Vanmassenhove, Chris Emmery, and Dimitar Shterionov. Neutral rewriter: A rule- based and neural approach to automatic rewriting into gender-neutral alternatives. arviv preprint ar (rXin:2109.06155 2021.
Angelina Wang, Solon Barocas, Kristen Laird, and Hanna Wallach. Measuring representa- tional harms in image captioning. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency, pp. 324-335, 2022.
Hao Wang, Luxi He, Rui Gao, and Flavio Calmon. Aleatoric and epistemic discrimination: Fundamental limits of fairness interventions. Advances in Neural Information Processing Systems, 36, 2024.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-617, 2018.
Sean Williams and James Huckle. Easy problems that llms get wrong, arXiv preprint arXie:2405.19616, 2024.
Kyra Wilson and Aylin Caliskan. Gender, race, and intersectional bias in resume screening via language model retrieval. In Proceedings of the AAAT/ACM Conference on AI, Ethics, and Society, volume 7, pp. 1578-1590, 2024.
Jeongrok Yu, Seong Ug Kim, Jacob Choi, and Jinho D Choi. What is your favorite gender, mim? gender bias evaluation multilingual masked language models. Information, 15(9): 549, 2024.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th international conference on world wide web, pp. 1171-1180, 2017.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 15-20, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18633/w1/N11-20 URL https: hawlss/thollaaaooooooooomo org/N18 8 8-2003.
15

---

Published as a conference paper at COLM 2025
A
Appendix
A.1
Radar plots
This section supplements the discussion augmented coreference resolution using referent augmentation (R-Aug) ř 4.1. Coreference confidence (averaged over all prompts) with non-referent augmentations (NR-Aug) is shown in Fig. 5 (Type-1) and Fig. 6 (Type-2), and with contrastive augmentations (C-Aug) in Fig. 7 (Type-1) and Fig. (Type-2).
Models are more robust to non-referent augmentations than referent and contrastive aug- mentations, and on syntactically unambiguous Type-2 sentences than ambiguous Type-1 sentences, visually evident from Fig. 6 being the least skewed, and closest to the ideally fair brown polygon for most demographic attributes, compared to other augmentations and sentence type combinations.
lama3 70s InGrruEE
Mitral 8x78 Instruct
Mistral 7B inztrut V0.2
Pytha 126
Falcon 40e Inszrct
TI4E
Figure 5: Average coreference confidence on Type-1 sentences with non-referent augmen- tation (NR-Aug). In each radar plot,fem and masc represent single-axis gender attributes, whereas all other attributes are intersectional. The ideal coreference confidence but omitted from the plot to maintain visual clarity as all models are underconfident and have maximum coreference confidence 0.6
Pytha-176
Tale nak-on-Ale--mmttui
WOOD
Southian
Figure 6: Average coreference confidence on Type-2 sentences with non-referent augmen- tation (NR-Aug). In each radar plot,fem and masc represent single-axis gender attributes, whereas all other attributes are intersectional. The ideal coreference confidence 1, and is shown in brown.
16

---

Published as a conference paper at COLM 2025
Pyllcw11h
044
028
8:3
Figure 7: Average coreference confidence Type-1 sentences with contrastive augmentation (C-Aug). each radar plot, fem and mase represent single-axis gender attributes, whereas all other attributes are intersectional. The ideal coreference confidence is 1, but is omitted from the plot to maintain visual clarity as all models are underconfident
Iama3 70e instruct
Matro 9x78 Instruct
Micral 78 instruct
Pythi 12b
Falcon 400 Instruct
1003
MOPST
L.00
LES
2.50
1.00
Figure 8: Average coreference confidence Type-2 sentences with contrastive augmentation (C-Aug). each radar plot, fem and mase represent single-axis gender attributes, whereas all other attributes intersectional. The ideal coreference confidence is 1, and is shown in brown.
A.2 Line plots
In this section, we compare the effect of different demographic augmentations on coreference confidence on Type-1 sentences in Fig. 9 and for Type-2 sentences in Fig. 10, supplementing the discussion about hegemonic augmentations in $ 4.1. The 'baseline' indicates performance without demographic augmentation. We can see that coreference confidence decreases with referent augmentation even for hegemonic markers, for all 10 attributes (subplots), on both Type-1 and Type-2 sentences, and for all models with the exception of Pythia, which has very low coreference confidence before augmentation itself and is thereby more stable to referent and contrastive augmentations than other more performant models.
17

---

Published as a conference paper at COLM 2025
Condoroncc Carficnns
Caro CLNRANTEE
auprertations Wereeces
Conterecc centreence
rebg sugree molian
C
B phong
ain Mnreapood
Easettin
Calip Hayente
C dha
Colig
Corelrrance Contalerce
Deoder ideatity as gwwereetooncccommmm Type sederres
recus prerittion ecpeentttoon cn perfences
Canofonence ce Conference
12
R.
Keopp fonsanneer
an-tug trasperieer
Colige cepender
Saviine
RAli Areetcan
Kalig paigrimangnen
pisti Airertaa
Aaga iomiinnnn
CAugi heggimmmeen Coigiemmrrnt
Conteoecce Carfeeee
NST www.ytare Type SrEEE
Coneloriina Confidencce
Eype augmentatioss gydel
Conaeorncc Contiancce
spcis eceromic STATU augmertidsss typel centenoos
8.
Aosatire
Rauggt fet
Aay oot
der
Cltign etio
Aulay ooor
aataag prur
Caligrid
Calop
tabiitt suçmentations typel cenbences
Conetrrrcee Confience
Earma Tee-----ttttuuteerr inttructffer
8
Motral 8x78 instruct
8x70 insiiistucctfo feom
Batatner
ettatiee
Reugerr
tet dide
wn diatted
4R-ue-t0 4RAug-47
C-up able
Caupullabled
Golgg w7
Cotagial
Falcon-400 instrutt masc instrutt
Figure 9: Coreference confidence (averaged over all prompts) for all augmentations on Type-1 sentences for all models (colors) and both masculine (bold) and feminine (dashed line) pronouns. Baseline indicates performance without any augmentation.
Ilama3
mixtral
mistral
pythia
falcom
Ace
CC
Acc
CC
Acc
CC
Acc
CC
Acc
CC
No aug
0.699
0.322 0.693
0.669
0.259 = 0.677
0.637
0.252 = 0.855
0.570
0.064 = 0.397
0.711
0.334 = 0.558
typel
NR-Aug
R-Aug
0.202
0.708
0.129
0.36 = 0.541
0.491
0.608
0.701
-0.005
0.351
= 0.495
0.485
0.695
0.553
0.438
-0.142 a 0.521
0.528
0.628
0.487
-0.035
0.112 0.293
2.255.0... 0.287
0.803
0.372
0.395
-0.002 a: 0.38
0.403
C-Aug
0.647
0.629
0.554
0.697
No aoa
0.969
0813 = 0,285
0.926
0.629 = 0.353
0.875
0.689 = 0.545
0.831
0.291 = 0.284
0.877
0.487 = 0.375
type2
R-Aug
NR-Aug
0.937
0.259 0.28
0.944
(1872
0.267
0.2 0.294
0.767
0.144 0:407
0.765
0.12 0.199
0.229
0.698
0.945
0.104 0.283
C-Aug
0.916
0.225 0.251
0.933
0.33 a 0.238
0.887
0.244 a 0.307
0.843
0.146 : 0.171
0.872
0.18 0.2
Table 3: Coreference accuracy (Acc) and confidence (CC) under different augmentations. The mean and Std coference confidence is reported. No aug indicates performance on WinoBias base prompts without demographic augmentation
A.3
Accuracy versus uncertainty comparisons
A.3.1
Coreference confidence reveals bias, while accuracy obscures it
In this section we compare accuracy and uncertainty behaviors before and after demographic augmentations, corroborating the discussion in $ 4.1. In Tab. we report the average coreference accuracy (over all base prompts) and coreference confidence (additionally averaged over all demographic markers) under different augmentations. Surprisingly, the highest accuracy that models are able to reach Type-1 sentences before demographic
18

---

Published as a conference paper at COLM 2025
Corcfrrocc corfdooo T. 1
Canwereeee Centeerece
Conttrence Contsrecc
CHANTIN
Pilao pranty
Rebion old
Roor stea
tot Caty 4eaco
Colip titeeee
Kuliig chrosien
No deu Hostim
all pootd
As chestan
Calo Cronston
Clip
Ciratrrence Corfience x
Mentiy gmvetaitne centanoes
Confalecce
Condluuunc Coll dennce Cronbeecco Contoorr
ralionatryy Rigmer noperlaalzra Tebiora on hyp2 ce perleeoee
roppende
est Aigh Augeingennt
Elgenisder
Caught thrggeoee
Bavettse
NR pey
Galog hertrtienna
Calag Armetiaa
aegange Aug DJzz ONDWCIEE
Conferette Cie Bnderte
type
7p0?
Corfiderne
SOC Peeeccoomii strtis asgnoratioss on byoz sentences
culog www.uuhpppaiin sosalaon
Edlog Caggrtanttggiiiccoo oo
Ruuapittr
suge
set Ai-eopotioo
Caligg op
Caloge
desciio
Riiig stor
Gabo rid
Coreterance Confeence
Coubiily wugnenial 2pa2
Bamaa Instrcct
Motral www.-nccrtttrrtnnno.o
Hottrll Be si---rst crtffm
Histral 75 instruct v0.2 masc
22
Pythle 12b ane
Mistral-7e instruct-v..2
Pythlo 12b fo-ttem
Oneelie
Rekig diontled
Buligg Af
Rilagg RD
ARt degdg attle
MR-Aag NT
MiAuppp0
Glug adie
Cats stsatta
Catiger
Coup
Polcon 400 insbuct
Reiiog
Nalcon-tt Micen-tb-iraauue -b-iraaruuttbrrr
Figure 10: Coreference confidence (averaged over all prompts) for all augmentations on Type-2 sentences for all models (colors) and both masculine (bold) and feminine (dashed line) pronouns. Baseline indicates performance without any augmentation.
augmentations is 73% (Falcon), with Pythia doing as poorly 57%. Additionally, models are highly uncertain on Type-1 sentences without any demographic augmentation, with mean coreference confidences under 30,, and more accurate models generally having higher mean coreference confidence. Models are more accurate (83-96%) on Type-2 sentences that are syntactically unambiguous, but models whose accuracies differ by less than 5% have confidences that differ by 20% or more.
Accuracy and confidence always decrease under referent augmentation, as discussed pre- viously. Surprisingly, accuracy increases under non-referent augmentation for all models (with the exception of Ilama3) on both Type-1 and Type-2 sentences. Coreference confidence, however, increases only for Type-1 sentences and decreases for Type-2 sentences with non- referent augmentations, for all models. This is an indication of bias because it suggests that prepending a demographic marker to the non-referent makes the model less likely to pick it, which explains why confidence would increase only in the ambiguous case. This finding demonstrates that imcertainty-base evaluations able to detect bias better than accuracy, which increases in both ambiguous and unambiguous cases and could be misinterpreted as improved performance rather than increased bias. Notably, the best-performing model i.e., Ilama3, is the only model that does not show this spurious increase in accuracy under non-referent augmentation Type-2 sentences.
19

---

Published as a conference paper at COLM 2025
Ilama3
mixtral
mistral
pythia
falcon
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
no auginentation
0.097
0.065
0.001
0.068
0.056
0.178
0.007
0.055
0.023
0.019
non-demographic
0.081
0.052
0.016
0.052
0,049
0.165
0.002
0.039
0.006
0.023
age
0.192
0.026
0.117
0.115
0.147
0,1
0.016
0,028
0,069
0.04
body type
0.259
0.251
0.153
0.157
0.243
0.392
0.072
0.053
0.152
0.118
disability
0.125
0.203
0.251
R-Aug
language
gender identity
0.145
0.172
0.151
0:113
0.136
0.349
0.141
0.079
0.255
0.269
0.063
0.108
0.102
0.101
0.075
0.176
0.064
0.152
nationality
0.185
0.016
0.112
0.109
0.094
0.119
0.014
0,028
0.09
0.041
sexual orientation
0.25
0.346
0.346
0.194
0.23
0.204
0.11
0.087
0.227
0.18
socio-rconomic status
0.382
0.213
0.271
0.389
0.134
0.292
race
0.127
0.098
0.242
0.329
0.212
0.16
0.087
0.122
0.121
0.136
zeligion
0.105
0.054
0.042
age body type
0.089
0.105
0.104
0.102
0.103
0.177
0.019
0.111
0.062
0.118
0.134
0.083
0.138
0.155
0.167
0.067
0.109
0.089
0,06
disabiiiity
0.113
0:109
0.111
0.194
0.125
0.251
0.056
0,081
0.218
0.228
gender identity
0.224
0.265
0.384
0.228
0.275
0.252
0.111
0.088
0.28
0.317
NR-Aug
nationality
language
0.052
0.103
0.122
0.078
0,098
0.136
0.132
0.04
0,361
0.159
0.033
0.076
0.068
0.075
0.047
0.122
0.038
sexual orientation
0.107
0.32
socio- economic status
0.203
0.141
0.12
0.164
0.317
0.201
0.101
0.061
0.173
0.209
0.152
0.161
0,232
0.123
0.145
0,278
0,082
0.092
0.129
0,157
religion
0.019
0.132
0.061
0.08
0.102
0.205
0.024
0.096
0.093
0.111
age body type
0.091
0.029
0,087
0,056
0.067
0.067
0.019
0.033
0.052
0.049
0.165
0.086
0.14
0.104
0.317
0.243
0.075
0.048
0.195
0.136
disability
0.196
0.252
0.154
0,181
0.163
0.409
0.059
0.087
0.261
0.202
gender identity
0.091
0.196
C-Aug
0.155
0.061
0.1
0.082
0,206
0.37
0.069
0.059
0.088
0.147
0.24
0.09
0.057
0.063
0.072
0.053
0.016
0.034
0.1
0.044
sexual orientation
0.286
0.287
0.469
0.189
0.284
U.149
0.169
0.162
0.124
0.U74
socio-economic status
0.19
0.151
0.194
0.211
0.403
0.294
0.097
0.05
0.273
0.213
race religion
0.055
0.172
0.047
0.058
0.095
0.045
0.09
0.105
0.026
0.043
0.056
0.072
Table 4: Maximum subgroup coreference confidence disparity, defined in Eq. (2), for all augmentations and all models. Values greater than 0.2 are shown in bold. Lower values indicate fairer model behavior.
A.3.2 Accuracy disparities corroborate confidence disparities
In this section, we compare maximum subgroup disparities in coreference confidence (re- ported in Tab. 4) with those in accuracy (reported in Tab. 5), under different augmentations. We find significant disparities (20-46%) in both accuracy and coreference confidence. While there isn't consistent trend of one being larger than the other, significant accuracy and uncertainty disparities generally coexist in the same demographic attribute (for example, sexual orientation, gender identity, etc)
A.4 Occupation analysis
In this section, we provide additional evidence of stereotyping harm towards intersectional identities, for Llama3 in Fig. 11, Mixtral in Fig. 12, Falcon in Fig. 13 and for Pythia in Fig. 14, corroborating the discussion in $ 4.2.
20

---

Published as a conference paper at COLM 2025
1y502 S01E27A0
Jamas-/tb-rstou tppsz SentErees
0.33
0.37
6.11
622
65.5
02.8
611
41007
40003
vorter
006
0333
10
0.50
0.28
022
022
035
04
3.50
325
015
425
18
0.25
0.340
5.25
00013
0.54
1.021
23
0.22
1.000
11.0
5.00
22
0002
02:4
0.23
0.3
0:23
032
pard
0.19
08.3
0.54
0.13
033
andor
0.24
3.50
00:1
0.24
0.290
0.23
055
3.50
001
0.25
5.8
103
0110
0.29
0.1
044
54
621
0.18
0.2
0.25
033
0.00
indepereo
081
117
0222
6.77
ACOVSRIEE
1224
6.1
550
0.54
E2
000
100
032
-0335
2117
0141
0.33
00
333
000
33
0.14
0.41
0.3
005
14.00
1121
-3.00
0.34
11
110
0110
0119
100
1007
-0.79
Ieaduger
320
0.77
hirthaue
002
100
0.33
-L.0
140
gec
(a) Gender identity
(b) Sexual orientation
Figure 11: Llama3 average subgroup coreference confidence on Type-2 sentences with contrastive augmentation (C-Aug), broken down by referent occupation. Values close to 1 indicate that the model is correct and confident, values around 0 indicate the model is highly uncertain, negative values indicate the model is wrong.
Midra
Bx75/"StIKt
Eype2 sortte
Bedsttrttruct..mmm typt2 senterces
redaaee
639
831
mechanc
0.20
0.35
0.25
1.95
0.35
Wetter
0.20
0.341
033
6:28
6.2
639
0.75
diver
3.30
5.10
175
837
175
18
mwwe
8638
0.79
6:78
632
Tover-
072
0.30
3.30
3.30
0.38
0.314
33
dewdkger
0.25
0.24
0.38
0.38
01:1
0.35
1.25
6117
22
18:08
08:0
1124
pardd
0:14
0.30
0668
022
0.18
050
0.79
0.5
32
0.30
0004
0.41
0.55
0.5
0.17
0.54
0.33
phydece
1.11
100.
16.0
0.37
0.31
634
0.1
1.51
1.34
3226
0.7
0.344
6.3
3.61
sufopencc
0.62
0.64
0.37
04
042
0.1
11:0
11:0
01:
1.247
aconmiion
3.5
0.43
6:2
033
6.59
000
9.2
1020
622
022
-0335
0.022
10
01:1
1.10
6.1
040
1000
0
1:05
10
1002
0.0
042
027
037
-0.50
1.6%
83
6.34
6.71
0:11
11
0.33
42
desrer
81
-0.99
6.44
5.51
6.55
0111
30
hartooe
042
secieary
0.01
-100
ge:
(a) Gender identity
(b) Sexual orientation
Figure 12: Mixtral average subgroup coreference confidence on Type-2 sentences with contrastive augmentation (C-Aug), broken down by referent occupation. Values close to 1 indicate that the model is correct and confident, values around 0 indicate the model is highly uncertain, negative values indicate the model is wrong.
21

---

Published as a conference paper at COLM 2025
hikaar-40--inu Jilcn typec Jeniencos
hicon- Erstict bypcz pcriee1cs
0:38
0:21
021
0.11
566
06.2
13.0
0:48
41068
0.12
0
0:00
81.3
61:4
032
0.350
0.15
1.43
63
0.04
0.350
0.40
012911
333
9.23
425
18
0.11
0.28
0.24
0.77
0:11
8.10
1.00
00.03
18:0
23
0.037
0.15
13:0
4.014
0.35
0.507
22
part
0011
0.33
022
0:3
021
0.50
park
0.44
1825
18
0.34
021
0.34
3.40
3.50
023
0.23
0.23
08.0
1.24
6:048
0.2
0155
0.21
0233
0.16
0.28
6.008
199
0.25
0.10
1631
0.24
123
00300
0.13
012
9.25
1.90
0.55
012
0.16
0.16
6:3
8.59
0.11
0.31
0.33
0.28
1340
021
4.00
inlepperce
011
6127
012
01:3
0.311
10403
0.74
0.40
0.33
0.300
0331
5.20
1.22
031
0.0
5.51
E23
100
0.66
1322
033
-0.35
01801
1.591
14.5
1832
0.25
0.21
0.01
0.11
023
0.54
1.04
143
10
068
-3.00
0.34
0.30
627
120
10:3
-0.79
Imadugger
0000
6.50
nara
8.17
hirthaue
0.20
2.355
66603
0.1
0.3
100
-L.0
(a) Gender identity
(b) Sexual orientation
Figure 13: Falcon average subgroup coreference confidence on Type-2 sentences with contrastive augmentation (C-Aug), broken down by referent occupation. Values close to 1 indicate that the model is correct and confident, values around 0 indicate the model is highly uncertain, negative values indicate the model is wrong.
Pythib
12:
bypc temoes
Pythib
on bycc scriencess
6.11
677
mechancc
0.11
1.20
0.31
0.3
(9.73
00:03
0.14
00637
0.36
3.331
0.38
02
0.300
1.51
0:381
6.22
0081
5397
175
18
0:013
5.305
2.17
5.200
0.0.0
0.30
0.055
0329
1,300
00033
0.30
33
develger-
0.300
0.32
0.009
22
1.811
parrd
01945
6133
0.33
30043
6228
022
0.50
0.2
03
0.25
0137
100k
0.41
0.24
0.055
1.34
phytice
11000
000
1301
0.45
132
326
653
8.74
635
0.25
1330
30000
612
0.30
0.22
0.34
0.355
615
NAST
5.13
1326
1.75
53117
0.20
0341
6,01
130
535
617
0.12
0.20
1336
6003
627
032
-0235
(35)
0.3
6.522
10
011
0.11
117
1113
6:3
6.3
131
200033
6397
6000
02
006
10
atandert
0014
022
0331
6002
612
045
-0.00
83
0.18
0.1
0.00
0290
637
30681
11
0311
001
104
42
desrer
040354
199
-073
C54)
040
4017
10.
100
exceptorie
1838
0.32
0.681
632
633
30
hartooee
3.45
315
(35)
024
011
seciearry
4.001
002
-1.0
(a) Gender identity
(b) Sexual orientation
Figure 14: Pythia average subgroup coreference confidence on Type-2 sentences with contrastive augmentation (C-Aug), broken down by referent occupation. Values close to 1 indicate that the model is correct and confident, values around 0 indicate the model is highly uncertain, negative values indicate the model is wrong.
22

---

Published as a conference paper at COLM 2025
Ilama3
mixtral
mistral
pythia
falcon
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
Type-1
Type-2
boseline augmentationn)
0.082
0.019
0.009
0.06
0.041
0.079
0.008
0.022
0.023
0.055
age
0.096
0.064
0.101
0.21
0.083
0.197
0.067
0.071
0.054
0.0K3
body type
0.119
0.092
0.113
0.141
0.151
0.258
0.143
0.085
0.115
0.12
disability
0.105
0.046
0.123
0.127
0.117
0.235
0.144
0.086
0.174
gender identity
0.161
0.017
0.338
0.09
0.248
0.099
0,232
0.228
0.254
0.224
R-Aug
language
0.092
0.034
0.072
0,103
0.092
0.198
0.115
0.09
0.067
0.046
0
0.042
0.104
0.139
0.005
0.197
0.052
0.066
0.113
0.108
sexual orientation
0.147
0.062
0.252
0.22
0.225
0.188
0.172
0.24
0.249
0.239
socio-economicc status
0.145
0.073
0.109
0.159
0.172
0.244
0.104
0.077
0.256
0.271
0.18
0.077
0.169
0.18
0.201
0.172
0.151
0.172
0.19
religion
0.081
0.087
0.075
0.148
0.068
0.14
0.106
0.063
0.09
0,046
0.077
0.068
0.081
0.067
0.07
0.067
0.043
0.045
0.034
body type
0,071
0.043
0.094
0.081
0.092
0.135
0.089
0,075
0.049
disabifity
0.08
0.057
0072
0,065
0.123
0.055
0.105
01002
0.111
0.07
gender identity
0,231
0.109
0.334
0.177
0.39
0.216
0.235
0.119
0.239
0.177
NR-Aug
language
0,097
0.037
0.096
0,122
0.098
0.213
0.134
0.103
0.066
0.042
nationality
0.089
0.049
0072
0,077
0.004
0.48
0.055
0.09
0.777
0.053
sexual orientation
0,233
0.026
0.189
0,044
0.26
0.041
0.177
0.068
0.233
0.031
sucio economic status
0.105
0,059
0.081
0.136
0.178
0.112
0.103
0.056
0.133
0.138
race
0.152
0.201
0,088
0.126
0:112
0,202
0.069
0.101
0.048
religion
0,055
0.044
0.069
0.047
0.073
0.038
0.089
0.022
0.083
0.012
age body type
0.132
0.078
0.167
0.165
0.108
0.122
0.08
0.054
0.068
0.119
0.083
0.126
0,127
0.216
0.166
0.176
0.103
0.212
0.136
disability
0.167
0.085
0.125
0.056
0.137
0.062
0.14
0.155
0.265
0.131
gender identity
0.236
0.093
0.24
0.084
0.187
0.171
0.362
0.343
0.385
0.292
C-Aug
nationality
language
0.145
0.11
0.032
0.08
0.117
0.106
0.117
0.067
0.119
0.247
0.078
0.047
0.11
0.124
0.013
0.1145
0.046
0.126
0.08
sexual orientation socio-economic status
0.185
0.085
0.391
0,152
0.457
0.156
0.312
0.295
0.218
0.131
0.127
0,171
0.171
0.159
0.133
0.094
race
0.164
0.07
0.078
0.09
0.146
0.099
0.118
0.04
0.1
0.12
religion
0.073
0.099
0.11
0.053
0.223
0.072
0.046
0.058
0.085
0.038
Table 5: Maximum subgroup accuracy disparity, computed by replacing the per-sample coreference confidence CC(L,) in Eq. (2) with the per sample accuracy Acc(L;)
I[CC(Li) > Oj. Values greater than 0.2 are shown in bold. Lower values indicate fairer model behavior.
A.5 Chain of Thought (CoT) Prompting with Mistral
Implementation details We add a single reasoning step: First the model is shown the sentence and prompted "Let's think step by step and first show your reasoning steps." The reasoning is then appended with role of assssstant' and the model is re-prompted to perform coreference resolution.
Example
[('role': 'user', 'content': 'The developer argued with the designer and slapped him in the face. Which occupation does the pronoun "him" refer to? Let\'s think step by step and first show your reasoning steps."), {' ('role': 'assistant', 'content': 'The pronoun "him" in this context refers to the designer. The reason for this is that the argument was between the developer and the designer, and the developer was the one who slapped the designer. Therefore, him" is the person who received the action of being slapped, which was the designer .
'},
{' ('role': 'user', 'content': 'Based on the reasoning above, the pronoun "him" refers to the }]
Results The average coreference confidence with referent augmentation (R-Aug) and non- referent augmentation (NR-Aug) is reported in Tab. 6 and Tab. 7 respectively. The maximum subgroup confidence disparities are reported in Tab. 8 (R-Aug) and Tab. (NR-Aug).
We find that (i) Disparities are generally lower with CoT, although some significant dispari- ties (larger than 20%, as bolded in Table 2) remain, for example along socio-economic status (ses), shown in the confidence disparity tables (ii) CoT prompting is more likely to decrease the coreference confidence than increase it, for both Type1 and Type2 sentences, shown in the average coreference confidence tables.
23

---

Published as a conference paper at COLM 2025
In summary: while CoT can be an effective mitigation for intersectional unfairness, it trades off parity in confidence with overall confidence in coreference resolution.
Type-2
Type-1
Category
fem
masc
fem
masc
CoTl
W/o
CoTT
W/o
CoT
W/o
CoT
W/o
no augmentation
0,48027
0.59998
0.53173
0.77778
0.14356
0.28199
0.11929
0.22315
gender identity
0.02577
0.15221
0.01180
0.13230
-0.06301
0.07407
-0.06133
-0.18426
sexuality
0.06895
0.05746
0.05455
0.15630
-0.16814
-0,19958
+0.03594
-0.10934
disability
-0.04992
0,20129
0.04390
0,28150
-0.13679
-0.05867
-0.08525
-0.14185
race
0.00932
-0,00407
-0,00543
0,01631
-0.15457
-0.16929
-0.08663
-0.27509
body-type
0.15377
0.14944
0.11684
0.30316
-0.12688
-0.13755
-0.06916
-0.17015
ses
0.15426
0.24634
0.11537
0.35428
-0.13218
-0.06511
-0.03368
-0.00642
Table 6: Average coreference confidence with referent augmentation (R-Aug), by gender (fem/ (emm/ and reasoning condition (CoT W/o reasoning).
Type-2
Type-1
Category
fem
masc
fem
masc
Co'T
W/o
COT
W/o
CoT
W/o
CoT
W/o
no augmentation
0.48027
0.59998
0.53173
0.77778
0.14356
0.28199
0.11929
0.22315
gender identity
0.38616
0.21503
0.43318
0.45693
0.13134
0.06436
0.10511
0.32744
sexuality
0.49109
0.55585
0.51137
0.64240
0.27908
0.45412
0.13668
0.32019
disability
0.55391
0.65620
0.58666
0.76951
0.25809
0.37970
0.20240
0.43301
race
0.51507
0.61718
0.56587
0.74405
0.26808
0.46534
0.19041
0.51683
body.typ type ses
0.53912
0.72838
0.59956
0.81667
0.27891
0.51217
0.20496
0,49052
0.52142
0.67460
0.54984
0.74359
0.29115
0.47249
0.17603
0.34264
Table 7: Average coreference confidence with non-referent augmentation (NR-Aug), by gender /maac)) and reasoning condition (CoT / W W/o reasoning).
Marker
Type-2 CoT
Type-2 W/o reasoning
Type-1 CoT
Type-1 W/o reasoning
no augmentation
0.05146
0.178
0.02427
0.056
gender identity
0.04610
0.063
0.04358
0.269
sexuality
0.05377
0.204
0.17917
0.230
disability
0.16975
0.382
0.09655
0.222
race
0.16975
0.160
0.10027
0.212
body_type ses
0.16424
0.392
0.12838
0.243
0.21580
0.400
0.23435
0.389
Table 8: Confidence disparities under referent augmentation (R-Aug), with and without reasoning.
A.6 Non-demographic augmentation baseline
We supplement the no augmentation baseline with non-demographic baseline. We include this baseline in order to distinguish between brittleness that is deeooraphicallyysaliiee (and therefore an ethical concern) and brittleness that is inherent. Put differently, the goal of this experiment is to evaluate whether any additional information about the referent- demographic or otherwise-is treated as relevant by virtue it being conveyed. Here we compare with referent augmentations (R-Aug) using non-demographic markers ('confused', 'unfocused', 'relaxed', 'focused', 'unperturbed', 'regular', 'usual', 'routine', 'normal'). All other experimental settings are the same with demographic augmentations.
Results We find that non-demographic augmentations do decrease the coreference confi- dence, but (i) comparably for both male and female pronouns, shown in Tab. 2, and (ii)
24

---

Published as a conference paper at COLM 2025
Marker
Type-2 CoT
Type-2 W/o reasoning
Type-1 CoT
Type-1 W/o reasoning
augmentation
0.05146
0.178
0.02427
0.056
gender.identig
0.10952
0.252
0.06099
0.275
sexuality
0.09723
0.163
0.19421
0.284
disability
0.16354
0.251
0,09291
0.175
race
0.12641
0.278
0.14985
0.145
_tpe
0.10828
0.166
0.11788
0.167
ses
0.03676
0.201
0.19483
0.317
Table 9: Confidence disparities under non-referent augmentation (NR-Aug), with and without reasoning.
to lesser extent than demographic augmentations, shown in Tab. 10. This additional experiment shows that models are sensitive to any augmentation, demographic or not, further supporting our claims of memorization and invalidity. Secondly, it shows that intersectional unfairness does exist, since demographic augmentations affect pronouns disparately, whereas non-demographic augmentations do not.
llama3
mistral
mixtral
pythia
falcon
typel mase
type2
typel
type2
typel masc
type2 fem
typel
type2
type1 masc
type2
fem
mase
fem
masc
fem
fem
no augmentation
0.369
0.272
0.845
0.780
0.223
0.282
0.778
0.600
0.259
0.258
non-demographic
0.273
0.193
0.502
0.450
-0.031
0.022
0.467
0.305
0.050
0.065
disability 0.153
0.091
0.260
0.249
-0.142
-0.059
0.282
0.201
0.024
0.052
Table 10: Non-demogiaphic augmentation: Coreference confidence (Equation 1) averaged over all sentences, as reported in the radar plots (Fig. and Fig. 3). Values are further averaged over all respective markers.
25


Reviews:
[{"review_id": "TG0LmszORX", "reviewer": "Reviewer_LZzW", "summary": "**Empiricism, Data, and Evaluation** and **Technological Impact**- the data augmentation technique proposed in this paper is quite important and would be impactful for synthesizing intersectional examples on top of existing social bias datasets.\n\n**Understanding Depth, Principled Approach**- the paper and the data construction demonstrated a deep understanding of WinoBias and related NLP techniques.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Innovative approach to mitigate the \"memorization\" issue. While the original dataset might have been included in LLM training data, the synthetic/augmented version addresses this concern- as seen in elongated shape of the polygons in this paper (wider in the horizontal/memorized direction, narrower in the synthetic/slanted directions.) \n\nInnovative technique for visualization intersectional fairness features- the polygon, as noted above, are quite useful for comparing intersectional fairness properties at a glance. This design could be of great significance to the community.", "reasons_to_reject": "While the paper mentioned the potential difference between \"reasoning\" and \"memorization\" in the introduction, the paper itself didn't seem to use chain-of-thought for reasoning (L228, \"next-token probabilities\" only). Maybe the authors could consider including chain-of-thought reasoning and o1/R1-styled reasoning models could be a follow-up work.", "questions_to_authors": "Are there ways to avoid the combinatorial explosion? For example, would some random subsampling technique be sufficient without sacrificing too much precision?", "extra_scores": {}}, {"review_id": "xHUg2mey6C", "reviewer": "Reviewer_z92p", "summary": "The paper introduces WinoIdentity, an expansion of the WinoBias evaluation set to measure potential biases across intersectional identities. It includes 25 demographics markers across 10 attributes, intersected with gender, producing 245,700 prompts to evaluate bias patterns. The authors also introduce a bias metric to test the relative confidence models have for identity combinations within these prompts. Their evaluations find notable confidence discrepancies for some intersectional identities, especially in anti-stereotypical settings.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper offers many positives, including:\n\n* It is clearly written and well visualized, making the analysis easy to follow.\n* The authors have thought deeply about the different ways to derive new prompts/settings that explore intersectional identity biases. The combinations expose complex identity relationships that expand significantly on WinoBias.\n* The authors have been thoughtful about validity and value misalignment, in assessing predictive systems in critical social contexts. This discussion was helpful in understanding the failure modes.", "reasons_to_reject": "While the benchmark and analyses have a clear design, there are some broader questions about how realistic and helpful these types of static prompt evaluations really are in measuring real-world bias.\n\n* These types of synthetic evaluation bias benchmarks, where prompts are constructed to play on stereotypes, and the model is asked to select from an essentially multiple choice answer, saw a great deal of popularity in the last few years. However, one of their core critiques is because of their contrived nature, they remain ungrounded in real bias, discrimination settings, and may not provide realistic measures for these real-world cases.\n* Among these types of synthetic benchmarks, WinoIdentity seems to expand markedly on prior resources, which is positive, but I would urge the authors to discuss its real-world applicability to the extent they can.\n* Aside from the synthetic prompt construction, it isn’t clear the coreference confidence disparity metric translates to real harm behaviors?", "questions_to_authors": "* To what extent do you see “coreference confidence disparity” reflecting real harm?\n* Are findings robust: multiple variants of the phrasings consistently yield similar metrics?\n* Based on your findings, what concrete recommendations do you have for the model developer or broader community?", "extra_scores": {}}, {"review_id": "aNzuqcPWiy", "reviewer": "Reviewer_rGgN", "summary": "This paper presents an analysis of intersectional bias in LLMs. A new dataset based on WinoBias, named WinoIdentity, is proposed as a testbed and a new group fairness metric, named Coreference Confidence Disparity, is developed to measure bias. Five recent LLMs are evaluated using this dataset and the metric. They find high confidence disparities along various demographic attributes for the models and models being most uncertain about doubly-disadvantaged identities. They also find similar confidence decreases for hegemonic markers, which they attribute to LLM memorization.\n\nThis is a well-written paper that makes a significant contribution to the literature on bias/fairness by focusing on intersectional bias. The dataset and the metric are well-justified. The evaluation is extensive and the analysis is generally insightful. Because there are many results, the discussion gets quite dense at times and the tables, figures become hard to interpret. Overall, the findings could be presented a bit more clearly. While the dataset and the metric seem useful, the work is limited to an investigation of bias in LLMs and doesn't offer an intervention to reduce LLM bias.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Dataset seems broadly useful to study LLM bias, and include many demographic attributes. \n- Intersectional bias is understudied. \n- Focus on model uncertainty for bias evaluation is sensible and the Coreference Confidence Disparity metric could be broadly useful. \n- The evaluation is quite extensive  and detailed.\n- The paper is well-written and the methodology is clearly explained and justified.", "reasons_to_reject": "- The paper is limited to measuring bias in existing LLMs, not reducing bias.", "questions_to_authors": "- Are there hallucinations in generating tokens for occupations? If so, how are they handled?", "extra_scores": {}}, {"review_id": "qT4b7Eectr", "reviewer": "Reviewer_orsS", "summary": "This paper presents an extension on a dataset and an evaluation framework to measure the interesectional bias of LLMs in coreference resolution. The paper evaluates on 10 demographic attributes, with 11 markers in the privileged group and 14 in the disadvantaged group. The WinoBias dataset is extended in 3 ways: prepending the demographic marker to the referent entity, prepending to the non-referent entity, and prepending to both entities where a contrastive marker is used for the non-referent entity. For model evaluation, greedy decoding is used and the log probability of each candidate occupation word is measured. The paper proposes the metric of coreference confidence as the difference between the probabilities of the correct and incorrect occupation words. The coreference conference disparity is defined as the maximum difference of the corefence confidences between the privilege and unprivileged groups for a given demographic attribute. The paper finds that intersectional bias exists across all evaluated LLMs, in particular when adding on to the gender attribute. Interestingly, the paper finds that adding advantaged markers to the referent entity (but not the non-referent entity) also results in drops in coreference confidence scores, especially for the more challenging Type 2 sentences. When examining results by male or female-dominated occupations, the paper also finds evidence reinforcing these biases in the confidence scores, even though the answer is correct.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clearly written with a comprehensive explanation of the prior work and how this work builds on it.\n\nThe paper contributes a dataset with three types of extensions to the WinoBias dataset. \n\nThe paper presents an evaluation with strongly motivated metrics to measure disparity across several LLMs and several demographic attributes.", "reasons_to_reject": "The premise of the task is problematic. In particular, expecting there to be little/no change in probabilities after adding a demographic marker may not be correct. In human communication, people follow the Gricean maxims of conversation, and in particular the maxim of relation. If a person is giving more information than required (i.e., the demographic marker), then the other person assumes it is relevant and would give more weight and meaning to this extra information. Thus, even a human would be expected to change their confidence when given extra information that would normally be considered irrelevant.\n\nThe paper lacks evidence that the LLM probabilities should be equated with their confidence.", "questions_to_authors": "1. How is this task reconciled with the human expectation that probabilities should change, if following the maxim of relation?\n\n1. Do the probabilities need to be calibrated? This paper may be useful: A Survey of Confidence Estimation and Calibration in Large Language Models (Geng et al., NAACL 2024).", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel method of augmenting existing social bias evaluation datasets (primarily WinoBias) to then effectively evaluate intersectional biases. Intersectional biases are harder to examine, an are under studied, making this an important contribution. Reviewers also note this novelty in approaching intersectional bias as a strength of the work.
The paper is well written and extremely well visualized and will be a useful read to the community.
Some potential things to add in camera ready include (i) how do such prompts relate to natural conversation with a chatbot? Some discussion on this would help understand the impact of this evaluation dataset, (ii) human annotation is indeed expensive. A clearer reasoning and acknowledgement of this in the  body of the work can help users of the work walk away with clearer next steps.
Decision: Accept