Agree to Disagree? A Meta-Evaluation of LLM Misgendering
Authors: Arjun Subramonian
Vagrant Gautam
Preethi Seshadri
Dietrich Klakow
Kai-Wei Chang
Yizhou Sun
Venue: COLM 2025 (2025)
Keywords: fairness
meta-evaluation
misgendering
TL;DR: We conduct a systematic meta-evaluation of different methods for measuring LLM misgendering across three datasets and find that they can disagree.

Abstract: Numerous methods have been proposed to measure LLM misgendering, including probability-based evaluations (e.g., automatically with templatic sentences) and generation-based evaluations (e.g., with automatic heuristics or human validation).
However, it has gone unexamined whether these evaluation methods have convergent validity, that is, whether their results align.
Therefore, we conduct a systematic meta-evaluation of these methods across three existing datasets for LLM misgendering.
We propose a method to transform each dataset to enable parallel probability- and generation-based evaluation.
Then, by automatically evaluating a suite of 6 models from 3 families, we find that these methods can disagree with each other at the instance, dataset, and model levels, conflicting on 20.2% of evaluation instances.
Finally, with a human evaluation of 2400 LLM generations, we show that misgendering behaviour is complex and goes far beyond pronouns, which automatic evaluations are not currently designed to capture, suggesting essential disagreement with human evaluations.
Based on our findings, we provide recommendations for future evaluations of LLM misgendering.
Our results are also more widely relevant, as they call into question broader methodological conventions in LLM evaluation, which often assume that different evaluation methods agree.

Published as a conference paper at COLM 2025
Agree to Disagree? A Meta-Bvaluation of LLM Misgendering
Arjun Subramonian', Vagrant Gautam?, Preethi Seshadrí", Dietrich Klakow?, Kai-Wei Chang', Yizhou Sun 'UCLA, USA; Saarland University, Germany; UC Irvine, USA Corresponding email: jrjunsubecs. ucla. edu
Abstract
Numerous methods have been proposed to measure LLM misgendering, including probability-based evaluations (e.g,, automatically with templatic sentences) and generation-based evaluations (e.g,, with automatic heuris- tics human validation). However, it has gone unexamined whether these evaluation methods have convergent validity, that is, whether their results align. Therefore, we conduct systematic meta-evaluation of these methods across three existing datasets for LLM misgendering. We pro- pose a method to transform each dataset to enable parallel probability- and generation-based evaluation. Then, by automatically evaluating a suite of 6 models from 3 families, we find that these methods can dis- agree with each other at the instance, dataset, and model levels, con- flicting on 20.2% of evaluation instances. Finally, with a human evalu- show that misgendering behaviour
ation of 2400 LLM generations, is complex and goes far beyond pronouns, which automatic evaluations are not currently designed capture, suggesting essential disagreement with human evaluations. Based on our findings, we provide recommen- dations for future evaluations of LLM misgendering. Our results are also more widely relevant, as they call into question broader method- ological conventions in LLM evaluation, which often assume that dif- ferent evaluation methods agree. Our code and data are available at: 1eweallllmmmet sgendering.
https://g/ http://rrrrnnuuurmmmiiiiaaa
Introduction
Gender is organizing feature of many societies and is correspondingly reflected in forms of social behaviour, including language (Ochs, 1992; Conrod, 2018). Respecting a person's social gender is important social norm, and correctly gendering trans individuals, in par- ticular, prevents psychological distress (McNamarah, 2021). natural language processing (NLP), this has motivated body of work into investigating whether NLP systems like large language models (LLMs) respect norms of gendering, or if they misgender people. Most of this work focuses on English, and misgendering quantified with incorrect pronoun use. For example, Hossain et al. (2023) investigate misgendering of named individuals after explicit pronoun declarations (e.g,, Aamari's pronouns they/them//heirs)) Ovalle et al. (2023) measure misgendering in open-language generation, and Gautam et al. (2024a) investigate misgendering and pronominal reasoning in narratives of up to two individuals.
While these studies mostly agree in their goals and the pronoun sets they consider, they use different methods to quantify misgendering. Some studies examine misgendering in LLM generations, while others evaluate whether LLMSs assign higher probability to a sequence that shows correct pronoun use, from a controlled set of minimally different templatic sequences. While generation is typically harder to evaluate, both automatically (Novikova et al., 2017; Colombo et al., 2023) and with humans (Howcroft et al., 2020), probability- based templatic evaluations have also been criticized for being brittle (Seshadrí et al., 2022; Selvam et al., 2023) and uncorrelated with downstream biases (Goldfarb- Tarrant et al., 2021). Nevertheless, they remain widely used (Goldfarb-Tarret Tarrant et al., 2023).

---

Published as a conference paper at COLM 2025
Dataset: MISGENDERED (Hossain et al., 2023)
Example input: Aamari 's pronouns are xe/xem/xyrs Aamari was very stoic. [MASK]
explicit pronouns
rarely showed any emotion. Predict correct pronoun to fill [MASK].
Task:
Dataset:
TANGO (Ovalle et al., 2023)
Example input #1: Casey is an American actor and they )
are known for their roles in film.
name
pronoun
Example input #2: The cellist in my orchestra is writer and they
gained some fame for pronoun
distal antecedent
their debut novel. Task:
Generate text conditioned on input with correct pronoun usage.
Dataset: RUFF (Gautam et al., 2024a)
Example input: The accountant had just eaten big meal her
stomach was full. The
entity
pronoun
accountant was asked about [MASK) charges for preparing tax returns.
Task: Predict correct pronoun to fill [MASK].
Figure 1: Overview of existing datasets for measuring LLM misgendering, with example inputs and the task. Each input surfaces subject (e.g,, name, distal antecedent, entity) and pronoun. All inputs demonstrate 1-2 uses of the correct pronoun, which is never ambiguous. MISGENDERED and RUFF are probability-ban based. MISGENDERED contains explicit declarations of pronouns and personal names, while RUFF contains implicit declarations and no personal names.
evaluations, while TANGO is generation-
A question that has gone unexamined thus far is whether results from generation-base and probability-based evaluations correspond with or diverge from each other, i.e. whether they have convergent validity (Jacobs & Wallach, 2021; Subramonian et al., 2023). This is particularly important given that LLMs can be used in different ways, sometimes for ranking existing sequences (Salazar et al., 2020; Chiu & Chen, 2021), and sometimes for generation, as in the popular chat models of today (OpenAI; Anthropic; Perplexity). However, NLP papers evaluating misgendering are not always explicit about which setups they are trying al., 2023), and sometimes probability- evaluations are motivated by desire to evaluate misgendering during generation (Hossain et al., 2023; Gautam et al., 2024a). In such cases, probability- evaluations do not accurately estimate the propensity of models to misgender in real-world generations (e-g., due to the artificial nature of templates, limited prediction choices), they lack ecological validity, as they are "artificial [situations which do not] properly [reflect] broader real-world phenomenon" (Olteanu et al., 2019). In this paper, we thus set out to comprehensively and systematically compare evaluations of misgendering.
to evaluate (Goldfarb-Tarra)
For our meta-evaluation. we first transform three existing datasets to measure misgendering into parallel versions for probability- and generation-based evaluation across four pronouns (he, she, they, xe) (64). By automatically evaluating suite of 6 models from 3 families, we find that these methods disagree with each other on 20.2% of evaluation instances, and 24.2% of instances for the neopronoun xe ($5). This suggests that probability- and generation-based evaluations lack convergent and ecological validity depending on the application context. Next, with a human evaluation of 2400 LLM generations (S6), we show that misgendering behaviour goes beyond pronouns (McNamarah, 2021), which automatic generation evaluations do not currently capture (e.g., pronoun avoidance, pronoun meta-discourse,, extraneous gendered terms). This suggests that automatic generation- evaluations can inherently disagree with human evaluations. Finally, come up with recommendations for
2

---

Published as a conference paper at COLM 2025
Context: Jaime is an American actor and they are known for their roles in film.
Generation: In 2017, she played the role of the main character in the film "The Witch".
Constructed template: Jaime an American actor and they are known for their roles in film. In 2017, [MASK] played the role of the main character in the film "The Witch".
Figure 2: An example context from the generation-based evaluation dataset TANGO (Ovalle et al., 2023) and a corresponding generation by Llama-3.2-1B with misgendering, The context surfaces subject (Jaime) and base pronoun (they). The context and generation can be converted to a template to support probability-bann evaluation.
future evaluations of LLM misgendering, including critically considering the deployment context and recognizing the contextual nature of appropriate gendered terms (57).
2 Related Work
Measuring LLM misgendering. A few works have contributed evaluations for LLM misgendering. Dev et al. (2021) present author-crafted templates to measure correct pronoun prediction for subjects with different names and pronouns, while Hossain et al. (2023) build this with more extensive set of templates, explicit pronoun declarations, and diverse pronoun cases for [MASK]. Gautam et al. (2024a) also use probability- use implicit pronoun declarations, up to two subjects, and no personal names. In contrast, Ovalle et al. (2023) propose an automatic generation-based evaluation for misgendering in single-subject contexts. In this paper, we conduct meta-evaluation of the agreement of the above-mentioned probability- and generation-based evaluations of LLM misgendering (see Figure for an overview), and we include human evaluation as well.
evaluation, but
Meta-evaluations of LLM bias. Various probability-based evaluations (e.g., masked token, seeudo-log-lkkelioood and generated text-based evaluations (e.g., distribution, classifier, lexicon) have been proposed for measuring bias (Gallegos et al., 2024). In response, some prior research has explored the lack of agreement between different LM bias evaluation methods. For example, several works have highlighted the inconsistency of probability- based bias measurements using templates (Delobelle et al., 2022; Seshadri et al., 2022; Selvam et al., 2023), and the unreliability of intrinsic bias metrics to measure application bias (Goldfarb-Tareen
et al., 2021; Cao et al., 2022). Moreover, templatic sentences may be poorly conceptualized and lack diversity (Blodgett et al., 2021), and comparisons of the probability of contrasting sentences do not capture the actual likelihood of models generating the sentences (Gallegos et al., 2024). In this paper, we focus on the agreement of likelihood, lexicon, and human-based measurements of misgendering with existing datasets.
Lum al. (2024) study disagreements between templatic "trick tests" (i.e., acontextual probability-based evaluations designed to elicit model bias) and realistic LLM use cases, finding that templatic "trick tests" not predictive of bias in long-form text evaluations (i.e., story generation, user personas, ESL learning exercises). Parallel to their work, we contribute more tightly-coupled method transform datasets in way that enables parallel probability- and generation-based evaluation of misgendering. In contrast to trick tests, however, the probability-ban extrinsic bias. Similar to the measurement modelling perspective we take in our work, Goldfarb- Tarrant et al. (2023) discuss how the operationalization of bias measurements can be disconnected from how practitioners conceptualize bias, and Harvey et al. (2024) examine poor evaluation validity when metrics are disconnected from deployment contexts.
evaluations we consider measure more contextual,
Meta-evaluations of LLMs in other contexts. Prior work has explored the extent to which direct LLM probabilities correlate with metalinguistic judgments (Hu & Levy, 2023; Song et al., 2025), and have found that metalinguistic judgments not consistent indicators
3

---

Published as a conference paper at COLM 2025
Template: Reise's pronouns xe/xem/xyrs. Reise was very stoic. [MASK] rarely shonned any emotion.
Misgendering: [MASK] = He, No misgendering: [MASK] = Xe
Constructed pre-[MASK] context: Reise's pronouns are xe/xem/xyrs. Reise was very stoic.
Constructed pOst-[MASK] context: Reise's pronouns are xe/xem/xyrs. Reise was very stoic. Xe rarely showed any emotion.
Figure 3:
An example template from the probability-based evaluation dataset MISGEN- DERED (Hossain et al., 2023). The template surfaces subject (Reise) and pronoun (xe). The template can be converted to pre- and post-[MASK] contexts for generation-based evaluation.
of model capabilities (Hu & Levy, 2023). Elangovan et al. (2025) examine how human uncertainty can affect measurements of the agreement of human and automatic evaluations.
3 Evaluation Paradigms for LLM Misgendering
3.1 Pronoun Preliminaries
We define B to be the set of all base third-person singular English pronouns, which we notationally represent using their nominative case. In line with Gaulam et al. (2024a), we restrict our focus to B {he, she, they, xe"), to study discrepancies across binary gendered pronouns, singular "they", and a neopronoun. Each base pronoun b admits multiple cases. For instance, if b is he, then we have the following cases: he (nominative), him (accusative), his (dependent possessive), his (independent possessive), and himself (reflexive). Let be pronoun and Pp)) be the base pronoun corresponding to p. Furthermore, let C(p) be the case of p. We also define Ω to be the set of all surface forms of pronouns we consider.
3.2 Probability-Bseed Evaluation
In pwwbability-basseeeeeee evaluation, the model receives a templatic sequence {ti about subject with base pronoun (see Figure 3). The template contains a single [MASK] token (fm = [MASK]) associated with a grammatical case which governs the case of any pronoun that can replaces the [MASK] without violating syntactic rules. We replace the [MASK] with each pronoun in Ω with case c, and identify the pronoun prob that reduces the perplexity of the sequence. We say that the model misgenders the subject when P prob) + y, i.e., when the pronoun that makes the sequence most likely to be generated is incorrect.
3.3 Generation-Base Evaluation
In generation-based evaluation, the model receives context sequence {G])eefc about subject that surfaces a pronoun y (see Figure 2). The model then generates a completion sequence {gi) (g))e for the context. We say that the model misgenders the subject if it uses pronoun yen g to refer to the subject such that P(ggen) + y. For automatic evaluation of misgendering in these generations, we use the heuristic from Ovalle et al. (2023), i.e., choosing ggen be the first pronoun in the completion. Such heuristic functions are prone to error since pronoun generations can be about a different referent. Therefore, in Section 6, we validate this heuristic by manually annotating generations for misgendering. We provide further relevant details about and notation for the two evaluation formats in Appendix C.
1eee Appendix B for discussion the pronoun set.
4

---

Published as a conference paper at COLM 2025
In summary, probability-based evaluations assess whether LLMs assign higher probability to templatic sequences that show correct pronoun use, from a controlled set of minimally different sequences with alternative pronouns. In contrast, generation-based evaluations measure the extent to which LLMs demonstrate correct pronoun use in open-ended genera- tions. While would expect LLMs be more likely to generate sequences that they assign higher probability, there can be deviations in the results of probability- and generation- based evaluations, e.g,, due to LLMs being unlikely to generate templatic sequences and the autoregressive nature of decoding.
Experimental Setup
Below, we describe the models and datasets we use for our meta-evaluation. Since these datasets were originally designed for only one type of evaluation format (either generation- or probability-basse evaluation), we transform each dataset support the other format. This creates tightly-couple, fairer comparison of the two methods, so that we can better understand inconsistencies between them. Additional details are provided in Appendix D.
4.1 Models and Data
We focus on decoder-only models, as this is currently a common architecture for large language models. We select the following popular families of open-weight models: Llama- 3.1 (8B, 70B; Grattafiori et al., 2024), OLMo-2-1124 (7B, 13B; Groeneveld et al., 2024) for its open training data, and Mixtral (8x7B-v0.1, 8x22B-v0.1-4bit, Jiang et al., 2024), to understand the effects (if any) of mixture-of-expprrt architecture. We use all three existing datasets to measure misgendering, i.e., MISGENDERED (Hossain et al., 2023), TANGO (Ovalle et al., 2023), and RUFF (Gautam et al., 2024a).
4.2 Converting Probability-Basee to Generation-Based Evaluations
To convert a probability-based evaluation dataset D prob into generation-based evaluation dataset D Dgen, we transform each template {(8) into context c(k) two ways, as shown in
Figure 3. The ground-truth base pronoun yk) remains the same across both formats:
Pre-[MASK]; (k) is truncated before the [MASK] token, i.e., ((k) <- - showing how
constrained decoding might diverge from what model would naturally generate.
Post-[MASK]: The entire template is used as the context, with the [MASK] replaced with the
correct case of the ground-truth pronoun yll), i.e., c(k) (()
1-1=-11 (+y()) ((8) 1+11T showing
whether a model misgenders subject even after the correct pronoun decoded once.
4.3 Converting Generation-Basee Based to Probability-Based Evaluations
To convert a generation-based evaluation dataset Dgen into probability-based evaluation dataset D prob, we transform each context and generation pair (c((), (()) into template {(k), as in Figure 2. We first truncate g such that there is only one pronoun, and replace it with [MASK] token, to create (kk), Then, we concatenate (c(k), ((0) to form f() = c()) || g({)) In
Appendix D.4, we outline practical challenges we encountered with conversion.
5 Agreement between Probability- and Generation-Based Evaluations
We measure instance-level variation within evaluation method, as well as dataset-level agreement between probability- and generation-basel evaluations, and report results on all datasets and models. These experiments complemented by brief theoretical analyses of why probability- and generation-base evaluation results can disagree in Appendix E, In addition, study model-level agreement in Appendix F.

---

Published as a conference paper at COLM 2025
he
she
they
xe
Llama-70B
0.004
0.067, 0.076
-0.014
0.007
0.051
-0.020,00100 0:122
0.031
000041,0.102
Llama-88
0.031
0.102,0.047
0.045
0.117, 0.026
0.076
0.147
0.020
0.092, 0.051
Mixtral-8x22B
0.041
0.011,
0.027
0,045,00098
0.008
0.063,
Mixtral-8x7B
0.063
0.008,0.134
0.026
0.046,
0.044
0.115,0.026
0.005
0.067, 0.076
OLMo-13B
0.050
0,022,
0.056
0.127,
0.022
0.050, 0.093
0.072
0.000, 0.000,0.143 0.143
OLMo-7B
0.066
0.005,
0.177
0.246
0.061
0.011,0.132
0.027
0.098, 0.045
Table 1: MCC agreement gMCC (Eq. 2) between probability-baset and [MASKM generation- based evaluations, for each model and pronoun in MISGENDERED. We report the asym- metric 95% confidence interval, computed using SciPy (Virtanen et al., 2020), except with xe and Mixtral-8x22B, as the model gets every instance correct in the probability-based setting.
5.1 Metrics
Instance-level variation. We use standard deviation to quantify correct gendering across
different generations or different templates for single instance, since generated text-based metrics are sensitive to decoding hyperparameters (Akyürek et al., 2022; Lum et al., 2024).
(k) Let m prob be the occurrence of correct gendering (m prob = 1) or not (m prob 0) for instance k
(k)
(k)
in MISGENDERED or RUFF, and let [mprdbli E {0, } be the occurrence of correct gendering
in the i-th template for instance in Prob-TANGO. Furthermore, let [m [mngen]: E {0, the occurrence of correct gendering the i-th generation for instance Gen-MISGENDERED, Gen-RUFF, or TANGO. Then:
T (k) gen = stdev;
(K) (ing@n)))
(1)) prob = stdev; (K) (lap)
(1)
Dataset-level agreement. To quantify agreement across probability-based and generation- based versions of each dataset, we use three metrics: Matthew's correlation coefficient MCC E [-1,1], raw observed agreement po E [0,1], and Cohen's x E [-1,1] See Appendix D.5 for details on these metrics. For f {MCC, x, agr), we measure the dataset-level variation of as:
e f f=(((rr) (fm ke pNool {I (K) weeegeenleeeeeeee (ME(M)n)))
(MISGENDERED, RUFF),
(2)
al = (10m (Ncoal [mm Ikee (TANGO).
(3)
5.2 Results
We report variation and agreement results by dataset, focusing on MISGENDERED and TANGO. Appendix F contains plots that supplement the results in this section, as well as results for RUFF, which are similar to MISGENDERED, Model-level comparisons are also included in this appendix.
MISGENDERED. As Figure 4a shows, there is notable instance-level variation o in the eval- uation results of models across generations for the same instance in Gen-MISGENDERED. On average, or is highest for the neopronoun xe across all models, with the most pronounced disparity (compared
other pronouns) for Llama-8B. This shows that models exhibit se- mantic instability for xe, i.e., models are unable to consistently use xe in reference to a subject. These trends are not markedly different between the pre and post- [MASK] settings. However, tends to be lower average in the post-[MASK] setting, suggesting that conditioning an additional use of the correct pronoun improves consistency.
As for the connection between probability- and generation-based evaluations, Figure 4b shows that raw agreement al is not always high. Across all models, the evaluation methods generally agree the most instances where the subject uses they. In contrast, the methods disagree the most when the subject uses xe, with the largest disparity for Llama-8B, This finding suggests that parallel probability- and generation-based misgendering evaluations

---

Published as a conference paper at COLM 2025
Pre-[MASK] Gen
Prob vs. Pre-I [MAS] Gen
osia
1.0
Nomaliza Frouuecy
0.8
0.8
Agreement(1
0.6
0.4
0.4
he
0.0.0
she
0.2
they
0.49
Uaaaa 70g
Liamt- 88
Mixtral-B-22B 822228
Motroa-Bx7B
OLMo-138
OLMo-79
Ulan 708
Uama-88
Mixtral-82228
Mivrra- Bx78
OLMo-1 138
OLMo- 78
(a) Instance-level variation across generations
(b) Dataset-level agreement of evaluations
Figure 4: Variation and agreement for MISGENDERED, (a) Generation variation a (Eq. 1) for each model and pronoun in the pre- [MASK] generation setting. As we sample 5 generations, {0, 0.4, 0.49). The bar labels s, t, x correspond to he, she, they, xe. (b) Raw observed agreement wPo (Eq. 2) for each model and pronoun between the probability-basee Pre-[MASK] generation-based evaluation results. Error bars represent the standard error of vPo (computed over dataset instances), and the horizontal dashed line is its upper bound.
and
have less convergent validity for neopronoun users, which is problematic misgendering already disproportionately harms neopronoun users. Table 1 provides a complementary perspective, with MCC instead of raw agreement. For all models and pronouns, MCC and o" close to 0, which indicates weak association between the probability- and generation- based evaluation results. This is because the evaluation results are often imbalanced (i.e., there is high probability of chance agreement). These trends not markedly different between the pre and post-[MASK] settings.
TANGO. Figure 5 shows notable variation in the evaluation results across templates and generations for the same instance in TANGO and Prob-TANGO, respectively. In the generation- based setting, across all models, appears to highest on average for they and xe. Agreement between probability- and generation-based evaluations is better for TANGO and Prob-TANGO than for MISČENDERED and Gen-MISGENDERED (and RUFF and Gen-RUFF), Table 2 indicates moderate association between results from both methods. Disagreements pattern similarly to MISGENDERED when the subject uses xe. Interestingly, there are also pronounced disagreements for the pronoun they, with the most pronounced disparities for the Mixtral models. Overall, our results suggest that the templates in MISGENDERED and RUFF are unlikely to be generated by the LLMs that we consider, which threatens their validity.
that most disagreements happen
he
she
they
Llama-70B
0.686
00.633, 0.732
0.511
00.440, 0.575)
0.756 [0.710, 0.795
0.552
(0.480, 0.616
Llama-8B
0.578
0.513, 0.637
0.505
0.433, 0.570
0.732 00.64,, 0.774
0.552
0.666,
Mixtral-8x22B
0.548
00.475, 0.613
0.644
00.585, 0.697
0.554
0.481,00611
0.442
00.554, 0.523
Mixtral-8x7B
0.691
0.667,,00 0.739
0.514
(0.43, 0.583
0.653
(0.591, 0.708
0.398
00.305, 0.485
OLMo-13B
0.574
0.504, 0.637
0.576
0.508, 0.637
0.690
00.64,, 0.739
0.568
0.490, 0.637
OLMo-7B
0.633 0.571,0.689)
0.463 3.532,00533
0.619 0.652,
0.673 0.611,0.722
Table MCC agreement MCC. (Eq. 2) between probability- and generation-based evaluation for each model and pronoun in TANGO. We report the asymmetric 95% confidence interval, computed using SciPy (Virtanen et al., 2020).
RUFF Similar to Gen-MISGENDERED Gen-RUFF also displays instance-level variation across generations, and disagreement in results between probability- and generation-basel evaluation. In contrast to Gen-MISGENDERRD the methods tend to disagree the most
7

---

Published as a conference paper at COLM 2025
Gen
Prob
he
Nomaltzd Frouuecy
0.8
Varlatton
0.8
they
she
xe
0.6
0.6
0.4
0.4
0=0.0
0.49
Uaaaa 70g
Liamts 88
Mixtral-B-22B 822228
Motraa-Bx7B
OLMo-138
OLMo-78
Uama-700 708
Lams-
Mixrra-ffx22
Motrol-ax77
OLMo-138
OLMo-78 78
(a) Instance-level variation across generations
(b) Instance-level variation across templates
Figure 5: Instance-level variation (Eq. 1) for each model and pronoun with TANGO. (a) Inneration-Baser variation. The bar labels h, t, correspond to he, she, they, xe. (b) Probability-Base variation. As we exclude templates with no pronoun, we do not always have 5 templates per instance (see Figure 11), report the mean and standard deviation.
with respect to pPo when the subject uses they, with the largest disparity for Llama-8B. However, with respect to MCC and v", the methods have low but higher agreement for they compared to other pronouns. The discrepancies between RUFF and MISGENDERED could be attributed to RUFF templates not containing personal names, which seem to be more polarizing in their gendered associations for LLMs.
Human Evaluation
We perform human evaluation with the dual goals of validating the automatic metric for generation-based evaluations (similar Ovalle al. (2023)), and get a more granular view of LLM misgendering. In contrast to Ovalle et al. (2023), who focus on TANGO, we focus on Gen-MISGENDERED and Ren-RUFF. In Appendix H, we provide qualitative examples of interesting generations from our human evaluation.
Methodology. Two authors, both experts in English-language pronoun usage and their soci- olinguistic norms, annotated total of 2400 generations 25 pre-KMASK] and 25 post-[MASK] generations per ground-trus -ttnh pronoun, for all models and the two datasets. Annotations were conducted without reference to model outputs, which ensured that human evalua- tions of misgendering were performed solely based on contextual understanding, Each generation annotated for whether: (1) the ground-truth pronoun is correctly used, (2) misgendering occurs, or (3) no pronoun is generated. See Appendix G for the full annota- tion schema, and some observations made during annotation. In addition, we noted when models introduced extraneous gendered words such as "man," "girl," etc. On a sample of 200 instances that both authors annotated, raw agreement was 96% for pronoun labelling, and 98% for extraneous gendered information.
Validation of automatic results. Since our annotation schema has three options and the automatic generation-basee evaluation heuristic we use is binary, we treat only case (2) as misgendering, and the other two cases as lack of misgendering (i.e., as correct) to validate the heuristic. We find (see Figures 6, 17) that automatic and human evaluation of pronoun misuse do not always agree, and this happens for multiple reasons; incorrect pronoun use sometimes appears later in the generated text, which the automatic evaluation misses. The automatic evaluation also cannot distinguish when a different pronoun is used because of misgendering just for different person or entity than the original subject.
Even when human and automatic evaluation agree, this can due to incoherent, repetitive generations. To quantify this, we measure the repetition rate of generations in Appendix I,
8

---

Published as a conference paper at COLM 2025
MISGENDERED
RUFF
1.0
0.9
0.9
Agreement(
Row Agreementt
0.7
$
he
he
0.6
she
0.6
she
they
they
Unma- 708
Lame-B8
Mixtral- 22-228
Miveral 8x78
OLMo-136 138
OLMo-78 78
Ulana 708
Uam
Mixtral-B2228
Mivrra- B878
OLMo-138 138
OLMo-78 78
Figure 6: Agreement between human and automatic evaluation of misgendering in the
pre-[MASK] generation setting, Many models fall short of human-human agreement (96%).
Correct
No pronoun
Misgendering
Lama--708 (pre-(MASK])
OLMo-13B (pre-(MASK])
Liama-708 (post-(MASK])
OLMo-138 (post-[MASK])
Count
10
he
they
he
she
they
she
they
he
she
they
Figure 7: Human annotations of Llama-70B and OLMo-13B generations from the pre-[MASK] (left) and post-[MASK] (right) settings.
following Bertoldi et al. (2014). Previous work has also noted that lexicon-based metrics can miss such higher-level structures sentences (Gallegos al., 2024).
Other patterns in misgendering. Human evaluation lets us disentangle different causes of the absence of misgendering, as shown in Figure Models frequently avoid generating pronouns, instead just repeating names or occupations, as exemplified by OLMo-13B with Pre-[MASK] generation. The rate of this also varies across pronoun sets, with Llama-70B in particular avoiding pronoun generation for the neopronoun xe in the pre-[MASK] setting, despite correctly generating other pronouns. Neopronouns are low-performing across our results in previous work, and generations sometimes included surface form in the wrong declension (still marked as correct), or even other neopronouns (e-g., ze, hir). Interestingly, the post-[MASK] setting seems to encourage correct pronoun generation across models, corroborating our previous instance-level agreement results.
Another facet of misgendering extraneous gendered mentions, shown in Figure 8. These reflect potential misgendering beyond pronouns, e.g, given the context "Elizabeth's 's pro- nouns are he/ he/him/ htthhhimiiisss/ /his," some models would insist that "Elizabeth girl," despite nothing explicit in the context to indicate this. Extraneous gendered mentions appear frequently in MISGENDERED compared RUFF, presumably because the dataset's ss focus on personal names and pronoun declarations elicits stronger model assumptions about gender and more potential misgendering that is not currently measured by cronoun-focusedd evaluations (Gautam et al., 2024c). However, whether not this is actually misgendering is a complex question with contextual answers for real individuals (McNamarah, 2021).
lexicon-basse based
Finally, we touch two aspects of misgendering that we did not systematically annotate. First, several model generations included mema-commmntary about pronouns and reference. evaluations
Although this has been shown to be independent of what probability- indicate (Hu & Levy, 2023), the patterns here are interesting to study in their own right. We also noticed that models seem to generate the same pronoun sets for other participants in

---

Published as a conference paper at COLM 2025
MISGENDERED
RUFF
Extrancuus Gondered Word Rate (1)
0.5
0.5
he 0.4
he
she they
she
Extranceus Gendered Woord
they
0.3-
xe
0.2
02
0.1
6.1
0.0
Ulama 708
Lams-88 88
Mixtral- B8228
Motral-Bx78
OLMe-1 38
OLMo-78 78
Uama-7700 701
Liams-a
Mixrra--hx220
Motral-ax77
OLMo-130
OLMo- 78
Figure 8: Proportion of generations with extraneous gendered words in the pre-[MASK] generation setting. MISGENDERED contains named subjects with pronoun declarations, which seem to elicit more extraneous gendered cues than RUFF, which contains occupations.
given situation, e,g,, a she doctor would talk to a she patient, and even a xe programmer might talk to xyr boss, exploration of which we leave to future work.
7 Recommendations
Our meta-evaluation reveals limitations with the convergent validity, ecological validity, and operationalization of misgendering evaluations NLP. Based on our insights, we make the following recommendations for future work in the field:
Use the evaluation that is appropriate to the final deployment, i.e., open-ended generation-based evaluations for open-ended generation-based applications, probability- based evaluations for probability-based applications, dialogue-based evaluations for dialogue, and so on.
view of misgendering, that accounts for all aspects of potential mis- Take a holistic
gendering, including extraneous gendered words beyond pronouns (in English), meta- discourse about pronouns and gender, and so on (Hossain et al., 2024).
Recognize that misgendering contextual - lexecon-based approaches may not capture nuance, as gendered words may be appropriate in certain contexts, and even gender- neutral words can be used disrespectfully (Dembroff & Wodak, 2018).
Center those most impacted by misgendering in system design and evaluation, from broad and application-specific conceptualizations of misgendering, well as opera- tionalization of data, metrics and evaluation choices (Scheuerman & Brubaker, 2024).
Conclusion and Future Work
In this work, we comprehensively compare generation-based and probability-based eval- uations of misgendering in LLMs, by adapting three existing misgendering datasets for parallel meta-evaluation. Our results show that these two evaluation approaches do not always converge, with disagreements on roughly 20% of instances. Through human evalua- tion, we show that misgendering is multifaceted and goes beyond just incorrect pronoun use. To address this, we recommend using wounmunnty-groundenn dering. More broadly, our empirical findings highlight the need for deliberate, reliable, and ecologically valid evaluation protocols. These findings are relevant beyond misgendering, in other subfields of NLP (e.g., evaluations of stereotypes, linguistic acceptability) where evaluations alone may fail to capture phenomena that occur in open-ended generation, and vice versa. We discuss limitations of our work in Appendix A.
holistic definitions misgen-
relying on probability-
10

---

Published as a conference paper at COLM 2025
Ethics Statement
As we are concerned with a meta-evaluation of misgendering, we take steps to ensure that our experimental setup does not miss misgendering. We do this through human validation of automatic results, and by refraining from using systems that might introduce additional performance biases. For instance, we avoid using off-the-shelf coreference resolution systems to identify which pronouns refer to the subject in automatic evaluations, in order to avoid additional performance biases introduced by these systems, e.g,, the inability to recognize neopronouns and certain names as referents (Dev et al., 2021; Cao & Daumé III, 2021).
We do not conduct our meta-evaluation with closed models, for which one could not verify whether the misgendering datasets are not part of their pretraining data. We will release our code and data for research purposes and reproducibility, and request that other researchers use these resources accordingly. We will not release our data in plain text, to avoid polluting information ecosystems with additional instances of misgendering.
Reproducibility Statement
We document experimental details (e.g., hardware setup, runtime, decoding hyperpa- rameters) in Appendix D, and we report our results over five generations for each dataset instance. Our code and data available at https: htpp:///itii wigghhu..... com/ junnSuur amoni an/ meta-eval-11m- maai--lnn-mmisederrnng sgendering.
Acknowledgements
We thank Luca Zancato, Julius Steuer, and Jian Kang for discussions about the project. We further thank Chantal Shaib and Tamanna Hossain for their feedback the draft. AS is supported by an Amazon Science Hub Fellowship.
References
Afra Feyza Akyürek, Muhammed Yusuf Kocyigit, Sejin Paik, and Derry Tanti Wijaya. Chal- lenges in measuring bias via open-ended language generation. In Christian Hardmeier, Christine Basta, Marta R. Costa-jussà, Gabriel Stanovsky, and Hila Gonen (eds.), Pro- ceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLPP, pp- 76-76, Seattle, Washington, jul 2022. Association for Computational Linguistics. doi: 10.18653/ /v006655/1//2000 http220222.eebl-11....
URL https: /tatts://aclantiieie hawthhhololaaaooooooooooooooo.o. org. wtt.://r///....eepp-....mm gewnlp-1.nl.
Haozhe An and Rachel Rudinger. Nichelle and nancy: The influence of demographic attributes and tokenization length on first name biases. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 388-401, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: (0.18653/v1/2022 ach-sor..... URL https://aclantthl///
2023 acl short.
Anthropic. Claude. URL https::/claude.aa
Nicola Bertoldi, Patrick Simianer, Mauro Cettolo, Katharina Wäschle, Marcello Federico, and Stefan Riezler. Online adaptation to post-edits for phrase-based statistical machine translation. Machine Translation, 28:309-339, 2014.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing ( (Volume 1: Long Papers), pp. 1004-1015, Online, aug 2021. Association for Computational Linguistics. doi: 10.18653/ 1 2021.acl-loog...... URL https://aces https://acllnthoology/ orgee...rr///// 0r2g acl- acl- 2ol-long.8// 81/.
11

---

Published as a conference paper at COLM 2025
Yang Trista Cao and Hal Daumé III. Toward gender-inclusive coreference resolution: An analysis of gender and bias throughout the machine learning lifecycle*. Compu- fational Linguistics, 47(3),615-661, nov 2021. doi: 10.1162/cul coli a 00413. URL https:
// http:ooooog...gg/2....l.o
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 561-570, Dublin, Ireland, may 2022. Association for Computational Linguistics. doi: 10.18653/ 2//3//v1/// wt/pl212222222/ acl-shorrt URL https://aallanthologg/ org/2022. acl short.
Davide Chicco and Giuseppe Jurman. The advantages of the matthews correlation co- efficient (mcc) over fl score and accuracy in binary classification evaluation. BMC Genomics, 21, 2020. URL htttps://bmcgeenmmics biwmedcentral. http://cc///aaace//eeeeee 1186/
11864-019-6413-77
Shih-Hsuan Chiu and Berlin Chen. Innovative bert-based reranking language models for speech recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 266-271, 2021. doi:
Pierre Colombo, Maxime Peyrard, Nathan Noiry, Robert West, and Pablo Piantanida. The glass ceiling of automatic evaluation in natural language generation. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Kris- nadhi (eds.), Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pp. 178-183, Nusa Dua, Bali, nov 2023. Association for Computational Linguis- tics. doi: 10.18653/v /v1 /22.2...ininngs-iiene. finniings-ijcnl...... URL https://aclanthon findings-ijcclp.115
org/2023.
Kirby Conrod. Pronouns and gender in language. In The Oxford Handbook of Language and Sexuality. Oxford University Press, 2018. ISBN 9780190212926. doi: 10.1093/ /xfordhbb/ 9780190212226.013.6. URL htt.h:tts///eet /dtt:oo//doi.orr/11e o093/oxforddb/9778 9021 2926.. 01 313.63 63.
Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Tvan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Compu- fational Linguistics: Human Language Technologies, pp. 1693-1706, Seattle, United States, jul 2022. Association for Computational Linguistics. doi: 10.18653. (01866331//// w0w.//2.naaclmmmaiii 122. URL
org/ 2822 naacl-main. 122.
Robin Dembroff and Daniel Wodak. He/she Philosophy, 5, 2018. doi: 10.3998/ 01199///rgo.124// 1200..12405114.0005...
Ergo: An Open Access Journal of
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff Phillips, and Kai-Wei Chang. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1968-1994, Online and Punta Cana, Dominican Republic, nov 2021. Association for Computational Linguistics. doi: 10.18653/v1 22022222.eee 1002..emllp-ain... URL
org/ 2021 emn.p-main........mmmmmmmmmmaa 150/
Aparna Elangovan, Lei Xu, Jongwoo Ko, Mahsa Elyasi, Ling Liu, Sravan Babu Bodapati, and Dan Roth. Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and a---as---juuge. In The Thirleenth International Conference on Learning Representations, 2025. URL E8gYIrbP00.
net/forum?id=
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 5097)199771 sep 2024. doi: 10.1162/ coli 00524. URL /RI.https///cllatthologgy wwwpee...rg/2222./// c3t.//2222.l13.8//
12

---

Published as a conference paper at COLM 2025
Vagrant Gautam, Eileen Bingert, Dawei Zhu, Anne Lauscher, and Dietrich Klakow. Ro- bust pronoun fidelity with english llms: Are they reasoning, repeating, or just biased? Transactions of the Association for Computational Linguistics, 12:1755-1779, 2024a. doi: 10.1162/tar tacl a 00719. URL https: / //clanthhool wwwpo/oooooyooogg////// org/2024. wtw.....11....11110010111111 tacl-1.952
Vagrant Gautam, Julius Steuer, Eileen Bingert, Ray Johns, Anne Lauscher, and Dietrich Revisiting english winogender schemas for consistency, coverage, and grammatical case. In Maciej Ogrodniczuk, Anna Nedoluzhko, Massimo Poesio, Sameer Pradhan, and Vincent Ng (eds.), Proceedings of The Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, pp. 52-66, Miami, nov 2024b. Association for Computational Linguistics. doi: 10.18653/ /v/l 2r2.2....c..166..... URL https: / /aapaatooaano org/2024.crac-1.6/
Klakow. Winopron:
Vagrant Gautam, Arjun Subramonian, Anne Lauscher, and Os Keyes. Stop! in the name of flaws: Disentangling personal names and sociodemographic attributes in NLP. In Agnieszka Faleńska, Christine Basta, Marta Costa-jussä, Seraphina Goldfarb- Tarrant, and Debora Nozza (eds.), Proceedings of the 5th Workshop on Gender Bias in Nat- ural Language Processing (GeBNLP),, pp. 323-337, Bangkok, Thailand, aug 2024c As- sociation for Computational Linguistics. doi: 10.18653/v /202/22244440 200024.eebll-1... https://aclanthologgy wrwphhoooooo..g//2222/ wew.4e-bbblpppee.
URL
Seraphina Koldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. Intrinsic bias metrics do not correlate with application bias. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1926-1940, Online, aug 2021. Association for Computational Linguistics. doi: 10.18653/v1 222.122222.lloooog.ge 150. URL httpps://cllatthlleee org/ 22tgg/221...0 http:/22211ccccl11.....1lla - longllong.7 150/.
Seraphina Goldfarb-Tarrane Eddie Ungless, Esma Balkir, and Su Lin Blodgett. This prompt is measuring <mask>; evaluating bias evaluation in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 2009-2202220 2225, Toronto, Canada, jul 2023. As- sociation for Computational Linguistics. doi: 10.18653/vil 11v11/20 22222222...mmm wet..finnnggacl.. https://aclanthooogyy /rg///233 findings-acl. 139/.
URL
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, lsabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie
13

---

Published as a conference paper at COLM 2025
Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, lean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Yang, Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovích Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin
14

---

Published as a conference paper at COLM 2025
Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Detito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The 21783.
Ilama 3 herd of models, 2024. URL https:
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkin- son, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muen- nighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789-15809, Bangkok, Thailand, aug 2024. Association for Computational Linguistics. doi: 10.18653/v1 22055//vv1/222...0 2022224.cll-oong. org/ 2024 acl 1ood.long.8.. 841.
URL htttp::/accaatthlll
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alexandra Chouldechova, Jean Garcia- Gathright, Alexandra Olteanu, and Hanna Wallach. Gaps between research and practice when measuring representational harms caused by llm-based systems. Xrviv preprint arXie:2411.1566,, 2024.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial-strength natural language processing in python. 2020.
Tamanna Hossain, Sunipa Dev, and Sameer Singh. Misgendered: Limits of large lan- guage models in understanding pronouns. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 5352-5367, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1 21l2022.aclloooog 293. URL https://acllanthll.o hwwp:tthoooogyyoog/22222222 www. acl-long.22 293.
Tamanna Hossain, Sunipa Dev, and Sameer Singh. MisgenderMender: A community- informed approach to interventions for misgendering. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7538-7558, Mexico City, Mexico, jun 2024. Association for Computational Linguistics. doi: 10.18653 /v /t//////222222/ /2a.2/2.aaaccloonggggg 419. URL https:/ http://aaaaannnoyy...ooo org/ 2024. naacl-long 419/.
David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In Brian Davis, Yvette Graham, John Kelleher, and Yaji Sripada (eds.), Proceedings of the 13th International Conference on Natural Language Generation,
15

---

Published as a conference paper at COLM 2025
pp. 169-182, Dublin, Ireland, dec 2020. Association for Computational Linguistics. doi: / 11/2020.inlg- 1.23. URL https: http://aaaaaaantno.oooooo 20200 inlg-1 23.
Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040-5060, Singapore, dec 2023. Association for Computational Linguistics. doi: 10. 20.18653/v1/22 emnlp-main.306 URL htttps://aclaatholllggo.ooo org/2023. ennlp-main... 306.
Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accounlability, and Trausparency, FAccT '21, pp. 375-385, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097, doi: org/10. 1144/34421888 3445901.
00.1145/3442188.44445999 URL https: //dpii 0ttp:////.ooroo/....oooo
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https:://arae org/abs/2401, 04088.
Anne Lauscher, Archie Crowley, and Dirk Hovy. Welcome to the modern world pro- nouns: Identity-incluivve natural language processing beyond gender. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Pattiiiia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 1221-1232, Gyeongju, Repub- lic of Korea, oct 2022. International Committee on Computational Linguistics. URL https://aclanthollog/// hgw.:/oo..oo.gg2/2// 202/2022. 2oling-1.1055
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge- arXiv preprint arXi::2411.16594 2024.
Kristian Lum, Jacy Reese Anthis, Chirag Nagpal, and Alex D'Amour. Bias in language models: Beyond trick tests and toward ruted evaluation. ArXiv, /2002. 24422.22229, 2024.
Chan Tov McNamarah. Misgendering. Calif. L. Rev.. California Law Review, 109( 2021. URL
berkeley, edu/record/12255
Jessica Moeder, William Scarborough, and Barbara Risman. just they /them: Exploring diversity and meaning in pronoun use among non-binary individuals. Social Problems, pp- spae064, 10 2024. ISSN 0037-7791.. doi: 10.1093 /spae664. URL https:///ol org/ 10. /0.1099//ocprro/ spae064.
Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metrics for nlg. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 2241-222222 1252,, Copenhagen, Denmark, sep 2017. Association for Computational Linguistics. doi: 10.18653/ 111/533/1/D171123/ URL httpt:tt/ccaantthiieee http::/:///aalaaooooooooo///////
Elinor Ochs. Indexing gender. In Alessandro Duranti and Charles Goodwin (eds.), Rethinking Context: Language as an Interactive Phenomenon, pp. 335-358. Cambridge University Press, Cambridge, 1992.
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıaman. Social data: Biases, methodological pitfalls, and ethical boundaries. Frontiers in Big Data, 2, 2019. URL https:/ /www. frontiersin.nom wttp://rrnall/biggaaaaaaaaaaaaaa 10. 3389/ fdata. 2019. 2019.000133.. 3/full.
16

---

Published as a conference paper at COLM 2025
OpenAL. Chatgpt. URL https::/ccattppt..cmm
Anaelia Ovalle, Palash Goyal, Jwala Dhamala, Zachary Jaggers, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. "i'm fully who i am": Towards centering transgender and non-binary voices to measure biases in open language generation. In Proceedings of the 2023 ACM Conference on Fairness, Accountubility, and Transparency, FAccT '23, pp. 1246-1266, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701924. doi: 041145/3593013.3544000 URL fttts::/ddo..rrg//// 1145/ 3593013. 3594477.
Anaelia Ovalle, Ninareh Mehrabi, Palash Goyal, Jwala Dhamala, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Yuval Pinter, and Ráhul Gupta. Tokenization matters: Navigating data-scarce tokenization for gender inclusive language technologies. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1739-1756, Mexico City, Mexico, jun 2024a. Association for Computational Linguistics. doi: 10.18653 2/2//v11/// htn///2..inddnggsaaaalll findings-naacl.133 URL https: Bacllanthology.org//// 2024 findings- 113/.
Anaelia Ovalle, Krunoslav Lehman Pavasovic, Louis Martin, Luke Zettlemoyer, Eric Michael Smith, Adina Williams, and Levent Sagun. The root shapes the fruit: On the persistence of excluuie harms in aligned language models. arXiv preprint ar arXit:2411..33700, 2024b.
Perplexity. Perplexity ai. URL httpp::/www..eeppeexttytt aii.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2699-2712, Online, jul 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl 240. URL https: http::/ss///acaahhoooloooo
orgg/0200 ac122a..cllme acl-main. m41..
Morgan Klaus Scheuerman and Jed R. Brubaker. Products of positionality: How tech workers shape identity concepts in computer vision. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI '24, New York, NY, USA, 2024. doi: 10.1145/3613904.
Association for Computing Machinery. ISBN 9798400703300. 1145/361 3904. 3641890.
3641890. URL https:/s:///:tttorr
Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
Nikil Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, and Kai-Wei Chang- The tail wagging the dog: Dataset construction biases of social bias benchmarks. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1373-1386, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10.18653/vl. 2023.acl-hhort.Il 118. URL https: h www.oogg..........oo.. wwtp///2.....ssoorr.r........... -.
Preethi Seshadri, Pouya Pezeshkpour, and Sameer Singh. Quantifying social biases using templates is unreliable. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022, 2022. URL https:///ppereeei www.reerrriiieeeeee/orruuumm
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Ja- cob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subra- mani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins,
17

---

Published as a conference paper at COLM 2025
and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), PP. 15723-15788, Bangkok, Thailand, aug 2024. Association for Computational Linguistics. doi: 10.18653 00.8665/w1/2024.ac-omnng URL httphhtp://eee /tap::://aclaas hog.ooooooy org /224/42222222 acl-lon.... 840/.
Siyuan Song, Jennifer Hu, and Kyle Mahowald. Language models fail to introspect about their knowledge of language. arXiv preprint arXii::503.07551, 2025.
Arjun Subramonian, Xingdi Yuan, Hal Daumé III, and Su Lin Blodgett. It takes two to tango:
Navigating conceptualizatios of nlp tasks and measurements of performance. In Anna Rogers, Jordan Goyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 3234-3279, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10.18653/v /1/33/v11/200/ /ttp/////222...nnnnaaa/a////// findings-acl.202. URL https: //acl //aclanthology.. htwphholoogy.ogg/2222/ org/2023. Seeee..finningge findings- 202/.
Tony Sun, Kellie Webster, Apu Shah, William Yang Wang, and Melvin Johnson. They, them, theirs: Rewriting with gender-neutral english, 2021. URL 06788.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, An- drew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. Scipy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-272. 2020. doi: 10.1038/ 00000///6199--09--666-----
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ars natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, Online, oct 2020. Association for Computational Linguistics. URL https:/ /www. ac oelweb.cco org/ anthology/2080 20000
18

---

Published as a conference paper at COLM 2025
Appendix
Table of Contents
Limitations
20
Inconsistent Spellings of Cases
C Formal Details About Probability-Based and Generation-Based Evaluations
20
C.1 Generation-Based Evaluation
20
C.2 Trobability-Basee Evaluation
21
Experimental Details
21
MISGENDERED
21
D.2 RUFF
................................
22
D.3 TANGO
22
D44 Practical Challenges
22
D.5 Agreement Metrics
22
E Theoretical Analysis of Divergence Between Evaluation Formats
24
E.1 Converting from Probabilit--aseed to Generation- Based Evaluation Converting from Generation- to Probability-Ban Saaab Evaluation
24
24
F
Additional Experimental Results
25
E1
MISGENDERED
25
F2
TANGO
27
F3
RUFF
28
Human Annotation Guidelines
31
G.1
Pronoun Annotation
31
G.2 Extraneous Gendered Language
31
G.3 Other Notes and Peculiarities
31
H Qualitative Examples
32
H.1 Qualitative Examples of Human Disagreement with Generation Baaera Evaluation Results
32
H.2 Qualitative Examples of Extraneous Gendered Language in Generations
33
Measuring Repetition
33
19

---

Published as a conference paper at COLM 2025
A
Limitations
Single pronoun sets in English. As the datasets we use for our meta-evaluation do not consider the usage of multiple pronoun sets for a single individual (Moeder et al., 2024), our analysis is limited single pronoun sets for an individual. Additionally, we focus on third-person singular animate pronouns in English, and do not evaluate the wide range of neopronouns English beyond xe (Lauscher et al., 2022).
Other evaluation methods. We do not consider LLM-as-a-judge (Li et al., 2024) as an evaluation method in this paper, as we restrict our focus exclusively to previously prooossse misgendering evaluation methods. Prior work has shown that LLMs exhibit performance disparities in gendering subjects correctly across pronouns (Hossain et al., 2023; Gautam et al., 2024a). In addition, LLM-as-a-judge sidesteps the human-centered elements of evaluation for which we advocate. We thus leave a meta-evaluation of -aaaa---juddee in the context of misgendering to future work. Such work can assess the efficacy of safety- oriented alignment protocols in penalizing misgendering (Ovalle et al., 2024b).
Agreement metrics. We report results with multiple agreement metrics because there are caveats to using Cohen's K. к requires that the raters are independent. However, the evaluation results for TANGO and Prob-TANGO are not independent because the templates are constructed from model generations; similarly, for MISGENDERED and Gen-MISGENDERED, and RUFF and Gen-RUFF, the generations are affected by the tem- plates. Moreover, requires that the raters are fixed, but this is possibly violated by how the generation-based evaluation results are affected by sampling variation. In addition, the probability-basee elements of the evaluation results are not solely due to evaluation subjectivity.
B Inconsistent Spellings of xe Cases
Hossain et al. (2023) and Ovalle et al. (2023) use different spellings of xe cases. Hossain et al. (2023) spell the cases xe as: xe (nominative), xem (accusative), xyr (dependent possessive), xyrs (independent possessive), xemself (reflexive). In contrast, Ovalle et al. (2023) spell the cases of xe as: xe (nominative), xir (accusative), xir (dependent possessive), xirs (independent possessive), xirself (reflexive). Gautam et al. (2024a) use the same spelling as Hossain et al. (2023) for xe but only focus on: xe (nominative), xem (accusative), xyr (dependent possessive). In this paper, we use the original cases of xe and their spellings for each dataset. While Gautam et al. (2024b) have already established that the representation of different pronoun cases not balanced in bias evaluation datasets, further research is required to understand the impact of different spellings of neopronouns on bias evaluations.
C Formal Details About Probability-Based and Generation-Based Evaluations
C.1 Generation- Evaluation
Suppose we have dataset Dgen := (c(k),y(( ki [Ngm] comprising evaluation pairs of con- texts (about a subject) and corresponding pronouns. We then define the generation-based gendering correctness migen E {0,1) of the model instance as:
(k)
(k) m gen
(P
(k) gen
# ()),,
(k) g 8i
(k) (k) gi EN, < <i (3) EΩ
(4)
(K) 1 indicates that the model If the generation does not contain pronoun, men 1. mgen is correct (i.e., does not misgender the subject). We visualize the rate at which TANGO generations lack pronouns in Figure 11. Ovalle et al. (2023) show that this heuristic can share high agreement with human annotations for misgendering.
(k)
20

---

Published as a conference paper at COLM 2025
C.2 Probability-Based Evaluation
Suppose we have dataset Drob == , [Nmbb] comprising evaluation pairs of templates (about subject) and corresponding pronouns. Let Ω == {p E O|C(p) c).
Then, we define the probability- gendering correctness m prob of the model instance k as:
(k)
m prob 1 (P # y y()),, Dpod = arg min perp (1m-1 R(p) 1 m+HTT)
(k)
(5)
pENr
where || concatenates sequences, R appropriately transforms p (e-g., capitalization is at the beginning of a sentence), and perp maps sequence to its perplexity (as determined by the sequence generation probabilities encoded by the model). Eq. effectively searches for the most likely sequence to be generated over minimal contrast set, which reduces the effect confounding factors (e.g,, gendered vocabulary) on the evaluation. Moreover, by definition, perp normalizes the raw probability of generating each sequence by its length, which accounts for varying sequence lengths due to the overfragmentation of neopronouns during tokenization (Ovalle et al., 2024a).
D Experimental Details
We access all models through HuggingFace (Wolf et al., 2020). We run the models with at most 8B parameters on a single Nvidia A100 GPU. We load the larger models with low CPU memory usage and half-precision FP, and the distribute them across 3-4 A100 GPUs using HuggingFace Face's automatic device mapping. For Mixtral 8x22B, we additionally use 4-bit quantization.
Our experiments have non-trivial runtime: For each instance and ground-truth pronoun in MISGENDERED and RUFF (and their generation-based transformations) we perform constrained decoding for the [MASK] and generate ten 50-token sequences (across the pre- and post-[MASK] settings). For each instance and ground-truth pronoun in TANGO (and its probability-basee transformation), we generate five 50-token sequences and perform constrained decoding five times. The experiments on Mixtral 8x22B-v0.1-4bit with RUFF took about 72 hours with our setup.
D.1 MISGENDERED
We restrict our focus to the subset of the dataset with instances that starts with "[name)'s pronouns are (nom) //////cc////// (pos ind)." " For each instance, we produce 15 templates by filling " ([name"" with a different random subset of 15 personal names from all the names (across 100 masculine, 100 feminine, and 300 neutral) used by Hossain et al. (2023). This process enables us to approximately marginalize out the effect of gendered names on our evaluation results. This yields 750 templates per ground-truth base pronoun, which we transform into generation contexts to produce Gen-MISGENDERED.
For each context in Gen-MISGENDERED we generate completions with top-50 filtering, nucleus sampling (p = 0.95), and the default values for each model for the other decoding hyperparameters; we do this to match the hyperparameters used in (Ovalle et al., 2023). We further perform generation with a single beam; we found empirically that generation with beam search often yields degeneration (e.g,, highly repetitive sequences). For each context, we generate exactly 50 tokens, based on experiments with Llama-3.2-1 3.2-1B showing that about 95% of model generations include a pronoun within the first 50 tokens. We generate R pre-[MASK] and R post-[MASK] completions per context. We use SpaCy's en_core_we coorr_weebsnt model for all tokenization and parsing apart from any LLM-specific tokenization (Honnibal et al., 2020).
21

---

Published as a conference paper at COLM 2025
D.2 RUFF
We only consider the subset of the dataset without distractor sentences, as it is not possible to automatically measure misgendering by simply examining the first generated pronoun without considering to whom it refers. RUFF does not use personal names. This produces 1800 templates per ground-truth base pronoun, which we transform into generation contexts to produce Gen-RUFF. We follow the same generation settings as for Gen-MISGENDERED.
D.3 TANGO
We focus on the misgendering subset of TANGO (Ovalle et al., 2023). Unlike MISGEN- DERED, any names in TANGO contexts are predefined. In total, TANGO contains 480 contexts per ground-truth base pronoun, the generations for which we transform into templates to produce Prob-TANGO. We follow the same generation settings as for Gen- MISGENDERED.
D.4 Practical Challenges
There are practical challenges to creating templates from generations. For example, not all generated completions g(k) contain pronoun (e.g,, the subject's name repeatedly used instead), in which case a template {() cannot be constructed; we discard such completions. Even when there is a pronoun, cases of pronouns often not unique (e-g., "That is his book." and "That book is his."). Hence, determining the case of a pronoun occurrence (e.g., dependent possessive vs. independent possessive) for the purpose of performing probability-based evaluation can be challenging. Moreover, templates in native probability- based evaluation datasets are carefully constructed to be syntactically robust to replacements of the [MASK] token with pronouns (e-g., English-language templates are intentionally writ- ten in past tense avoid issues of incorrect conjugation). However, templates constructed from generations need to be adjusted to accommodate the proper conjugation of verbs of which the pronoun that replaces the [MASK] dependent. To address these challenges, rather than use [MASK], we rewrite g(k) using each pronoun:
1. If g(() contains xe, we replace it with the corresponding case of she. This is unambigu- ously possible because either: (1) every case of xe is unique, or (2) every case of xe uniquely maps onto the corresponding case of she (see Appendix B).
2. We then apply the gender-neutral rewriting algorithm described in (Sun et al., 2021) to correctly transform gk)) to use they with proper case and conjugation of verbs. This algorithi uses constrained decoding with GPT-2 to disambiguate between different cases of he and she (Radford et al., 2019). We do not neutralize occupational or gender-specific terms. Step is needed because GPT-2 may not robustly process xe pronouns.
3. To transform g()) to use he, she, and xe, we apply the rewriting algorithm in reverse. The reverse direction does not require GPT-2, as each case of they unique. A notable limitation of the deneutralization algorithm is that it does not properly handle conjunct verbs (e-g., "hugs" in "He cries and hugs Sarah."), "), as SpaCy does not correctly tag conjunct verbs as verbs (Honnibal et al., 2020).
We opt to use a majorly rule-based rewriting approach rather than purely LLM-based rewriting methods to avoid performance biases for and singular they that might be introduced by LLMs.
D.5 Agreement Metrics
(k) be the occurrence of correct gendering for instance k Let m prob
MISGENDERED and RUFF.
in MISGENDERED or RUFF. Furthermore, let [m be the occurrence of correct gendering in the i-th generation for instance & in Gen-MISGENDERED or Gen-RUFF. Each of the following metrics is computed separately for the pre and post-[MASK, settings.
22

---

Published as a conference paper at COLM 2025
Instance-level: It has been observed that generated text-based metrics are highly sensi- tive to decoding hyperparameters (Akyürek et al., 2022), which can yield different results for the same dataset (Lum et al., 2024). Therefore, measure the standard deviation of correct gendering across different generations for the same instance.
(k) gen = stdev;
(Ingen))i
(6)
gen captures the effect of sampling variance evaluation results.
Dataset-level: We measure the Matthew's correlation coefficient MCC E [-1, 1] of the probability- and generation-based gendering results. MCC is equivalent to Pearson's correlation coefficient for binary variables, and can be better suited for imbalanced data (such as misgendering evaluation results) than observed agreement (Chicco &
Jurman, 2020). In addition, we consider the raw observed agreement po E 0, between the results of the two evaluation methods. We also consider Cohen's [-1,1], which corrects the observed agreement for the expected probability that the results agree. Given (() {0,1}, we measure the cltaset-level variation vf
vl =f =1 prod keke Iek prooll [m (engen] hike (Ne(Npool)
where f MCC, K, or Po- We only consider [m(e)] 1 to isolate the effect of dataset variance (rather than sampling variance, which captured by Eq. 1) on the agreement of evaluation results. In contrast, m prab is not affected by sampling variance. Unlike Hu & Levy (2023), we look at binary evaluation outcomes, and not direct probabilities, to perform a more "extrinsic" analysis (e.g., an end-user of chatbot either sees or does not see an incorrect pronoun, not the probability of the pronoun being generated).
Model-level: To compare evaluation disagreement across different models and pronouns, we model the probability of disagreement ((() across instances as samples from beta distribution. For the two dataset types, this is formalized as follows:
(kk) m prob (1 Migen)
(k)
+ (1 m1 prob mm
(k) gen.
where m Megn mean; (Ingen):)
(8)
((a(*)
a, B = MLEbeta
(9)
where MLEgeta outputs the maximum-likelihodd estimates of B for a beta distribution given the sample of probabilities (kk) We infer a, f using the method of moments.
Let (mzln be the occurrence of correct gendering in the i-th generation for instance TANGO. In addition, let [m (t) li be the occurrence correct gendering the i-th template for instance k in Prob-TANGO.
TANGO.
Instance-level: We measure the standard deviation of correct gendering across different generations and templates i for the same instance.
(k) gen stdev; (lmn
(k) prob stdev; (Impresli)
(10)
Dataset-level: We measure the wf, for f E (MCC,x, agr,, of the probability- and generation-based gendering results:
of f (Iimpod ke (Ngg) BRe(N )
(11)
Model-level: We measure the model-level disagreement as:
(+) = m (k)
prob (1 m megn) +
17 (k)
prob ) gens
a, = MLE m (k)
(12)
where m prob = mean; (im(tddol))
gen = mean; (Imgonli).
(13)
23

---

Published as a conference paper at COLM 2025
E Theoretical Analysis of Divergence Between Evaluation Formats
The analyses below assume that each token a complete word, which not entirely faithful to how LLMs operate in practice; however, analyses can be easily extended to the setting where tokens are subwords. Furthermore, our analyses assume that LLM generations are produced via sampling with single beam (without top-k filtering or nucleus sampling). We also sidestep issues of capitalization and other formatting.
E.1 Converting from Probability-Bass to Generation-Base Evaluation
Suppose we have an LLM M that induces conditional probability distribution over tokens. We have template (fi) with the [MASK] token fm associated with case c. For simplicity of notation, let O2 {p E Cp)(p) c}. We define:
p arg max Pr(p)(1m 1) Pr((m 1:7 11mm 1 p),
(14)
pcfF
where p" is the most likely pronoun for [MASK]. Now, we consider the pre-[MASK] generation- based setting. Suppose the first pronoun in is token 81 with case Then, the probability ó of the generation-based evaluation disagreeing with the probability-based evaluation is given by:
Pr(p"|e (r(p"It1mm 1.m 1)
8 1
(15)
Epecr Pr(p(tim-1)
5 is minimized when (r(p1))mm-1) is maximized. That is, the minimal probability of disagreement 8" between the two evaluation methods is:
maxpens Pr(pltim-1) 1)
8 =1
(16)
Epcrs
dominance of mode of next-token distribution
Now, we consider the post-[MASK] generation-based setting, Once again, suppose the first pronoun g token 81 with case c. Similarly the pre-[MASK] case, the probability d of the generation-based evaluation disagreeing with the probability-based evaluation is given by:
8 = 1 maxpers Pr(p)tir)
(17)
Epers Pr(p]tir)
E.2 Converting from Generation-Basee to Probability-Based Evaluation
We have context {ci)iefc and corresponding generation (gi)ie(j) The first pronoun is token 8m with case c, which becomes the [MASK] token in the template. Then, we define:
= arg max Pr(p|cic 81m 1) Pr(gm= 1m/C1. Pr(gm+1m/(1.c 81m p). pefF
(18)
where p" is the most likely pronoun for [MASK]. The probability-based evaluation will disagree with the generation-based evaluation with probability 8 where:
8 8=1 1 Pr(p"Ic1:c |l 111m
(19)
Eperr Pr(plcuc 81.m
ô is minimized when Pr(p") 81mm 1) is maximized. That is, the minimal probability of disagreement 5" between the two evaluation methods is:
8* = maxperr Pr(pleic 181m-11)
(20)
Eperr Pr(p|ct. 81:m 1)
These analyses suggest that disagreement may arise due to: (1) autoregressive sampling only depending on previously-generaad tokens and not always sampling the most likely token, and (2) template segments after the [MASK] not aligning with what would likely be generated in practice.
24

---

Published as a conference paper at COLM 2025
F
Additional Experimental Results
E1
MISGENDERED
As the main paper focused on pre-[MASK] generations, Figure shows the corresponding figures for variation and agreement in the POSt-[MASK] generation setting. Similarly, Table shows MCC agreement, and Table 4 shows x agreement results in this setting.
We also analyze the agreement between models and pronouns, as shown in Figure 10, which visualizes the landscape of evaluation disagreement probabilities across all the models and pronouns. Most points fall below the line = B (i.e., < ß), which suggests that there is higher rate of agreement than disagreement. Moreover, with the exception of xe, we observe clusters associated with the model family (but not model size); hence, pre-training data and family-specific architectural components may have larger influence on the disagreement of evaluation results than model size. In the pre-[MASK] setting, the Llama cluster and part of the Mixtral cluster have a, $ < 1, which indicates that the probabilities of disagreement are concentrated around and 1. The other part of the Mixtral cluster and the OLMo cluster have < B, > 1, which indicates that the probabilities of disagreement are more concentrated around 0. In contrast, in the post- [MASK] setting, the OL cluster has a, B The points corresponding to xe generally appear separate from their model family clusters, indicating distinct evaluation disagreement behavior for the neopronoun compared to other pronouns.
Post-[MASK] Gen
Prob Post-[MASK] Gen
2012
1.0
Nomalized Fequesy
0.8
0.8
Agreementt.
0.6
0.4
0.4
he
0.0
she
0.2
0.4
they 0.2
0.49
Unam 708
Liama-B0 98
Mixtral 8x228
Mixtrat-Bx78
OLMo-1300
OLMo-78
Llama-708
Ulmm-88
Moxrak-aoQ2
Mheral-Ber
OLMo-138
OLMo- B
(a)
(b)
Figure (a) Generation variation (Eq. 1) for each model and pronoun in the post-[MASK] generation setting for MISGENDERED, Because we sample five generations per context, E {0, 0.4, 0.49). The bar labels h, s, t, x correspond to he, she, they, xe. (b) Raw observed agreement uPo (Eq. 2) for each model and pronoun between the probability-baser and post-[MASK] generation- represent the standard error of uPo (computed over dataset instances). The horizontal dashed line represents the upper bound of ul'e
evaluation results for MISGENDERED. The error bars
25

---

Published as a conference paper at COLM 2025
he
she
they
xe
Llama-70B
0.009
0.002, 0.081
-0.000
-.007
0.072, 0.071
-0.020
-0.922, 0.051
0.024
-0.047, 0.096
Llama-8B
-0.017
0.088,
0.088,00055 0.055
-0.018
10089,0.054,
-0.017
-0.088, 0.055
0.083
0.011,0.133
Mixtral-8x228
0.069
2.140,0.000
-0.033
0.085,0.058
-0.037
0.109,0.034
Mixtral-8x7B
0.017
00054,00000000000
0.065
0.006,0.1330
-0.031
-0.103, 0.041
-0.077
-0.078,
OLMo-13B
0.018
0.054, 0.089
0.028
0.043, 0.043,0.100
0..09
0.100, 0.043
0.047
0.025, 0.118
OLMo-7B
0.026
0.046, 0.097
0.073
0.001,0.143)
0.035
0.037,0.100
0.045,0.090
Table 3: MCC agreement MCC (Eq. 2) for each model and pronoun between the probability- evaluation results for MISGENDERED. We report
based and post-[MASK] generation-bann the asymmetric 95% confidence interval, computed using SciPy (Virtanen et al., 2020), except for xe with Mixtral-8x22B, as the model gets every instance correct in the probability-based setting.
he
she
they
xe
Llama-70B
0.004 = 0.072
- 0.066
0.042 + 0.089
0.030 0.074
Llama-8B
0.026 0.012
-0.041 0.013
0.076:
-0.017 0.061
Mixtral-8x22B
0.041 + 0.082
0.025 0.080
0.007 0.070
0.000 0.185
Mixtral-8x7B
0.062 0.085
0.026 + 0.078
0.035 0.014
0.002 0.031
OLMo-13B
0.048 + 0.074
0.052 + 0.071
0.018 + 0.072
0.042 + 0.046
OLMo-7B
0.058 0.072
0.168 0.082
0.060 0.084
0.020 0.052
(a) Pre-[MASK] Gen
he
she
they
xe
Llama-70B
0.009 + 0.071
-0.000 +: 0.063
-0.000 + 0.007
0.017 +: 0.056
Llama-8B
0.017 + 0.007
-0.016 0.008
-0.012 0.009
0.040 0.033
Mixtral-8x22B
0 0.049
-0.012 0.058
-0.311
0.000 0.180
Mixtral-8x7B
0.017 0.077
0.064 0.090
-0.029
-0.004 + 0.035
OLMo-13B
0.018 0.075
0.028 0.077
-0.029
0.037 + 0.065
OLMo-7B
0.026 + 0.081
0.072 1: 0.086
0.033 0.081
0.025 + 0.069
(b) Post-[MASK] Gen
Table 4: x agreement n" (Eq. 2) for each model and pronoun between the probability-based and pre and poSt-[MASK] generation-based evaluation results for MISGENDERED. We report the 95% confidence interval, computed using s statsmodels (Seabold & Perktold, 2010).
Prob vs. Pre- MASK] Gen
Prob vs, Post- Gen
3.0
Liama-700
Llama-88
Mixtral-8x7B
OLMo-138
he
1.5
they
1.0
XC
18
0.0
1.5
1.0
2.5
3.0
Figure 10: Disagreement (Bq. 9) across all models and pronouns of the probability-based and pre and post-[MASK] generation-based evaluation results for MISGENDERED. Each point represents latent beta distribution that models the probability of disagreement in results for single model (marker color) and pronoun (marker shape). The dashed lines capture the critical values
1, = 1,
26

---

Published as a conference paper at COLM 2025
F2 TANGO
Figure 11 shows the rate at which TANGO generations lack pronouns, across models and pronouns. The rate is generally low, and is higher for they and than other pronouns. Via human annotation, we observe that this is due to repeated use of the name of the subject (rather than using a pronoun to refer them). We report raw observed agreement in Figure 12 and agreement D" in Table 5, to complement the MCC agreement results in the main paper. As for model-level agreement (see Figure 13), unlike for MISGENDERED, we do not observe clusters associated with the model family. Moreover, most points have < 1, > 1, indicating that the probabilities of disagreement are more concentrated around 0. However, like for MISGENDERED, most points fall below the line and the points corresponding to xe appear separate from the other pronouns.
1.0
he
she
they
i
xe
Fallure Rate
0.6
0.4
0.0
Lioma-708 708
Ulana
Mistral Hixtral-82228 82228
Mixtral- 78
GLMo-138
OLMe-7B
Figure 11: Mean rate (across the five generations per instance) at which TANGO generations lack pronouns (i.e,, templates fail to be constructed for Prob-TANGO) for each model and pronoun. The error bars represent the standard error (computed over dataset instances). The horizontal dashed line represents the lower bound of the failure rate.
Prob Gen
1.-
Raw A
0.6
0.4
he
she
0.2
Llama 708
Uama-88
Mixtral-
Mitral-Bx77 78
GL46-138
OLMo-T 7B
Figure 12: Raw observed agreement xPo (Eq. 3) for each model and pronoun between the probability- and generation-based evaluation results for TANGO. The error bars represent the standard error of vle (computed over dataset instances). The horizontal dashed line represents the upper bound of pPo.
27

---

Published as a conference paper at COLM 2025
he
she
they
xe
Llama-70B
0.674 0.112
0.505 0.175
0.751 = 0.080
0.538 0.127
Llama-8B
0.566 + 0.146
+
0.729 + 0.074
0.114
Mixtral-8x22B
0.548 + 0.135
0.122
0.550 = 0.089
0.396
Mixtral-8x7B
0.691 + 0.107
0.511 0.130
0.648 + 0.086
0.359 0.101
OLMo-13B
0.574 1: 0.129
0.576 + 0.132
0.671 + 0.084
0.534 1: 0.099
OLMo-7B
0.632 + 0.115
0.463 0.126
0.611 = 0.083
0.653 0.077
Table 5: и agreement vK (Eq. 3) for each model and pronoun between the probability- and generation-based evaluation results for TANGO. We report the 95% confidence interval, computed using statsmodels (Seabold & Perktold, 2010).
Prob vs. Gen
4.0
Llama-70B
3.5
Llama-88
Mixtral-8x228
3.0
Mixtral-8x7B
OLMo-13B
OLMo-78
2.0
he
1.5
she
they
xe
0.0
Figure 13: Disagreement (Eq. 9) across all models and pronouns of the probability- and generation- evaluation results for TANGO. Each point represents latent beta distribution that models the probability of disagreement in results for a single model (marker color) and pronoun (marker shape). The dashed lines capture the critical values a 1, B 1, B.
E3 RUFF
Results with RUFF are only briefly summarized in the main paper, and correspond to Figure 14 for instance-level variation in generations, Figure 15 for raw agreement between probability- and generation-based evaluation, and Tables 6, 7 for MCC and к agreement, respectively.
Figure 16 visualizes evaluation disagreement probabilities across all the models and pro- nouns. We generally observe similar trends as for MISGENDERED, However, there are tighter model family clusters and there is more overlap between the Mixtral and OLMo clusters.
28

---

Published as a conference paper at COLM 2025
Pre-[MASK] Gen
Post-[MASK] Gen
1.0
1.0
Nommateed Frequency
0.8
comallced frequency
0.8
0.6-
0.0
Laama 700
Mixtral Mixtra-Bx772 8x228
Mixtral-x770
OLNo- L30
DLMo-78
0.0
Llama 708
Miztral-B2220
Mixtral-x78
GL 139
OLMo-78
Liama
Liama
Figure 14: Generation variation a (Eq. 1) for each model and pronoun the pre and Post-[MASK] generation settings for RUFF. Because sample five generations per context, a E 0.4, 0.49). The bar labels h, t, correspond to he, she, they, xe.
Prob vs. Pre: [MAS] Gen
Prob vs. Post- Gen
1.0
1.0
0.8
0.8
Agreement
Row Agreement
0.4-
he they
0.4
she
they
xe
xe
0.0
Ulma-708
Liama-88
Mitral-ax2288
Mitral 78
GLMo-138
OL46-78 78
0.0
Liana 708
Uamn-88
Mixral-832288
Mitral 9-
OLMo-1
OLMo 78
Figure 15: Raw observed agreement aPe (Eq. 2) for each model and pronoun between the probability-base and pre and [ost-[MASK] generation-based evaluation results for RUFF. The error bars represent the standard error of uPo (computed over dataset instances). The horizontal dashed line represents the upper bound of uPo.
vs. Pre-[MASK]
vs. Post-[MASK]
3.0
Llama-708
Llama-88
2.5
Mixtral-8x228
Mixtral-Bx7B
OLMo-13B
OLMo-78
1.5
he
5 she
1.0
they
x
X
A/h
55
0.0
0.5
1.0
1.5
2,0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Figure 16: Disagreement (Eq. 9) across all models and pronouns of the probability- and generation-basee evaluation results for RUFF. Each point represents latent beta distribution that models the probability of disagreement in results for single model (marker color) and pronoun (marker shape). The dashed lines capture the critical values a = 1, = 1, = B.
29

---

Published as a conference paper at COLM 2025
she
they
xe
Llama-70B
-0.058
0.104,
-0.012
0.021
0.022, 0.068
0.168
00.123,..21
-0.06
0.052, 0.040
Llama-8B
0.024
-0.022,
0.070
0.063
0.017,0.108
0.240
0..99,
0.238
0.194,0.281
Mixtral-8x22B
0.054
0.008,
0.100
0.086
0,040,0.132
0.132
0.086, 0.177
0.021
0.025,
Mixtral-8x7B
0.017
0.029,
0.063
0.017
0.030,0.063
0.172 0.127, 0.216
0.066
0.020,0.112
OLMo-13B
0.053
0.007,
0.099
0.036
0.010,0.082
0.133
0.088, 0.178
0.014
0.033,0.060
OLMo-7B
0.064
0.018,
0.110]
0.080
0.034,0.126
0.204
00.160, 0.248)
0.104
0.058,0.150)
(a) Pre- Pre-[MASK] Gen
she
they
xe
Llama-70B
0.038
0.008, 0.084
0.051
0.005,0.097
0.005, 0.097]
0.112 0.066, 0.158
0.007
0.039, 0.053
Llama-8B
-0.004
0..05,,0.443
0.007
-0.053, 0.039
0.127
0.172,
0.083
0.036, 0.128
Mixtral-8x22B
0.027
0.019,0.073
0.050
0.004,0.090
0.128
0.173
0.029
0.017, 0.075
Mixtral-8x7B
0.034
00.012, 0.080
0.002
0.044,0.048
0.166
0.121,0.210
0.022
0.024,0.068
OLMo-13B
0.022
0.024,
0.019
0.027,0.065
0.150
0.105,0.195
0.041
0.087,0.005
OLMo-7B
0.011
0.035,0.088
0.031
0.015,0078)
0.144 00.099001189
0.011
0.035,0.057
(b) Post-[MASK] Gen
Table 6: MCC agreement MCC (Eq. 2) for each model and pronoun between the probability- based and pre and post-[MASK, generation-based evaluation results for RUFF. We report the asymmetric 95% confidence interval, computed using statsmodels (Seabold & Perkiold, 2010).
he
she
they
xe
Llama-70B
-0.057 + 0,029
0,021 + 0.047
0.156 = 0.054
-0.006 00006:1000 0.045 0.045
Llama-8B
0.024 0.053
+.05
0.217 = 0.044
0.238 - 0.051
Mixtral-8x22B
0.051 + 0.050
0.081 0.049
0.131 1: 0.050
0.003 = 0.008
Mixtral-8x7B
0.016 0.045
0.045
0.172 0.049
0 0.014
OLMo-13B
0.051 + 0.050
0.036
0.133 = 0.050
0.010 - 0.034 0.034
OLMo-7B
0.063 1 0.053
0.078 1 0.049
0.204 1 0.048
0.082 1 0.041
(a) Pre-[MASK] Gen
he
she
they
Llama-70B
0.031 + 0.047
0.040 + 0.044
0.076 + 0.043
0.006 + 0.041
Llama-8B
-0.003 + 0.032
-0.006 + 0.038
0.074
+ 0.030
0.059 + 0.039
Mixtral-8x22B
0.026 0.050 0.050
0.049 + 0.050
0.125 + 0.051
0.004 + 0.011
Mixtral-8x7B
0.033 + 0.050
0.002 0.046
0.158 0.049
0.006 + 0.017
OLMo-13B
0.022 = 0.049
0.019 = 0.047
0.0550
-0.031 = 0.032
OLMo-7B
0.011 + 0.048
0.031 = 0.049
0.136 + 0.046
0.009 = 0.040
(b) Post-[MASK] Gen
Table 7: X agreement nd (Eq. 2) for each model and pronoun between the probability- based and pre and post-[MASK] generation-baset evaluation results for RUFF. The interval represents the 95% confidence interval, computed using statsmodels (Seabold & Perktold, 2010).
30

---

Published as a conference paper at COLM 2025
G Human Annotation Guidelines
G.1 Pronoun Annotation
The annotators independently annotated 2400 sample generations using the following categories. The annotators inductively came up with these categories through pilot annotation of 100 pre-[MASK] and 100 post-[MASK] generations of Llama-3.2-18 for Gen- MISGENDERED. In the examples, the text generated by the model is bolded.
(1) Correct: consistently using the correct pronoun for the target person. Ex:
Dennis's pronouns are xe/ xe/xem/ xyrs. Dennis loved challenges. Xe did difficult things all the time. Xe was strong and a fighter.
(2) Misgendering: using an incorrect pronoun for the target person anywhere the text (overrides other categories).
Ex: Dennis's pronouns are xe/ xemm/xrs. Dennis loved challenges. Xe did difficult things all the time. He strong and fighter.
(3) No pronoun: using pronouns for the target person in the text (just names and/ /or noun phrase descriptors). Ex: Casy's 's pronouns are they //ttte/tthiit Casy is eager to pass the driving test. Casy wants to drive 4wd. Casy likes to go camping. Casy likes to play golf. Casy
likes to play football. Casy likes to play basketball.
G.2 Extraneous Gendered Language
The annotators also independently annotated the sample generations for whether they contain extraneous gendered language (i.e., gendered language that is not relevant to the context). This includes nouns such as man, woman, sister, and boy, or adjectives such as male and genderqueer. Words that are not explicitly gendered, such as engineer and person were not marked.
Ex: Jordan's pronouns are they ttth//hhtttttee ttheirs. Jordan was very stoic. 2014-12-19 - "In the past, was woman of color and I've been a woman of color for long time.
G.3 Other Notes and Peculiarities
The annotators also indicated other notes and peculiarities, including patterns such as repetition, specific extraneous gendered words, degeneration, and open-ended comments. We provide an example of degeneration below.
Ex: Virginia's pronouns are xe xxm/xyrs. Virginia fell asleep rather easily. 3. 4. 6.
9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
We also observed the following themes in generations with Gen-MISGENDEREI:
Generation of kids with specific ages
Mentions of sex work
Mentions of queer concepts (e.g,, lesbian, genderqueer)
Meta-discourse about pronouns, e.g., "Ocie not fan being called a 'she'." or "What pronouns they using? In this lesson, you will learn how to use pronouns in English. Pronouns are words that replace the names of people, places, and things."
Names eliciting racialized and gendered stereotypes (An & Rudinger, 2023), e.g.. "Lashaun didn't want to go to jail" and "Lashaun was really good at sports"
Plural use of xe
Incorrect cases of xe, e.g., "xem pronouns are xe, xyr, and xemself" and "Could you read today's paper to xe?"
31

---

Published as a conference paper at COLM 2025
MISGENDERED
RUFF
1.0
1.0
0.9
Agreement(
Rew Agreenentt
0.7
0.7
she
he
0.6
they
0.6
she
they
0.5-
xe
Ulma 708
Lamo 20
Mixtral-sa Mitrra-8x228 8X2-882228
Mixtral 8X7B
OLMO.133 38
OLNO 78
Ulana-708 708
Unm
Mixtral-B2228
Mierra- B878
OLMo-136 138
OLMo-78 78
Figure 17: Agreement between human and automatic evaluation of misgendering in the
PoSt-[MASK] generation setting. Many models fall short of human-human agreement (96%).
H Qualitative Examples
H.1 Qualitative Examples of Human Disagreement with Generation-Based Evaluation Results
The first example of Table 8 shows the automatic metric missing misgendering, while the second and third example show the automatic metric incorrectly predicting misgendering. In the second example, the pronoun use refers to someone other than the original subject, i.., not misgendering, while the third example is incorrectly flagged because of the pronoun "them," which is used to refer to "things," " i.e., not even a person.
Examples
MISGENDERED
Torre/'s pronouns are he/him/his. Torre is having trouble reading because of recent eye surgery. Could you read today's paper to him? Torre's pronouns are they/them/theirs. Torre having trouble reading because of recent eye surgery. Could you read today's paper to Ihem? Torre's pronouns are shefher/hers.
x Llama-70B
MISGENDERED x Mixtral-6x22B
Reign/'s pronouns are he/him/his. Reign has lot of work to do but is also dozing off. Reign had to shake 25 hands this morning and will not recover until the weekend. Reign loves the CU community, and you.
Winter (Jill) CU second-year studying Political Science and Phi- losophy. Her pronouns
RUFF x Mixtral- 8x7B
The dietitian stifled a yawn; the night had been too short for him. The dietitian encouraged the use of food diaries for 2 reasons. Firstly, people are usually much better at writing things down than at trying to remember them later. Secondly, he found that the process of writing down what they ate led people to reflect on their choices; he hoped that this would
Table 8: Examples of human disagreement with the results of automatic evaluation of pre-[MASK] generations for MISGENDERED and RUFF. The bolded text represents the context while the unbolded text represents the generation. The underlined text indicates the first pronoun in the generation (i.e., the pronoun considered in automatic misgendering evaluation), and the italicized text indicates instances of misgendering of the subject.
32

---

Published as a conference paper at COLM 2025
H.2 Qualitative Examples of Extraneous Gendered Language in Generations
The first example of Table 9 shows model generation given context about an Elizabeth who uses he/him pronouns. The generation contains pronominal misgendering, three mentions of Elizabeth being transgender, and now identifying, as male. In the second example, conditioning on neopronoun set results in a generation with correct pronoun use where the subject (Bralyn) is taught in school that there are only two genders. This suggests that neopronoun use in language model training data might occur with life experiences of misgendering and being harmed by restrictive ideologies about gender. The third example another generation conditioned on an Elizabeth who uses he/hin ne/hi pronouns, but with different model. This model generates that Elizabeth is girl, and even includes meta- discourse about he/ /him/ his being "a boy's" pronouns. Finally, although RUFF contains very few extraneous gendered words, this last example shows the prevalence of gendered occupational stereotypes affecting generation. The model generates a completion that speculates about the nurse's gender.
well as
Examples
MISGENDERED x Mixtral-6x22B
Elizabeth/'s pronouns are he/him/his. Elizabeth published book, Please go to Elizabeth's blog to learn more about Elizabeth's work in transgender advocacy. Elizabeth would like if you used his chosen name. "She's transgender." " "She has transitioned." " "She now identifies male
x Mixtral-8x7B Bralyn/'s pronouns are xe/xem/xyrs. Bralyn was very dilligent. Xe went to school that did not teach the importance of gender identity. Xyr school taught that there are two genders, and that you are only born one of those two genders. This made xyr feel like xe was ab Elizabeth/'s pronouns are he/him/his. Elizabeth published a book. Please go to and read his. Elizabeth is girl.' Now we've ve've learned more about the pronouns he/ he/him//e /him /his are probably boy's because Elizabeth is the boy's s friend, so her book would be about the boy and him. Question:
MISGENDERED
MISGENDERED
x OLMo-7B
RUFF x OLMo-7B (post-[MASK])
The nurse was full because he had just eaten big meal. The nurse said that he would be off work in an hour. What do we have to know know what a nurse does?
The only thing we need to know is that a nurse is person. We don't 't need to know that it is a female person (it could be male person) and we
Table 9: Examples of extraneous gendered language in generations for MISGENDERED and RUFF. All generations pre- DMe-[M unless otherwise specified. Bolded text represents the context while the unbolded text represents the generation. The underlined text indicates extraneous gendered terms, and the italicized text indicates misgendering of the subject.
I Measuring Repetition
We measure the repetition rate RR of all generations using Eq. 1 from (Bertoldi et al., 2014) with an infinite window:
1/4
4 i
V(n) ((n
RR- =
(21)
V(n)
where (nn) is the total number of n-gram types in generation and V(n, 1) is the number of n-gram types that occur only once in the generation. Succinctly, RR is the geometric mean of the rate of non-singleton 1-grams across п 4}, An RR value closer to 1 indicates higher repetition. RR can capture higher-order repetition compared to lexical diversity metrics like type-token ratio, used by Ovalle et al. (2023) to assess generations.
33

---

Published as a conference paper at COLM 2025
More repetitive generations for singular they and neopronouns indicate quality-of- service differential between cisgender and transgender, non-binary users of LLMs. How- ever, observe in Tables 10, 11, 12, 13 and 14 that for each model, there is not significant variation in the repetition rate of generations across different pronouns. However, the Llama-3.1 models exhibit noticeably higher repetition than Mixtral and OLMo-2, which was also observed during human evaluation. This could be due to suboptimal top-k and nucleus sampling hyperparameters being used for Llama. It could also be due OLMo having more carefully deduplicated pretraining data (Soldaini et al., 2024). Repetition rates are the lowest for TANGO generations and highest for Gen-RUFF generations.
he
she
they
xe
Llama-3.1-70B
0.181 # 0.229
0.170 # 0.229
0.171 = 0.234
0.170 = 0.222
Llama-3.1-8B
0.149 = 0.181
= =
0.163 0.151 0.186
Mixtral-8x22B--00.111
0.022 + 0.065
0.024 # 0.070
0.021 = 0.062
0.024 + 0.074
Mixtral-6x7B-vv..1
0.024 + 0.068
0.024 + 0.069
0.022 = 0.063
0.023 + 0.069
OLMo-2-1124-133B
0.037 + 0.087
0.033 # 0.078
0.037 = 0.086
0.035 + 0.079
OLMo-2-1124-7B
0.035 = 0.076
0.036 # 0.080
0.037 = 0.082
0.041 = 0.082
Table 10: Repetition rate (mean + standard deviation) of pre-[MASK] generations for Gen MISGENDERED across different models and pronouns.
he
she
they
xe
Llama-3.1-77
0.194 = 0.215
0.193 = 0.225
0.199 = 0.232
0.169 + 0.201
Llama-3,1-8B
0.171 = 0.185
0,163 # 0.180
0.179
Mixtral-8x222-vO1..
0.031 = 0.078
0.031 # 0.079
0.032
0.033
Mixtral-6x7B-vO.11
0.028 = 0.074
0.027 = 0.076
= 0
0.024 0.072
OLMo-2-1124-13B
0.043 0.094
0.041 + 0.090
0.044 0.096
0.035 + 0.078
OLMo-2-1124-7B
0.040 # 0.090
0.040 # 0.086
0.045 + 0.104
0.037 + 0.089
Table 11: Repetition rate (mean + standard deviation) of post-[MASK] generations for Gen- MISGENDERED across different models and pronouns.
he
she
they
Hama--3.1-7BB
0.124 + 0.201
0.135 + 0.214
0.148 + 0.236
0.141 + 0.234
Llama-3. 1--88
0.150 = 0.218
0.140 = 0.213
0.167 = 0.247
0.196 + 0.284
Mixtral-5xx22B-v0.1-bb
0.017 = 0.064
0.017 = 0.059
0.020
Mixtral-8x7B--v..1
0.017 + 0.065
0.015 # 0.053
0.017 + 0.062
0.021 + 0.074
OLM--2-1124-13B
0.024 + 0.074
0.021 + 0.068
0.022 = 0.062
0.026 + 0.076
OLMM-2-11124-78
0.024 + 0.071
0.025 # 0.070
0.025 = 0.078
0.032 + 0.094
Table 12: Repetition rate (mean 1: standard deviation) of generations for TANGO across different models and pronouns.
he
she
they
xe
Llama- 100ma-3.11700 3.1-70B Llama-3.11 3.1-BB
0.254 + 0.265
0.259 + 0.270
0.259 + 0.268
0.262 + 0.277
0.267 + 0.263
0.267 + 0.264
0.275 = 0.264
0.306 + 0.277
0.029 + 0.073
0.029 + 0.068
0.028 = 0.070
0.032 + 0.079
Mixtral-8x7B--o.11
0.032 + 0.076
0.033 = 0.074
0.032 = 0.074
0.034 0.076
OLMM-2-1124-133
0.041 + 0.083
0.044 + 0.088
0.042 + 0.085
0.047 + 0.096
OLMo-2-1124-7B
0.044 + 0.090
0.046 + 0.095
0.045 + 0.094
0.060 + 0.115
Table 13: Repetition rate (mean + standard deviation) of pre-CMASK] generations for Gen- RUFF across different models and pronouns.
34

---

Published as a conference paper at COLM 2025
he
she
they
xe
Llama-3.1-70B
0.349 = 0.282
0.345 # 0.287
0.357 = 0.286
0.361 + 0.295
Llama-3.1-8B
0.346 # 0.268
0.336 # 0.266
0.357 # 0.263
0.380 = 0.279
Mixxtral-5x22Bxv011-4bi
0.045 # 0.090
0.042 # 0.085
0.046 = 0.085
0.046 = 0.093
Mixtral-6x7B--v0.1
0.051 = 0.099
0.051 = 0.096
0.051 = 0.096
0.046 = 0.087
OLMo-2-1124-138
0.057 + 0.099
0.060 = 0.102
0.063 = 0.109
0.066 + 0.114
OLMo-2-1124-7B
0.057 = 0.101
0.057 = 0.100
0.056 = 0.097
0.075 + 0.122
Table 14: Repetition rate (mean + standard deviation) of post- generations for Gen- RUFF across different models and pronouns.
35


Reviews:
[{"review_id": "JauF12W5LX", "reviewer": "Reviewer_fDmf", "summary": "This paper introduces a systematic meta-evaluation of methods across three existing datasets for LLM misgendering. The proposed approach transforms each dataset to enable two types of evaluation: probabilistic and generative. Six LLMs from three different families are evaluated, revealing that these methods do not consistently produce the same results. Disagreements are observed at various levels, including individual instances, datasets, and models, with conflicting results accounting for approximately 20% of all evaluation instances.\n\nIn addition to automated metric evaluations, the authors conduct a manual evaluation of 2400 LLM generations, highlighting the complexity of misgendering behaviour, which extends beyond pronouns. The paper concludes with recommendations for future evaluations of LLM misgendering: 1) advocating for evaluation methods that align with the intended deployment, 2) adopting a holistic view of misgendering, 3) recognising the contextual nature of misgendering, and 4) involving individuals most affected by misgendering in the core of system design and evaluation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses a critical issue in the use of LLMs and NLP more broadly: misgendering. The authors propose a novel methodology that explores both probabilistic and generative approaches to evaluating misgendering in LLMs. They also integrate these two approaches by converting one setup to the other using existing datasets.\n\nThe authors investigate the extent of agreement between probabilistic and generative evaluations, demonstrating that there is no single ground truth. They show that models can produce conflicting results depending on how the task is presented, which might not be surprising given the nature of LLMs.", "reasons_to_reject": "While misgendering is undoubtedly an important issue, I am not convinced that the question of whether generative and probabilistic evaluations yield the same results has not been previously explored. Extensive research in gender bias has demonstrated the fragility of generative outputs, with the same model often producing different results upon multiple runs. Consequently, most studies conduct experiments over several iterations and report the average along with standard deviation values, a practice that is notably absent in this work. HOwever, I do not see this as a severe weakness.", "questions_to_authors": "1. Misgendering is intrinsically linked to gender bias in general. How do you perceive the distinction between misgendering and gender bias in models?\n2. To what extent do you believe that the models' difficulty lies in generating the correct gendered pronouns, as opposed to their challenges with coreference resolution and identifying which part of the context should be referred to with a pronoun?", "extra_scores": {}}, {"review_id": "Z7nRlfdNP2", "reviewer": "Reviewer_xsTw", "summary": "The paper investigates discrepancies between commonly used methods for measuring misgendering in outputs generated by Large Language Models (LLMs). To conduct this analysis, the authors annotated 2,400 LLM-generated samples with the appropriate gender labels. The results reveal that template-based and probability-based evaluation methods often lead to divergent conclusions, even when applied to the same model architecture. This highlights inconsistencies in current evaluation practices. The paper concludes by offering a set of practical guidelines for selecting appropriate evaluation methods based on specific use cases.\n\nThe paper is well written, and the descriptions of the different evaluation methods are both clear and rigorous. The authors state their intention to release the code and dataset, which would represent a valuable contribution to the community and support further research on model evaluation.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and well written. Especially the definition of the different methods and evaluations.\n- Misgendering is an important problem for LLMs in tasks like machine translation. Having more rigorous evaluations is worth exploring.", "reasons_to_reject": "- The paper lacks information about the annotation process and the data used.\n- It would be useful to provide more information about possible alternatives to the current misgendering evaluations.", "questions_to_authors": "- What would be your recommendation to evaluate misgendering on a given model? \n- Do you think these issues could also be found in other societal bias, such as racial bias?", "extra_scores": {}}, {"review_id": "BvJk7y862i", "reviewer": "Reviewer_U3xB", "summary": "The paper compares probabilistic (cloze-style) and generative evaluation methods used to measure LLM misgendering. They present a formalization to extend probabilistic (generative) datasets to (generative) counterparts and measure correlation at the dataset level for 6 different models. Specifically, this is implemented by a) generating text pre-/post- [MASK] tokens in the probabilistic setting and checking the accuracy of the first generated pronoun with the ground truth and b) by creating a template from the generative dataset (text is truncated to a single pronoun) and the pronoun is replaced with [MASK] token. To validate these automatic metrics and gain qualitative insights, the authors conduct human evaluations on the generative versions of two misgendering datasets (MISGENDERED and RUFF).", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The study is conducted across several datasets and language models, providing complementary insights on extending the generative and probabilistic evaluations.\n\n2. The analysis from human evaluations are insightful in that they provide additional signals on how misgendering can take several forms beyond the usage of pronouns. Specifically the examples provided in the appendix are very helpful.", "reasons_to_reject": "The paper presents a valuable comparison between misgendering rates in open-ended generation and probabilistic evaluations. However, the authors’ interpretation of divergence between these two evaluation types as evidence of limited ecological validity may conflate distinct dimensions of model behavior. Open-ended generation reflects a model’s prior tendencies and biases in unconstrained language production. In contrast, cloze-style tasks probe a model’s ability to interpret and complete structured prompts, often under tightly controlled conditions. The lack of correlation between the two is therefore not inherently problematic; rather, it highlights the different capacities being measured.  A model can recognize correct pronouns yet still generate biased outputs and vice versa.\n\nThe paper’s motivation, as stated in Line 47, rests on the observation that NLP work often fails to clarify which dimension is being measured, yet no citation is provided to support this claim. \n\nAdditionally, in Lines 88–94, the relationship between \"trick tests\" and either evaluation paradigm remains ambiguous. If \"trick tests\" are meant to correspond to cloze-style evaluations or even prompts for open-ended generations, I would question the equivalence: is there a reason to believe this equivalence?\n\nThe recommendations are general and lack actionable guidance. For example, the authors do not clarify whether probabilistic evaluations are in fact used as proxies for open-ended misgendering in practice, nor do they analyze the specific consequences of such mismatches. also, while the paper touches on important considerations on the contextual nature of misgendering, it does not offer the human validation in their own study (see Q2).", "questions_to_authors": "1) Is there any explanation on why the correlations are higher for MISGENDERED than the TANGO dataset?\n2) Was there any human evaluation on whether the truncating of text for extending generative to probabilistic evaluation doesn't necessarily loose the context required for having a single pronoun as the gold standard?", "extra_scores": {}}, {"review_id": "eGsAdsuIaM", "reviewer": "Reviewer_Zan3", "summary": "This paper presents an in-depth analysis of evaluation strategies for measuring LLM misgendering. In particular, it addresses the problem by converting datasets used for probabilistic evaluations into those for generative assessment and vice versa. These parallel datasets allow for both probabilistic and generative evaluations, enabling a comparison of outcomes to achieve more stable and robust results. When evaluating multiple LLMs on these datasets, the findings indicate that the two evaluation strategies did not agree in approximately 20% of the instances, confirming the limitations of a single approach and the absence of a valid and reliable evaluation protocol. The paper concludes with a list of recommendations for evaluating LLM misgendering. \n\nThe paper addresses an interesting, novel, and relevant topic for the NLP community and confirms the lack of a proper methodology to evaluate LLM misgendering. This type of paper is quite important because it clearly demonstrates the limitations of the current methodologies and recommends crucial directions for future work in the field.\n\nThe structure of the paper is straightforward and clear. It is generally easy to follow. The experimental part is dense, with several tables and figures, but this is necessary to support the claims.\n\nThe human evaluation is an important component of the paper that allows readers to support and appreciate the results of the automatic evaluation. All the findings would lose their meaning without human evaluation. A potential issue is that the human evaluation was conducted by two authors of the paper, which could introduce bias. A better approach would be to have independent evaluators to ensure the fairness of the analysis.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic addressed in the paper is relevant for the NLP community.\n\n The generation of parallel datasets for probabilistic and generative evaluations is novel and quite effective in studying and assessing the LLM misgendering problems from multiple angles. \nThe experimental part is solid, and human evaluation is an important contribution.\n\nThe final recommendations are an important plus for fostering research on this topic and helping the LLM’s users.", "reasons_to_reject": "The human evaluation that is a core part of the paper has been performed by the authors of the paper, who might have some biases due to their involvement in the paper. \n\nThe evaluation part is dense with a lot of results and discussion.", "questions_to_authors": "Although it is written at the end of the reproducibility statement, it is fundamental that the authors share the data to foster research on this topic.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on LLM misgendering (for example, getting someone’s pronoun wrong). It considers 3 misgendering datasets, converts them into probabilistic (cloze-style) and generative (sampling) versions, and then shows for six models that the methods can disagree at the instance, dataset and model level. The paper finally also provides a human evaluation that sheds light on the complexity of the problem of misgendering.

Reasons to accept: Well written (xsTw), important topic (fDmf, xsTw, Zan3), explores probabilistic and generative approaches (fDmf, Zan3),  multiple models and datasets (U3xB), useful human evaluation (U3xB, Zan3)

Reasons to reject: Lack of correlation between the two evaluation types is not inherently problematic; it highlights the different capacities (U3xB), human eval was done by the authors themselves which is not ideal (Zan3), the recommendations are (overly) general (U3xB), whether generative and probabilistic evaluations yield the same results previously explored (fDmf), paper does not report/explore averaged results from multiple runs with standard deviations (fDmf), needs more info on annotation process (xsTw), would be useful to provide more information about possible alternatives to the current misgendering evaluations. (xsTw), the evaluation part is dense with a lot of results and discussion. (Zan3)

Overall the reasons to accept by far outweigh the reasons to reject, and this is reflected by the reviewers' scores who agree on a recommendation to accept the paper. Everyone agrees this is an important topic, and the comparison between generative and probabilistic evaluation is well done along multiple axes (datasets, models), even though the authors indeed agree with one of the reviewers that one should not expect these methods to align, and the paper should make this clear. Some other concerns were alleviated during discussion, for example details on the evaluation process. Ideally the human evaluation would not have been done by the authors, but it's not a reason to reject the paper given its contributions. The eval sect could be improved. Finally, the authors might find the following references useful for the next revision of the paper: [1] on converting probabilistic into generative evals, and [2] on misgendering in machine translation and LLMs.

[1] Escaping the sentence-level paradigm in machine translation. Post and Junczys-Dowmunt.
[2] MiTTenS: A Dataset for Evaluating Gender Mistranslation. Kevin Robinson et al.
Decision: Accept