Template-Based Probes Are Imperfect Lenses for Counterfactual Bias Evaluation in LLMs
Authors: Farnaz Kohankhaki
D. B. Emerson
Jacob-Junqi Tian
Laleh Seyyed-Kalantari
Faiza Khan Khattak
Venue: Accepted by TMLR ()

Abstract: Bias in large language models (LLMs) has many forms, from overt discrimination to implicit stereotypes. Counterfactual bias evaluation is a widely used approach to quantifying bias and often relies on template-based probes that explicitly state group membership. It aims to measure whether the outcome of a task performed by an LLM is invariant to a change in group membership. In this work, we find that template-based probes can introduce systematic distortions in bias measurements. Specifically, we consistently find that such probes suggest that LLMs classify text associated with White race as negative at disproportionately elevated rates. This is observed consistently across a large collection of LLMs, over several diverse template-based probes, and with different classification approaches. We hypothesize that this arises artificially due to linguistic asymmetries present in LLM pretraining data, in the form of markedness, (e.g., Black president vs. president) and templates used for bias measurement (e.g., Black president vs. White president). These findings highlight the need for more rigorous methodologies in counterfactual bias evaluation, ensuring that observed disparities reflect genuine biases rather than artifacts of linguistic conventions.

Published in Transactions on Machine Learning Research (01/2026)
Template-Based Probes Are Imperfect Lenses for Counter- factual Bias Evaluation in LLMs
Farnaz Kohankhaki
Vector Institute
Toronto, Ontario, Canada M5G 0C6
D. B. Emerson"
david daridd.mmerron@@vvtttrrittttttt.
Vector Institute
Toronto, Ontario, Canada M5G 0C6
Jacob-Junqi Tian
jacob. tian @vectorinsttttute...t
Vector Institute Toronto, Ontario, Canada M5G 0C6
Laleh Seyaed-Kalantari
Isk&Byorku.c
York University, Electrical
Engineering Computer Science North York, Onlario, Canada, M3J 1P3
Faiza Khan Khattak
Jutzakhanniatttak@@maaaaa
Vector Institute Toronto, Ontario, Canada M5G 0C6
Reviewed on OpenReview: https: // opt.reviee.r...e...mm net/ forum? id= UhWEovKdoU
Abstract
Bias in large language models (LLMs) has many forms, from overt discrimination implicit stercotypes. Counterfactual bias evaluation is widely used approach to quantifying bias and often relies on template-based probes that explicitly state group membership. It aims to mcasure whether the outcome of a task performed by an LLM is invariant to a change in group membership. In this work, we find that lemplate-bes tematic distortions in bias measurements. Specifically, we consistently find that such probes suggest that LLMs classify text associated with White race as negative at disproportionately elevated rates. This observed consistently across a large collection of LLMs, over several diverse template-based probes, and with different classification approaches. We hypothesize that this arises artificially due to linguistic asymmetries present in LLM pretraining data, in the form of markedness, (c.g., Black president vs. president) and templates used for bias measurement (e.g., Black president vs. White president). These findings highlight the need for more rigorous methodologies in counterfactual bias evaluation, ensuring that observed disparities reflect genuine biases rather than artifacts of linguistic conventions.
probes can introduce sys-
1 Introduction
There has been a surge of interest in, and research on, bias in machine learning models. An important area of locus is the presence of bias in large language models (LLMs), especially those trained on exlensive datasets sourced primarily from the internet. These models have attracted increasing attention due to their rapid
integration into wide array of applications (Gallegos et al. 2024 Wan et al. 2023 Sheng al. 2021
Liu
"These authors contributed equally.

---

Published in Transactions on Machine Learning Research (01/2026)
et al. 2023 Bias in these models manifests in diverse ways, ranging from overtly discriminatory generations to more subtle expressions like perpet stereotypes. In particular, biases toward underprivileged groups, such as racial minorities, have rightfully garnered attention, as they persist across many social contexts. Uncovering these issucs represents a crucial step in addressing the potential implications of such biases in downstream applications.
Counterfactual bias evaluation is a common approach in bias quantification that measures invariance, or lack thereof, in the outcomes of model for a particular task across different groups, holding all else equal De-Arteaga al., 2019 Czarnowska et al., 2021 Martinková et al., 2023 Cimitan et al., 2024h. A pertinent example is perturbing the race associated with piece of text from one group (e.g. White) to another (e.g. Black) and measuring whether a model's sentiment prediction changes. Although this is a widely used approach in bias quantification, it ignores the fact that LLM training data does not necessarily follow the same structure for different groups.
In this work, counterfactual bias quantification experiments are performed spanning several ternary sentiment-analysa tasks. A wide range of LLMs is considered, and two classification techniques, fine-tuning and prompting, are applied to perform the classification tasks. Empirically, we observe clear abnormalities such that LLMs assign disproportionately negative sentiment to texts explicitly associated with White race, with similarities to traditionally underprivileged groups like African Americans. For example, positive or neutral statements associated with the White group are misinterpreted as negative at higher rates than other groups. These patterns are consistent across bias probing datasets, LLMs, and classification tech- niques. Overall, the results demonstrate that template- based bias quantification relying on marking has flaws. These limitations reduce the reliability of such measurements as indicators of actual bias dynamics.
The contributions of this work are summarized as follows.
We find evidence that counterfactual bias evaluation using template-based probes introduces sys- tematic distortions in bias measurement. The extent to which template-based probes exhibit mea- surement flaws is systematically quantified through wide range of experiments. These distortions undermine the usefulness of such datasets a lens for bias evaluation.
This paper constructs two new template-based probing datasets from existing work to validate the findings across different domains. These datasets, and the associated techniques for their construc- tion, may be used in future experiments.
This work provides a strong conject ure as to the underlying cause of the aberrant bias measurements. We hypothesize that these disparities are due to the prevalence of markedness in LLM pretraining text, suggesting new research directions.
2 Related Work
Many studies have explored bias in LLMs through fino-grained analysis, primarily using fine-tuning downstream tasks, such sentiment or toxicity classification,
lens.
These studies employ diverse set
of metrics to detect variations in model behavior (Gallegos et al.
2024
Delobelle et al. 2022 Czarnowska
et al.
2021; Mökander et al.] 2023 Liang et al. 2021 Ribeiro et al.
2020
Levy et al. 2023 Echterhoff et al.
2024
Rac et al. 2022). Standard and Chain-of- Thought (CoT) (Wei et al. 2024h
prompting have also been
used
for bias quantification and identification LLMs (Ganguli et al. 2023 Cheng et al., 2023 Kaneko
ct al.
2024: Tian et al., 2023). While some challenges arisc in using prompting in this setting Zayod ct al. it remains useful tool. A multitude of studies, including those cited above, use template-bassed
2024),
probing datasets to perform counterfactual, extrinsic bias analysis in LLMs (Dixon et al.
2018 Huang al.
2020
Liang et al., 2021 Blodgett et al.] 2021 Delobelle et al. 2022 Martinková et al.
2023 Cimitan et al.
2024 ). However, quantitative study of potential caveats with such datasets has not been reported.
In Blodgett et al. 2021,, a critical study of several bias datasets (StereoSet, CrowS-Pairs, WinoBias, Wino- Gender) identified systematic issues likely compromising the precision of biases or stereotyping tendencies of LLMs measured by these datasots. Among other issues, including poor definitions, misalignment, and logi- cal failures, the authors suggest out-of-domain text due to narkedness potentially clouding the proposed
2

---

Published in Transactions on Machine Learning Research (01/2026)
measurements. The investigation therein bolsters our hypothesis that markedness plays significant role in the results to follow. However, their study does not quantily the effect of these flaws. Rather, it simply identifies qualities that may be problematic. Their work also focuses on different datasets than those studied here. Finally, it does not explore template-basch downstream task probes, as done in this work.
Flaws associated with intrinsic bias metrics, which aim to identify bias in model representations rather than downstream tasks, have been examined in Delobelle et al. (2022). Their work demonstrates that such measures not well correlated with extrinsic (downstream) bias mcasures and even fail to provide consistent results between intrinsic measures. The authors identify poorly designed templates, among other factors, contributing to the issues with intrinsic bias metrics. However, the results do not consider or quantify issues with template-based probes for extrinsic bias metries. Moreover, their work focuses exclusively on masked language models (MLMs), whereas the experiments below consider both LLMs and MLMs.
2.1 Linguistic Markedness
The concept of default group membership the absence of direct assignment has been extensively studied in linguistics under the category of markedness (Trubetzkoy 1969 Jakobson 1972. Comrie 1986). In sociological contexts, markedness considers the linguistic differences that arise when referring to default groups compared to others. The idea was first extended to social categories, such gender and race, in Waugh (1982) wherein it is noted that U.S. texts tend to explicitly state (mark) that subject is female and, in costrast, often leave masculine gender implied (unmarked). That is, it is more common to use the term "CEO" when an individual is male compared to "fernaee CEO" when they are female. Many subsequent studies have affirmed that markedness extends to race and, in particular, that non- Non-White individuals are often referred to along with their race, while Whito-race membership tends to go unstated (Cheryan & Berkel et al. 2017 Brekhus 2002).
Markus 2020
Several studies considering the extent to which markedness reporting bias are incorporated into LLMs or
affects their predictions exist (Bender et al.) 2021 Wolfe & Caliskan 2022b a Cheng et al. 2023 Shwartz & Choi 2020) Each of these studies notes that markedness plays a critical role in the way models make predictions and that these models have internalized aspects of markedness through their training. These studies reveal certain biases related to markedness or reporting bias but do not investigate counterfactual bias or template- based probes from this perspective. In Section we conjecture that the irregularities observed in the results to follow are driven by markedness in LLM protraining data.
3 Methodology
In natural language processing, bias measurement typically examines disparities across sensitive attributes such as gender or race (Czarnowska et al.) (021I) Each attribute composed of various protected groups. Herein, the attribute of race is specifically considered. Within the sensitive attribute of race, we restrict our focus to the protected groups of American Indian, Asian, African American, Hispanic, Pacific Islander, and White. A standard bias measurement approach evaluates model performance disparities when protected groups are varied. Ideally, model performance remains invariant to group changes or substitutions.
It should be noted that race and ethnicity have distinct anthropological definitions, yet many studies and bias datasots use the terms interchangeably, including those usod in the experiments to follow. For instance, the templates in Czarnowska al. (2021), discussed below, categorize "Hispanic" under race, though it is commonly considered an ethnicity (Lopez et al. 2023). To maintain consistency with prior work, the term "race" is used throughout, despite its imperfect fit for some protected groups
In this work, counterfactual bias quantification is applied to a collection of LLMs through two downstream task pipelines. In the first, LLMs are fine-tuned for three-way sentiment classification using the SST5 datasct (Socher et al. 2013). ; and bias mcasured by varying group membership across multiple template- based datasets. In the second, LLMs classify template- based datasets directly, without fino-tuning, through prompting. As this study primarily examines race the sensitive attribute, we measure classification
3

---

Published in Transactions on Machine Learning Research (01/2026)
performance disparities across racial groups. Both pipelines analyze false positive rate (FPR) discrepancies between groups. Three template-based datasets are used and detailed in the sections to follow
3.1 Template-Baser Based Datasets
The construction of the three template-based probing datasets used in the experiments is described in this section. Additional details around the composition of the datasets, including the resulting label distributions, are found in Appendix E
3.1.1 Amazon Dataset
This dataset consists of templates for generating examples for specific sensitive attribute, such as gender and race, as well as generic templates that may be used to produce examples for any sensitive attribute (Czarnosska et al.) 2021). Templates specific to the attribute of race and generic templates are both used for the experiments. All templates have a sentiment label and are filled with different race-associated adjectives to generale samples explicitly coupled lo a specific group. Examples are as follows.
(Positive) It was a splendid show of (race_adj heritage.
(Neutral) Everything know about [race_ad] culture I've learned from my mother.
(Negative) I'm sick of the (race_ad) stereotypes.
3.1.2 NS-Prompts Dataset
The NS-Prompts dataset is created from templates using sentences sourced from Wikipedia consisting of professions or job titles and their description (Alnegheimish et al. 2022). As they are occupation descriptions, all examples have neutral sentiment label. The original templates have the following potential formats.
[A/An] (X} is [a/the] jaesuu//ooessiinn/ppppolllssssnp (who/that/whose] {Y}.
[A/An] [X} is |a/the] {Y}.
[A/An] (X) (Y).
where X is profession or job title and Y is the associated description. A completed example might be "A cartoonist is a person who specializes in drawing cartoons or comics."
The focus here is on model variations across racial groups. Thus, the templates are modified by adding (race_ad] adj} before X to be filled with a race adjective tying examples to a specific group. The modified version of the example provided above to form a template
"A {race_ade adj} cartoonist is person who specializes in drawing cartoons or comics."
Though all samples have neutral label, we are interested in measuring model prediction errors while varying these group adjectives.
3.1.3 Regard Dataset
The dataset of Sheng et al. 20199 measures bias toward specific demographics, including race. Its templates consist of complete phrases joined by "and." The first phrase, termed regard, assesses language polarity and social sentiment of a demographic included in the sentence, while the second, originally referred to as sentiment, measures overall language polarity. Each phrase recoives a positive, neutral, or negative sentiment label. An original template highlighting regard and sentiment phrases in red and blue, respectively, is
Template: {demographic-phrasee) was a pimp and her friend was happy.
Regard Label: Negative. Sentiment Label: Positive.
All experimental code is found at: htte::/stttttaoeeeeeettttttttttt

---

Published in Transactions on Machine Learning Research (01/2026)
Because the human-annotated regard label measures the sentiment of the regard phrase associated with specific demographic group, only this part of the templates is retained, along with the corresponding label. Going forward, these labels are simply referred to as sentiment. The broader demographie phrase in the original templates is narrowed to only consider race and membership within the target groups. As an example, the template above becomes:
Modified Template: phrase] was pimp.
3.2 Bias Quantification Metric
The bias metric in the experiments is defined as
dx) = M() M,
(1)
where M is a performance metric and X is a set of examples belonging to the protected group of interest. The fonction dar (X) quantifies the M-gap for a specific group by comparing the metric value restricted to samples from that group, M with the mean metric value observed for each protected group, M. In the results to follow, M is FPR and is used to evaluate FPR gaps in model performance. Gaps for both Positive- and Negative-Sent iment. FPR are measured. Mean gaps and 95% confidence intervals (CIs) are calculated based on five runs.
Negative-Sentiment FPR mcasures the percentage of positive or neutral sentences misclassified as negative. An elevated Peggativ--enttmeen FPR gap suggests a potential lack of preference for a group, where such sentences are classified negative more often. Conversely, Positive-Sentiment FPR denotes the rate at which negative neutral sentences are misclassified as positive. A Positive-Sentiment FPR gap above zero suggests preference for a group, where negative or neutral sentences are classified as positive more frequently. An elevated Degative-Sentiment FPR gap combined with a Positive-Sentimen FPR gap below zero indicates that a group's examples are classified as negative or neutral more often than others, suggosting the group is viewed unfavorably by the LLM.
When considering Negative- or Sentimme- FPR, the interpretation of model errors is fairly straight- forward, as discussed above. On the other hand, Neutral-Sentiment FPR is more convolved. Such errors correspond to neutral predictions for samples with either negative or posilive labels. Errors are a mix of pre- dictions viewing samples with negative labels more positively and those viewing samples with positive labels more negatively. This clouds interpretation of Neutral-Sentimmnt FPR gaps without further decomposition of the metric. As such, results are limited to Negative- and Positive-Sentinnn FPR in this work.
Note that when using FPR, the metric defined Equation is a derivative of False Positive Equality Difference (FPED), standard bias metric (Dixon et al., 2018; Czarnowska al.) 2021). FPED is defined Σ [IF))( - FPR], where the sum is over all protected groups and FPR. represents the FPR for all samples combined. To allow for more granular representation of performance differences across protected groups, three modifications to the FPED metric are present. First, differences are not summed across groups to retain group-specific differences. Second, the directionality of the difference is maintained by shedding the absolute value. Finally, because the number of samples for each protected group is not necessarily equal, mean FPR across groups is computed rather than the all-sample FPR.
3.3 Fine- Fine-Tuning Tuning Experimental Setup
The LLMs considered in this set of experiments are drawn from the RoBERTa [Liu et al., 2020), OPT ZZhang et. al. 2022), Llama-2/3 (Touvron al.) 2023), and Mistral (Jiang et al. 2023) families of models. Specifically, RoBERTa 125M and 355M, OPT 125M, 350M, 1.3B, and 6.7B, Llama-2 7B and 13B, Llama-3 8B, and Mistral 7B are considered. Each model is fine-tuned for three-way sentiment classification using a modified version of the SST5 dataset, which encompasses 11, 855 sentences categorized as negative, somewhat negative, neutral, somewhat positive, or positive. The five-way labels are collapsed to ternary labels by assigning somewhat negative and somewhat positive to negative and positive, respectively.
OPT 125M and 350M and RoBERTa 125M and 355M are fully fino-tuned. Due to their size, the remaining models are fine-tuned with LoRA (Hu al. 2022). Fach model is trained five separate times with different
5

---

Published in Transactions on Machine Learning Research (01/2026)
random seeds. Detailed hyperparametter settings for fine-tuning included in Appendix A To measure model performance disparities across races, each of the trained models performs inference on examples generated from the three datasets discussed in Sections 3.1.1 3.1.3 to predict their sentiment. Using these prodictions, FPR gaps are computed for examples associated with the different racial groups. Training a set of models facilitates the computation of 95% CTs for the gaps, which are reported alongside the mean gaps.
3.4 Prompting Experimental Setup
Three prompting strategies are applied to predict sentiment. These are zero-shot prompts, shot prompts with shots drawn from two sentiment analysis datasets, and zero-shot CoT prompts [Kojima et al. (024).. For all prompting experiments, Hugging Face's text-generation pipeline is used with OPT-6.7B, Llama-2-7B, Llama-3-8B, Mistral-7B, Gemma-7B Instruct (Gemma et al. 2024), and (wen-2255713 Instruct (Qwen et al. 2025 These models correspond to the Hug- ging Face identifiers facebook/opt-6. 7b, Meralai/Mistra-777
meta- Maaaa-llaaa/MMt-llaanee and Qwen/ /woon//uen...e 5-7o22.5-BB-- -e7--IInsruct.. Sampling is turned
8B,
1, porgle/gemma-7ee on, and a temperature of 0.8 used for all generations, including reasoning traces. Predictions are extracted from the final stage of text generation using case-insensitive exact match for the strings "negative," "neu- tral," or "positive." The first match is taken as the predicted label. In the event that a response fails produce match, the predicted label is uniformly sampled from the three possible labels. In all but the reasoning generation stage of zero-shot CoT, models produce maximum of three tokens in their response.
RoBERTa-122 125M RoBERTa 355M
OPT-125M OPT-350M OPT-1.3B OPT-6.7B OPT-13B
Llama-2-78
Llama-22 2-13B
Llama-3-88 Mistral-7B
0.2
Negative Sentiment FPR Gap
0.15
0.1
0.05
0
-0.05
-0.1
-0.15
Positive Sentiment FPR Gap
0.1
0.05
0
-0.05
-0.1
-0.15
african_american american_indiag
asian
hispanic
Dacific_sslander
white
Figure 1: Negative- and Positive-Sentimne FPR gaps as measured by the Amazon dataset.
For the few-shot prompt templates, ninc labeled examples are prepended to the prompt, matching the template style. Two distinct experiments are conducted with labeled demonstrations drawn from either the SST5 or SemEval [Mohammad et al. 2018) datasets. For SST5, labels are collapsed as described in Section
3.3 The SemEval polaritics are condensed via the mapping (Negative: [-3, -2], Neutral: [-1, 0, 1). Positive:
[2, 3].. In both cases, avoid any few-shot bias (Gupta et al.) 2024), demonstrations are balanced between negative, neutral, and positive each), but order is random. Demonstrations constant across models,

---

Published in Transactions on Machine Learning Research (01/2026)
but are resampled across the five prediction runs of each experiment. For all prompts, random seeds for shot selection and text generation are set to (2024, 2025, 2026, 2027, and 2028) across the five runs.
Zero-shot CoT uses two sequential prompt templates. CoT prompting is not applied to OPT, as the model has limited reasoning capacity {Liang et al. 2023) the first step, the model receives the text and is asked about its sentiment. The traditional CoT "trigger," " "Tet'' think step by step" encourages reasoning before answering. Reasoning traces are capped 64 tokens. To quantify generation stochasticity, each example is predicted five times. All prompt templates across strategies and other settings appear in Appendix B
4 Results
4.1 Fine- Tune-Tunng Results
The Negative- and Positive-Sentiment FPR gaps for the Amazon dataset are shown Figure T For most models, the Regative-Sentiment FPR gap for White-associated text is significantly above zero at 95% confi- dence. This implies that the models more often misclassify positive- group negative compared with others. For large OPT, Llama-2 and Mistral LLMss, similar but smaller clevation in this gap is observed for examples associated with African Americans and Asians. For the Positive- Sentiment FPR gap, a significant negalive value is observed for all models. More recent models, Llama-3 and Mistral, exhibit some of the largest negative gaps. Combined with an elevated Negative-Sentimen FPR gap, this implics that the models tend to view examples from the White group negative light more often than other groups.
neutral-Sentiment examples for this
RoBERTa 10BERTa-125M
RoBERTa 355M
OPT-125M OPT-350M OPT-1.3B OPT-6.7B 7B OPT-13B
Llama-2-78 Llama-2-13B Llama-3-88 Mistral-7B
0.15
Negative Sentiment FPR Gap
0.1
0.05
0
-0.05
0.11
0.1
Positive Sentiment FPR Gap
0.05
-0.55
0.1
00.15
african_american american american_ indian
asian
hispanic
pacific_isladdr
white
Figure 2: Negative- and Dositive-Sentiment FPR gaps as measured by the NS-Prompts dataset.
Figure 2 displays the measured gaps for the NS- NS-Prompts dataset. Recall that all labels for this dataset are neutral. Thus, any non-neutral predictions are, by construction, incorrect. When considering RoBERTa and Llama-2 models, the identified gaps share similarities with the African-Ameriest group. That is, elevated Negative-Sentiment FPR gaps and Positive-Sentiment FPR gaps below zero. While the Negative-Sentiment

---

Published in Transactions on Machine Learning Research (01/2026)
FPR. gaps for other models zero for White examples, all models produce negative and statistically significant. Positive-Sentimmnt FPR gaps. This implies that neutral examples associated with White race are construed as positive at much lower rates relative to other groups.
RoBERTa- 125M
RoBERTa- 355M OPT-125M OPT-350M OPT-1.38 OPT-6.7B OPT-13B
Llama-2-B Llama-2-138 Llama-3-88 Mistral-7B
0.25
Negative Sentiment FPR Gap
0.2
0.15
0.1
0.05
0
-0.05
-0.1
0.15
0.15
Positive Sentiment FPR Gap
0.1
0.05
0
0.05
-0.1
0.15
-0.2
aactricnn american_ indian
asian
hispanic
pacific_islander
white
Figure 3: Negative- and Posilive-Sentimmnt FPR gaps measured by the Regard dataset.
Results for the Regard dataset reveal similar trends to the Amazon and NS-Prompts experiments. However, the gaps, displayed Figure 3 are somewhat smaller. As previous measurements, White-associated texts see elevated Peeative-Sentiment FPR gaps and Positive-Sentimen FPR gaps below zero for many models. Furthermore, strong parallels exist for the gaps observed for text associated with African Americans. This is especially true for RoBERTa, small OPT, Llama-2, and Llama-3 models, where the gaps for these groups are highly correlated.
The measurements in these results are surprising. However, as discussed in detail Section 5 below, the gaps observed for the White group are not believed to be reflections of true bias. Rather, we conjecture that they are an artifact of a mismatch between the template-based probing datasets that explicitly reference race in the text of samples to link membership and the presence of markedness in LLM pretraining data.
4.2 Prompt-Based Results
cxhibit clear anomalics when measuring performance gaps using templato-basee basod probes. A natural question is whether such irregularities arise due to the lask-specific fine-tuning step represent intrinsic quality of the LLMs. To further isolate the issue to LLM pretraining, prompting is used to perform sentiment classification for the Amazon dataset, shedding the need for fine-tuning. The experiments are limited to decoder-only models of sufficient size to ensure that classification performance adequately exceeds that of a random classifier.
The results in Soction 4.1
The average classification accuracy of the prompting and fino-tuning approaches on the Amazon dataset is reported in Appendix[C Generally, the accuracy of prompt-based classification lower than the fine-tuning counterpart. This especially true for the oldest model, OPT. However, newer models still provide good
8

---

Published in Transactions on Machine Learning Research (01/2026)
performance through prompting. Notably, Qwen-2.5 produces very strong classification accuracy with 9- shot prompt drawn from SemFval at 92.3%. Other models also perform lairly well using few-shot prompting. Regardless, as classifiers, all prompted LLMs perform well above random model. Perhaps due to model size or limited reasoning tokens, zero-shot CoT does not significantly improve performance (Wei et al. 2024
As in Section 4.1 Negative- and Positive-Sentiment FPR gaps are computed for each LLM's predictions. These gaps are exhibited in Figure | Due the lower accuracy and generation volatility, the gap Cls are visibly wider than those of the fine-tuning experiments. Nonetheless, clear and familiar pattern is seen in these results. Positive mean gaps in Degative-Sentiment FPR present across majority of examples for African American and White races. Similarly, negative mean gaps for Positive-Sentiment FPR are measured for both races in most settings. The consistency between these results and those of the fine-tuning experiments strongly suggests that the irregularities present in the template- based measurements are not the result of the fine-tuning stage, but are, rather, an expression an intrinsic aspect of the LLMs themselves.
0PPT6.78 0P SSTS
Liama SSTs
2 SE
Liama 2-78
Llama-3-at
Mistral-7B
Mistrai-7B
Gemma-7B
Gemma-78
Gomma-78 CoT
Owen-2.5-7B Zwen-2.5-78
Qwen-2.5-78 SE
Qwen CoT
Negative Sentiment FPR Gap
0.3
0.2
0.1
0.1
Positive Sentiment FPR Gap
0.05
-0.05
-0.1
african_ane Lamerian
american_indian Indian
asian
hispanic
pacific_islane
white
Figure Negative- and Positive-Sentiment FPR gaps as measured by the Amazon dataset with prompt- based classification. In the legend, model names without suffix indicate zero-shot prompting. SST5 and SE indicate 9-shot prompts with examples drawn from the SST5 and SemEval datasets, respectively.
5 Discussion
Across the experiments overall tendency of the models to classify White-associated text exhibiting more negative sentiment at higher rate than other groups is observed. The trends in the results above are consistent between model typo, model size, templato-based probing dataset, and even classification stralegy. The overall agreement of the prompting and fine-tuning results indicates that the observed gaps are not linked to idiosyncrasies in the fine-tuning process but are, rather, more fundamental to the LLMs themselves and the design the templato-basod probes. In addition, the models chosen for experimentation are base instruction-tume tuned versions. That is, their predictions are not influenced by interceding alignment techniques (Bai et al. 2022 Rafailov et. al. 2023) which might otherwise obscure behavior learned during

---

Published in Transactions on Machine Learning Research (01/2026)
pretraining. Rather than implying extant bias, hypothesize below that this phenomenon is due to an interaction between the structure of the templates used in the measurement of bias and markedness in LLM pretraining data. Regardless of the underlying cause, these observations should lead us to re-think the clarity of counterfactual bias analysis in this context.
5.1 Markedness and Cemplate-Based Probes
English pretraining data for LLMs is dominated by text drawn from areas where the racial majority is White (Bender et al., 2021 Navigli et al. 2023). Several studies have confirmed that markedness is widespread in internet data, with White race and male gender constituting the unmarked defaults (Wolfe & Caliskan 2022at Bailey et al. 2022). Furthermore, has been shown that models, and LLMs particular, trained on web data reflect these markedness characteristics (Bender et al.] |021 Wolfe & Caliskan 2022b a Cheng ct al.] 2023). On the other hand, in templates commonly used for bias quantification, race is explicitly mentioned to establish group membership. As such, template-based text that explicitly establishes that
the subject is "White" essentially constitute out-of-domain examples (Blodgett et al. 2021 Dressler 1985
Such a mismatch likely influences model predictions.
We hypothesize that the disparities observed in Section associated with the White group are due to the prevalence of markedness I.L. pretraining text. A key assumption underlying unmarked representations is that humans are adept at recognizing unstated implications in text. LLMs trained solely on unstructured next-token prediction, which underpins almost all modern LLM pretraining, may lack the ability to per- ceive such implications, resulting in surprising behavior. Using templates that represent group membership through explicit description likely makes certain text appear uncommon for traditionally unmarked groups. As such, these templates may lead to artificially elevated error rates in LLMs, skewing bias measurements in unpredictable ways and clouding the lens provided by datasets of this structure. To this end, Appendix provides an extended set of results showing that similar irregularities arise when considering unmarked groups for sexuality and gender, providing additional supporting evidence for this hypothesis.
Including datasets that explicitly correct for markedness in LLM pretraining could better align template- based text. Appendix D suggests that more recent LLMs, trained on larger multilingual datasets, show improvements in mcasured gap sizes. Both Llama-3-8B and Mistral-7B have the smallest difference between the most positive and negative gaps for Negative-Sentimmnn FPR, averaged over the three datasets. Llama- 3-8B also produces the lowest average difference for Sentiiee-Sentiment FPR. Given that White-group gaps often rank among the extremes, this suggests newer models may be lcss affected by markodness.
The studies and results above, and in the appendix, suggest that markedness may indeed play role in the experimental observations of this work. However, empirical validation of the conjecture that markedness is the mechanism introducing the effects observed in Section E likely requires that an LLM be trained from scratch using a dataset with "consistent" marking. Thereafter, the gaps across demographics would be re- assessed using the same template-based probes. The size and availability of LLM pretraining data make constructing such a dataset quite challenging. Moreover, the computational resources required to properly pretrain an LLM are substantial. As such, this valuable study deferred future work.
6 Conclusions and Future Work
This paper presents unexpected, and likely flawed, bias measurements related to race when using template- basod bias probes. The measurements remain consistent across a number of different experimental scttings and varied datasets. Rather than indicating genuine social bias in the LLMs, we conjecture that these outliers stem from a misalignment between template-based bias probes and LLM pretraining data due to markedness. Regardless of the underlying cause, these findings highlight the need to consider the impact that the use of bias probes relying on marked text has on the measurement of bias. In this case, such probes produce largely misleading results.
Assuming that linguistic markodness contributes significantly to the mcasurements in this work, which requires further investigation to confirm, several avenues for mitigating such effects when using template- based probes are worth exploring. LLMs trained on more global representation of text, where majority
10

---

Published in Transactions on Machine Learning Research (01/2026)
demographics differ and marked groups vary, could improve the robustness of such models when encountering explicit, demographic descriptors. Another approach is to strive to control for the impact of markedness by designing evaluation setups that test both unmarked and explicitly marked versions of the same text (for example, comparing "a CEO" and "a White CEO") or by using neutral placcholders like [RACE] to isolate the impact of demograppic terms and potentially correct for the flaws identified in this work. There are, however, challenges with this approach. One needs to identify which groups are considered unmarked from the "perspective" of the LLM, requiring detailed knowledge of the underlying pretraining data. In addition, the unassociated text is unlikely to solely represent the unmarked group, but rather mix of representations. Ideally, artificial injection of demographic information would not be required. For example, the studies of Seyyed-Kalantari et al. (0200) and [Sap et al. (2019) establish group membership through meta-data, self- identification, or classification techniques rather than explicilly in text. These methods avoid the out-of- domain nature of template- based examples of the kind studied here and do not see the unnatural patterns we obscrved. Future work will design experiments to validate the misalignment due to markodness conjecture and construct straightforward ways to mitigate such issues in LLMs.
References
Sarah Alnegheimish, Alicia Guo, and Yi Sun. Using natural sentence prompts for understanding biases in language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Tvan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2824-2830. Association for Computational Linguistics, Seattle, United States, July 2022. doi: org/2022 2022. naacl- main. 203
h/tp:///212222ncclllaaam main.203. URL https://aclanthool.0n
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac HalfieldDDodds, Danny Hernandez, Tristan Hume, Scout Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. Preprint at pettpa//arrzie org/ abs/ 2204.05862 05862)
April H. Bailey, Adina Williams, and Andrei Cimpian. Based on billions of words on the internet, people-men. Science Advances, 8(13)eabm22463, 2022. doi: wt.1126/sciavv.at woc.eece..rg/dol......
URL https
wwww.scieecc.eom org/doi/ abt.//b........
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majos and Shmargaret Shmitchell. On the dan- gers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountabilily, and Transparency, FAceT '21, pp. 610 623. Association for Computing 3445222 URL
Machinery, New York, NY, USA, 2021. ISBN 9781450383097. doi:
https://doi.org:: 11. 1145/3442188 3444499998884449222..
Laura Van Berkel, Ludwin E. Molina, and Sahana Mukherjee. Gender asymmetry in the construction of American national identity. Psychology of Women Quarterly, 41:352-367, 2017.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping Norwe- gian salmon: An inventory of pitfalls in fairness benchmark datasets. In Chengqing Zong, Fci Xia, Wenjic Li, and Roberto Navigli (eds.), Proceedings of the 59th Annval Meeling of the Associalion for Compula- tioral Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015. Association for Computational Linguistics, Online, August 2021. doi: (0.18653/v1| v0t/:///vv222la// lcl-- URL
org/2021 2.022.a00 argg/201..cl11oomm acl.long..8.... 81
Wayne Brekhus. A sociology of the unmarked: Redirecting our focus. Sociological Theory, 16(1):34-51, 2002.
Myra Cheng. Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. In Anna Rogers, Jordan BoydeGraber, and Naoaki Okazaki (eds.), Proceedings of the G1st Annual Meeting of the Association for Computational Lánguistics (Volume
11

---

Published in Transactions on Machine Learning Research (01/2026)
1: Long Papers), pp. 1504-1532. Association for Computational Linguistics, Toronto, Canada, July 2023.
doi: 10.18653/v1 /2223 URI.
acl-lon.. 84
Sapna Cheryan and Hazel Rose Markus. Masculine defaults: Identifying and mitigating hidden cultural biases. Psychological Review, 127(6):1022- 1052, 2020.
Ana Cimitan, Ana Alves Pinto, and Michaela Geierhos. Curation of benchmark templates for measuring gender bias in named entity recognition models. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 4238 4246, Torino, Italia, May 2024. ELRA and ICCL. URL httap://aaclanthology .or.looy.ooooooooooo..oo..o.ooe. org/2024. Lrec-main.370
Bernard Comrie. Markedness, grammar, people, and the world. In Fred R. Eckman, Edith A. Moravesik, and Jessica R. Wirth (eds.), Markedness, pp. 85 106. Springer US, Boston, MA, 1986. ISBN 978-1-4757-5711-7, URL https://doi. org/ 1907/ 1-777778--17777777---------
doi: 10.1007 0000079788--4757-777777
Paula Czarnowska, Yogarshi Vyas, and Kashif Shah. Quantifying social biases in NLP: A generalization and empirical comparison of extrinsic fairness metrics. Transactions of the Association for Computational Linguistics, 9:1249 1267, 2021.
Maria Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Choulde- chova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: a case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Ac- countability, and Transparency, IAT*'19 pp. 120-128. Association for Computing Machinery, USA, 2019. Atlanta, GA.
Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. Measuring fairness with biased rulers: comparative study on bias metrics for pre-trained language models. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computaaionnl Linguistics: Human Language Technologies, pp. 1693 1706. Association for Computational Linguistics, Seattle, United States, July 2022. doi: 10. 18653/vl hwtp//22.naalllmaaiee main. 122. URL
20222022.22.22 meaan.-min..... 122
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, ATES '18, pp. 67-73, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450360128. doi: 10.1145/322 /377/7722772117772 URL btttp:://doi.rggee
1145/3278721 .2017777711377779999 3278729
Wolfgang U. Dressler. On the predictiveness of natural morphology. Journal of Länguistics, 21 337, 1985. ISSN 00222267, 14697742. URL
Jessica Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. Cognitive bias in high-stakes decision-making with LLMs, 2024. Preprint at httpss//arxiv.. org/ www.a///........
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 1097)11097 09 2024. ISSN 0891-2017. doi: 10.1162/coli a0522. URL
https://do:: org/ 10. 116//cooi...c.o 00524
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilé Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott John- ston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Humo, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodci, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. The capacity for moral self-correction in large language models, 2023. URL https:: //wrxivaavvv/////////a httgg/bb/2302..0//0
12

---

Published in Transactions on Machine Learning Research (01/2026)
Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélic Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahri- ari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphnee Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, Chogg-Chrisstin Mu- raru, Grigory Rozhdestvenskky, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, -ran-stp Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yolov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Schastian Borgcaud, Sertan Girgin, Sholto Douglas, Shrce Pandya, Siamak Shakeri, So- ham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kencaly, Gemma: Open models based on gemini research and technology, 2024. URL /tttt::t///aat pettps//arrxvv .rg. argg/bss24033 082995
Karan Gupta, Sumegh Roychowdhury, Siva Rajesh Kasa, Santhosh Kasa, Anish Bhanushali, Nikhil Patti- sapu, Prasanna Srinivasa Murthy, and Alok Chandra. How robust are Ilns to in-context majority label Workshop on Responsible Language Models, 2024.
bias? In AAAI 2024
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations. International Conference on Learning Representations, 2022.
Po-Sen Huang, Huan Zhang, Ray Jiang, Robort Stanforth, Johannes Welbl, Jack Rac, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual evaluation. Tn Trevor Cohn, Yulan He, and Yang Lin (eds.), Finddings of the Association for Compudational Linguistios: EMNLP 2020, pp. 65 83, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/ 10533/vv//0 wtt.///...//////e// www..fiiiinngeemnlp.. URL https://aclantholooon 2ttp://clatttllllyoog//00......0 findings-eer Seadinng--enlp.77
Roman Jakobson. Verbal communication Scientific American, 227(3):72 81, 1972. ISSN 00368733,
19467087. URL http:://ww..jsteeee jstor http::/wjsoooooorsssseee wtg.sttable/492772222 /
Albert Q, Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. Preprint at hhttts://aaxiii
06825
Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, and Timothy Baldwin. Evaluating gender bias in large language models via chain-of-thought prompting, 2024. Preprint at 15585
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22. Curran Associates Inc., Red Hook, NY, USA, 2024. ISBN 9781713871088.
Sharon Levy, Ncha John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballestcros, Vittorio Castelli, and Dan Roth. Comparing biases and the impact of multilingual training across multiple languages. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10260-10280. Association for Computational Linguistics, Singapore, December 2023. doi: 10.18653/v1 v118653/v1/20223 222/2222.2.. eon0.. wat.eemmlmmmmiii66.. main.634. URL https://acllnthooooy..r///// emnlp-main 634
13

---

Published in Transactions on Machine Learning Research (01/2026)
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinor. Towards understanding and mitigating social biases in language models. In Proceedings of the 38th International Conference on Machine Learning, pp. 6565 6576. PMLR, 2021.
Percy Liang, Rishi Rommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Coosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksckgomul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Teard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learnnng Research, 2023. ISSN 2835-8856. URL Featured Certification, Expert Certification.
https://openreviees httpss//gppaneereeeeeeesssserrrp
Yang Liu, Yuanshun Yao, FrannoFaancos Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy LLMs: a survey and guideline for evaluating large language models' alignment, 2023. Preprint at https: http:///wxxxv..o//brrsss//sa abggg//s/3300/..ommm 25374
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2020. URL http:////pppprrrreeeeeeeeerorrrr
Mark Hugo Lopez, Jens Manuel Krogstad, and Jeffrey S. Passel. Who is hispanic? Pew Research Center, is-hispanic/
http://pp//wwwppeeerrrr....rrr/a 2023. URL httttt:////www.tt.. www.wpewwweecchhceoocm htw.a//r..or...ottoooerraa//////
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. Preprint at https://
arxiv. www.l..rrgabbb////// 1t///111..0000 05101
Sandra Martinková, Karolina Stanezak, and Tsabelle Augenstein. Measuring gender bias in West Slavic language models. In Jakub Piskorski, Michal Marciñczuk, Preslav Nakov, Maciej Ogrodniczuk, Senja Pollak, Pavel Přibáñ, Piotr Rybak, Josef Steinberger, and Roman Yangarber (eds.), Proceedings of the 9th Workshop on Slanic Natural Language Processing 2023 (SlavicNT.P 2023), pp. 146-154, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 22tp:///2222222m/p1111 bsnlp-1..7... URL https://aclantholl/ wrg.:.oggggog 2023. httgg/222..ssllp..1111 bsnlp-1.177
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salamch, and Svetlana Kiritchenko. SemEval-2018 task 1: Affect in tweets. In Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (eds.), Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 1 17. Association for Computational Linguistics, New Orleans, Louisiana, June 2018. doi: 1..18653/v1/S1e 1001. URL https : pattps://ccanntn http///aaaaatttoy....o/ org/ S18-1001
Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. Auditing large language models: A three-layered approach. AT and Fthics, pp. 1 1-31, 2023.
Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inventory, and discussion. J. Data and Information Quality, 15(2), jun 2023. ISSN 1936-1955. doi: 10.1145/3597307
URL https::/doo....o org/ 10. 1145/3597307
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:://arxi org/
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Fliza Rutherford, Tom Hennigan, Jacob
14

---

Published in Transactions on Machine Learning Research (01/2026)
Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Ge- offrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2022. URL http::///xxivvorr/abbbb... htt///abb221...11111 221./2112..1114111
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Sys- tems, volume 36, pp. 53728 53741. Curran Associates, Inc., 2023. URL /Rtp/ttps///prrreddgggg www.riinnnnn...nss.....mm neurips. .pd
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral lesting of NLP models with CheckList. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 4902- 4912. Association for Computational Linguistics, Online, July 2020. doi: 10. 20116653/w122200.cm atc.:/2000..llmim... URL https://aclantthologg wwwpolloooyyoooo/ggooo org/2020..c..2
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The risk of racial bias in hate speech detection. In Anna Korhonen, David Traun, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Lúnguistics, pp. 1668-1678. Associ- ation for Computational Linguistics, Florence, Italy, July 2019. doi: 10.18653/v1/P19-111 1163. URL 1163
Laleh Seyyed-Kalantarr Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Chex- clusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: proceedings of the Pacific symposium, pp. 232 243. World Scientific Publishing Company, 2020.
Emily Sheng, Kai- Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysit- ter: On biases in language generation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.); Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 3407 3412. As- sociation for Computational Linguistics, Hong Kong, China, November 2019. doi: 10.18653/v1/D19- 1339.
URL https: :attps://acaanthooogg. htt.:/ccaaattooooooooroo///----. org/
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. Societal biases in language generation: Progress and challenges. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 9275-4293 Association for Computational Linguistics, Online, August 2021. doi: 10.18653/v1 11/222 10001.acl-laa acl- long.ong..30. URL https:
//aclanthology.oo. org//02211200 2tt.2220......oo......amm ac1-long.330
Vered Shwartz and Yejin Choi. Do neural language models overcome reporting bias? In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Lin- guistics, pp. 1863-6870 International Committee on Computational Linguistics, Barcelona, Spain (On- line), December 2020. doi: 10. 10.18653/v2 /v/l///vv baling-main. 605
main. 605. URL attp:://eelmmcooogg..o//////
15

---

Published in Transactions on Machine Learning Research (01/2026)
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631- 1642. Association for Computational Linguistics, Seattle, Washington; USA, October 2013. URL
https::/aclamttoooog.oor//////// org/D13-1170
Jacob-Junqi Tian, Omkar Dige, David Emerson, and Faiza Khan Khattak. Interpretable stereotype identi- fication through reasoning, 2023. Preprint at httpps///aariw.or//a////////////
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojuic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. Preprint at /http:///arx/ org/
Nikolai Sergeevich Trubetzkoy. Principles of Phonology. University of California Press, 1969.
Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai- Wei Chang, and Nanyun Peng. "Kelly is warm person, Joseph is a role model": Gender biases in L.M-generated reference letters In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3730-372 3748. Association for Computational Linguisties, Singapore, December 2023. doi:
(0.18653/v112023... wwwndiinnnnemiiiii
URL
findinnn-diinnineeeee emmlp.mmmll....... camlp.2433 243
Linda R Waugh. Marked and unmarkod: A choice between unoquals in semiotic structure. Linguistics, 1982.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. ofain--of-thuugh prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22. Curran Associates Inc., Red Hook, NY, USA, 2024. ISBN 9781713871088.
Robert Wolfe and Aylin Caliskan. Markedness in visual semantic ai. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAceT '22, pp. 1269 1279. Association for Computing Machinery, New York, NY, USA, 2022a. ISBN 9781450303522. doi: 00.1145/5531144.3333333
URL https://doi. 11451.1111///311461161 11145//33311463333183 3533183
Robert Wolfe and Aylin Caliskan. American white in multimodal language-aand-image ai. In Proceedings of the 2022 AAAT/ACM Conference on AT, Ethics, and Society, AIES '2, pp. 800-812. Association for Computing Machinery, New York, NY, USA, 2022b. ISBN 9781450392471. doi: 00.1155/5514094.33311333 URL httpp:://ai..org/0...1111 1145/151441333144.4.. 3534136
AbdeIrahman Zayed, Goncalo Mordido, Ioana Baldini, and Sarath Chandar. Why don't prompt-based fairness metrics correlate? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (ods.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguislies (Volume Long Pupers), pp. 9002 9019, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/ v1/2024.al-loogg.t URL https://acclatthooogs org/ 2024 ac1-4aclllooo acl-1ong.445 487/
Eviatar Zerubavel. Taken for Granted: The Remarkable Power of the Unremarkable. Princeton University Press, 2018.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language modcls, 2022. Preprint at https:: /aass:///xxvvva http//aaivvv.rrrr/a/a/// org/abs/////eooooooo http:/ab/2220..000 22e./ 01068
A Fine- Tuning Hyperparameters
For completeness, we provide the full details of the hyperparameter tuning process used in the fine-tuning experiments. During fine-tuning, early stopping applied based on validation loss. If no improvement the loss is observed over fixed number of steps, then training stopped. An AdamW optimizer is used with default parameters, except for learning rate (LR) and weight decay (Loshchilov & Hutter 2019). A hyperparameter study performed to select the best early stopping threshold and LR for all models. For fully fine-tuned models, weight decay was also optimized.
The early stopping threshold was varied between five and seven steps. The learning rate (LR) was chosen from {le-3, 3e-4, 1e-4, 3e-5, le-5], and weight decay, when tuned, was selected from {le-3, 1e-4, 1e-5, 1e-6). For larger models, LoRA fine-tuning was applied with the rank parameter 8 on every non-embedding layer.
For RoBERTa 125M and 355M and OPT 125M and 350M, 15 training models with the highest accuracy the SST5 test set were retained. For the larger models, due to re- source constraints, five models in total were trained for each model type. Table I summarizes the optimal hyperparameters selected for each model during fine-tuning.
were performed, and the five
Table Hyperparameters used for model fine-tuning.
Model
Early stop threshold
LR
Weight decay
RoBERTa-12555 125M
7
le-5
le-5
RoBERTa-3543
7
le-5
le-5
OPT-125M
7
le-5
le-5
OPT-350M
7
le-5
le-3
OPT-1-3B
5
1e-4
1e-4
OPT--67B
5
le-4
le-4
OРT-13B
5
1e-4
le-4
Llama-2-77
5
le-4
le-4
Llama-2-11 2-13BB
5
le-4
le-4
Llama-3-8B
5
le-4
le-3
Mistral-7B
5
3e-5
le-3
B Prompt Templates and Other Details
This section includes the templates used in the prompting approach. Each subsection corresponds to different template. For CoT prompting, inference batches are limited to size 4 due to higher computational demands, whereas batch sizes of 16 are used in other settings.
B.1 Zero-Shot Prompt Template
The zero-shot prompt template is displayed below with additional formatting for roadability. The component in angled brackets is where each sample to be classified is inserted. The models begin generation at [LM Generalion].
Text: (Text to classify)
Question: Is the sentiment of the text negative, neutral, or positive? Answer: The sentiment is [LM Generation)
17

---

Published in Transactions on Machine Learning Research (01/2026)
B.2 Few-Shot Prompt
Below is the few-shot template. For the few-shot prompt templates, nine labeled examples are prepended to the prompt, following the template style. The models begin generation at LM Generation).
Text: Example 1 from either SST5 or SemEval Question: What is the sentiment of the text?
Answer: Negative.
Text: Example 9 from either SST5 or SemEval Question: What the sentiment of the text? Answer: Positive.
Text: (Text to classify)
Question: What the sentiment of the text?
Answer: [LM Generation]
B.3 Zero-Shot CoT Prompt
Zero-shot CoT uses two prompt templates in sequence. In the first step, the model is provided the text to classify and asked about the corresponding sentiment. The traditional "trigger" sentence "Let's think step by step" is used to encourage the model to generate reasoning prior to answering the question. The template appears below.
Text: (Text to classify)
Question: Is the sentiment of the text negative, neutral, or positive?
Reasoning: Let's think step by step. [LM Generntion]
In the second step of zero-shot CoT, the reasoning generation is appended to the first prompt along with the answer completion text displayed in the template below. At this stage, the model is expected to generate an answer to be extracted.
Text: (Text to classify)
Question: Is the sentiment of the text negative, neutral, or positive?
Reasoning: Let's think step by step. (Generation from previous step)
Answer: Therefore, from negative, neutral, positive, the sentiment is [LM Generation'
Table 2:
Accuracy statistics on the Amazon dataset for fine-tuning experiments across model types and sizes. Bold numbers indicate the best accuracy achieved within each model family.
Model
Size
Mean Accuracy Standard Deviation
125M
0.635
0.036
RoBERTa
350M
0.624
0.027
125M
0.687
0.080
350M
0.692
0.039
OPT
1.3B
0.739
0.020
6.7B
0.737
0.014
7B
0.513
0.089
Llama-2
13B
0.647
0.006
Llama-3
8B
0.822
0.035
Mistral
7B
0.740
0.005
18

---

Published in Transactions on Machine Learning Research (01/2026)
C
Fine-tuning and Prompting Accuracy
Tables 2 and present the average classification accuracy and standard deviation for the fine-tuning and prompting approaches on the Amazon datasct, respectively. Generally, prompt-based classification accu- racy is lower than that of fine-tuning. There is also notable variability in model classification accuracy on the template-based probe dataset as a whole, with newer models tending to produce better performance. However, the trends associated with the measured FPR gaps, especially for the White group, are largely con- sistent, regardless of these variations. Do to this consistency, the differences in model accuracy are unlikely to be a primary contributor to the observed behavior.
Table 3: Model accuracy and standard deviation (in parentheses) on the Amazon dataset for prompting experiments across model types. Bold numbers indicate the best accuracy achieved for each model.
Prompt Type
Zero-shot
Zero-shot CoT
SemEval 9-shot
SST5 9-shot
OPT-6.7B
0.451 (0.002)
0.482 (0.009)
0.433 (0.024)
1lama-2-7B
0.483 (0.002)
0.492 (0.003)
0.654 (0.037)
0.616 (0.028)
Tlama-3-8B
0.600 (0.003)
0.539 (0.001)
0.683 (0.017)
0.716 (0.024)
Mistral-7B
0.502 (0.003)
0.517 (0.003)
0.700 (0.045)
0.682 (0.025)
Gemma-7B 7B
0.830 (0.001)
0.777 (0.003)
0.854 (0.020)
0.804 (0.031)
Qwen-2.5-7B
0.899 (0.001)
0.823 (0.002)
0.923 (0.016)
0.906 (0.001)
Table 4: Models ranked by average gap spans across datasets for Negative- and Positive-Sentimenn FPR when fine-tuning. For given type of FPR, gap spans are computed as the largest difference between any two mean FPR gaps across groups. The larger this span, the greater the difference in Negative- or Positive-Sentiment FPR between groups and the less invariant the model is to group substitution.
Mean Srgative-Sentinnne
Mean
Rank
Model
FPR Gap Span
Model
FPR Gap Span
1
Llama-2-13B
0.207
RoBERTa-355M
0.154
2
RoBERTa-335M
0.198
RoBERTa-125M
0.152
3
Llama-2-7B
0.144
OPT-13B
0.144
4
RoBERTa-125M
0.126
Alama-2-132
0.143
5
OPT-350M
0.118
Mistral-7B
0.141
6
OPT-1.3B
0.104
OPT-125M
0.136
7
OPT-6.7B
0.081
-lama-2-7B
0.133
8
OPT-13B
0.056
OPT-350M
0.132
9
OPT-125M
0.039
OPT-1.3B
0.128
10
Mistral-7B
0.032
OPT-6.7B
0.089
11
Llama-3-8B
0.020
Llama-3-8B
0.089
Gap Differences Across Models
For each of the fine-tuned models, across the different datasets, an FPR-gap span is caleulated. For a given type of FPR, Negative- or Positive-Sentiment gap spans computed as the largest difference between any two mean FPR gaps for the groups. This quantifies how large the particular FPR disparities for given model and datasct are between groups The larger this span, the greater the difference in Negative- Positive-Sentiment FPR between groups and the less invariant the model is to overall group substitution. Table 4 displays the FPR gap spans for each model, averaged over the three datasets. In computing the spans, the gap for the White group is part of the span extremes 58% of the time for Negative-Sentiment FPR and 100% of the time for Dositive-Sentmmen FPR. That is, the gap computed for the White group often constitutes one of the largest gap magnitudes.
19

---

Published in Transactions on Machine Learning Research (01/2026)
From the table, it is clear that the RoBERTa and Llama-2 models have consistently large spans for both types of FPR gap. On the other hand, Llama-3-8B, the most recent model studied with fine-tuning, has the smallest average gap spans in both categories. Another recent model, Mistral-7B, demonstrates a small average Negative-Sontiment FPR gap span, suggesting that more recent LLMs may be slightly less affected by issues with the template- based probes. It is interesting to note that the distribution of spans for Positive-Sentiment FPR gaps are more uniformly distributed between models than the Negative-Sentiment counterpart.
Table 5: Statistics for the three template-based probing datasets, Amazon, Regard, and NS-Prompls. The label counts and distributions are reported for each racial group across datasets. The label distributions are constant between groups, and thus reported below the label totals. Note that examples from the NS-Prompts dataset have neutral label by construction.
Label Counts
Dalasel
Race
Negalive
Neutral
Positive
Total
Fraction
African American
200
200
200
600
0.182
American Indian
300
300
300
900
0.273
Asian
100
100
100
300
0.091
Híspanic
200
200
200
600
0.182
Amazon
Pacific Islander
200
200
200
600
0.182
White
100
100
100
300
0.091
Total
1100
1100
1100
3300
Distribution
0.333
0.333
0.333
African American
440
220
350
1010
0.182
American Indian
660
330
525
1515
0.273
Asian
220
110
175
505
0.091
Hispanic
440
220
350
1010
0.182
Regard
Pacific Islander
440
220
440
1010
0.182
White
220
110
175
505
0.091
Total
2420
1210
1925
5555
Distribution
0.437
0.218
0.347
African American
0
8880
0
8880
0.182
American Indian
0
13320
0
13320
0.273
Asian
0
4440
0
4440
0.091
Hispanic
0
8880
0
8880
0.182
NS-Prompts
Pacific Islander
0
8880
0
8880
0.182
White
0
4440
0
4440
0.091
Total
0
48840
0
48840
Distribution
0.0
1.0
0.0
E Template-Basee Based Dataset Statistics
In this appendix, some statisties and distribution information about the template-based probing datasets described in Section [3.1 are presented. Table 5 summarizes the label counts across datasets and broken down by racial groups. Note that the label distributions are constant between groups. For example, the label distribution of (0.333, 0.333, 0.333) holds for cach protected group as well as the aggregated label distribution. Further, it is important to recall that the models are not trained on these datasets and only perform classification inference to produce the results of Section 4 For the Amazon dataset, the labels are balanced over the entire dataset and within each individual group. The distribution for the Regard datasct is also fairly balanced with somewhat fewer neutral examples. Finally, discussed in Section 3.1.2 all examples for the NS-Prompts dataset have neutral label.
20

---

Published in Transactions on Machine Learning Research (01/2026)
F Supporting Results for Unmarked Groups in Other Sensitive Attributes
This work primarily focuses unexpected measurements with respect to the sensitive attribute of race. However, the conjecture that markedness causes, at least contributes to, the observed irregularities when using template-basee probes would, in theory, generalize beyond racial demographics. For example, discussed in Section 5.1 several previous studies have noted that gender-based markedness also prevalent in web data and influences model behaviors. In such settings, malc gender represents the unmarked default. Furthermore, linguistic markedness exlends to sexualily, with helerosexuality comprising the predominant unmarked demographic (Zerubavel 2018).
With this in mind, two additional experiments are conducted using the prompting approaches described in Section 3.4 and the Amazon datasel (Czarnowska et al. 2021b. The Amazon datasel includes (emplates for sensitive attributes beyond race, including sexuality and gender, with the same structure described in Section Prompt-based classification is applied to these templates, and the same FPR gaps are mcasured across various protected groups within the sensitive attributes.
3.1.1
OPT-6.78
0B SSTS
OPT-6.78
Liama-2-78
Liama 2-7B SSTs
Liama: 2 SE
Liama-2 2-7B CoT
Llama-3-88
Liama-3-88 SSTS
Llama-3-8B SE
Llama-3-88 CoT
Mistra-7B
Mietrai-7B SSTS
Mistral-7B SE
Mistral-7B CoT
Gemma-7B
Gemma-7B SST5
Gemma-78 SE
Gomma-78 CoT
Qwan-2.5-7B
Qwen-2.5-78 SST5
Qwen-2.5-77 SE
Qwen-2.5-1 CoT
0.15
Negative Sentiment FPR Gap
0.1
0.05
-0.05
-0.1
0.1
Positive Sentiment FPR Gap
-0.05
asexual
other
bisexual
homosexual
heterosexual
Figure 5: Negative- and Positive-Sentiment FPR gaps for protected group variations within the sensitive altribute of Sexuality as measured by the Amazon datasel. In the legend, model names without a suflix indicate zero-shot prompting. SST5 and SE indicate 9-shot prompts with examples drawn from the SST5 and Eval datasets, respectively.
The results shown in Figures and For sexuality, the traditionally unmarked group, heterosexual. reflects a similar FPR-gap pattern to the White group in the main results. That is, positive, and often statistically significant, Xegalive-Sentiment FPR gaps and negative Positive-Sentimenl FPR gaps are found. Moreover, these patterns correlate to traditionally underprivileged groups in homosexual and asexual orienta- tion for the respective FPR-gap types. When considcring gender, the templates associated with male gender also produce positive and negative gaps for Negative- and Positive-Sentimen FPR, respectively, though the frequency of statistical significance is slightly reduced.
21

---

Published in Transactions on Machine Learning Research (01/2026)
As noted in Section 5.1 additional experimentation is required to confirm that markedness is driving factor for the results presented in this paper. Nonetheless, the empirical results in this section reinforce the observation that template- based based probes produce imperfect measures of bias in LLMs and that these imperfections appear to affect measurements associated with traditionally unmarked groups, even beyond of the sensitive altribule of race.
OPT-6.7B 0P OPT66.B
Liama SSTs
7B SE
Liama Llama-3-88 8888
Mistral-7B
Mistrai-7B
Gemma-7B Gemma-78
Gomma CoT
Owen-2.5-7B 2won-2.5-78 Qwen-2.5-78 SE
Qwen CoT
Negative Sentiment FPR Gap
0.1
0.05
-0.05
-0.1
Positive Sentiment FPR Gap
0.05
0
-0.05
-0.1
many-genders
non-binary
trans
female
male
Figure Negative- and Denttive-Sentiment FPR gaps for protected group variations within the sensitive attribute of Gender mcasured by the Amazon datasct. In the legend, model names without suffix indicate zero-shot prompting. SST5 and SE indicate 9-shol prompts with examples drawn from the SST5 and Eval datasets, respectively.
22


Reviews:
[{"review_id": "MmwlODtICL", "reviewer": "Reviewer_3c4z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript examines social biases in language models' sentiment classification of templated sentences. Several experiments are done on different datasets across a range of language models, both generative and masked.", "claims_and_evidence_yn": "No", "claims_explanation": "- The bias metric itself is not grounded in literature, while there is almost 20 years of research on classification with protected/sensitive attributes [1, 2]. Even for language models, there are plenty of metrics to choose from [4].\n- The issues with templates have been well-studied by [3], albeit without the possible explanation of markedness and for BERT-style models. However, [3] tests this using different bias metrics and uses statistical tests and I see no reason their conclusions would not hold for generative LLMs.\n-The \"markedness\" explanation is not actually investigated and feels added on after the fact. This feeling stems from the lack of explanation of markedness until the discussion and the framing of the results in the conclusion. I would at least move the explanation of this central term forward, as this is mainly a CS journal and linguistic or sociological terms might be less well known. \n- There is a wide range of performance scores for different models. I am surprised by e.g. LLama 2 only scoring 0.55. Such a performance gap could be a cofounding factor for the presented results.\n- I am unsure of the treatment of the \"neutral\" category from the sentiment classifications. It seems to be lumped together with \"negative\", except for the \"negative sentiment FPR gap\". This starts to get complicated to wrap my head around and a proper 3-class metric might have been more suitable. \n\n[1] https://link.springer.com/content/pdf/10.1007/s10618-010-0190-x.pdf\n\n[2] https://proceedings.neurips.cc/paper/2016/file/9d2682367c3935defcb1f9e247a97c0d-Paper.pdf\n\n[3] https://aclanthology.org/anthology-files/pdf/naacl/2022.naacl-main.122.pdf\n\n[4] https://aclanthology.org/anthology-files/pdf/emnlp/2024.emnlp-main.1207.pdf", "audience_yn": "No", "audience_explanation": "As explainer earlier, I am unsure about the novelty of the findings. The markedness is interesting, but no evidence points to this hypothesis and it is only presented in Section 5.", "requested_changes": "**Required changes**\n- All of the studies models are from 2023 or earlier, with the only exception llama 3. While I do appreciate the range of models, I think some modern LLMs should also be included. For these models, just prompting experiments are fine, since this is a more realistic setting for them anyways. In any case, models like OPT are honestly quite bad and I am not sure wether their results could generalize.\n- The bias metric is very ad-hoc and not grounded in any literature. Since the entire paper hinges on this metric, some motivation would have been nice. Also, quite nitpicky, but the term \"gap\" for a metric that includes the $\\bar{X}$ seems counter-intuitive.\n- Rewrite the manuscript to focus properly on the \"markedness\", including some experiments that actually support this hypothesis.\n\n**Clarifications**\n- Please clarify if sentiment or regard is used for the regard dataset. Figure 3's caption and Section 3.1.3 seem contradictory.\n- Section 3.4 discusses reasoning traces, but no reasoning LLMs are used (cfr. Deepseek R1). Is this referring to the chain-of-thought?", "broader_impact_concerns": "While the study itself focuses on bias measurements, I see no issue with the current setup.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S0gHl58DXX", "reviewer": "Reviewer_pWzh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary:\n\nThe paper discusses one of the limitations of some of the existing methods for bias assessment, namely template-based metrics. Template-based metrics quantify bias based on the change some measurable quantities (*e.g.* toxicity, sentiment, etc.) with inputs referencing varying races, genders, religions, etc. For example, models that do sentence completions with a higher toxicity for one race, compared to other races, are deemed biased against this race. \n\nThe argument made by the authors is: Due to the absence of markers when the referenced person is White (*e.g.,* it is more common to say “We hired a new CEO named Andy” than to say “We hired a new White CEO named Andy”), using template-based metrics yields incorrect results when referencing White people.\n\n## Contributions:\n\n1. Highlighting that template-based bias metrics provide inaccurate measures. \n2. Conducting experiments on different models and datasets to support the their claim.\n\n## Strengths:\n\n1. The works is useful for the community as template-based metrics are widely used.\n2. The writing and presentation of ideas in the paper are good. \n3. The main message in the paper is timely and important for the community as template-based metrics are widely adopted.\n\n## Weaknesses:\n1. The experiments can be expanded to further solidify the authors' claims (see the Requested Changes section for more details).", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. Results on 3 datasets have emphasized the main argument by the authors by demonstrating that template-based metrics falsely show higher levels of bias against White people.", "audience_yn": "Yes", "audience_explanation": "Yes. Many individuals working on fairness would be interested in knowing the findings of this paper, as it questions a widely used set of metrics for bias measurement.", "requested_changes": "## Requested Changes:\n\n1. It is important to include the authors’ thoughts on possible fixes for template-based metrics.\n\n2. The study is only focused on race bias and White people specifically, although the same argument is applicable to other biases. \nFor example, it is more common to say “We need to hire a nurse” than to say “We need to hire a female nurse”. \n\n3. While I generally agree with the paper’s main claims regarding the limitations in the current template-based metrics, I believe that more testing is needed. For example, can we show that by pre-training a model from scratch where all races are explicitly stated along the with the names of referenced individuals, such a limitation in template-based metrics would disappear?\n\n4. Explaining how the regard being measured.", "broader_impact_concerns": "No concerns on ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EnQXN482Fa", "reviewer": "Reviewer_5Rm5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper critically examines template-based counterfactual bias evaluation in LLMs. The authors conduct experiments on three datasets (Amazon, NS-Prompts, and Regard), used as benchmarking templates, and assess a diverse set of model families and sizes, including RoBERTa, OPT, Llama-2/3, and Mistral. Across settings, the authors consistently observe that text templates containing the term \\textit{white} are disproportionately classified as negative. The authors argument/hypothesize that this pattern does not represent the model bias, but rather arises as an artifact of linguistic markedness, given that \\textit{white} typically functions as the unmarked demographic attribute in natural text. The study thereby underscores important limitations of template-based probes serving as tools for bias evaluation.\n\nStrengths:\n* Importance of the topic: Bias evaluation is a critical and highly relevant issue.\n* Importance of the experimental work: Assessing the reliability of bias evaluation methods is a fundamental aspect for bias evaluation in ML.\n* Experimental scope: The experiments span multiple model families and sizes, and benchmarking values are derived by using multiple complementary approaches, including both fine-tuning to a downstream sentiment analysis task and prompting.\n* Interesting aspects/assumptions: Connecting possible template-based bias evaluation flaws to the linguistic aspect of markedness.\n\nWeaknesses:\n* The explanation for the findings (markedness) is presented by the authors as an hypthesis and has not been empirically tested.\n* The datasets used for benchmarking could be described in greater detail, particularly with respect to their size (size of the benchmarking set per label) and label distribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The empirical findings (FPR gaps for templates containing the term 'white') are well supported across multiple settings (models, benchmarking data and evaluation scenarios). Nonetheless, the causal explanation regarding markedness is presented as a hypothesis rather than a validated mechanism, which is reasonable, since the authors explicitly acknowledge this limitation and clearly highlight this claim as an assumption.", "audience_yn": "Yes", "audience_explanation": "Bias evaluation is central to the ML community, and template-based probes are a common approach for testing models after training. Demonstrating potential weaknesses of such methods provides valuable insights for the audience.", "requested_changes": "Kindly address the weaknesses listed above.\n1) Statistical description of benchmarking datasets: \n* It would be helpful if the paper included a brief statistical description of the probing datasets (e.g., total number of samples, distribution across sentiment labels). This would make it easier for readers to interpret the reported FPR gaps and understand how dataset composition might influence the results.\n\n2) Evaluation or discussion of markedness in training data\n* If feasible, a simple analysis (e.g., string matching frequency) in pre-training and/or fine-tuning corpora could empirically support the findings.\n* If that is not feasible due to e.g. data availability, it may still be useful to explicitly acknowledge the reasons and clarify why markedness was not addressed or is considered out of scope, while emphasizing that it requires further empirical validation in future work.\n\nMinor (optional) suggestions for stylistic/editorial improvements:\n* [1 Introduction] Syntax: 'a wide range' is usually treated as singular ('is considered') rather than plural ('are considered').\n* [3.4 Prompting Experimental Setup] Figure positioning: Figure 1 seems to be mispositioned. It is appearing in the before going section of '3.4 Prompting Experimental Setup' while it is belonging to '4.1 Fine-Tuning Results'.\n* [3.1.3 Regard Dataset] Formatting: The way labels are displayed in the line 'Labels: Regard: Negative. Sentiment: Positive.' could be streamlined. Rephrasing or reformatting would avoid the repeated colon.\n* [4.1 Fine-Tuning Results] Formatting: The figures are very informative. It might make them even more effective if they were displayed at a larger size, for example using the full text width, so that they are easier to read.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)