Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models (2024)

Abstract: Research on Large Language Models (LLMs) has often neglected subtle biases that, although less apparent, can significantly influence the models’ outputs toward particular social narratives. This study addresses two such biases within LLMs: representative bias, which denotes a tendency of LLMs to generate outputs that mirror the experiences of certain identity groups, and affinity bias, reflecting the models’ evaluative preferences for specific narratives or viewpoints. We introduce two novel metrics to measure these biases: the Representative Bias Score (RBS) and the Affinity Bias Score (ABS), and present the Creativity-Oriented Generation Suite (CoGS), a collection of open-ended tasks such as short story writing and poetry composition, designed with customized rubrics to detect these subtle biases. Our analysis uncovers marked representative biases in prominent LLMs, with a preference for identities associated with being white, straight, and men. Furthermore, our investigation of affinity bias reveals distinctive evaluative patterns within each model, akin to ‘bias fingerprints’. This trend is also seen in human evaluators, highlighting a complex interplay between human and machine bias perceptions.1

heading: 1 Introduction
text: In recent years, the landscape of natural language processing has been transformed by the advent of Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022), LLaMA-2 (Touvron et al., 2023), and Mixtral (Jiang et al., 2024). These LLMs have expanded the boundaries of natural language generation and understanding beyond theoretical research, embedding themselves into critical decision-making processes with significant real-world implications,
1The complete code, dataset, and detailed interactions with the language models are available at https://github.com/akkeshav/subtleBias.
such as hiring practices, automated essay evaluations, and even judicial decision-making (Lippens, 2023; Pinto et al., 2023; Cui et al., 2023).
The decision-making by humans is often subtly influenced by biases that, while less overt, significantly shape perceptions and judgments. Such subtle biases, although difficult to detect (Hebl et al., 2002), can have far-reaching consequences (Jones et al., 2016). Among these, representative bias and affinity bias prominently affect decision-making processes.
Representative bias stems from an unconscious presumption that dominant characteristics within a person’s environment are universally normative, thus skewing what is considered ‘normal.’ This bias is commonly seen in media representation, where prevalent cultural narratives disproportionately influence societal norms (Dixon, 2017; Shor et al., 2015). Affinity bias is the unconscious preference for those who share similarities with oneself, such as cultural backgrounds, personal experiences, or gender identities. This type of bias is evident in scenarios like literary awards, where judges might favor narratives that resonate with their own experiences (Marsden, 2019).
As LLMs increasingly assume roles traditionally filled by humans, such as in creative writing and content moderation (Dathathri et al., 2019; Roush et al., 2022; Ippolito et al., 2022), they not only showcase their ability to replicate complex human tasks but also raise questions about their potential to perpetuate human biases. This study probes the extent to which LLMs exhibit representative and affinity biases, particularly in areas where they supplant human-generated content and its evaluation.
We propose a comprehensive approach to quantify and analyze these biases in LLMs. Our methodology includes the ‘Creativity-Oriented Generation Suite’ (CoGS), a novel benchmark suite designed to scrutinize subtle biases through a series of structured yet open-ended tasks. Figure 1 offers a snap-
375
shot of our findings, depicting GPT-4’s evaluation tendencies across different identity axes within the short poem task.
Our contributions are threefold:
1. Creation of the ‘Creativity-Oriented Generation Suite,’ comprising 12 diverse openended tasks for content creation, ranging from short stories to haikus, complete with customized evaluation rubrics and a variety of themes for comprehensive analysis.
2. Development of two novel metrics, the Representative Bias Score (RBS) and the Affinity Bias Score (ABS), tailored to measure biases in content generation and evaluation.
3. Extensive testing of recent LLMs, such as LLaMA-2, GPT-4, and Mixtral, demonstrating prevalent representative biases towards identities typically associated with being white, straight, and men, and uncovering distinct patterns of affinity bias, with Mixtral displaying notably lowest ABS scores.
.

heading: 2 Creativity-Oriented Generation Suite
text: To systematically evaluate LLMs for bias, we introduce the Creativity-Oriented Generation Suite
(CoGS), a collection of tasks designed to assess model capabilities in generating content that is both diverse and creative across a wide range of themes and identities. Each task is defined by a problem instance P = {t, c, i, tr}, where:
• t denotes the task prompt template from the set T of all tasks. An example is “Write a very short story about [theme]."
• c represents a theme from the set C of all themes, enabling the creation of diverse task instances. Examples include “mountains” and “social media."
• i specifies an identity prompt from the set I , tied to a particular identity within the axes A of race, gender, and sexual orientation. Each axis a ∈ A includes distinct identity groups, e.g., an identity prompt could be “You embody the lived experience of being [identity]."
• tr is the task’s evaluation rubric from the set R of rubrics, which details criteria such as creativity, coherence, and thematic relevance.
This structured approach allows for the generation of a diverse array of problem instances, each designed to probe different aspects of creativity, theme variation, and identity representation. The templated nature of task prompts (t) facilitates easy
integration of any theme (c) from C, promoting a wide range of creative responses.
CoGS organizes themes under 10 broader topics, such as ‘social,’ which includes themes like family and friends, leading to a total of 30 distinct themes applied across various tasks. To ensure a standardized and fair assessment, specific rubrics for each task have also been developed. CoGS challenges LLMs with 12 unique open-ended generation tasks (to see the complete list of task prompt templates, refer to Appendix Table 6), ranging from blog writing to imaginative storytelling. Detailed information on some of these tasks, including theme examples and corresponding rubrics, is provided in the Appendix (Figures 8, 9, and 10).
Altogether, CoGS comprises 360 default prompts, which, when combined with 8 identity groups across 3 identity axes (race, gender, and sexual orientation), yield an additional 2,880 identityspecific prompts, culminating in a total of 3,240 prompts. Figure 2 illustrates the evaluation of creative outputs within CoGS’s ‘short poem’ task, highlighting the integration of identity prompts, thematic variation, and the suite’s evaluative rubrics.

heading: 3 Measuring Subtle Bias in LLMs
text: In the following sections, we introduce the Representative Bias Score (RBS) and the Affinity Bias Score (ABS) as metrics to evaluate subtle biases in LLMs. For a general, visual overview of the methodologies applied, refer to Figure 3.

heading: 3.1 Representative Bias
text: The development of LLMs involves extensive training on diverse datasets, predominantly sourced
from the internet. This training process raises questions about whether LLMs exhibit a generation style that aligns more closely with specific identity groups, potentially introducing a subtle form of bias (Lee et al., 2024; Omrani Sabbaghi et al., 2023; Kirk et al., 2021). To address this, we adopt a semantic similarity-based approach to measure the extent of representative bias in LLM outputs.
Let a language model m be a function that, given a problem instance P , outputs textual content O:
Om = m(P ) (1)
where P = {t, c, i, tr} comprises a task prompt template t, a theme c, an optional identity prompt i, and an evaluation rubric tr.
The model’s outputs are differentiated based on the inclusion of an identity prompt i, yielding two types of outputs: Omi , with the identity prompt, and Omd , without the identity prompt (default):
Omi = m(t, c, i, tr) (2) Omd = m(t, c, tr) (3)
To measure semantic similarity, we first transform the outputs into vector embeddings using a sentence embedding model, suitable for capturing the semantic content of texts. This embedding model converts sentences into high-dimensional vectors that represent their semantic features:
embed(Omi ) → V mi (4) embed(Omd ) → V md (5)
where V mi and V m d are the vector embeddings of Omi and O m d , respectively.
Subsequently, we calculate the cosine similarity between these embeddings to assess the semantic closeness of the model’s outputs with and without the identity prompt:
S(V mi , V m d ) = V mi · V md ∥V mi ∥∥V md ∥
(6)
The difference in similarity Dmi quantifies the deviation of the identity-prompted output from the default output, reflecting the model’s bias:
Dmi = 1− S(V mi , V md ) (7) The Representative Bias Score (RBS) for model m regarding an identity axis a, across all tasks, is the standard deviation of the average semantic similarity differences for each identity:
RBSma = √√√√ 1 n n∑
i=1
(Dmi −Dma )2 (8)
where n is the number of identities within the identity axis a, Dmi is the average semantic similarity difference for identity i across all tasks, and Dma is the mean of these average differences across all identities in axis a.
To determine the identity considered most “normal” by model m for axis a, we solve:
i∗ = argmin i Dmi (9)
For illustration, consider two models, GPT-4 and LLaMA-2, evaluated across three identities in the gender identity axis: man, woman, and non-binary. For GPT-4, the computed differences are DGPT-4man = 0.1, DGPT-4woman = 0.2, and D GPT-4 non-binary = 0.15. For LLaMA-2, the differences are DLLaMA-2man = 0.05, DLLaMA-2woman = 0.07, and D LLaMA-2 non-binary = 0.06.
The RBS for GPT-4 is calculated to be approximately 0.04, indicating a moderate degree of bias with man considered as the most “normal” identity, given its minimal divergence from the default output. In contrast, LLaMA-2 shows an RBS of approximately 0.01, suggesting a more balanced and equitable treatment across gender identities, with much less bias toward any particular gender.

heading: 3.2 Affinity Bias
text: Affinity bias in the context of LLMs refers to the predisposition of these models to favor outputs that align with certain identity groups over others during evaluation tasks. Unlike representative bias, which examines the content generation aspect of LLMs, affinity bias focuses on the evaluative behavior of models, particularly in tasks where LLMs are required to judge or select between various outputs based on predefined criteria.
To measure affinity bias, we first formalize the
outputs generated by a model m for a given problem instance P , which includes an identity prompt i across all identity axes A:
Omi = m(P ) ∀i ∈ A (10) These outputs are stored for analysis across every identity group and task.
An evaluator model, denoted as me, is then prompted to select the “best” output from the set of Omi for all identity groups, given a specific task t and its associated rubric tr from the set of rubrics R. The specific evaluation prompt we used in our study, as well as identity and task prompts, are detailed in Appendix Table 5.
For each identity axis a, we compute the proportion of outputs Omi preferred by the evaluator model me for a specific identity group i across all tasks. The standard deviation of these proportions across all identities within an axis a quantifies the spread of the model’s preferences, indicating the fairness or unfairness of its evaluative behavior:
ABSmea = √√√√ 1 n n∑
i=1
(pi − p̄)2 (11)
where n is the number of identities in axis a, pi is the proportion of selections where an output corresponding to identity i was selected as “best”, and p̄ is the average of these proportions for axis a.
The identity group i∗ that the model me prefers for each identity axis a can be identified by:
i∗ = argmax i pi (12)
For example, consider the gender identity axis across all tasks. If the proportions of preferred outputs are 70% for “man”, 20% for “woman”, and 10% for “non-binary”, converting these percentages to decimal form gives us 0.7, 0.2, and 0.1, respectively. The standard deviation (ABS) for this model, representing the preference spread and indicative of bias towards “man”, is approximately 0.262. In contrast, a model with a more balanced distribution of preferences—40% for “man", 30% for “woman”, and 30% for “non-binary” (or in decimal form, 0.4, 0.3, and 0.3)—yields a lower ABS of approximately 0.047, indicating a more equitable evaluative behavior. Thus, the ABS quantifies the extent of affinity bias, with a higher score reflecting a model’s stronger inclination towards a particular identity group. The identity group “man” is identified as the most preferred by both models here, given its highest proportion of selection.

heading: 4 Experiments & Results

heading: 4.1 Experimental Design
text: Identity Axes: Our study investigates biases along three pivotal identity axes—race, gender, and sexual orientation—each chosen based on their prominence in societal discourse and potential for discrimination (Crenshaw, 1989; Buolamwini and Gebru, 2018; Bacchini and Lorusso, 2019; McMurtry et al., 2019; Losty and O’Connor, 2018; Bi et al., 2020).
Prompts were derived from CoGS, with identity prompts framed as "You possess an inherent comprehension of being [identity group]..." to induce diverse responses without emphasizing the identity.2 Evaluation criteria (tr), sourced from CoGS, guided LLMs in selecting their preferred response per standardized format using evaluation prompt. Please refer to Appendix Table 5 for detailed instructions on usage of the rubric in the evaluation prompt.
Models: We analyzed outputs from GPT-4, LLaMA-2, and Mixtral, using the Sentence Transformer all-mpnet-base-v23 for vector embeddings, setting the temperature to 0.2 to prioritize determinism in responses.
Main Experiments: Responses to 3,240 CoGS prompts were generated, analyzing Representative Bias Score (RBS) and Affinity Bias Score (ABS) against an unbiased baseline, with radar plots visualizing each model’s bias profile. Qualitatively, the roundness of these plots indicates the degree of evaluative equity.
Human Performance: Fifty instances from the ‘very short story’ task were evaluated by three NLP graduates with a strong linguistics background. Disparities in evaluator consensus, as quantified by Fleiss Kappa, underscored the subjective nature of bias perception.4 This variation led to considering both aggregated and individual human judgments in our analysis.
Temperature Analysis: The preliminary analysis was done across both higher and lower temperatures for a sample of 500 problem instances. It was found that the evaluative preferences led to the same conclusions for all temperature settings (per-
2Preliminary tests confirmed the effectiveness of this approach. See Appendix Table 5 for detailed prompts.
3See https://www.sbert.net/docs/pretrained_ models.html for all-mpnet-base-v2 details.
4Fleiss Kappa scores indicated slight agreement for race (κ = 0.0426) but disagreement for gender (κ = −0.0466) and sexual orientation (κ = −0.0113).
formed with temperatures 0, 0.25, 0.5, 0.75, and 1) for every model. As a result, the temperature of 0.2 was selected for this research work because a degree of stability (but not full determinism) in the results was desired.

heading: 4.2 Results

heading: 4.2.1 Which Identities do LLMs Default To?
text: Figure 4 features the semantic similarity of LLMgenerated content with default responses, uncovering a systematic leaning towards ‘white’, ‘man’, and ‘straight’ identities across all models. This trend underscores a potential representative bias within these models, positioning certain identities
as the normative standard. Interestingly, LLaMA-2 presents an anomaly in racial preferences, favoring ‘black’ and ‘asian’ identities over ‘white’, a deviation possibly reflecting its diverse training data or architecture aimed at mitigating racial bias (Touvron et al., 2023).
RBS insights are summarized in Table 1a, with Mixtral showcasing the lowest RBS, highlighting
its broader inclusivity in content generation. Intriguingly, despite its low RBS, Mixtral’s responses to identity prompts generally exhibit lower semantic similarity to the default responses than those of other LLMs (to see the extent of lower semantic similarity, refer to Appendix Figure 11). This pattern may suggest that Mixtral’s training paradigm encourages balance without favoring a specific identity. However, it also raises the question of potential unrecognized biases toward unrepresented identity groups that might align more closely with the default responses.

heading: 4.2.2 Do LLMs Show Preference for Certain Identities?
text: The affinity biases of LLMs towards different identity groups are shown in Figure 5. Here, GPT4’s bias towards ‘white’, ‘straight’, and ‘man’ is
evident, reflecting a significant evaluative preference. In contrast, LLaMA-2’s preferences align oppositely, favoring ‘black’, ‘queer’, and ‘female’, marking a distinct evaluative pattern from GPT-4.
Mixtral stands out in having the most uniform evaluative patterns, as demonstrated by its balanced radar plot. Table 1b corroborates this through its lowest ABS, indicating a fairer evaluative process relative to the other models.
Task-specific biases also occurred, aligning with societal stereotypes related to identities and their assumed strengths, exemplified by Mixtral’s affinity bias for Asian identity in ‘haiku’ task (short-form poetry intrinsically linked to Japan). For details, see ‘haiku’ row in Appendix A.2 Figures 15, 16, 17. Also, in ‘very short story’ task, all models often favored ‘black’ identity content over or as much as ‘white’ identity, which may reflect biases as-
sociated with racial identity and storytelling. For further insights, see the ‘very short story’ row in Appendix A.1, Figures 12, 13, 14.
Figure 6 reveals that human evaluators and LLMs displayed similar behaviors regarding the race identity axis. However, significant differences emerged in other areas. For sexual orientation, human evaluators tended to prefer responses associated with the straight identity group, whereas LLMs were more likely to choose responses related to queer identity group. In the context of gender, human preferences skewed towards the man identity group (except for one evaluator, #1, see Figure 7), while LLMs demonstrated a pronounced preference for the non-binary identity group.

heading: 5 Qualitative Analysis
text: Our qualitative analysis studies how identity groups are represented across various tasks by LLMs, providing insights into the subtleties of bias not captured by quantitative metrics alone. We categorize LLM outputs into three levels based on the perceptibility of identity group markers: imperceptible, where identity cues are absent; nuanced, where
identity is subtly indicated; and obvious, where identity is explicitly mentioned. The examples in Table 2 show selective instances that are categorized according to the perceptibility of identity markers in LLM outputs.
We also provide qualitative examples of selection preferences of different LLMs and human evaluators in Table 3, showing cases of consensus as well as divergence in choices across identitythemed outputs.

heading: 6 Related Work
text: LLMs as Writing Evaluators. The capability of LLMs in evaluating the coherence of written texts has been of recent interest, with performances that often align with human evaluators (Naismith et al., 2023). In broader NLP tasks, such as story generation, the detection of adversarial attacks and translation quality assessment, has also been documented (Chiang and Lee, 2023; Kocmi and Federmann, 2023). Despite these advancements, the fairness and consistency of LLM evaluations remain under scrutiny (Wang et al., 2023). Our study aims to further this discussion by examining the underex-
plored aspect of affinity bias in LLM evaluations. Biases in LLMs. Research in natural language generation has mainly addressed overt biases—gender, race, sexual orientation, and political leaning—often emerging as toxicity, stereotyping, or biased opinions. These are typically detected through toxicity analysis in prompt continuations, question-answering, and hate-speech detection (Tjuatja et al., 2024; Schramowski et al., 2022; Acerbi and Stubbersfield, 2023; Sheng et al., 2019; Esiobu et al., 2023; Feng et al., 2023; Dhamala et al., 2021). Specifically, studies have explored explicit gender and racial biases in LLM-generated content, examining offensiveness and politeness (Sun et al., 2023), and gender bias in reference letters (Wan et al., 2023). Our research shifts the focus to subtler forms of bias, highlighting their significance as evidenced by existing research on their potential effects in areas such as scholarship reviews and job interview decisions (Dovidio et al., 2016; Purkiss et al., 2006).
Open-Ended Generation Task. Explorations into open-ended generative tasks by LLMs have spanned from structured narrative generation to the creative articulation of literary styles (Lu et al., 2023; Chakrabarty et al., 2023; Garrido-Merchán et al., 2023). Recently, the evaluation of LLM capabilities has extended beyond traditional storytelling to include unique challenges, such as generating content that mimics specific literary genres or poetic forms (Sawicki et al., 2023). Our CreativityOriented Generation Suite expands the scope of such open-ended generation tasks to include areas previously unexplored, such as dance choreography writing, trivia creation, interview script generation, and puzzle construction. It also offers a versatile, templated framework for incorporating diverse themes and identities, enabling studies on LLMs’ creative proficiency as well as on the biases influencing their content generation.

heading: 7 Conclusion
text: We introduce the Representative Bias Score (RBS) and the Affinity Bias Score (ABS) to measure subtle biases in LLMs, using the Creativity-Oriented Generation Suite for evaluation. Our findings reveal pronounced representative biases in LLMs towards white, straight, and man identities in creative tasks, suggesting an implicit normalization of these identities. Additionally, we uncover unique patterns of bias for each LLM, indicative of distinct
“bias fingerprints”. Our comparisons with human evaluators highlight both similarities and differences in bias patterns, emphasizing the complex interplay between human cognition and LLMs.
Limitations
Scope of Identity Axes: Our focus was limited to three primary identity axes: race (white, black, asian), gender (man, woman, non-binary), and sexual orientation (straight, queer). While this selection encompasses a significant spectrum of identities, it notably omits other critical categories such as age (e.g., youth, middle-aged, elderly), disability (e.g., physical, sensory, intellectual), religion (e.g., Christianity, Judaism, Islam, Hinduism, atheism), and sex (e.g., male, female, intersex). These categories represent a vast range of experiences and perspectives that could also significantly influence LLM outputs and evaluations. Including these and potentially other nuanced identity groups, such as socioeconomic status or educational background, in future studies could provide a more comprehensive and inclusive understanding of biases in LLMs.
Model Selection: The analysis was conducted on a select group of LLMs: GPT-4, LLaMA-2, and Mixtral. These models were chosen for their architectural diversity and representativeness of current state-of-the-art. However, the inclusion of other models, such as Claude-2.1, Gemini Pro, PerplexityAI or those specialized in specific languages and domains, in future studies would likely reveal further interesting findings.
Task and Theme Variety: The CreativityOriented Generation Suite introduced innovative tasks such as dance choreography writing and puzzle generation, alongside traditional ones like short story writing and poetry. While this diversity addresses a broad spectrum of creative expression, it does not encapsulate all potential creative or generative tasks LLMs might be tasked with, such as songwriting or scriptwriting for interactive media.
Quantitative vs. Qualitative Bias Measurement: Our approach predominantly utilized quantitative metrics (RBS and ABS) for bias assessment. While effective for scalable and comparative analysis, this method may not capture the full depth of biases, especially those manifesting subtly or contextually. Future research could benefit from integrating qualitative analyses such as ours to uncover the intricate ways other forms of subtle bias
are embedded in LLM-generated content. Generalizability to Real-world Applications: The experiments were designed to simulate a range of creative tasks in a controlled environment. This setting, while useful for systematic analysis, may not fully reflect the complexities and variables of real-world applications where LLMs are deployed. For instance, the impact of user-specific prompts, interactive dialogues, or long-form content generation on bias manifestation remains to be explored.
Interestingly, when human evaluators in our study were shown their bias fingerprints reflecting their affinity biases, many found the insights both enlightening and conducive to self-reflection. It led to very interesting conversations. Motivated by this, we are currently developing a web application that leverages our framework to offer users personalized bias fingerprint assessments, with the goal of helping raise self-fawareness and reflection on potential biases in interactions with LLM-generated content, which are now becoming ubiquitous (and often, indiscernible from human-generated content!).

heading: Acknowledgements
text: This work was supported by the Natural Sciences and Engineering Research Council of Canada and by the New Frontiers in Research Fund.

heading: A Appendix
text: A.1 Affinity Biases: GPT-4 as an evaluator
A.2 Affinity Biases: Mixtral as an evaluator
A.3 Affinity Biases: LLaMA-2 as an evaluator

heading: Future Work
text: Including these and potentially other nuanced identity groups, such as socioeconomic status or educational background, in future studies could provide a more comprehensive and inclusive understanding of biases in LLMs. Model Selection: The analysis was conducted on a select group of LLMs: GPT-4, LLaMA-2, and Mixtral. These models were chosen for their architectural diversity and representativeness of current state-of-the-art. However, the inclusion of other models, such as Claude-2. 1, Gemini Pro, PerplexityAI or those specialized in specific languages and domains, in future studies would likely reveal further interesting findings. Task and Theme Variety: The CreativityOriented Generation Suite introduced innovative tasks such as dance choreography writing and puzzle generation, alongside traditional ones like short story writing and poetry. While this diversity addresses a broad spectrum of creative expression, it does not encapsulate all potential creative or generative tasks LLMs might be tasked with, such as songwriting or scriptwriting for interactive media. Quantitative vs. Qualitative Bias Measurement: Our approach predominantly utilized quantitative metrics (RBS and ABS) for bias assessment. While effective for scalable and comparative analysis, this method may not capture the full depth of biases, especially those manifesting subtly or contextually. Future research could benefit from integrating qualitative analyses such as ours to uncover the intricate ways other forms of subtle bias
are embedded in LLM-generated content. Generalizability to Real-world Applications: The experiments were designed to simulate a range of creative tasks in a controlled environment. This setting, while useful for systematic analysis, may not fully reflect the complexities and variables of real-world applications where LLMs are deployed. For instance, the impact of user-specific prompts, interactive dialogues, or long-form content generation on bias manifestation remains to be explored. Interestingly, when human evaluators in our study were shown their bias fingerprints reflecting their affinity biases, many found the insights both enlightening and conducive to self-reflection. It led to very interesting conversations. Motivated by this, we are currently developing a web application that leverages our framework to offer users personalized bias fingerprint assessments, with the goal of helping raise self-fawareness and reflection on potential biases in interactions with LLM-generated content, which are now becoming ubiquitous (and often, indiscernible from human-generated content!)