Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models (2024)

Abstract: LLMs are increasingly powerful and widely used to assist users in a variety of tasks. This use risks introducing LLM biases into consequential decisions such as job hiring, human performance evaluation, and criminal sentencing. Bias in NLP systems along the lines of gender and ethnicity has been widely studied, especially for specific stereotypes (e.g., Asians are good at math). In this paper, we investigate bias along less-studied but still consequential, dimensions, such as age and beauty, measuring subtler correlated decisions that LLMs make between social groups and unrelated positive and negative attributes. Although these subtler biases are understudied they follow people as much as gender and ethnicity do. So, we want to see whether they also follow one with LLMs. We introduce a template-generated dataset of sentence completion tasks that asks the model to select the most appropriate attribute to complete an evaluative statement about a person described as a member of a specific social group. We also reverse the completion task to select the social group based on an attribute. We report the correlations that we find for 4 cuttingedge LLMs. This dataset can be used as a benchmark to evaluate progress in more generalized biases and the templating technique can be used to expand the benchmark with minimal additional human annotation.1

heading: 1 Introduction
text: Alongside the impressive new capabilities of recent language generation models such as ChatGPT (Brown et al., 2020), GPT-4 (OpenAI, 2023), and Llama-2 (Touvron et al., 2023), these systems are increasingly involved in consequential decisions made in the real world. This includes job hiring and performance reviews, with tips for hiring
1Our dataset and code is available at https://github.com/kamruzzaman15/ Identifying-Subtler-Biases-in-LLMs.
managers appearing across the internet. Even before these recent advancements, AI has been used in the criminal justice system leading to the amplification of social inequities (Moy, 2019). In order to manage these biases prior research has investigated the most salient dimensions of bias in word embeddings and LLMs, such as gender and ethnicity (Bolukbasi et al., 2016; Caliskan et al., 2017; Kurita et al., 2019). Prior work also focuses particularly on whether these AI systems produce specific stereotypes of underrepresented minorities, such as associating Middle Eastern people with perfumes or Jewish people with greed (Nangia et al., 2020; Nadeem et al., 2021).
Social scientists found that human biases extend beyond simple stereotypes and can lead to general associations of positive attributes to members holding (or perceived to hold) certain key characteristics. For example, Dion et al. (1972) found that people are more likely to infer a plethora of other desirable characteristics to people that are judged more attractive—a result that has been confirmed and elaborated upon for the present day by more recent studies (Commisso and Finkelstein, 2012;
8940
Peng et al., 2019; Maurer-Fazio and Lei, 2015; Weber et al., 2013).
In this paper, we extend the evaluation of bias in LLMs in the following key ways: we investigate whether LLMs make general associations between stereotyped categories and unrelated positive, negative, and neutral attributes rather than specific stereotype inferences. In addition, we investigate dimensions of bias that have been largely overlooked: age, beauty, academic institution, and nationality. Although understudied, these dimensions of bias follow people as much as gender and ethnicity do. Furthermore, the few existing studies regarding these biases in LLMs all study them in terms of specific stereotypes. Figure 1 shows an example of how we formulate the completion task for LLMs in the bias domain of academic institutions.
The contributions of this paper are the following.
1. We formulate a task to investigate biases along generalized positive-negative sentiment rather than specific stereotypes and present a procedure for semi-automatically collecting a large dataset for this task.
2. We investigate both directions of biased association: generating unrelated attributes given a bias-triggering description and generating biased triggering descriptions from unrelated attributes.
3. We find that current LLMs show a pattern of bias in the domains we considered save for a few specific model-domain combinations.

heading: 2 Motivating Studies from Social Science
text: Many studies demonstrate the pervasive impact of social biases in various spheres of life. Dion et al. (1972) found that people who are considered attractive are more likely to be believed to hold socially desirable traits and higher occupational status. This attractiveness bias also occurs inversely, as shown by Gross and Crofton (1977), where known desirable qualities influence our perceptions of an individual’s beauty. These two papers inspired us to set up our experimental design. More recent psychological research has confirm using modern scientific standards that such biases extend in specific real-world settings like employee termination (Commisso and Finkelstein, 2012), hiring practices (Peng et al., 2019), interview callbacks (Maurer-Fazio and Lei, 2015), and even in victim blaming scenarios (Weber et al., 2013).
An opposite bias effect for beauty exists for people in particular contexts, in which a highly attractive person is dispreferred (Agthe et al., 2010). These effects are subject to interactions with sexual motive, social competition, culture, and the nature of the interaction making them more difficult to study (Wan and Wyer, 2015; Li et al., 2020). Generally, this opposing beauty bias effect appears when a preference for beauty would introduce a social threat (e.g., hiring a same-sex person, an embarrassing purchase from an opposite-sex person, etc.). In this paper we focus on the most general beauty bias where beauty is unduly associated with positive attributes leaving the opposite bias effect for future work.
Age-related stereotypes similarly impact societal perceptions and actions, with studies by Perdue and Gurtman (1990), Marques et al. (2020), and Donizzetti (2019) indicating a general propensity to infer negative traits based on age. This aligns with the foundational work on ageism by Butler (1969) and has been found to have implications in professional settings (Ng and Feldman, 2012).
Organizations and institutions have been found to be associated with specific personality traits, both within the US (Slaughter et al., 2004) and internationally (Anderson et al., 2010). Rutter et al. (2017) found that universities leverage specific personality traits when branding and Humburg (2017) student personality traits play an important role in student choice of university and alternatives (e.g., vocational education). Such bidirectional effects of personality traits result in hiring biases that prioritize students from academic institutions with particular reputations (Morley and Aynsley, 2007; Mavridopoulou and O’Mahoney, 2020).
Nationality bias has found to affect student interactions in multicultural online learning environments (Morales-Martinez et al., 2020), consumer perception towards products (Insch and McBride, 2004), academic philosophy (Seabra et al., 2023), and peer evaluation (Tavoletti et al., 2022). Tavoletti et al. (2022) identified the economic development of a person’s country of origin as an important factor within this bias, overshadowing individual qualities when one evaluates their peers.
As LLMs are trained on data created by humans, we hypothesize that they are prone to similar biases to those identified in people. If present, we must measure the degree to which such biases are present so that we can appropriately account for them when LLMs are used for consequential de-
cisions, such as recruiting and hiring. In order to do this, we measure our bias in terms of “representational harms” (using the terminology from Blodgett et al. (2020)), that is, an LLM is biased if it makes general associations between stereotyped categories and unrelated positive, negative, and neutral attributes. Following the guidance of Blodgett et al. (2020), our objective is to define and measure these biases, particularly how they manifest and potentially perpetuate existing social hierarchies.

heading: 3 Related Work
text: Moving on to related work in NLP specifically, bias in models have been studied for word embeddings using cosine similarity (e.g., the Word Embedding Association Test) and sentence embeddings using templates such as “This will [target]” (e.g., the Sentence Encoder Association Test.) (Bolukbasi et al., 2016; Caliskan et al., 2017; May et al., 2019). Nangia et al. (2020) created Crowdsourced Stereotype Pairs (CrowS-Pairs), a dataset that studied nine different types of social biases (e.g., age, nationality, physical appearance, etc.) in masked language models. This work is the closest to our own in its study of ageism, beauty, and nationality and the use of intrasentence biases. However, it differs in both the model type (autoregressive vs. masked) and in the generality of the associations that are studied. Nadeem et al. (2020) introduced a dataset called StereoSet to measure stereotypical bias in both masked and autoregressive pretrained language models. Nationality bias has been studied before in GPT-2 by Venkit et al. (2023). They generated stories and analyzed how other factors (e.g., number of internet users, economic condition, etc.) affect nationality bias.
Czarnowska et al. (2021) explored seven social biases in three RoBERTa-based models, including age, gender, and nationality. They used GDP to categorize countries for nationality bias. Sun et al.’s (2022) systematic study on pretrained language model-based evaluation metrics revealed that they exhibit greater bias than traditional metrics on attributes such race and gender. Zhang et al. (2021) designed cloze-style example sentences to explore performance differences across demographic groups (age, race, etc.). Smith et al. (2022) created HOLISTICBIAS, a dataset for measuring bias across various demographic identities in language models.

heading: 4 Task Definition
text: Unlike previous studies which have focused on identifying bias in a single direction, we take a more general measurement by considering two directions of bias. For example, Nadeem et al. (2021) measured stereotypes of LLMs by giving the race description (e.g., Hispanic, Ghanaian etc.) or a profession (e.g., physicist, tailor, etc.) and asked LLMs to choose between stereotypicallty associated attributes (e.g., poor, creative, etc.), but they did not measure the bias in reverse context (e.g., by giving attributes and asking LLMs to choose the associated race or profession). In our study, we study both directions, inspired by Dion et al.’s (1972) and Gross and Crofton’s (1977) work showing that people demonstrate beauty bias in both directions. In our experiment, we use fill-in-the-blank style sentences which evoke the biases in the provided text using a description of a person based on the bias category we are studying. We will refer to this description as simply the stimulus. For example, in Figure 1 (a), the stimulus is “Hinds Community College” for institutional bias.
Stimulus to Attribute Inference (SAI): We provide a stimulus and ask the LLM to infer a related attribute. The LLM must choose between a set of three attributes: positive, negative, and neutral. In Figure 1 (a), the stimulus is “Hinds Community College” and positive, negative and neutral attributes are “innovative”,“unimaginative”, and “chummy”, respectively.
Attribute to Stimulus Association (ASA): We provide attribute and ask the LLM to choose a specific stimulus. The LLM must choose between a set of three stimuli: positive, negative, and neutral. In Figure 1 (b), the attribute is “energetic” and the positive, negative and neutral stimuli are “University of california, san diego”, “Ivy tech community college”, and “Wichita” respectively.

heading: 5 Dataset Creation
text: We consider four domains of bias in creating our dataset: age, beauty, academic institutions, and nations. We refer to academic institutions as institutions throughout our writing.2
2We focused on educational institutions, where rankings and quality classifications are readily available, but institution quality lacks a uniform metric across other types.

heading: 5.1 Dataset Statistics
text: Our dataset contains 11,940 test instances: 2,154 for ageism bias (SAI: 858, ASA: 1296), 3,684 for beauty bias (SAI: 1,938, ASA: 1,746), 3,600 for institutional bias (SAI: 1,950, ASA: 1,650), and 2,502 for nationality bias (SAI: 1,710, ASA: 792).3 We further divide the beauty bias into two parts in our analysis. One part is called beauty bias (excluding professions) and another part is called beauty profession measuring the interaction between beauty and professions specifically. There are 2,016 items for beauty (non-profession) bias (SAI: 1,026, ASA: 990) and 1,668 for beauty profession (SAI: 912, ASA: 756). For the exact list of stimuli and attributes, please see the supplementary materials.
We collected positive and negative attributes from Anderson (1968), Perdue and Gurtman (1990), and Cross et al. (2017). Neutral attributes were mostly sourced from primary personality lists.4,5 The remaining neutral attributes were manually curated by the authors.

heading: 5.2 SAI Data Creation
text: In this case, we measure the LLM selection of attributes in response to each stimulus. We divide stimuli into two groups (positive and negative) and attributes into three groups (positive, neg-
3Our original institutional category has 32,808 instances, and we sample 3,600 from that to keep a similar ratio to other categories.
4https://ideonomy.mit.edu/essays/traits.html 5https://liveboldandbloom.com/11/
personality-types/neutral-personality-traits
ative, and neutral). For a basic overview of attributes and stimuli in SAI direction, see Table 32 in Appendix F. For all bias categories, we use personality traits as attributes and divide them into three parts, namely positive traits (e.g., creative, adaptable, etc.), negative traits (e.g., unimaginative, rigid, etc.), and neutral traits (e.g., unpredictable, playful, etc.).
Age: We divide the stimulus ages into young (25- 35) and old (60-70). For the sake of our writing (not the actual representation) we call the young and old stimuli as positive and negative stimuli, respectively. We select these age ranges based on the experimental results from Cameron (1969) while pushing all age groups more towards middle age to make them relevant in the work setting.6
Beauty: Beauty stimuli are divided into positive (e.g., attractive, gorgeous, etc.), and negative (e.g., unattractive, plain, etc.).7 Only for beauty stimuli, we consider different professions as attributes (e.g., astronomer, security guard, etc.) in addition to personality traits, following the study of Dion et al. (1972). For the sake of our writing (not the actual representation) we consider the high, mid, and lowsalaried professions (e.g., high: surgeon, mid: tax examiner, low: security guard) as positive, neutral, and negative professions. We categorized profes-
6Many of our template sentences for ageism assume a work setting.
7We selected beauty words as ’Positive’ or ’Negative’ beauty terms based on their synonymy or antonymy to ’beautiful’, considering only their grammatical fit and context in our template sentences, thereby encapsulating the subjectivity of beauty within these terms.
sions based on income, drawing inspiration from Wong and Penner (2016), and using data from the U.S. Bureau of Labor Statistics.8 We consider annual mean wage of more than 100k as high-salaried professions, 50k-70k as mid-salaried professions, and less than 40k as low-salaried professions.
Institution: Stimulus institutions are divided into university (e.g., MIT, Harvard University, etc.) and community college (e.g., Houston Community College, Miami Dade College, etc.) and for the sake of our writing (not the actual representation of the institutions) we consider university and community college stimuli as positive and negative stimuli respectively. In this study, we select the top 100 best national universities and the top 100 community colleges based on enrollment in the USA according to U.S. News.9,10
We ensure each state is represented with at least one university and one community college by including the highest-ranked university or the community college with the highest enrollment from each state. We maintain balanced lists by truncating the resulting lists to 100.11 For more attributes and stimuli for institutions, see Table 34 in Appendix F.
Nation: Stimulus nations are divided into rich (Luxembourg, Norway, etc.) and poor (South Sudan, Gambia, etc.) in terms of GDP per capita12,13 and for the sake of our writing (not the actual representation) we consider rich and poor countries as positive and negative stimuli, respectively. We follow Tavoletti et al. (2022) in using economic conditions to categorize countries. We select the 15 countries with the highest and lowest (with available data) GDP per capita as positive and negative stimuli, respectively.

heading: 5.3 ASA Data Creation
text: In this case, we measure the LLM selection of stimulus when provided with attributes. Here, we
8https://www.bls.gov/oes/current/oes_nat.htm# 00-0000
9https://www.usnews.com/best-colleges/ rankings/national-universities
10https://www.usnews.com/education/ community-colleges/search?sort=enrollment& sortdir=desc
11We limit our focus to U.S. institutions to manage economic, national, and geographical biases, sidestep the lack of a uniform international equivalent to community colleges, and deal with the scarcity of global data on educational rankings and lists.
12We use GDP per capita values as reported by the International Monetary Fund (IMF) (as of August 2023).
13https://www.imf.org/external/datamapper/ NGDPDPC@WEO/OEMDC/ADVEC/WEOWORLD
divide the attributes into two parts, removing neutral attributes. On the other hand, we divide each set of stimuli into three parts by adding a neutral or relatively unrelated set of stimuli for the third group. For more details about ASA data creation, see Appendix A.

heading: 5.4 Iterative Data Collection
text: We create our dataset by iterating over the lists of stimuli and attributes. In SAI direction, we use every stimulus term with each sentence template. For example, consider the beauty bias sentence “He looks unattractive; it is obvious that he is (wise/stupid/freewheeling)” in Figure 2. We replace “unattractive” with every other positive and negative stimulus term (e.g., attractive, fashionable, unfashionable, etc.) from beauty bias list. When we select attributes in the SAI direction, we randomly pick one triple of positive, negative, and neutral attributes (e.g., friendly, unfriendly, and carefree). In ASA direction, we swap the stimuli and attributes. We use every term from the attributes list with each sentence template and randomly select one triple of positive, negative, and neutral stimuli.14
To avoid potential confounding effects of gender bias, which has been found in prior NLP systems (Bolukbasi et al., 2016), we exhaustively (and uniformly) include the three most common sets of pronouns: masculine (he/him), feminine (she/her), and non-binary (they/them). For example, in the sentence “Because he was (ambitious/unambitious/freewheeling), he ended up at Community College of Vermont, where he was a second-year student.”, we replace “he” with “she” and “they” to form new templates. We also use this to analyze the effects of gender bias in our dataset (Section 7.1).15 Similarly for institutional bias, we control for education level. We use “firstyear”, “second-year”, and “teacher” descriptors to force an education level. As community colleges offer two-year programs, we include only first-year and second-year students.
14We have adjectival and nominal variations of attributes to ensure grammaticality. For example, in “It is clear that the man, who comes from South Sudan, is friendly.”, we use the adjectival form “friendly”. In “He is known for his friendliness; he is from South Sudan.”, we use the nominal form “friendliness”.
15We generalize the gendered pronouns to gendered person descriptions as needed, i.e., “man” or “boy” for masculine, “woman” or “girl” for feminine, and “people” for non-binary.

heading: 6 Experimental Setup
text: We use GPT-4, PaLM-2, Llama-2-13B, and Mistral7B in our experiments.16,17 For model details, see Appendix B. Exact experimental details are in our supplementary materials, including scripts for replication. We examine how LLMs respond to positive and negative attributes and stimuli. Specifically, we calculate the conditional likelihood of a model selecting positive, negative, and neutral attributes in response to stereotypically positive and negative stimuli. We will refer to these as [stimulus]to-[attribute] likelihoods. For example, we call the likelihood of the model to select positive attributes (e.g., friendly, motivated, creative, etc.) in response to stereotypically negative stimuli (e.g., unattractive, Hinds Community College, South Sudan, 65 years old, etc.) the negative-to-positive likelihood (NPL). Our shorthand uses P for positive, N for negative, and Nu for neutral. We consider a system to be biased if the conditional likelihood of positive, negative, or neutral completions change when the polarity of the prompting text changes. That is, an unbiased system will have ∆PL, ∆NL, and ∆NuL values (defined in Figure 3) of 0. For example, in Figure 3, we find that ∆NL is systematically negative, so the negative predictions of the models are biased in the opposite direction of the
16We use 4-bit quantized versions for Llama-2 and Mistral 7B due to our resource constraints.
17We also have experimental results for GPT-3.5, which follow the same broad trends as the other models discussed here. Results for GPT-3.5 are only in the appendices due to space constraints.
prompt text changes. We further report correlations and statistical significance using Kendall’s τ test (Kendall, 1938).18 In the SAI direction, we calculate Kendall’s τ statistic between the binary positive and negative stimulus variable and the ternary positive, negative, and neutral attribute variable.
We reverse everything in the ASA direction, e.g., calculating the likelihood of selecting positive, negative, and neutral stimuli in response to positive and negative attributes.

heading: 7 Results and Discussion
text: Figure 3 shows the high-level trends in model predictions for each of the categories in both directions for GPT-4 and Mistral. The prediction rates are summarized in terms of the change of rates when
18We selected the Kendall’s τ test instead of the χ2 test because there is a natural order to negative, neutral, and positive categorical values.
moving from positive to negative independent variable values. If the stimuli and attributes are independent of each other in a model, we should find the plots to be close to 0 with minor random variations. We instead see the trend that positive generations are more frequent when the given provided unrelated information is positive and vice-versa for negative generations. The effect on neutral generations is small and the direction of the effect does not follow any obvious pattern. Only one modelcategory combination (Mistral Ageism in the ASA direction) breaks this pattern—the change in the dependent variable does not follow the same direction as the change in the independent variable.
This correlation between the stimuli and attributes for LLMs is statistically significant. Table 1 shows the results of the Kendall’s τ test for each model and in each direction. The null hypothesis is rejected in all eight settings. This serves as a clear indication of a pattern of bias in modern LLMs.
We next break down the results by bias category, where the τ -test results are presented in Table 2. Here we focus on the broad trends of our results. Complete results and additional discussions are available in Appendix C.
Ageism. Mistral is the only model where we fail to reject the null hypothesis in either SAI or ASA settings. The effect size for Llama-2 is considerably smaller than for GPT-4 and PaLM-2, suggesting that unknown engineering decisions made for proprietary models exacerbate age-related bias in LLMs or that quantization of LLMs suppress age-related bias.
Beauty. Beauty-bias results are statistically significant for all models in both SAI and ASA directions and the effect sizes are among the largest
across the board. This confirms the patterns we see in Figure 3 and points to a sorely overlooked bias in LLM development.
Beauty-Profession. Here we again see statistically significant results in every model-setting combination. The effect sizes here are smaller than for the beauty bias setting. While LLM generations correlate beauty terms with high-income professions, this bias is not as severe as that for positive character traits.
Institution. The institutional bias results are again significant across the board. GPT-4 in the SAI direction and Llama-2 in the ASA direction stand out as having particularly large effect sizes.19
19That is GPT-4 is very likely to predict positive character traits for people associated with high-ranking institutions while Llama-2 is very likely to predict a high-ranking institution for people with positive character traits.
Nationality. In the SAI direction, we fail to reject the null hypothesis for GPT-4 and Llama-2. For PaLM-2 and Mistral, we see relatively small effect sizes. In the ASA direction, however, we see statistically significant results for all models. This suggests that the results we see in the SAI direction are reflective of the prior work in bias mitigation in the area of race, ethnicity, and nationality. This work does not however carry over in the ASA direction. That is, LLMs have strongly biased predictions of a person’s nationality in response to given positive or negative character traits.

heading: 7.1 Addressing Possible Confounds
text: Education Level. A key possible confounding variable in our investigation of institutional bias is educational level. We see this in Figure 4 which shows the percentage of selecting positive, negative, and neutral attributes/stimuli averaged across all models for different educational levels. In the SAI direction, models select considerably more positive and fewer negative attributes for teachers compared to first and second-year students. So, there is a trend toward positive representation for teacher. Table 5 shows that the correlation between education level and attribute quality (SAI direction) is statistically significant for all 4 models. This pattern does not generally hold in the ASA direction, and in fact, there are small effect negative correlations for GPT 4 and Mistral predictions. We would
not expect the education level to be predictive of the type of institution since they all have first-year students, second-year students, and teachers.
Table 3 shows the τ -test results for institutional bias while controlling for educational level. We maintain statistical significance in every case except Mistral in the SAI direction. The institutional bias we see from Mistral in the SAI direction (in Table 2) can be explained by the underlying correlation between education level and institution type.
As most of the institutions contain distinct words “University" and “Community College", one of the reviewers wonders if those two words dominate the selection of attributes, instead of the actual university and community college. To address this we perform a follow-up experiment where we analyzed the results for positive institutions that do not include the word “university” (e.g., MIT) and
negative institutions that do not include the phrase “community college” (e.g., Dallas College). The overall trend is consistent with that of Table 2’s institutional bias, with the only difference being that in this experimental setup, in the SAI direction, the Mistral model rejects the null hypothesis, and there is no statistically significant result for that. See Table 31 in Appendix E for full results.
Gender. Another possible confound in our experiments is gender bias. Figure 5 shows the percentage of selecting positive, negative, and neutral attributes/stimuli averaged across all models for gendered pronoun settings. In both SAI and ASA directions LLMs are marginally more likely to select positive attributes and less likely to select negative attributes for feminine pronouns compared to masculine or non-binary pronouns. Table 6 partially confirms this observation with statistically significant results for GPT-4 in both inference directions and the other models in one of the directions. We find that the effect sizes are small even where results are significant.20
Table 4 shows the dataset-wide statistical tests while controlling for gender. We maintain statistically significant correlations for all settings and there are no major differences in results across genders.

heading: 7.2 Invalid LLM Responses
text: We excluded some examples for invalid responses. LLMs didn’t always select from our three choices. These responses were categorized into five groups, with trends detailed in Appendix D.
20This is promising regarding the progress of the field, but this is only a coarse-grained analysis. We only use pronouns to access this variable and our dataset does not focus on gender bias-specific attributes and stimuli.

heading: 8 Conclusion
text: We examine the behavior of four common LLMs across several less-studied domains of bias, looking at general positive and negative polarity associations rather than precise stereotypes. Our findings indicate that all four models exhibit statistically significant biases. Our dataset and experimental design draw upon prior literature on both social science and computer science. As the use of LLMs continues to grow and these models are increasingly employed in various tools, it becomes crucial to be vigilant about even subtle forms of bias. While much work has been done to investigate overt biases such as those related to race, gender, and religion, less attention has been paid to subtler biases such as ageism, beauty, and institution. Through the introduction of our dataset, we encourage the consideration of these overlooked biases when using LLMs. We hope that this dataset will help to further research and mitigate these types of biases. Future research is needed to extend our findings to other models and further biases that have been identified by social scientists.

heading: 9 Limitations
text: There are a few factors in our experiments that may limit the generalizability of our results and conclusions. While we selected four of the most common and powerful LLMs currently available, our experiments were far from exhaustive. Many LLM variants exist today and will be developed in the future. In constructing our dataset, we limited many of our stimuli and attributes to a relatively small number of options. For instance, in the case of institutional bias, we consider only 100 universities and 100 community colleges. It is possible that our specific set of prompts and template structures also affected the results that we saw.
In grouping our stimuli and attributes we used
several proxies to allow us to collect data efficiently. For example, economic conditions were used as proxies for grouping both professions and nationalities. We also used college rankings as a proxy for the sentiment polarity of academic institutions. All of these proxies are noisy approximations of the biases that we seek to measure and could have affected our results.
Our experiment was also only conducted in English. This means that the behavior of the LLMs, which are capable of working in multiple languages, may change in other languages. This could be due to technical reasons, such as the relatively smaller training data and development investment in other languages. Or this could be due to cultural or sociolinguistic reasons. That is, the prevalence, degree, and specific stimuli of each of the bias categories are culture-dependent and the way that the bias is realized in language is dependent on the sociolinguistic context—the linguistic and social norms of the community within which the utterance was generated.

heading: Acknowledgements
text: This project was fully supported by the University of South Florida. We thank the reviewers for their valuable feedback especially their suggestions to add different types of confounding variables to our experiments.

heading: A ASA Data Creation
text: Age: The stimuli here are divided into three parts, namely positive stimuli (age between 25-35), neutral stimuli (age between 42-52), and negative stimuli (age between 60-70). We chose these age ranges based on the study by Cameron (1969).
Beauty: We only consider the positive and negative professions and skip the neutral professions, similar to the other attribute groups in this direction. The positive and negative stimuli here are
the same as those from the SAI direction. For the neutral stimuli, we use the set of neutral attributes elsewhere in the dataset.21
Institution: Here we use cities in USA by population as neutral stimuli (e.g., New York, Tampa, etc.)22. Here we also select at least one city form each state.23 For more attributes and stimuli, see Table 34.
Nation: For the added neutral stimuli we select countries from the middle third of IMF’s report of GDP per capita (e.g., Thailand, Gabon, etc.).
For a basic overview of attributes and stimuli in ASA direction, see Table 33.

heading: B Model Details
text: We use four major language models for assessing our task: 1) The GPT-4 using checkpoint on the OpenAI API; 2) Google PaLM-2 using PaLM API and MakerSuite; 3) Llama2-13B via the TheBloke/Llama-2-13B-chat-GGML checkpoint on Huggingface; 4) mistral-7B via the TheBloke/Mistral-7B-Instruct-v0.1-GGUF checkpoint on Huggingface. We also use the GPT-3.5TURBO-INSTRUCT using checkpoint on the OpenAI API.

heading: C Detailed Result Section
text: We present all the statistical results for GPT-3.5 in Table 7. Most of these results are statistically significant. For the beauty bias category, the results are strong in both directions which we can also see from Figure 6. We can also say that in both SAI and ASA directions models are marginally more likely to select positive attributes and less likely to select negative attributes for feminine pronouns compared to masculine or non-binary pronouns. But GPT-3.5 didn’t show statistically significant results when we controlled the institutional category based on educational level.
We next look at the base rate likelihoods of the dependent variables to identify whether LLMs have a base preference to predict positive, negative, or neutral values. Table 8 shows a full list of these results. GPT-4 clearly favors predicting positive
21We found that almost every beauty term is charged with some degree of positive or negative force.
22https://en.wikipedia.org/wiki/List_of_United_ States_cities_by_population
23There are no obvious categories for neutral educational institutions. And cities are substitutable in the sentence constructions. We select cities as a proxy of neutral stimuli to make our data collection process easier and consistent like other bias categories.
values on average in every category. The GPT-3.5 and PaLM-2 results are more mixed, where the preference is dependent on category. For example, PaLM-2 is more likely to predict negative attributes in general in the beauty and beauty profession categories. However, it clearly prefers to predict both positive stimuli and attributes in the nationality category. In the SAI direction, the Llama-2 model clearly favors positive values, but in the ASA direction, results are more mixed.
We then use the English-gendered pronouns in the dataset to coincidentally investigate the degree to which LLMs show gender bias in our dataset. Table 9 shows a table of results for these models, in both directions and the gender of the pronoun. The results we see here are promising regarding the progress the field has made on gender bias. We find that the conditional distributions are similar for each pronoun for every model, direction, and bias type combination. As we would expect from Table 8, the positive inferences are more likely than negative and neutral inferences, but the number stay close across pronoun types. If anything, the results suggest that LLMs skew slightly more positively for feminine pronouns, in that PPL and NPL values for feminine pronouns typically exceed the
others while PNL and NNL values are typically exceeded by the others. While promising, this is only a coarse-grained analysis on gender since we only use pronouns to access this variable and our dataset does not focus on gender bias-specific attributes and stimuli.
We represent our complete experimental results for each bias type and direction in Tables 10 -29.
C.1 Tables in SAI direction
For ageism bias in Table 13, we can see that the positive-to-positive likelihood (PPL) is 75.82% while the positive-to-negative likelihood (PNL) is 2.84%, which means the GPT-4 is more inclined to select positive attributes (e.g., efficient, creative, etc.) in response to younger age (e.g., 26 years old, 28 years old, etc.) and less inclined to select negative attributes in response to younger age. The negative-to-negative likelihood (NNL) is 16.82%, which means the GPT-4 is also selecting negative attributes in response to older age (e.g., 65 years old, 68 years old, etc.). Given the values of PNL and NNL, it is evident that GPT-4 favors negative attributes more when dealing with older individuals compared to younger ones. So, we can say that there exists an ageism bias in the GPT-4. For
ageism bias, we also see the same kind of trends for PaLM-2 in Table 16, Llama-2 in Table 19, and GPT-3.5 in Table 10. But for Mistral in Table 22, this difference is less compared to other models and is not statistically significant.
When we observe the beauty bias, we can say that there is a very strong association of beauty bias in all of these models. In Table 10, the PNL and NNL of beauty bias are 5.06% and 79.22% respectively, which means that the GPT-3.5 selects negative attributes more when dealing with negative appearance (e.g., unattractive, etc.). The same thing is also happening for beauty bias with professions. Different values of beauty bias in professions suggest that GPT-3.5 selects negative attributes (e.g., security guard, taxi driver, etc.) more when dealing with negative appearance. We can see the similar trend like GPT-3.5 in all other models. In Table 13, for GPT-4, we see that PNL for beauty bias is 0%, which really indicates how much biased it is.
In the same way, in Table 13, the PPL of institutional bias is 90.54%, which is comparatively high compared to other models. The NNL and PNL of institutional bias suggest that GPT-4 chooses negative attributes more when dealing with community colleges. We also see the same kind of trends for GPT-3.5 in Table 10, PaLM-2 in Table 16, Llama2 in Table 19, and Mistral in Table 22. But for Mistral, this difference is less compared to other models.
In Table 16, for nationality bias, PaLM-2 picks negative attributes more when dealing with poor countries. This trend is also similar for all other models. But the difference between NNL and PNL, NPL and PPL, etc. are smaller compared to other types of bias for GPT-4, Llama-2, and Mistral. This also indicates that we made some improvement at least for nationality bias.
In Table 14, we showed our experimental results from a gender perspective and in the SAI direction with GPT-4. For ageism bias, in terms of negative attribute selections, we can see that GPT-4 selects fewer negative attributes for feminine pronouns. However, when it comes to institutional bias, we observed that the PaLM-2 model in Table 17 tends to select more negative attributes for masculine genders compared to non-binary and feminine. In the case of nationality and institutional bias, the PaLM-2 model also tends to favor more negative attribute selections for masculine compared to nonbinary and feminine.
In Table 25, 26, 27, 28, 29 we present our results
for different educational level for GPT-3.5, GPT-4, PaLM-2, Llama-2, and Mistral models respectively.
C.2 Tables in ASA direction
In this direction, we also notice that there is a strong likelihood between attribute and stimulus. We can see that the PPL is less compared to the SAI direction in most cases, but there are still some discrepancies among values. In Table 13, When we look at the NNL and PNL of ageism bias, we realize that GPT-4 selects negative stimuli (e.g., 66 years old, 69 years old, etc.) more when dealing with negative attributes. We also see similar trends for other models. For Mistral in Table 22, we see the trend of PPL and NPL is different for ageism bias compared to others.
In Table 13, for beauty bias in professions, we notice that the difference between NNL and PNL is 22.32%. This means that the GPT-4 picks the negative appearance more in response to the lowerincome individuals. This trend is also similar to other models. For institutional and nationality bias, the same kind of relationship exists for all models.
Table 12 presented our experimental results from a gender perspective and in the ASA direction with GPT-3.5. For beauty bias in professions, we observed that the GPT-3.5 model tends to select more negative attributes for males compared to females and non-binary genders.
In ASA direction we also see similar trends like SAI direction in gender settings. In Table 15, for beauty bias, we see that PPL is higher for feminine pronoun settings. From Table 21, we see that the gap between these different gender settings is even narrow.
D Invalid Results Categorized
Based on the broad invalid responses pattern, we divide the responses into five different categories. Category 1 contains just some number between 1 to 3, we named this category as “Numeric Selection”. Category 2 contains responses which are from context sentence but not from the option list, we named this category as “Non-Option Span”. Category 3 indicates that the sentence was not completed, meaning the response is either null or assistance cannot be provided, we named this category as “No Response”. Category 4 includes those responses which response stating that there will be some stereotype in the sentence and it is not appropriate to promote stereotype or none of the op-
tions are appropriate for the context, we named this category as “Stereotype Awareness”. Category 5 includes something outside of the context sentence and also outside of the option list, we named this category as “Out-of-Context Responses”. Table 30 represents the exact number of invalid responses in each category for all models. In a majority of cases the invalid responses contained some text from the provided prompt sentence not from the option list. In another common case, LLMs would respond with an answer that is closely related to the provided stimulus or attribute, but not one of the options in the prompt (e.g., filling the BLANK with “retired” for an stimulus of “66 years old” or “model” in response to the stimulus “gorgeous”). PaLM-2 would often respond with just an empty string. GPT-4 and Llama-2 would occasionally recognize that the nationality bias items could be offensive or stereotyping and respond that they could not answer such questions.

heading: E Statistical Tests for Educational Level

heading: F Lists of Attributes and Stimuli

heading: G Positivity and Bias
text: Figure 5 shows a global preference toward positive generations. Especially, in the SAI direction where the positive generation likelihood is about double the negative generation rate for all genders. This raises the question whether current LLMs avoid criticism not by avoiding bias altogether, rather by avoiding negative biases only. We explore this question by performing a τ -test between positive and negative binary given information (stimuli or attributes) and a dependent variable of whether the generated response is stereotypical. That is, a generated response is stereotypical if the polarity of the generation matches the given information. For example, producing a negative attribute for a negative stimulus would be a stereotype value of 1. And producing a positive attribute would be marked as a stereotype value of -1. This leads to a ternary dependent variable (anti-stereotype, neural, stereotype). This test would lead to positive τ values if an LLMs are more stereotypical when
the given information is positive. The results of the statistical test are presented in Table 35. We find statistical significance for most experimental settings. However, the direction of correlation varies by model, inference direction, and bias domain. In the SAI direction, Llama-2 has statically significant positive effects for all bias domains. So Llama-2 is more likely to produce stereotypical attributes when given positive bias stimuli. GPT-4 and Mistral show similar results, though GPT-4’s results are not statistically significant in the beauty profession domain and Mistral’s results are not statistically significant in the age and nationality domains. PaLM-2 in the SAI direction curiously has negative τ values for age and beauty domains. In the ASA direction, the results are much more mixed in τ positivity, though most results are still statistically significant.
The largest effect sizes are GPT-4 in the SAI direction for the nationality domain (0.913), GPT-4 in the SAI direction for the age domain (0.710), and GPT-4 in teh ASA direction in the institution domain (0.588). This suggests that something in the GPT training paradigm causes it to be much more biased when the setting is positive. Also notably, age, institution, and nationality are more well-studied and considered in the field of NLP bias. These results align with the hypothesis that efforts to mitigate bias in LLMs are leading to this generation pattern.

heading: Future Work
text: Many LLM variants exist today and will be developed in the future. In constructing our dataset, we limited many of our stimuli and attributes to a relatively small number of options. For instance, in the case of institutional bias, we consider only 100 universities and 100 community colleges. It is possible that our specific set of prompts and template structures also affected the results that we saw. In grouping our stimuli and attributes we used
several proxies to allow us to collect data efficiently. For example, economic conditions were used as proxies for grouping both professions and nationalities. We also used college rankings as a proxy for the sentiment polarity of academic institutions. All of these proxies are noisy approximations of the biases that we seek to measure and could have affected our results. Our experiment was also only conducted in English. This means that the behavior of the LLMs, which are capable of working in multiple languages, may change in other languages. This could be due to technical reasons, such as the relatively smaller training data and development investment in other languages. Or this could be due to cultural or sociolinguistic reasons. That is, the prevalence, degree, and specific stimuli of each of the bias categories are culture-dependent and the way that the bias is realized in language is dependent on the sociolinguistic context—the linguistic and social norms of the community within which the utterance was generated