Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender? (2024)

Abstract: We examine whether large language models (LLMs) exhibit raceand gender-based name discrimination in hiring decisions, similar to classic findings in the social sciences (Bertrand and Mullainathan, 2004). We design a series of templatic prompts to LLMs to write an email to a named job applicant informing them of a hiring decision. By manipulating the applicant’s first name, we measure the effect of perceived race, ethnicity, and gender on the probability that the LLM generates an acceptance or rejection email. We find that the hiring decisions of LLMs in many settings are more likely to favor White applicants over Hispanic applicants. In aggregate, the groups with the highest and lowest acceptance rates respectively are masculine White names and masculine Hispanic names. However, the comparative acceptance rates by group vary under different templatic settings, suggesting that LLMs’ raceand gender-sensitivity may be idiosyncratic and prompt-sensitive.

heading: 1 Introduction
text: Field experiments in prior social science research (Bertrand and Mullainathan, 2004; Cotton et al., 2008; Kline et al., 2022) have demonstrated that Black- or White-sounding names play a nontrivial role in influencing the hiring decision of candidates with similar qualifications. Their results suggest that applicants with names perceived as African American encounter significantly fewer opportunities in comparison to their counterparts with names perceived as European American. Following the rapid advancement of large language models (LLMs; Touvron et al., 2023a,b; OpenAI, 2023), a number of studies have examined the ways in which LLMs exhibit human-like behaviors and cognitive biases (Aher et al., 2023; Dillion et al., 2023; Argyle et al., 2023). In this work, we pose the following question: When prompted to make hiring decisions, do LLMs exhibit discriminatory
behaviors based on the race, ethnicity, and gender associated with a job applicant’s name?
There are several reasons to study this question: (1) To contribute to scientific understanding of the internal, representational biases of LLMs, (2) to demonstrate the potential harms of using LLMs in real-world hiring decisions, and (3) as further validation of LLMs as a tool for social scientists to cheaply test hypotheses prior to conducting costly, real-world studies. The research question has implications for understanding both representational and allocational harms of LLMs (Barocas et al., 2017; Crawford, 2017; Blodgett et al., 2020).
We design a series of prompts that ask an LLM to write an email (e.g., on behalf of a hiring manager) to inform a job applicant about the outcome of their hiring decision. In all settings, the prompt contains the instructions to the LLM and the first name of the applicant. We experiment with three additional variables: the job title (position sought), the candidate’s level of qualification, and template (para)phrasing. Crucially, all prompts do not spec-
386
ify whether to accept or reject the applicant; thus, to fulfill the instructions, the model must choose.
With large-scale analysis of these generations (over 2 million emails), we find that LLMs tend to favor White applicants in making hiring decisions. In contrast, models tend to disadvantage names associated with underrepresented groups. In particular, Hispanic names receive the least favorable treatment. While it is (hopefully) unlikely that employers would use LLMs in precisely this fashion, we believe that, by isolating the influence of names on hiring decisions, these experiments may serve as a “canary in the coalmine,” indicating the risk of possible fairness issues with the use of LLMs at other stages of the hiring pipeline, or in professional workplace settings more generally.

heading: 2 Experiment Setup
text: To study the influence of race and gender on LLMs’ hiring decisions, we develop a set of prompt templates instructing models to write an email to a job applicant informing them of a hiring decision. Each template contains a “[NAME]” placeholder, which we substitute with first names statistically associated with a particular race or ethnicity, and gender in the United States. We then measure the average rate of acceptance within each demographic group and compare it to the average acceptance rate over all groups. This methodology of first-name substitution is well established in the social sciences and in NLP research for measuring biased or discriminatory behavior in humans or models (Greenwald et al., 1998; Bertrand and Mullainathan, 2004; Caliskan et al., 2017).
Collecting first names We obtain 100 names that are most representative of each of the three races/ethnicities in our study (White, Black, and
Hispanic), evenly distributed between two genders (female and male) by consulting Rosenman et al. (2023) for race/ethnicity data and the social security application dataset (SSA1) for gender statistics. As a result, we have 50 names in each intersectional demographic group and 300 names in total. Detailed name selection criteria and a complete list of first names are available in appendix A.
Prompts We design 820 templates by enumerating all possible combinations of 4 qualification levels, 5 base templates, and 41 occupational roles, as shown in Fig. 2. To mitigate the model’s sensitivity to different template phrasings, we use ChatGPT 3.5 to paraphrase our initial template into four variations, resulting in five base templates. The 41 job roles include 40 occupations (38 are from WinoBias (Zhao et al., 2018) and we additionally include “CTO” and “software engineer” as they are frequently generated by Llama2 in our preliminary experiments) and 1 under-specified setting. We use under-specified inputs primarily to better isolate the influence of name demographics on hiring decisions. Including other applicant details (e.g., real-world or synthetic resumes) could confound the results or limit their generalizability, as it would introduce a large number of variables, making exhaustive and well-controlled experiments infeasible (Veldanda et al., 2023). Detailed information about template construction is illustrated in appendix B.
Models We carry out our experiments using five state-of-the-art instruction-tuned generative LLMs: Mistral-Instruct-v0.1 (Jiang et al., 2023), Llama2 (Touvron et al., 2023b) with three different model sizes (7b, 13b, and 70b), and GPT-3.5-
1https://www.ssa.gov/oact/babynames/
Turbo (Ouyang et al., 2022). Model hyperparameters are detailed in appendix C.1. For open-source models, we execute the experiments with 3 different random seeds for reproducibility and report the average results. We note that due to limited computational resources, we run the experiments on a smaller scale for Llama2-70b and GPT-3.5Turbo, obtaining 756, 000 emails for Mistral-7b and Llama2-{7b, 13b}, 48, 000 emails for Llama270b, and 19, 200 emails for GPT-3.5.
Generation validity We randomly sample 30 instances for each intersectional group and manually check the validity of the generated content in a total of 180 emails generated from Llama2-13b. An email is valid if it (1) follows a typical email communication format with fluent content and (2) clearly communicates the binary application outcome (accept or reject). By randomly sampling 180 emails per model (evenly distributed among gender and racial groups), we find that all models have high validity rates between 83% to 100% (Table 5 in appendix C.2). We also find that the validity rates for each intersectional group within a model have relatively small standard deviations (Table 6 in appendix C.2). Assuming a binomial distribution for valid email generations, we do not find statistically significant differences between any pair of groups within the same model setting (p > 0.05). These observations suggest that all intersectional groups have very similar validity rates.
Email classification Our experiments require labeling over 2M emails as acceptances or rejections. To automate this, we train a support vector machine (SVM) model with TF-IDF features (Ramos et al., 2003) using 1, 200 manually annotated instances evenly distributed across gender and race/ethnicity.
To further mitigate the risk of demographic bias in the classifier, applicant names are redacted during training and usage. The classifier achieves an F1 score of 0.98 on the 170 valid emails randomly sampled from Llama2-13b generations, showing that accept and reject emails are easy to distinguish. More details are described in appendix C.2.

heading: 3 Results and Discussion
text: We examine the generated emails from a variety of LLMs and elaborate how they relate to known labor market discrimination. We present the acceptance rates for every intersectional group in different templatic settings and models in Table 1 to Table 3. To measure the statistical significance in the difference between the email outcome distributions, we conduct a permutation test between each group’s acceptance rate and the population acceptance rate. Details about the permutation test are elaborated in appendix C.3.
Differences are small but statistically significant. We aggregate the acceptance over (1) a subset of 7 occupational roles2 and (2) all 41 occupational roles respectively for different models in Table 1. We observe that the absolute differences between the highest and lowest acceptance rate for different groups are generally small (between 1.60% and 3.78% across models). Despite the small magnitude, our permutation test testifies the statistical significance. A model that discriminates in a small
2The seven roles include the under-specified setting, software engineer, CTO, secretary, hairdresser, carpenter, and mechanician. We choose to experiment with these seven roles because of their strong gender association indicated in WinoBias (Zhao et al., 2018) or their frequent occurrence in Llama2 generations in our preliminary experiments.
but statistically significant manner can still be problematic. An absolute disadvantage of 3.78% based purely on the racial, ethnic, or gender associations of one’s name should be concerning, particularly as such differences, if systematic, can accumulate throughout a pipeline where a series of slightly discriminatory decisions are made (Alexander, 2011).
Acceptance rates are uniformly lowest for Hispanic male names.
Hispanic male applicants consistently receive the least favorable treatment in many settings across Mistral-7b (Tables 1, 2, 3), Llama2-{7b, 13b, 70b} (Tables 1, 2, 3), and GPT-3.5 (Table 1). Lower LLM-based acceptance rates for applicants with Hispanic names echoes prior findings of discrimination against Hispanic individuals in the labor market (Reimers, 1983; Chiswick, 1987; Cross, 1990; Kenney and Wissoker, 1994; Woods, 2000; Duncan et al., 2006). If deployed by employers for hiring decisions, LLMs could further entrench, systematize, and amplify hiring discrimination against Hispanic job applicants.
Some groups exhibit higher acceptance rates.
Table 1 shows that White male and Black female names receive above-average acceptance rates overall in two and three of five models tested, respectively. The trend that models often favor White male applicants reflects existing disparities in the U.S. labor market (Galgano, 2009; Ritter and Taylor, 2011; McDonald et al., 2014; Pedulla and Pager, 2019) and pose a risk of exacerbating them if LLMs are adopted for employment decisions. The
results observed for Black female names are interesting as they run counter to the real-world resume study of Bertrand and Mullainathan (2004). However, when occupations are grouped by education level3 (Table 2), we observe that higher acceptance rates for Black female names on Mistral-7b only applies to occupations in the “no formal education” and “postsecondary non-degree award” categories.
Llama2-70b shows least variation across demographic groups.
Llama2-70b appears to exhibit the least variation in acceptance rates across groups (Table 1), with a range of 1.6% between the groups with the highest and lowest overall acceptance rates. By contrast, the corresponding ranges for Llama2-13b and GPT-3.5 are 3.56% and 3.78%, respectively. This observation may suggest that larger models could be more robust and fair in the task of generating hiring decision emails in an under-specified setting. However, it is inconclusive which exact factors contribute to the minimal variations in Llama2-70b because the model training details are not fully available to the public.

heading: Qualifications matter.
text: In Table 3 we group results by stated qualification levels and observe a couple trends across models. When candidate qualification level is not specified, it appears that female names receive higher acceptance rates in general than male names; however,
3Data source: https://www.bls.gov/emp/tables/ education-and-training-by-occupation.htm
when candidates are described as only “somewhat qualified” or “not qualified,” White names, in particular White male names, appear most likely to receive acceptances. While our results do not offer an explanation for why these trends occur, we speculate that it could pertain to a (real or perceived) gender “confidence gap”: Partially-qualified female job seekers are less likely to apply for positions than their partially-qualified male counterparts due to lower confidence in their qualifications (Carlin et al., 2018; Sterling et al., 2020)
Some models exhibit human-like gender-occupation stereotypes. We find that some models, in certain cases, exhibit human-like stereotypes when making hiring decisions for masculine or feminine job roles. For instance, Table 4 shows that, for secretary, which is a stereotypically feminine occupation (Zhao et al., 2018), GPT-3.5 generates a lower number of acceptance emails for male candidates compared to their female counterparts across racial and ethnic groups. While we observe this trend for some female- or male-dominated jobs, it may not be universally applicable to all occupational roles across models, suggesting that LLM’s gender-sensitivity may be idiosyncratic and prompt-dependent.

heading: 4 Related Work
text: First names, demographic identities, and economic opportunities Researchers have been using first names that have strong correlation with some demographic attributes, such as gender, race/ethnicity, and age, to examine the problem of social bias in both social science studies and NLP systems (Greenwald et al., 1998; Nosek et al., 2002; Caliskan et al., 2017; An et al., 2022). Partially due to their association with demographic identities, first names often lead to inequitable distribution of economic opportunities as people build stereotypes in favor of or against names that reveal a person’s demographic identity (Bertrand and Mullainathan, 2004; Nunley et al., 2015; Goldstein
and Stecklov, 2016; Ahmad, 2020).
First name biases in language models While numerous recent works propose new benchmark datasets and algorithms to uncover social biases in language models (Rudinger et al., 2018; Zhao et al., 2018; Nangia et al., 2020; Nadeem et al., 2021; Parrish et al., 2022; Cheng et al., 2023; Hossain et al., 2023), some are particularly dedicated to the study of first name biases or artifacts in these models (Maudslay et al., 2019; Shwartz et al., 2020; Wolfe and Caliskan, 2021; Wang et al., 2022; Jeoung et al., 2023; Sandoval et al., 2023; Wan et al., 2023; An et al., 2023; An and Rudinger, 2023). We build upon previous research and examine the disparate treatment of names in email generation regarding job application outcomes.
Auditing LLMs in hiring Several contemporaneous works (Tamkin et al., 2023; Haim et al., 2024; Gaebler et al., 2024) also examine whether LLMs treat individuals of various demographic backgrounds differently in decision-making. Most related to our paper, Veldanda et al. (2023) and Armstrong et al. (2024) generate synthetic resumes for a limited number of job categories (≤ 10) and uncover hiring bias either during generation or in downstream tasks (e.g., resume summarization and assessment) using a smaller set of names (≤ 32). In contrast, our work studies implicit hiring discrimination in LLMs by conducting large-scale experiments using 300 names and 41 occupational roles in under-specified inputs, without introducing other confounders from synthetic resumes.

heading: 5 Conclusion
text: Through the use of 820 templates and 300 names, we generate as many as 756, 000 job application outcome notification emails per model that we use to measure LLMs’ discriminatory behavior in labor market decisions. Our analyses demonstrate the presence of such discrimination in some LLMs against some traditionally underrepresented groups, such as Hispanic, as their acceptance rates are systematically lower than the average in multiple cases. White applicants, however, are often portrayed in a more positive light with a higher chance of getting accepted. Our findings alert the community to be concerned about the implicit biases within the model as they could cause both representational and allocational harms to various demographic groups in downstream tasks.

heading: Limitations
text: Incomplete representation of demographic identities Due to the limited data availability of first names, we are only able to thoroughly study names representing three races/ethnicities (Black, White, and Hispanic) and two genders (female and male). Getting a large number of names from the underrepresented demographic groups is a common challenge in research on first name biases (An et al., 2023; An and Rudinger, 2023; Sandoval et al., 2023). In addition, it is essential to recognize that our diverse community encompasses numerous other racial, ethnic, and gender identities, not to mention various demographic attributes such as nationality, religion, disability, and many more. We acknowledge that some of these attributes are not strongly correlated with first names and thus it is less feasible to use names as a proxy to represent these demographic traits. While our study focuses on a small subset of demographic identities inferred from first names, our findings on first name biases in email generation underscore the need to use LLMs fairly and responsibly.

heading: Incomplete representation of occupations In
text: this paper, we have studied 40 different occupational roles on a coarse-grained level. However, the 2018 Standard Occupational Classification (SOC) system4 contains 867 occupations. There remains a large number of occupational roles not being tested. It is inconclusive, although likely, that LLMs would also have differential treatment towards different first names for other occupations. Additional extensive experiments would need to be conducted in order to assess the validity of this hypothesis.
A wider range of LLMs could be tested In our experiments, we have tested 5 state-of-the-art models of considerably very large model sizes (all ≥ 7b). However, the discrimination and biases in smaller language models are not studied in our work. Since these smaller models typically have weaker instruction-following abilities, our hypothesis is that they may exhibit different behavior from the larger models, especially when the input prompt states the candidate is not qualified. We leave the study of smaller models as future work.
Not simulating the entire hiring process Our prompts are designed to study LLMs’ discriminatory behavior in labor market with little to no ad-
4https://www.bls.gov/soc/
ditional information about the applicant. This simulation is different from a realistic hiring process in real life where substantially more information about a candidate would be made available to the hiring team. Despite a much simplified processing of getting to know a job applicant, the short but focused input prompt could directly reveal the representational biases in LLMs without the distraction of additional applicant details. Finally, we note that our experiments do include specifying an applicant’s degree of qualification for the position, which can be seen as a summary judgment in place of other application details such as a resume.

heading: Ethics Statement
text: As the widespread adoption of LLMs continues, prioritizing responsible usage of these tools becomes paramount, particularly in contexts where they are employed to allocate social resources and economic opportunities. Our study sheds light on the potential risks associated with integrating LLMs into the hiring process. Notably, these models have learned to correlate distinct first names with varying rates of job application acceptance. This underscores the necessity of vigilant consideration when deploying LLMs in decision-making processes with significant societal implications.
Though we believe studying the discriminatory behavior of LLMs is an important social and scientific endeavor, our study is not without potential risk. Studies of race, ethnicity, and gender have the potential to themselves essentialize or misconstrue social categories in ways that flatten or misrepresent individual members of those groups. Additionally, while it is our belief that the harms of LLMs for hiring practices outweigh the potential benefits in part due to scalability concerns, employers and policy-makers must also weigh the harms of the alternative; in this case, human decision-making is also known to be biased. While warning of the potential harms of AI usage in decision-making is beneficial if it prevents harmful usage, there is a potential risk that the resulting stigmatization of LLMs could prevent its future adoption in settings where it could be used to advance social equality.

heading: Acknowledgements
text: We thank the anonymous reviewers for their constructive feedback. We are grateful to Kaiyan Shi and Tianrui Guan, who helped us with data collection in the early stages of this project.

heading: A First Names

heading: A.1 Selection Criteria
text: Of our name data sources, Rosenman et al. (2023) provide the racial/ethnic distribution among five categories: “White”, “Black”, “Hispanic”, “Asian”, and “Others”. This categorization of race/ethnicity primarily follows the U.S. Census Bureau’s definition of race and ethnicity. For robust results, we only include names that have more than 1,000 occurrences in the data source provided by Rosenman et al. (2023). We assign the majority race (> 50%) as the race associated with a name. No names in the dataset meet the inclusion criteria for the category “Others” and there are fewer than 15 names
for “Asian”. As a result, our study only involves the other three racial/ethnic categories. With reference to the SSA dataset,5 we use the majority gender (> 50%) to approximate the gender associated with a name. We only include a name in our study if it appears in both of the data sources.
Within each racial and gender subgroup (e.g., Black female), we then rank the names by their percentage of the majority race and select the top 50 ones for our experiments.

heading: A.2 Names Used
text: We list all 300 first names used in our experiments.
White female names Abbey, Abby, Ansley, Bailey, Baylee, Beth, Caitlin, Carley, Carly, Colleen, Dixie, Ginger, Haley, Hayley, Heather, Holli, Holly, Jane, Jayne, Jenna, Jill, Jodi, Kaleigh, Kaley, Kari, Katharine, Kathleen, Kathryn, Kayleigh, Lauri, Laurie, Leigh, Lindsay, Lori, Luann, Lynne, Mandi, Marybeth, Mckenna, Meghan, Meredith, Misti, Molly, Patti, Sue, Susan, Susannah, Susanne, Suzanne, Svetlana
White male names Bart, Beau, Braden, Bradley, Bret, Brett, Brody, Buddy, Cade, Carson, Cody, Cole, Colton, Conner, Connor, Conor, Cooper, Dalton, Dawson, Doyle, Dustin, Dusty, Gage, Graham, Grayson, Gregg, Griffin, Hayden, Heath, Holden, Hoyt, Hunter, Jack, Jody, Jon, Lane, Logan, Parker, Reed, Reid, Rhett, Rocco, Rusty, Salvatore, Scot, Scott, Stuart, Tanner, Tucker, Wyatt
Black female names Amari, Aretha, Ashanti, Ayana, Ayanna, Chiquita, Demetria, Eboni, Ebony, Essence, Iesha, Imani, Jalisa, Khadijah, Kierra, Lakeisha, Lakesha, Lakeshia, Lakisha, Lashanda, Lashonda, Latanya, Latasha, Latonia, Latonya, Latoya, Latrice, Nakia, Precious, Queen, Sade, Shalonda, Shameka, Shamika, Shaneka, Shanice, Shanika, Shaniqua, Shante, Sharonda, Shawanda, Tameka, Tamia, Tamika, Tanesha, Tanika, Tawanda, Tierra, Tyesha, Valencia
Black male names Akeem, Alphonso, Antwan, Cedric, Cedrick, Cornell, Cortez, Darius, Darrius, Davon, Deandre, Deangelo, Demarcus, Demario, Demetrice, Demetrius, Deonte, Deshawn, Devante, Devonte, Donte, Frantz, Jabari, Jalen, Jamaal, Jamar, Jamel, Jaquan, Jarvis, Javon, Jaylon, Jermaine, Kenyatta, Keon, Lamont, Lashawn, Malik, Marquis, Marquise, Raheem, Rashad, Roosevelt,
5https://www.ssa.gov/oact/babynames/
Shaquille, Stephon, Sylvester, Tevin, Trevon, Tyree, Tyrell, Tyrone
Hispanic female names Alba, Alejandra, Alondra, Amparo, Aura, Beatriz, Belkis, Blanca, Caridad, Dayana, Dulce, Elba, Esmeralda, Flor, Graciela, Guadalupe, Haydee, Iliana, Ivelisse, Ivette, Ivonne, Juana, Julissa, Lissette, Luz, Magaly, Maribel, Maricela, Mariela, Marisol, Maritza, Mayra, Migdalia, Milagros, Mireya, Mirta, Mirtha, Nereida, Nidia, Noemi, Odalys, Paola, Rocio, Viviana, Xiomara, Yadira, Yanet, Yesenia, Zoila, Zoraida
Hispanic male names Agustin, Alejandro, Alvaro, Andres, Anibal, Arnaldo, Camilo, Cesar, Diego, Edgardo, Eduardo, Efrain, Esteban, Francisco, Gerardo, German, Gilberto, Gonzalo, Guillermo, Gustavo, Hector, Heriberto, Hernan, Humberto, Jairo, Javier, Jesus, Jorge, Jose, Juan, Julio, Lazaro, Leonel, Luis, Mauricio, Miguel, Moises, Norberto, Octavio, Osvaldo, Pablo, Pedro, Rafael, Ramiro, Raul, Reinaldo, Rigoberto, Santiago, Santos, Wilfredo

heading: B Prompts
text: We write one template to begin testing the behavior of LLMs in making hiring decisions in an underspecified context. To mitigate the model’s sensitivity to different template phrasing (Shin et al., 2020; Sancheti and Rudinger, 2022; Lu et al., 2022), we use ChatGPT 3.56 to paraphrase our first template into 4 variations, resulting in 5 base templates in total. The instruction we use for the paraphrasing task is
Help me find four ways to paraphrase the following sentence. Keep the placeholder terms like "[NAME]", "pronoun_poss", and "pronoun_subj".
Write an email informing [NAME] about pronoun_poss application decision for the role pronoun_subj has applied.
Note that the root template used for paraphrasing is slightly different from our first root template in Fig. 2 as this one contains pronouns. We later choose to experiment with a modified template without any pronouns so that we can control any potential influence on model generation exerted by different pronouns like “she” and “he.” This would
6https://chat.openai.com/
allow us to focus on studying the model behavior towards different first names.
For each base template, we add additional information about the job role to probe model behavior under the influence of 40 occupations in addition to an under-specified setting. In total, we have 41 occupational roles for each template, including the one where the occupation is not specified. Furthermore, we attempt to give an LLM information about candidate qualification and test if it makes more informed decisions following this additional hint. We prepend a sentence directly describing one of the three levels of qualifications (“highly qualified,” “somewhat qualified,” and “not qualified”) to the templates for each role. As a result, we have a total number of 820 templates, as shown in Fig. 2.

heading: C Additional Experiment Setup Details

heading: C.1 Models
text: We specify the model hyperparameters used in our paper. For fair and controlled comparisons, we keep the hyperparameters consistent across models when possible throughout our experiments. We note that the use of every model follows its original intended use because all of the selected models are specifically fine-tuned to follow human instructions like our designed prompts.
Because Llama2-70b and GPT-3.5-Turbo require heavier computational cost that exceeds our budget, we run the experiments on a smaller scale by reducing the number of occupations to 7 for both, having only one random seed for Llama2-70b, and having only two templates for GPT-3.5-Turbo. In the end, we obtain 756, 000 emails for Mistral-7b and Llama2-7b, 70b, 48, 000 emails for Llama2-70b, and 19, 200 emails for GPT-3.5.
Llama2 We mainly follow the hyperparameters recommended in the original Llama2 repository,7 where temperature = 0.6, top_p = 0.9, max_batch_size = 4. We set both max_seq_len and max_gen_len to be 256. The same set of hyperparameters is used for all thre model sizes (7b, 13b, and 70b). Note that even if temperature is nonzero, our experiments are reproducibility because we have set the random seed (1,42,50) to obtain the experimental results.
GPT-3.5-Turbo We keep a consistent temperature with Llama2, temperature = 0.6, max_tokens
7https://github.com/facebookresearch/llama
= 256, frequency_penalty = 0.9 and presence_penalty = 1.9. We leave other hyperparameters to be default values.
Mistral-Instruct-v0.1 The model size of MistralInstruct-v0.1 is 7b. We use temperature = 0.6, max_new_tokens = 256, do_sample = True, top_p = 5 as hyperparameters for generation. Note that even if temperature is non-zero, our experiments are reproducibility because we have set the random seed (1,42,50) to obtain the experimental results.
Terms of use for each model We carefully follow the terms of use provided by the model authors or company.
• Llama2: https://ai.meta.com/llama/ license/
• GPT-3.5-Turbo: https://openai.com/ policies/terms-of-use
• Mistral-Instruct-v0.1: https://mistral. ai/terms-of-service/
Computing infrastructure For offline models (Llama2 and Mistral-Instruct-v0.1), we conduct our experiments using a mixture of NVIDIA RTX A5000 and NVIDIA RTX A6000 graphic cards. For each experiment involving Llama2, we use one A6000, two A6000, and eight A5000 GPUs respectively for each model size 7b, 13b, and 70b, and we use one A6000 GPU for Mistral-Instructv0.1.

heading: C.2 Email Classification
text: To label the application outcome stated in the generated emails, we adopt a combination of manual and automatic annotation. We manually label 1, 200 application outcome emails in the early iterations of our experiments, evenly distributed across genders and races/ethnicities. We then train a support vector machine (SVM) model with TF-IDF features (Ramos et al., 2003) using 840 samples from
the manually labeled data. We use 180 for validation, and 180 for testing. This classifier achieves 0.97 accuracy on the test set containing 180 samples, also evenly distributed across demographic groups. Given the good performance of the classifier, we use it to label other generated emails.
Because the classifier is not trained on the exact phrasing of all our base templates, we further manually annotate the application decision in the same random subset used for validity analysis and check the human labels with the model predictions. The classifier performs extremely well even though the input template contains variations, achieving an F1 score as high as 0.99 for Llama2-70b, shown in Table 5.

heading: C.3 Permutation Test
text: To measure if a group is treated significantly more or less favorably in comparison with the overall acceptance rate, we conduct an adapted version of the permutation test (Caliskan et al., 2017; An et al., 2023). Considering one demographic group A out of the whole population in our study, our null hypothesis is that A has the same acceptance rate as the global population under the same setting. We first compute d, which is the difference between the average acceptance rate of group A and that of the global population. We then permute the identity labels of the whole population, obtaining A′, which has the same cardinality as A. We find d′, the new difference between the average acceptance rate of A′ and that of the global population. The p-value is estimated by repeating the permutation step for a large number of times (5, 000 in our experiments) and calculating P (d′ > d).
We note that in Table 1, we conduct separate permutation tests for each individual job first, and then combine the p-values using Fisher’s
method (Fisher, 1928) to obtain the aggregate statistical significance across multiple occupational roles.

heading: Future Work
text: While warning of the potential harms of AI usage in decision-making is beneficial if it prevents harmful usage, there is a potential risk that the resulting stigmatization of LLMs could prevent its future adoption in settings where it could be used to advance social equality