Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

1 Introduction . The advent of deep learning and massive growth in training data have led to natural language models surpassing humans on numerous benchmarks [ 1 , 22 , 39 , 40 ] . However , as Bender et al . [ 7 ] states , these models can exacerbate existing biases in data and perpetuate stereotypical associations to the harm of marginalized communities . Simultaneously , pre-trained models have become readily accessible via open source libraries such as HuggingFace , allowing non-experts to apply these tools in their own applications . These developments in generative language models substantiate a need to understand the potential for biases towards protected classes , such as gender and ethnicity . This paper considers potential biases present in the most popular and most downloaded versions of large-scale , open sourced text generation models applied ‘ out-of-the-box ’ . Despite the release of newer and larger models often redirecting researchers ’ attention , there exist important research gaps in existing models . Bearing in mind that the potential negative total impact from biased models is correlated with number of downloads of that model , this paper tests the biases in the small GPT-2 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . model , which is the most downloaded text generation model on HuggingFace with over half a million downloads in the month of May 2021 alone . These numbers motivate further research on the biases of these models given their increased use in hiring related downstream tasks , such as chatbots or unsupervised scanning of CVs and applications [ 30 ] . Within this context , specifying which biases to analyze is crucial ; Blodgett et al . [ 9 ] find that a majority of NLP papers investigating bias are unclear in their articulations of bias . In this paper , we consider both representational and allocational harms [ 4 ] . We attempt to elucidate representational harms , or those harmful in their own right , by highlighting occupation-related stereotypes that may propagate negative generalizations about particular social groups . For example , women ’ s higher likelihood of being associated with care-oriented occupations may perpetuate unwanted stereotypes . Especially within the context of occupations , such associations may lead to allocation harms . Frequent stereotypical association of certain demographic groups with a subset of occupations may lead to conditioned expectations in job hiring where a certain individual is predicted to be well-suited for a job based on their demographics [ 20 ] . In this paper , we generate 396K sentence completions using GPT-2 with default parameters to assess which occupations GPT-2 preferentially associates with intersections of gender and protected classes . We further compare these to real-world occupation data from the US Labor Bureau to map model biases to systemic societal biases . This paper provides the following contributions : ( a ) a detailed data collection protocol for studying intersectional biases in generative language models ; ( b ) the analysis of biases present in GPT-2 for gender intersected with ethnicity , religion , sexuality , political affiliation , and continent name origin ; and ( c ) a comparison of GPT-2 ’ s predictions with ground truth occupation distribution as observed in US labor market data.1 2 Related Work . Bias in NLP models . Negative generalizations , stereotypes , or misrepresentations of particular social groups can be learned by generative language models . Extensive research has shown that unrestricted training of natural language models can inherit human biases and , in some cases , amplify them [ 11 , 13 , 19 , 43 ] . Previous papers have worked to identify , quantify , and de-bias context-independent word embeddings such as Word2Vec and GloVe [ 11 , 15 , 44 ] . Researchers have also attempted to quantify and mitigate biases in transformer-based generative language models , such as ELMo [ 44 ] and BERT [ 8 , 26 ] . Nadeem et al . [ 28 ] evaluate stereotypical bias in various language models , including XLNet and GPT-2 , related to domains of gender , profession , race , and religion , although they do not consider the intersections of these domains . We extend these domains by additionally considering continental name origin and political affiliation , intersected with gender . Probing language models . Our work utilizes what Solaiman et al . [ 35 ] term ‘ bias probes ’ - a battery of inputs to large language models to produce certain outputs in order to map the biases learned by these models . The authors generate hundreds of samples for several potentially problematic prompts ( i.e. , “ Police describe the suspect as ... ” , “ The victim was ... ” ) and determine the gender and race attributes for the resulting outputs . Most similar to our work , Sheng et al . [ 33 ] utilize prefix templates ( i.e. , “ X worked as ... ” , “ X had a job as ... ” ) , including those related to occupation contexts . They generate 100 samples for each prefix template and demographic combination ( gender intersected with ethnicity and sexuality ) and analyze bias in GPT-2 by using sentiment score as a proxy for bias . We extend such work by conducting an empirical analysis of the sentence completions within the specific context of bias towards occupational associations . In our paper , we focus on one sentence template to reduce variation in returned occupations while keeping sentence semantic structures fixed . Unlike [ 33 ] , we do not introduce potentially noisy sentiment classification , instead directly analyzing the statistical distributions of returned occupations . Further , we generate an order of magnitude more samples than [ 33 , 35 ] for greater statistical robustness . Lastly , unlike previous work , we compare the returned occupational associations from our completed prefix templates to real-world US labor market data . We choose the proposed protocol to evaluate biases in text as it is best suited for probing generative language models in their most “ natural ” form , in which sentence completions are generated . In contrast to this approach , embedding association tests , such as the Word Embedding Association 1Materials and data are available at https : //github.com/oxai/intersectional_gpt2 . Test ( WEAT ) [ 13 ] , would require more heuristic choices , as they have been found to be highly dependent on the initial selection of seed words [ 2 ] . Coreference resolution methods , such as Zhao et al . [ 44 ] , suffer from frequent ambiguities and unstated assumptions [ 10 ] . Finally , information theoretic approaches , such as Rudinger et al . [ 32 ] , require a pre-generated corpus and thus would confound the ( template-based ) generation with the bias measurement . Intersectional biases . As Crenshaw [ 14 ] explains , intersectional biases are a necessary consideration because a single axis of analysis treating gender and race as mutually exclusive categories distorts the reality of marginalized communities ( such as Black women ) . More recently , Foulds and Pan [ 17 ] provides definitions of fairness in machine learning systems informed by the framework of intersectionality . The intersections between gender and racial biases have been studied in sentiment analysis [ 25 ] and language models such as BERT and GPT-2 [ 36 ] . As well as race and gender , we extend our analysis to intersections with other legally protected categories that have historically been subject to discrimination : religion , sexuality , and political affiliation . 3 Methods . 3.1 Model Choice . As of May 2021 , the 124M-parameter version of GPT-2 was the most downloaded text generation model on HuggingFace2 , with 526K downloads ; the second most downloaded model , XLNet [ 42 ] , had 167K downloads ( see Appendix B ) . Therefore , we focus our analysis on the small GPT-2 model , licensed under the Apache License , Version 2.0 . Our intent is not to show how an optimized model with tuned hyperparameters predicts job distributions , but how an ‘ out-of-the-box ’ default model used by non-expert users could unintentionally propagate bias . Therefore , we keep the inference hyperparameters fixed to their default values ; in particular , the top_k parameter and the decoder temperature . For completeness , we conduct a brief ablation of these hyperparameters to assess their effect on the diversity of the returned jobs in Appendix C. As a further robustness check , we analyze XLNet , the second most downloaded text generation model , with the same prefix-templates and verify that our results are consistent across models ( see Appendix E ) . 3.2 Data collection Table 1 : Summary table of data collection showing the number of calls per category and per variant ( Var ) . The total number of calls is 396,000 . Category Var Calls Total Calls Cum.Share Base 2 7,000 14,000 81 % Ethnicity 8 7,000 56,000 82 % Religion 10 7,000 70,000 84 % Sexuality 4 7,000 28,000 83 % Political 4 7,000 28,000 82 % Continent 200 1,000 200,000 76 % Our data collection pipeline is shown in Fig . 1 . We prompt GPT-2 using prefix templates similar to those introduced by Sheng et al . [ 33 ] . Identity-based templates . Our prefix templates are of the form “ The [ X ] [ Y ] works as a . . . ” , where X is one of the following protected classes : ethnicity , religion , sexuality , and political affiliation , and Y is ‘ man ’ or ‘ woman ’ . For a baseline to intersectional effects , we leave X blank ( i.e . “ The man/woman works as a . . . ” ) .3 The ethnicity and religion classes used in this paper correlate with the top ethnicities and religions in the US , as we situate our analysis with US data . Using these 28 unique templates ( Tab.1 ) , we generate 7,000 sentences using GPT-2 . Generated sentences are limited to a maximum length of 10 words to capture immediate occupation associations . Name-based templates . An additional prefix template is created of the form “ [ Z ] works as a . . . ” , where Z is a name sampled from the most popular male and female first names per country , obtained from Wikipedia [ 41 ] . We aggregate names into five geographic groups : Africa , Americas , Asia , Europe , Oceania . We sample 20 names for each geographic group and gender pair , yielding 200 unique templates , from which we generate 1,000 sentences each . By prompting GPT-2 with templates devoid of inherently gendered or racialized terms , such as ‘ man/woman ’ or ‘ Asian/Black ’ , we can 2https : //huggingface.co/models ? pipeline_tag=text-generation 3We discuss the implications of the binarization of gender in Sec . 5 and Appendix A better examine the latent associations when GPT-2 estimates the ethnicity and gender from first names . Occupation entity recognition . For each generated sentence , we use the Stanford CoreNLP Named Entity Recognizer ( NER ) [ 27 ] to extract job titles . NER was unable to detect titles for some sentences which were removed from the dataset , losing 10.6 % of identity-based sentences and 19.6 % of name-based sentences . We then create a one-hot encoded frequency matrix for returned job tokens , combining duplicate jobs ( e.g . nurse/nurse practitioner ) . However , we do not merge job tokens with inherent hierarchies ( e.g . assistant professor/professor ) or implicit gender associations ( e.g . salesman/salesperson , waitress/waiter ) . Sentences returning multiple titles ( e.g . “ The woman works as a waitress and a maid ” ) were treated as two separate entries in the frequency matrix given that individuals can have more than one job .

Summary: This paper focuses on probing GPT-2, as a text generative model, for the potential bias in its generated content. The authors used a template-based data collection approach (feeding GPT2 with pre-designed sentence parts and asking the GPT2 model to complete the sentence, using the default parameter). Particularly the authors presented results on: 1. Bias found in occupations between women and men 2. Intersectional interactions (such as religion + gender) have more significant effects on bias then just gender alone (shown by a series of logistic regression analysis) 3. Comparison with occupational statistics from US Labour Bureau showed that GPT2 even pulled back the effect of bias, considering the imbalance in occupational distribution in the real world.  While I appreciate the authors efforts to investigate bias (esp considering intersectional marginalized community), I recommend "a weak accept" of the paper in its present form, due to the lack of clarity in its presentation and several issues (please refer to weaknesses in the main review section). That said, I’d be willing to increase scores upon reading the authors’ rebuttal.