Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models
Authors: Hannah Rose Kirk
Yennie Jun
Filippo Volpin
Haider Iqbal
Elias Benussi
Frederic A Dreyer
Aleksandar Shtedritski
Yuki Asano
Venue: NeurIPS 2021 Poster (2021)
Keywords: Natural language processing (NLP)
generative language models
bias
GPT
occupations
intersections

Abstract: The capabilities of natural language models trained on large-scale data have increased immensely over the past few years. Open source libraries such as HuggingFace have made these models easily available and accessible. While prior research has identified biases in large language models, this paper considers biases contained in the most popular versions of these models when applied `out-of-the-box' for downstream tasks. We focus on generative language models as they are well-suited for extracting biases inherited from training data. Specifically, we conduct an in-depth analysis of GPT-2, which is the most downloaded text generation model on HuggingFace, with over half a million downloads per month. We assess biases related to occupational associations for different protected categories by intersecting gender with religion, sexuality, ethnicity, political affiliation, and continental name origin. Using a template-based data collection pipeline, we collect 396K sentence completions made by GPT-2 and find: (i) The machine-predicted jobs are less diverse and more stereotypical for women than for men, especially for intersections; (ii) Intersectional interactions are highly relevant for occupational associations, which we quantify by fitting 262 logistic models; (iii) For most occupations, GPT-2 reflects the skewed gender and ethnicity distribution found in US Labor Bureau data, and even pulls the societally-skewed distribution towards gender parity in cases where its predictions deviate from real labor market observations. This raises the normative question of what language models \textit{should} learn - whether they should reflect or correct for existing inequalities.

Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models
Hannah Rose Kirk"5, Yennie Jun', Haider Iqbal", Elias Benussi', Filippo Volpin', Frederic A. Dreyer', Alcksandar Shtedritski', Yuki M. Asano Oxford Artificial Intelligence Society, University of Oxford hannah.kirk ox..o...c.ukk
Abstract
The capabilities of natural language models trained on large-scale data have in- creased immensely over the past few years. Open source libraries such as Hugging- Face have made these models easily available and accessible. While prior research has identified biases in large language models, this paper considers biases contained in the most popular versions of these models when applied 'out-of-the-box for downstream tasks. We focus on generative language models as they are well-suited for extracting biases inherited from training data. Specifically, we conduct in- depth analysis GPT-2, which the most downloaded text generation model on HuggingFace, with over half million downloads per month. We assess biases lated to occupational associations for different protected categories by intersecting gender with religion, sexuality, ethnicity, political affiliation, and continental name origin. Using a template- based data collection pipeline, we collect 396K sentence completions made by GPT-2 and find: (i) The machine-predieteed jobs are less diverse and more stereotypical for women than for men, especially for intersections; (ii) Intersectional interactions are highly relevant for occupational associations, which we quantify by fitting 262 logistic models; (iii) For most occupations, GPT-2 reflects the skewed gender and ethnicity distribution found in US Labor Bureau data, and even pulls the societally-skewed distribution towards gender parity in cases where its predictions deviate from real labor market observations. This raises the normative question of what language models should learn whether they should reflect or correct for existing inequalities.
Introduction
The advent of deep learning and massive growth in training data have led to natural language models
surpassing humans on numerous benchmarks 1 22 39 40). However, as Bender et al. [7] states, these models can exacerbate existing biases data and perpetuate stereotypical associations to the harm of marginalized communities. Simultaneously, pre-trained models have become readily accessible via open source libraries such as HuggingFace, allowing non-experts to apply these tools in their own applications. These developments in generative language models substantiate need to understand the potential for biases towards protected classes, such as gender and ethnicity.
This paper considers potential biases present in the most popular and most downloaded versions of large-scale, open sourced text generation models applied 'out-of-the-box" Despite the release of newer and larger models often redirecting researchers' attention, there exist important research gaps in existing models. Bearing in mind that the potential negative total impact from biased models correlated with number of downloads of that model, this paper tests the biases in the small GPT-2
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

model, which is the most downloaded text generation model on HuggingFace with over half a million downloads in the month of May 2021 alone. These numbers motivate further research on the biases of these models given their increased use in hiring related downstream tasks, such as chatbots or unsupervised scanning of CVs and applications (30]
Within this context, specifying which biases to analyze is crucial; Blodgett et al. [9] find that a majority of NLP papers investigating bias are unclear in their articulations of bias. In this paper, we consider both representational and allocational harms 4 We attempt to elucidate representational harms, or those harmful in their own right, by highlighting occupation- propagate negative generalizations about particular social groups. For example, women's 's higher likelihood of being associated with care-oriented occupations may perpetuate unwanted stereotypes. Especially within the context of occupations, such associations may lead to allocation harms. Frequent stereotypical association of certain demographic groups with a subset of occupations may lead to conditioned expectations in job hiring where certain individual is predicted to be well-suited for job based on their demographics (00
stereotypes that may
In this paper, we generate 396K sentence completions using GPT-2 with default parameters to assess which occupations GPT-2 preferentially associates with intersections of gender and protected classes. We further compare these to real-world occupation data from the US Labor Bureau to map model biases to systemic societal biases. This paper provides the following contributions: (a) detailed data collection protocol for studying intersectional biases in generative language models; (b) the analysis of biases present in GPT-2 for gender intersected with ethnicity, religion, sexuality, political affiliation, and continent name origin; and (c) comparison of GPT-2's predictions with ground truth occupation distribution as observed in US labor market data!!
2 Related Work
Bias in NLP models. Negative generalizations, stereotypes, or misrepresentations of particular social groups can learned by generative language models. Extensive research has shown that unrestricted training of natural language models can inherit human biases and, in some cases, amplify them
[3 19 43). Previous papers have worked to identify, quantify, and de-bias context-independen
word embeddings such Werd2Vec and Glo [ [5 44). Researchers have also attempted to quantify and mitigate biases in transformer- generative language models, such as ELMo
44 and BERT 8 26). Nadcem et al. 28 evaluate stereotypical bias in various language models, including XLNNe and GPT-2, related to domains of gender, profession, race, and religion, although they do not consider the intersections of these domains. We extend these domains by additionally considering continental name origin and political affiliation, intersected with gender.
Probing language models. Our work utilizes what Solaiman et al. 335 term 'bias probes' battery of inputs to large language models to produce certain outputs in order to map the biases learned by these models. The authors generate hundreds of samples for several potentially problematic prompts (i.e.. "Police describe the suspect as..." "The victim was...") and determine the gender and race attributes for the resulting outputs. Most similar to our work, Sheng et al. [33 utilize prefix templates (i.e., "X worked as..." " "X had a job as..."), including those related to occupation contexts. They generate 100 samples for each prefix template and demographic combination (gender intersected with ethnicity and sexuality) and analyze bias in GPT-2 by using sentiment score as a proxy for bias. We extend such work by conducting empirical analysis of the sentence completions within the specific context of bias towards occupational associations.
In our paper, we focus on one sentence template to reduce variation in returned occupations while keeping sentence semantic structures fixed. Unlike [33. we do not introduce potentially noisy sentiment classification, instead directly analyzing the statistical distributions of returned occupa- tions. Further, we generate an order of magnitude more samples than [33 35 for greater statistical robustness. Lastly, unlike previous work, we compare the returned occupational associations from our completed prefix templates to weal-world US labor market data.
We choose the proposed protocol to evaluate biases in text as it is best suited for probing generative language models in their most "natural" form, in which sentence completions are generated. In contrast to this approach, embedding association tests, such as the Word Embedding Association
Matterials and data available at httppp///gtaaa..aac///////eeooet

---

Test (WEAT) [33, would require more heuristic choices, as they have been found to be highly dependent on the initial selection of seed words 2 Coreference resolution methods, such as Zhao
et al. [44, suffer from frequent ambiguities and unstated assumptions [10]. Finally, information theoretic approaches, such as Rudinger et al. [32], require a pre-generated corpus and thus would confound the (template-based generation with the bias measurement.
Intersectional biases. As Crenshaw [4] explains, intersectional biases are a necessary consideration because a single axis of analysis treating gender and race as mutually exclusive categories distorts the reality of marginalized communities (such as Black women). More recently, Foulds and Pan [17] provides definitions of faimess in machine learning systems informed by the framework of intersectionality. The intersections between gender and racial biases have been studied in sentiment analysis [25] and language models such as BERT and GPT-2 [36]. As well as race and gender, we extend our analysis to intersections with other legally protected categories that have historically been subject to discrimination: religion, sexuality, and political affiliation.
3 Methods
3.1 Model Choice
As of May 2021, the 124M-parameter version of GPT-2 was the most downloaded text generation model on HuggingFace(] with 526K downloads; the second most downloaded model, XLNet [42, had 167K downloads (see Appendix B.. Therefore, we focus our analysis on the small GPT-2 model, licensed under the Apache License, Version 2.0. Our intent is not to show how an optimized model with tuned hyperparameters predicts job distributions, but how an oou--of-the-boo default model used by non-expert users could unintentionally propagate bias. Therefore, we keep the inference hyperparameters fixed to their default values; in particular, the top parameter and the decoder temperature. For completeness, we conduct a brief ablation of these hyperparameters to assess their effect on the diversity of the returned jobs in Appendix C As a further robustness check, we analyze XLNet, the second most downloaded text generation model, with the same prefix-templates and verify that our results are consistent across models (see Appendix E.
3.2 Data collection
Our data collection pipeline is shown in Fig. [I We prompt GPT-2 using prefix templates similar to those introduced by Sheng et al. 33
Table I: Summary table of data collection showing the number of calls per category and per variant (Var). The total number of calls is 396,000.
Identity-based templates. Our prefix tem- plates are of the form "The [X][]] works where X is one of the following protected classes: ethnicity, religion, sexuality, and polit- ical affiliation, and Y is 'man' or 'woman'. : For a baseline intersectional effects, we leave X blank (i.e. "The man/woman works as a ...... " The ethnicity and religion classes used in this paper correlate with the top ethnicities and reli- gions the US, as we situate our analysis with US data. Using these 28 unique templates (Tab we generate 7,000 sentences using GPT-2. Generated sentences are limited to maximum length of 10 words to capture immediate occupation associations.
Category Var Calls
Total Calls Cum.Share
Base
2 7,000
14,000
81%
Ethnicity
8 7,000
56,000
82%
Religion
10 7,000
70,000
84%
Sexuality
4 7.000
28,000
83%
Political
4 7.000
28,000
82%
Continent 200
1,000
200,000
76%
Name-based templates. An additional prefix template is created of the form "[2] works as a where Z is a name sampled from the most popular male and female first names per country, obtained from Wikipedia [41]. We aggregate names into five geographic groups: Africa, Americas, Asia, Europe, Oceania. We sample 20 names for each geographic group and gender pair, yielding 200 unique templates, from which we generate 1,000 sentences each. By prompting GPT-2 with templates devoid of inherently gendered or racialized terms, such as "man/woman' or 'Asian/Black,, we can
hattpe://huginngface http://aac...o/ooe.eeeepppppeeee taagepppteiieettttteeeeetete baag-tex--geneaatiee generatiin
We discuss the implications of the binarization of gender in Sec. and Appendix A
3

---

Alac
Prequency for [X]] [xc]
Holk
Jonch
Modie
The
[X]]]] worlss
Seuuaity
Mhrghh
GPT-2
NER
worles
Arric
Cootbeet
(NAME]] [2]
Figure I: Data Collection Process. We collect 396K responses from GPT-2, and retrieve "titles" via Stanford CoreNLP's Named Entity Recognition (NER) analyze the predicted occupational distribution for various intersectional categories.
better examine the latent associations when GPT-2 estimates the ethnicity and gender from first names.
Occupation entity recognition. For each generated sentence, we use the Stanford CoreNLP Named Entity Recognizer (NER) (27 to extract job titles. NER was unable to detect titles for some sentences which were removed from the dataset, losing 10.6% of identity- based sentences and 19.6% of name-based sentences. We then create one-hot encoded frequency matrix for returned job tokens, combining duplicate jobs (e.g. nurse/nurse practitioner). However, we do not merge job tokens with inherent hierarchies (e-g- assistant professor/professorr) implicit gender associations (e.g. salesman//alessperson. waitress/waiter). Sentences returning multiple titles (e.g. "The woman works waitress and maid") were treated two separate entries in the frequency matrix given that individuals can have more than one job.
3.3 Empirical Analysis
The distribution of returned jobs is highly-skewed with long tails: few jobs comprise significant share and many jobs are mentioned infrequently. Therefore, we apply a lower-bound threshold to focus our analysis, removing tokens mentioned in fewer than 0.25% of total calls, which preserves approximately 80% of the sample (Tab[T). For jobs above the threshold, we run logistic regression on the one-hot matrix and output frequencies to predict p([job] 1 1|X, Y) for the input "The [X][]] works as a [job]". While GPT-2 is a 'black-box' model, this predictive modeling attempts to estimate how intersectional categories change GPT-2's prior on the probability of job associations. By using interaction terms, we can study whether intersectionality has additional influence beyond main effects (e.g. the isolated effects of gender and ethnicity). The logistic regression equation includes 'man' from the bascline case the reference group, with dummy variables added for woman, for each intersectional category C, and for interaction terms:
log odds(p(job,le)) = Bu + 31 Woman, + Σ Yie Category ic
Eec (Categogo Wooma;) + 6,+
where log odds(p) log(p/(1 p)) is the log-odds ratio of probability p.
3.4 Comparison with US Labor Market Data
comparison of GPT-2's predictions to the true labor market distribution requires recent data disaggregated by gender and intersection for granular set of occupations. The 2019 US Labor Bureau Statistics from the Current Population Survey [37] reports the gender and ethnicity shares of
workers in 567 occupational categories We recognize a number of limitations of this data, which we
WWe consider the 2019 data better comparison than 2020 as it excludes influences from the COVID-19 pandemic and GPT-2 has not been retrained since.

---

0.8
Wo thn (Average)
Nen (Average)
0.8
0.0
accountoor
0.00
0.0
excount 97% wamaon
acount 90%
0.0
10
36+
Log|Rank)
Figure 2: GPT-2 occupational stereotyping. GPT-2 stereotypes the occupational distribution of women more than that of men. The graph shows the share of occupations for each gender, sorted from most frequent to less frequent.
address in the discussion. However, using US data may provide an appropriate baseline comparison: 50% of Reddit traffic comes from the US, and a further 7% from Canada and the UK each (34). Given that US sources form majority in GPT-2's training material, and that no other major country had data available disaggregated by gender, ethnicity and granular job categories, consider the US dataset a satisfactory first benchmark.
We first select the 50 most frequently mentioned jobs by GPT-2. Then from these, match GPT-2's returned tokens to real US occupation titles, finding correspondences for 44/50 titles (sce Appendix [D. We compute GPT-2's predicted proportional representation for each gender-ethnicity pair, assuming the percentage of women equal across ethnicities. The 'predicted' labor force has equal representation across groups because generate the same number of sentence prompts per pair (n = 7,000). The real-world distribution is not so evenly balanced by demographic group, so the predicted proportions are scaled by the true distribution of gender and ethnicity reported in the
US Labor Statistics and summarized in Appendix The scaling factor is 7(c) G(c) where D(c)
G(c), E(c) are the gender- and ethnicity-shares of the US data, respectively and D(c) 12.5% is our artificial "population" -share. Hence the adjusted prediction given by:
adj. Pred(i, e) (c) Pred(i,
where Pred(i, c) is the share of job for characteristics c. For jobs reported in the US data, we calculate the difference between the predicted proportions and the true proportions.
4 Results
We analyze the effect of gender on returned occupational distributions in Sec. 4.1 and on particular occupations in Sec. 4.2 We extend these analyses intersectional associations in Sec. 4.3 with empirical results derived from logistic regressions. Finally, we compare and quantify the predicted distributions against ground truth US occupation data in Sec. 4.4
4.1 Gender differences in distributions
Fig. 2 ranks the frequency jobs against the cumulative share. While 16 jobs account for 50% of the outputs for men, only jobs account for the same share for women. Similarly, at the 90% level, men are associated with more jobs than women (66 vs 43, respectively). This suggests that GPT-2 predicts wider variety of jobs for men and narrower set of jobs for women. The Gini coefficients in Tab. confirm this more unequal distribution for women.
4.2 Gender differences in occupations
In addition to distributional differences, the set of returned jobs also differ by men and women. In Fig. we show the proportion of genders in all jobs mentioned more than 35 times for baseline
G = 12 (2i and i is the rank is ascending order.
(n2:)/(n En_ Iz), where x is the observed value, is the total values observed,
5

---

man and woman. We make two observations: first, there is greater number of jobs dominated by men as compared to women, reflecting the greater diversity occupations for men. Second, the occupations seem stereotypical: men are associated with manual jobs such as laborer, plumber, truck driver, and mechanic, and with professional jobs such as software engineer, developer and private investigator. Women are associated with domestic and care-giving roles such babysitter, maid and social worker. Furthermore, over 90% of the returns for "prostitute" were women, and over 90% of returns for 'software engineer' were men. We only find three jobs for which GPT-2's outputs suggest a gender-neutral prior over occupations: writer, reporter, and sales representative.
4.3 Intersectional analysis
The Gini coefficients (Tab. for gender- intersection pairs indicate a greater clustering of women into fewer jobs across all intersec- tions, especially for sexuality, religion and eth- nicity. We thus ask the question, how im- portant are gendered intersections in deter- mining the job returned by GPT-2? Tab. B presents summary results from 262 logistic re- gressions, which predict the likelihood of job being associated with the demographics in given sentence prompt. We focus on two met- rics indicating how often the addition of regres- sors adds explainability of the outcome: i) The proportions of regressions where the woman dummy and the interactions were significant (p 0.05), and ii) The change in Pseudo-P2 R the addition the woman dummy and the in- teractions Statistical results, including the efficients, for all regressions are in Appendix FF The aggregated results in Tab. 3 show that the woman dummy is frequently significant, most commonly so in ethnicity regressions (71%) and least commonly in political regressions (59%). Adding a woman dummy increases the model R² on average by +3.3% (percentage points), signifying that gender explains additional vari- ation in job prediction. Interactions are signifi- cant in approximately one third of regressions, but the additional increase to R2 is average smaller (+0.4%). There is some variation in the significance of interactions; for example, (women:hispane hispanic) ) and woman:black) are more frequently significant than (woman:white) and (woman:lesbian) is more frequently significant than { /woman:straigtt These results suggest that some intersections are more salient in chang- ing the returned job from given sentence prompt, and may anchor GPT-2 on a stereo- typical occupation set. In general, across wide range of jobs, gender and intersectional- ity are significant determinants of the job token returned by GPT-2. Knowing that gender and intersectional associations quantitatively important for conditioning GPT-2's probability distribution over jobs, we next ask what jobs are over-represented in one gender for each intersectional category? We calculate distance to the equi-proportion baseline
Table Gini coefficients of rank-frreuueee frequency distri- butions returned by GPT-2.
Gini
Relative Coeff
Gender
Intersec.
Coeff
Base M = 100 %
Man
Base
0.933
100
Man
Religion
0.929
99.57
Man
Sexuality
0.935
100.21
Man
Ethnicity
0.939
100.64
Man
Political
0.942
100.96
Woman
Base
0.951
101.93
Woman
Political
0.951
101.93
Woman
Ethnicity
0.956
102.47
Woman
Religion 0.956
102.47
Woman
Sexuality
0.958
102.68
Table 3: Aggregated logistic regression results.
We fit total of 262 logistic regressions and report the number of times the independent variables con- tributed significantly to the logistic model, well as their average contribution to the Pseudo-R2 R².
#Jobs Variable
Pet. Signif AR2
woman (w.)
0.71
3.22
waaasian
0.29
Ethnicity
w.rblack
0.36
0.40
w.thispanic
0.38
w.:white
0.16
woman (w.)
0.61
3.31
w.rbuddhist
0.19
w.cchristian
0.27
Religion
64
w.chindu
0.27
0.39
w.jewiih
0.33
w.mmuslim
0.25
woman (w.)
0.61
3.36
Sexuality
72
w.:leshian
0.35
0.45
w.sstraight
0,26
woman (w.)
0.59
3.47
Political
71
w.cconserv. 0.24
0.46
w.:liberal
0.30
"We use the McFadden H² which is calculated by comparing the log- likellhodood of a model with no predictors Lo. versus the log- likelihood of the estimated model LM:: Riger = In(Las) In(Lo)

---

100% Women
Male-dommintedd jobs
Gender Parity
Hemaleedorrnate jobs
100% Men
cash
pumte
cour
modratt
controtto
supervice
jounnalss
borber
bus drivar
tochiitton
cleck
sccumn quara
compunes prog pogrammet
walter
corpentes
conetr werket
polies office
mirr
decte
Bartani
Vitha
cook
caiter
ao aoce sone mas
modes
Figure 3: Fundamentally skewed GPT-2 output distributions. We show the gender proportions when querying for the base case, i.e. X = =0. Y (Man, Woman] } and present all jobs with greater than 35 n 0.25% mentions, making up 81 % of returned sentence prompts.
given by (1/lcl,0) 0) to (0, 1/|c),, where |e| the number of choices for intersection c. We normalize this baseline such that 1/lc| = 1x so that jobs lie on this line if adding intersections has no effect on the gender ratio.
For illustrative purposes, we compare the following two examples: religious intersection from the identity-template, which has the greatest man-woman dispersion to the equi-proportion baseline; and continental name-origin from the name-template, which has the least dispersion. We present the analyses for all remaining intersections in Appendix G We first consider religious intersections (Fig. 5). For Christian, Buddhist, and Jewish religions, GPT-2 generates occupations with a large wee-reppeeennattion factor towards one gender, especially for professional religious occupations: nuns are dominated by Buddhist women, rabbis are dominated by Jewish men, and monks, pastors, and priests are dominated by Buddhist and Christian men. Hindu men and women predominately have associations with non-religious professions (e.g. bouncers and massage therapists). We compare this with continent name origin intersections (Fig. (6), for which jobs are more closely distributed to the equi-proportion baseline. These findings suggest that name origin has less of an effect on the token returned by GPT-2 than when adding an explicit categorical intersection (e.g. ethnicity or religion).
From these quantitative and qualitative analyses, we have demonstrated that stereotypical jobs are associated with men and women, and that the of male- and female-dominated jobs changes with the addition of intersections like religion and sexuality. However, it remains to be seen whether GPT-22's 'stereotypical associations' directly reflect, exacerbate, or correct for societal skew given the unfortunate reality that jobs are not evenly distributed by demographic group.
4.4 Comparison to Labor Market Ground Truth
For given job, how well does GPTT2 predict the gender-ethnicity split? There three possible cases: GPT-2 overestimates the true representation of women in female-dominated jobs (exacerbates societal skew), GPT-2 matches the true proportional representation (directly inherits skew), or GPT-2 underestimates the true proportional representation (corrects for skew). In Fig. 4 we find that most predicted values lie close the ground-truth given by the identity line, indicating high accuracy in prediction. We use two quantitative measures of the relative deviation of GPT-2 predictions to US ground truth: mean-square error (MSE) and Kendall-Tau (K7) coefficient 22 For the baseline woman group, the KT coefficient is 0.628, indicating strong positive monotonous association, which is significant at the 1% level. The KT coefficients for all gender-ethnicity intersections also indicate strong positive association, and are all significant at the 1% level (see Appendix [}. The low MSEs shown Fig. corroborate the considerable degree of similarity between GPT-2's s predicted distribution and the ground truth distribution. Furthermore, GPT-2 pulls the distribution further from the extremes by ander-prediction the extent of occupational segregation. This is demonstrated by the fact that GPT-2 predicts higher proportion of women than the ground truth in male-dominated jobs with less than 25% women-share (on average +8)7%) and predicts lower proportions of women in jobs with more than 75% women-share (on average -6.5%). The exceptions to this pattern are courier, bus driver and photographer, for which GPT-2 under-predicts the proportion of women, and social worker and model, for which GPT-2 over-predicts the proportion of women.
7

---

Womeet
1.0
babysilter
022
receptional
Mayt
sácial warke
model
0.6
Cennoby
lecking
phetographir
Triarage therasit
Share Women
1o Za
bus driv
cook
Weman
[0.0238]
+sian [0.0004]
Figure 5: Man- Woman Occupational Split by Religion
garand
hee
0.2
+ Black [0.0013]
Woman
plutnue
truck priver
*Hispanic [0.0049]
Africa
Ameriis
0.0
mechanic
Carpenter
+ White [0.0182]
Mi
Eurapc
Octorra
0.0
0.2
0.4
0.6
0.8
1.0
Mays
Share (GPT-2 pred.)
Figure 4: GPT-2 Monte-Carlo prediction ground truth US population share for gender-ethnicity in- tersections. GPT-2's predictions with regards to inter- sectional characteristics are highly stereotypical yet they are closely aligned to the US population data. We show the predicted values for gender intersected with ethnicity along with the [Mean- -quared Errors] and annotate example jobs for the gender-only predictions.
64
Rolpttoii
Over per-repprsentation Pactor (Womes)
Figure Man- Woman Occupational Split by Continental Name Origin
For a given gender-ethnieity pair, how well does GPT-2 predict the top jobs? This question aims to answer the extent of stereotyping in GPT-2's predictions. Tab. fisows the top five predicted and ground truth jobs for each intersection. GPT-2 predicts high proportion of baseline women to be waitresses (14%) but only Hispanic women have waitress in the top five occupations, according to the US Labor data. While GPT-2 predicts 18% of Hispanic women to be waitresses, reality only 3% of Hispanic women in America work as waitresses. Some of this strong association may be because waitress is inherently gendered job. GPT-2 also over-predicts the number of nurses, predicting 1 11% of women to be nurses when in reality only about 4% of American women are nurses. Security guard is consistently over-predicted for men of all ethnicities. Yet security guard only appears as top job for Black men and at a lower frequency (2%) than the predicted frequency (8%). GPT-2 over-predice the proportion of janitors for all ethnicities, especially for White and Asian men, for whom janitor does not appear as top job.
The share taken up by the most popular occupation for each gender is significantly higher for women (waitress at 14%) than for men (security guard at 8%). The cumulative share of the top five occupations is 41 % for women, which is more than double the ground truth observation (17%). While GPT-2 also over-predicts the cumulative share of top five occupations for men, the discrepancy to US data is smaller (24% vs 10%). The comparison US data corroborates our previous finding of GPT-2's 2's tendency to associate women with small set of stereotypical jobs (Fig. 2 and Tab. 2
Discussion
Demographic distribution per occupation. Overall, we find strong differences in the occupational tokens returned by GPT-2 for gendered sentence prompts. At first glance, it may seem biased that GPT-2 predicts so many women to maids or secretaries and so few to be plumbers or truck drivers. However, in fact, the model predicts less occupational segregation by gender as compared to the US ground truth distribution. some cases, appears that GPT-2 is pulling the skews of the distribution found in reality towards gender parity.
8

---

Table 4: Top five jobs per intersectional category with associated proportions of cumulative
GPT-2
US
Jobs (Prop)
Sum
Johs (Prop)
WOMAN
hase
waitness (0.14), nurse (0.11), maid (0.06), receptionist
0.41
tencher (0.044, nurse (0.04), secretarylassistamn (0.03),
0.17
(0.05), teacher (0.05)
cashier (0.03), manuger (0.03)
Asian
waitress (0.14), maid (0. 11), nurse (0.08), teacher (0.05),
0.42
nurse (0.05), personal appearance worker (0.04), eashier (0.03). sccountan/auditor (0.03), manager (0.03)
0.18
receptionist (0.04)
Black
waitress (0.18), murse (0.19), maid (0.07), prostitute (0.05).
0.44
nursing/home bealth sid (0.07), cashier (0.04),
0.21
teacher (0.04)
(0.04), personal care (0.03), teacher (0.03)
Hisuaicc
waitress (0. 16), nurse (0.14), receptionist (0.07), maid
0.48
maidhoonekeeperkkeenee
(0.05), cashier (0.04),
0.18
(0.07), teacher (0.04)
waiter/waitres (0.03). secrtaayylassistant (0.03).
mussing/oome aide (0.03)
White
waitress (0.17), nurse (0.1 1),1), maid (0.07), teacher (0.05), receptionist (0.04)
0.44
teacher (0.04), murse (0.04), secrttarylassisaatt (0.04).
0.18
manager (0.03), cashier (0.03)
MAN
base
socurity guard (0.08), manager (0.05), waiter (0.04), jani- tor (0 (0.04), mechanic (0.05)
0.24 manager (0.04), truck driver (0.04), construction laborer
0.14
(0.02), retail sales supervisor (0.02), laborer/ material mover (0.02)
Asian
waiter (0.09), security guard (0.07), manager (0.04), jani-
0.27
software developer (0.11), manager (0.04). physi-
0.21
tor (0.04), chef (0.03)
ciankuuggon (0.02), teacher (0.02), engineer (0.02)
Black
socurity guard (0,08), waiter (0.07), bartender (0:05), jani-
0.29
truck driver (0.06), Jaborev/anatriil mover (0.04), janitor (0.03), manager (0.03), security guard (0.02)
0.18
tor (0:05), mechanic (0.04)
Hispanic
security guard (0.09), junitor (0.07), waiter (0.07), bur-
0.33
constraction laborer (0.06). truck driver (0.04), grounds maintenince worker carpeecrreeeee (0.03), janinoe (0.03)
0.19
tender (0.05), manager (0.05)
White
waiter (0.06), security guard (0.06), janitor (0.05),
0.25
manager (0.04), trick driver (0.04), construction laborer (0.03), retail sales supervisor (0.02), laborce/material mover (0.02)
0.15
chanic (0.04), bartender (0.04)
For ethnicity, GPT-2 accurately predicts the distribution of occupations in real world data with low mean-squared errors, especially for Asian and Black workers. In addition to gender and ethnicity, adding a religious intersection considerably changes the returned jobs, especially for men. For example, GPT-2 predicts 4% of Buddhist men to be monks. There are an estimated 3.75 million Buddhists in the US and approximately 1,000 Buddhist centers and monasteries [23] 29 A back of the envelope calculation shows each of these centers would need employ more than 70 monks each to reach the 4% threshold. Therefore, it is likely that GPT-2 infers too strong of an association between practicing a religion and working in a religious profession. However, the communicative intent of language choice might contribute to this result 66 in that there is difference between person practicing a religion versus being specifically called a Buddhist in text. Supporting this effect, we find intersections with continent-based names have returned occupations which are more similar to those of baseline man and woman. This finding indicates that prompting GPT-2 with explicit intersections like 'Buddhist man' or 'Black woman' changes the probabilities of returned tokens to greater extent than a name prompt where GPT-2 must independently ascertain the demographies of the individual.
The socictal consequences of this finding is a double-edged sword. On one hand, it is reassuring that demographic-speciff, stereotypes are less associated with an individual's name, thus reducing allocational harms from downstream applications such as automated CV screening. On the other hand, it suggests entire demographic groups face blanket associations with potentially damaging and unrepresentative stereotypes, therefore introducing representational harms.
Occupation distribution per demographic. Despite reflecting the gender-ethnicity proportions per real-world occupation, GPT-2 notably displays bias towards predicting greater occupational cluster- ing for women, who are associated with a smaller and less-diverse set of occupations. Occupational clustering is pattern observed in real-world data, For example, Waldman and McEaddy (38] found women were clustered into fewer jobs than men, and more recently, Glynn (18) reported 44.4% of women are employed in just 20 occupations, while only 34.8% men were employed in their top 20 occupations. Occupational clustering has adverse effects on the gender pay gap: semale-dominated industries have lower rates of pay than education so clustering has a devaluation effect on women's 's remuneration [16]. Some of the observed effect of occupational clustering may be artificially enhanced due to a 'coding' bias from official statistics, like the US Labor Bureau statistics, which do not capture women's work in the domestic or informal sector. Beyond statistical misrepresentation, a number of other mechanisms explain why occupational clustering exists in reality such as flexibility of hours, part-time work and career
industries requiring similar levels of skills or

---

breaks 21]; educational constraints [2]; and discrimination or stereotyping of female skills into 'uumele-suited" jobs [5).
Relevant to the last of these mechanisms, we find GPT-2 over-predicts occupational clustering for the top five jobs returned for women as compared to the true clustering present in the US labor force. This is true even if we hold the US labor coding bias fixed (i.e. comparing the same categories predicted by GPT-2 the same categories in the US data). The Gini coefficients confirm that the distribution is more unequal for women than for men. Gender-ethnicity predictions do not deviate much from the predictions for baseline man and woman. This signifies that GPT-2 predicts the occupations for women with less variety than for men, regardless of what ethnicity. Relevant to explaining why GPT-2 might be over- prerirreii occupational clustering, Zhao et al. [44 report that, in the 'OntoNotes' dataset, "male gendered mentions are more than twice as likely to contain job title as female mentions". This dataset includes news and web data, which are similar types of sources to those on which GPT-2 was trained.
Our findings on occupational clustering suggest GPT-2 encodes different kind of bias than that normally discussed in the algorithmic fairness literature. In reality, jobs such as secretaries, reception- ists, and maids do have large share of women, and mechanics, plumbers, and carpenters do have a large share of men. Therefore, GPT-2's bias is not in the jobs associated with women per se, but in the rate which it associates women with such small set of jobs, pattern exacerbated from the ground truth occupation data. In terms of propagating damaging and self-fulfilling stereotypes over cemale-suited jobs, we see this as problematic form of bias in a widely-used language model.
Limitations. This paper is subject to several limitations. First, our comparison to labor market data renders the ground truth baseline inherently US-centric. Second, without consistent, granular data on occupational splits by religion, sexuality, and political affiliation, we cannot comment on how accurately GPT-2 reflects the ground truth for these intersections. Third, we cannot compare jobs in the informal sector, such as 'prostitute', to real world incidences. If terms such as 'prostitute" are commonly used as slurs, GPT-2 may display a bias towards overestimating their proportion. Finally, by focusing only on two genders, the results do not adequately reflect occupational biases which may be associated with non-binary gender identities. Future research recommended to make ground examined
truth comparisons across a broader range of countries against the set of in this paper and comment on a broader spectrum of gender identities. Doing so would valuable in establishing potential areas of bias which risk being inherited by downstream applications of widely-downloaded generative language models such as GPT-2.
Conclusion
What should be the goal of generative language models? It is certainly appropriate that they should not exacerbate existing societal biases with regards to occupational segregation. It is less clear whether they should reflect or correct for skewed societal distributions. Compared to US data, identify a bias towards returning a small number of stereotypical jobs too many times, especially for women. However, for a given job, we find that GPT-2 reflects societal skew and, in some cases, errs on the side of correcting for it. One proposed reason for this observed pattern is over-represennaaioon in the training data towards 'exceptional cases'. If society expects women to be secretaries and nurses, it is possible that there are more training examples scraped from social media platforms or newspaper articles of when men occupy these stereotypes. or vice-versa with plumbers and software developers. This paper explicitly focuses on the most downloaded model for text generation, which potentially has greater tangible impact for inherited downstream biases than the most current and otate-of-the-art models, such as GPT-3, which requires a lengthy application process to be granted access. The contributions of this paper are thus two-fold: analyzing the most downloaded text generation models applied tii-off-thebbee and benchmarking the extent of bias relative to inherently skewed societal distributions of occupational associations. While both HuggingFace and the authors of the original GPT-2 paper (31 include discussion of bias in the model release, these discussions are limited to few illustrative examples intersecting only race with gender. Our paper advises that if such models are going to be made readily available, a greater discussion of their fairness and bias is required across more diverse intersectional associations. This will be necessary so that end users can be fully aware of the potential biases which risk being propagated when using these models "out-of-the-bos:
10

---

Funding Disclosure
This work has been supported by the Oxford Artificial Intelligence student socicty, the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems [EP/L015897/1] (A.S., Y.M.A.), the Economic and Social Research Council grant for Digital Social Science [ES/P000649/1] (..R.K.) and the ERC under the European Union's 's Horizon 2020 research and innovation programme [FUN2MODEL/8341155 (E.B). There are no competing interests.
Acknowledgements
We thank the four anonymous reviewers whose suggestions helped improve and clarify this article. also thank R. Maria del Rio-Chanona and Gesa Biermann for their useful comments.
References
[1] D. Adiwardana, Minh- Thang Luong, D. So, J. Hall, Noah Fiedel, R. Thoppilan. Z. Yang, Apoory Kulshreshtha, G. Nemade, Yifeng Lu, and Quoc V. Le. Towards numan-like open-domain chatbot. ArXiv, 0bb/2001.009977 2020.
[2] Maria Antoniak and David Mimno. Bad seeds: Evaluating lexical methods for bias measurement. In ACL/ICNLP, 2021.
[3] Roxana Barbulescu and Matthew Bidwell. Do women choose different jobs from men? mecha- nisms of application segregation in the market for managerial workers. Organization Science, 24(3):737-756, 2013.
[4] Solon Barocas and Andrew D. Selbst. Big data's disparate impact. Califormia Law Review, 104: 671, 2016.
[5] Andrea H Beller. Occupational segregation by sex: Determinants and changes. Journal of Human Resources, pages 371-392, 1982.
[6] Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, 2020.
[7] Emily M. Bender, Timnit Gebru, Angelina McMillan Majoor and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? . In Conference on Fairness, Accountabilify, and Transparency (FAceT 22). ACM, New York, NY, USA, 2021.
[8] Rishabh Bhardwaj. Navonil Majumder, and Soujanya Poria. Investigating gender bias in bert. ArXiv, abs/2009.05022 2020.
[9] Su Lin Blodgett, Solon Barocas, Hal Daum'c, and H. Wallach. Language (technology) is power: A critical survey of "bias" in nlp. In ACL., 2020.
[10] Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna M. Wallach. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In ACI/JCNNLP, 2021.
[11] Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and A. Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NewrIPS, 2016.
[12] Lex Borghans and Loek Groot. Educational presorting and occupational segregation. Labour Economics, 6(3):375-32) -395, 1999.
[13] A. Caliskan, J. Bryson, and A. Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356:183 186, 2017.
[14] K. Crenshaw.
Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. 1989.

---

[15] Mark Diaz, I. Johnson, Amanda Lazar, A. Piper, and Darren Gergle. Addressing age-related bias in sentiment analysis. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018.
[16] Paula England. Comparable worth: Theories and evidence. Routledge, 1992.
[17] Foulds and Shimei Pan. An intersectional definition of faimess. 2020 IEEE 36th International Conference on Data Engineering (ICDE), pages 1918-1921, 2020.
[18] Sarah Jane Glynn.
Explaining the Gender Wage Gap. may 2014. ///p:///eeeccccr///pppp/////////
URL
https://wwwt ameri org/ httpp//rrrrsrreeessee/////o/r/re wxtp//ipppaaaingggaoaam
90039/ http:/epllaaac aiiiigg the- gender- wage-- gap/
[19] H. Gonen and Y. Goldberg. Lipstick a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. ArXiv, abs// 0tss/193..0862,, 2019.
[20] María José González López, Clara Cortina Trilla, and Jorge Rodríguez. The role of gender stereotypes in hiring: a field experiment. European Sociological Review. 2019; 35 (2): 187-204, 2019.
[21] Anne Grönlund and Charlotta Magnusson. Family- friendly policies and women's 's wages is there a trade-of?? skill investments, occupational segregation and the gender pay gap in germany, sweden and the uk. European Societies, 18(1):91-11) 2016.
[22] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and W. Chen. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654. 2020.
[23] Institute for Genealogical Studies. US: Religious Records-Part 2, 2020. URL BOLLhtts::///
genealogicalstudiee. www. courseg. copu6o00
[24] M. G. Kendall. A New Measure of Rank Correlation. Biometrika, 301-))811 06 1938. ISSN 0006-3444. doi: (0.1093/biomet/3.11222 URL htt.t:tst//.o.org/////// 1tt.:0010////mmme/oee
1-2.81
[25] Svetlana Kiritchenko and Saif M. Mohammad. Examining gender and race bias in two hundred sentiment analysis systems. "SEM@NAACL-LII,, 2018.
[26] Keita Kurita, N. Vyas, Ayush Pareek, A. Black, and Yulia Tsvetkov. Mcasuring bias in contextualized word representations. ArXiv, hts//9906073377 2019.
[27] Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenip natural language processing toolkit. In ACL (System Demonstrations), pages 55-60. The Association for Computer Linguistics, 2014. ISBN 978-1-941643-00-6
[28] Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. ArXiv, abs/2004.094.0 09456, 2020.
[29] Pew Research. Religious Landscape Study, 2020. URL https://www http::/wwwwwwwwwo.uo.//m wwwforumm org/ - waa.ooaalaaaaaaaeestutrrrereeeee ttrp:iigiiiu-rriitiiiiiitiii buddhist/
[30] Jitendra Purohit, Aditya Bagwe, Rishabh Mehta, Ojaswini Mangaonkar, and Elizabeth George. Natural language processing based jaro-the interviewing chatbot. In 2019 3rd International Conference on Computing Methodologies and Communication (ICCMC), pages 33-1366, 2019. doi: 10.1 100.10C.CC...2011.00 200.C.......811111 1..200999881088
[31] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[32] Rachel Rudinger, Chandler May, and Benjamin Van Durme. Social bias in elicited natural language inferences. In EthNLP@ EACL, 2017.
[33] Emily Sheng, Kai-Wei Chang, P. Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. ArXiv, abs/1909.0 01366, 2019.
12

---

[34] Similarweb. Reddit.com traffic, ranking & marketing analytics. similarweb.com hwt.://///bbbbeeeeeddtt.to...
URL https://wwi
[35] Irene Solaiman, Miles Brundage, J. Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, and J. Wang. Release strategies and the social impacts of language models. ArXiv, 0et//908..92003 2019.
[36] Y. Tan and L. Celis. Assessing social and intersectional biases contextualized word represen- tations. In NeurIPS, 2019.
[37] US Labor Bureau of Statistics. Employed peons by detailed occupation, sex, race, and Hispanic or Latino ethnicity, 2019. htt.https:///aa bls. httpssoo//cps/pssaall...t
[38] Elizabeth Waldman and Beverly McEaddy. Where women work-an analysis by industry and occupation. Monthly Lab. Rev. 97:3, 1974.
[39] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In @ EMNLP, 2018.
[40] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: stickier benchmark for general-purpose language understanding systems. In NeurlPS, 2019.
[41] Wikipedia. List of most popular names, 2021. URL List_ wwwosstpoppllr__gie
ottpppeeiiarr....eeeeoooo wiki
names
[42] Z. Yang, Zihang Dai, Yiming Yang, Carbonell, R. Salakhutdinov, and Quoe V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurlPS, 2019.
[43] Jieyu Zhao, Yichao Zhou, Z. Li, W. Wang, and Kai-Wei Chang. Learning gender-neutral word embeddings. In EMNLP. 2018.
[44] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. ArXiv, abs/1904.03310, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See also the Discussion section in Sec. 5
(b) Did you describe the limitations of your work? [Yes] See "Limitations" subsection in
Sec. [S)
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We evaluate a set of pretrained models and highlight potential biases already present in these models. We do, however, note the gaps in our analysis in the Limitations section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Our code and results are provided in a URI..
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A] We did not train any models.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [N/A]
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A] Not relevant, as we did not train any models.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Methods section
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 'r
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rGW881IsvWw", "reviewer": "Reviewer_BGEN", "summary": "This paper tests occupational biases of GPT-2 (small) by generating many completions of \"The X Y works as\" (Y is either \"man\" or \"woman\", X is some ethnicity, religion, sexuality, or political affiliation word) and \"Z works as a\" where Z is a name associated with a gender and continent. The generations of GPT-2 are then compared against the US labor market (2019). Among other results, the paper finds that women are concentrated into a smaller number of professions, that some intersectional groups exacerbate this, and then generally the \"X Y\" prompt generates more skewed distributions than the \"Z\" prompt. Finally, the paper observes that the predicted probabilities roughly follow the US labor market, tending, in some cases, to \"correct\" for assymetries somewhat.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The setup is very carefully done. The authors deserve great praise for thinking through details (eg. the COVID detail) and being very up front about assumptions and their consequences. I wish all papers would do as well as this one -- it's truly great to see!\n\n2. The breadth of topics covered is very good, and the focus on intersectional effects is nice to see.\n\n3. The analyses are done well, motivated well, and explained well.\n\n4. The results give some insights into the behavior of these models.\n \nWeaknesses:\n\n1. I find myself missing what I should take away from this paper. It is a very nice study, and a very good reporting on the results of a study, but I don't know what to make of it. It's not surprising that GPT-2 encodes occupational stereotypes (we've seen similar results in many other models at this point) and, at least from eyeballing table 4, the intersectional effects seem fairly minor -- these lists look pretty consistent across each group, per GPT-2. It seems almost like GPT-2 is ignoring the race/ethnicity characteristic and just focusing on gender. Similar, in tables 2 and 3, the numbers seem really similar -- which are actually significant/meaningful? In contrast, there seems to be at least somewhat more difference in the actual US jobs proportions. The key question I'd like to see answered in the author response is: What are the take-aways from this study that we don't already know from prior work?\n\n2. Similarly, the consistent result that women occupy fewer jobs according to GPT-2 seems moderately interesting on its face, but it's not clear to me (a) why this is happening or (b) why it matters. Is there, for instance, work in social science, that would explain this phenomenon? Is it true in the real world? Why is this good/bad, and what does it tell us about GPT-2 that's interesting to know? As a counterpoint, I could imagine that this effect doesn't have anything to do with bias in the model, but rather with bias in the US Labor coding. For instance, it could be the case that occupations that are historically occupied by women get lumped together into one profession (eg \"nurse\" covering many different aspects of the nursing profession) while occupations that are historically occupied by men get fine grained distinctions (eg assistant professor, associate professor, full professor, emeritus professor, affiliate professor, etc.). Any claim about the spread of occupations seems like it would be substantially impacted by this sort of coding issue (see, eg, Bowker and Star's classic book \"Sorting Things Out\"). (Note that I'm not saying that this is true, just that it's plausible and would need to be accounted for.)\n\n\n\nLess significant comments:\n\n - while the inclusion of intersectional groups other than race/ethnicity is nice to see, because of the labor data the evaluation is really restricted to gender and race/ethnicity, which is unfortunate.\n \n - how is sampling done? how does it match with what other people do typically? (eg nucleus sampling, etc.) \n \n - The abstract states \"downstream tasks\" but generating text from a language model doesn't strike me as a \"downstream task.\" I'm not sure what's meant here. This is also stated around l47.\n\n - line 36 \"past month\" -- which month is that?\n \n - In the data processing (l125), 10% and 20% of sentences are removed. Are these evenly distributed across categories or is there bias?\n \n - Why only one template?\n \n - Figure 2 is not readable in black and white\n\n - For some of the occupations that have inherant lexical gender (eg nun, waitress, etc.) it's not at all surprising to see gender skew. Why is this interesting?\n \n - The captions of the figures could be more informative -- for instance, figure 3 should state explicitly that this is from GPT-2.\n \n - The observations around line 224 could be from a defaulting effect. No one says \"straight woman\" then just say \"woman\" because straight is the default. This is actually a more general flaw with all of these prompt-based approaches, and the results here suggest it's meaningful.\n \n - around line 272, how does this interact with the data matching and filtering (namely where occupation names are matched)?\n \n - around line 302, this is a deeper question, but there is a big difference between \"practicing a religion\" and \"being specifically called a Buddhist in text.\" That is, when people write text (which is used to train models like GPT-2) they have some communicative goal. They mention that someone is Buddhist if it is somewhat relevant for that communicative goal. Talking about someone's profession as a monk/nun would also plausibly be relevant to that communicative goal. So there's a big difference between \"a person is Buddhist\" and \"a person is Buddhist and is explicitly mentioned as such in text.\" This of course confounds all results, but some discussion of this (see also Koller & Bender 2020) would help situate this work. Generally speaking I think the claim around religion is overstated.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does an exemplary job of couching it's limitations. It briefly discusses societal impact.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "j4w06BNy1X", "reviewer": "Reviewer_Gm9u", "summary": "This paper focuses on probing GPT-2, as a text generative model, for the potential bias in its generated content. The authors used a template-based data collection approach (feeding GPT2 with pre-designed sentence parts and asking the GPT2 model to complete the sentence, using the default parameter). Particularly the authors presented results on:\n1. Bias found in occupations between women and men\n2. Intersectional interactions (such as religion + gender) have more significant effects on bias then just gender alone (shown by a series of logistic regression analysis)\n3. Comparison with occupational statistics from US Labour Bureau showed that GPT2 even pulled back the effect of bias, considering the imbalance in occupational distribution in the real world.\n\nWhile I appreciate the authors efforts to investigate bias (esp considering intersectional marginalized community), I recommend \"a weak accept\" of the paper in its present form, due to the lack of clarity in its presentation and several issues (please refer to weaknesses in the main review section). That said, I’d be willing to increase scores upon reading the authors’ rebuttal.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The consideration of intersecting gender with other associations and the investigation of their effects are novel and extensively explored. \n2. The authors collected a large number of data points, using GPT2-small, the most commonly used version, and therefore the results are particularly informative to many practitioners.\n\n\nWeaknesses:\n\n1. Comparing the occupational statistics computed by GPT2 vs those by the United States is very interesting and informative. However, the presentation on the methodology and the subsequent discussion is confusing to me. Particularly from section 3.4, I am not sure what “adj.” in equation (1) means and why “adj. Pred” is appropriate as a scaling factor. Would appreciate it if the authors could clarify and make this section clearer. \n2. The analysis of intersection effects is interesting but I fail to see a clear presentation on statistical significance of these results. It may be clearer if the authors could specify p-values on some regressors and offer some discussions. From Table 3, I also do not believe that average pseudo-R2 is necessarily a meaningful measure for the individual factor.\n3. The authors claim the contribution of “benchmarking the extent of bias relative to inherently skewed societal distributions of occupation associations”. However, I have some reservations as 1) the authors did not propose any quantitative measurement to the extent of occupation bias relative to real distributions in society; 2) the authors did not compare any models other than GPT2.\n4. Several sections of the paper read confusing to me. There is a missing citation / reference in Line 99, section 3.1. The notation \\hat{D}(c) from Line 165, section 3.4 is unreferenced.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors made great effort to acknowledge the limitations of their work. \n", "review_text": "", "extra_scores": {"ethical_concerns": "I commend the authors for their effort in analyzing potential bias inherent in a very popular model. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "GmPWmYQFeND", "reviewer": "Reviewer_Enfk", "summary": "This paper analyzes the intersectional occupational bias in GPT-2. The authors propose a data collection protocol involving the generation of sentences using GPT-2 using identity-based and name-based templates, and the subsequent extraction of occupations using named entity recognition. Bias is analyzed with respect to gender, intersected with ethnicity, religion, sexuality, political affiliation and continental name origin. By comparing the distribution of predictions with US labor market distributions, the authors find that GPT-2 reflects, and sometimes corrects, the skewed societal distributions found in US labour market data.\n\nThe paper's contributions are the following:\n- A bias probing and statistical analysis protocol\n- Analyzing more categories for intersectional bias: ethnicity, religion, sexuality, political affiliation and continental name origin\n- As far as I am aware, a new comparison of model predicted distributions with real-world societal distributions", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThe question of examining bias in pre-trained language models is not new. However, the authors extend the use of prefix templates for conditional language generation (Sheng et al., 2019) by including more categories, and including a statistical analysis. The comparison of pre-trained langauge model predictions to real-world societal distributions is also a new contribution to the community.\n\nI am chiefly concerned that the paper does not compare its proposed bias probing and statistical analysis protocol with existing methods (eg. embedding association tests, coreference resolution). While the authors do recognize this body of work exists and they cite them, I would have liked to see explanations of why their proposed protocol is better.\n\nN.B. A comparison to sentiment classification in [21] is done and I agree with the authors, but is still insufficient to me.\n\n- Can you evaluate your proposed bias detection protocol in relation to other methods (eg. embedding association tests)? Why do you think this is better?\n\n**Quality**\n\nThe methods used are appropriate, and experimental results are well substantiated with comprehensive data and suitable illustrations. This paper is a complete work. Justifications for certain methodological choices (like 2019 labor data, proportional scaling, US-centricity) are astutely included and the authors are transparent with limitations of the paper.\n\n- I am not convinced that one contribution of this work is to evaluate GPT-2 out-of-the-box. To my understanding, most other papers evaluating bias in large language models also do not tune hyperparameters before use, and the choice of model is often whichever was most salient at the time of experimentation.\n\n**Clarity**\n\nThe paper is generally well written and structured. The provided appendix is also comprehensive.\n\n- Line 99: missing link\n- Line 166: should this be Pred(i, c)?\n- Line 353: \"going to made readily available\"\n\n**Significance**\n\nComparison with real-world societal distributions is an important contribution for the community, and will inform the debate on whether generative language models should reflect the real world or an ideal type world. In particular, the following results are useful:\n- GPT-2 generated text indicates a greater clustering of women into fewer jobs\n- Name origin has less of an effect on the occupation returned by GPT-2 than an explicit categorical intersection\n- GPT-2 sometimes under-predicts the extent of occupational segregation, are interesting and useful to the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "A0urr2U1MMa", "reviewer": "Reviewer_FmSE", "summary": "This work presents an empirical study of GPT-2 regarding the biases in the generated texts. Based on the generated texts, this work compares the gender and ethnicity distribution with the US Labour Bureau data and found that the distributions in the generated distributions are skewed, which demonstrates the biases embedded in the GPT-2 model. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- My first concern about this method is the text generation approach employed when collecting the 396K sentences. How to make sure the data generation process can reflect the intrinsic bias in the GPT-2? In other words, how do we know the template-based method did not misguide the model and then either amplify or reduce the bias in GPT-2?\n- Regarding the selection of models, I agree GPT-2 is one of the popular models that has been used for text generation ever since it was released. However, it is unclear which observations are generic and which are only applied to GPT-2. I would like to argue a clear distinction between these two categories. \n- It is an interesting idea to compare with the US Labor Market Data. However, from the perspective of research, I am not how rigorous this method is. For example, how do we know that the generated texts describe real-world situations instead of fictional facts? Besides, even the underlying assumption of this work hold and the data from both sides are consistent, that does not mean the GPT-2 does not have gender bias --- the bias in language is more subtle than some statistics\n\nOverall, I think this work needs more justification on model selection and methodology selection. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are addressed in this paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates associations between occupations and demographic attributes in text generated by GPT-2 from a narrow range of templated prompts, and shows that GPT-2's generations indeed tend to mirror observed trends in the US labor market, opening up the potential for it to reinforce existing inequities. 

Reviewers raised significant concerns about the novelty and significance of this work, but all agreed that the core claims are sound. During discussion, a consensus formed that it would be helpful to publish this work to help continue conversations about bias in self-supervised models at NeurIPS. 
Decision: Accept (Poster)