Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper analyzes the intersectional occupational bias in GPT-2. The authors propose a data collection protocol involving the generation of sentences using GPT-2 using identity-based and name-based templates, and the subsequent extraction of occupations using named entity recognition. Bias is analyzed with respect to gender, intersected with ethnicity, religion, sexuality, political affiliation and continental name origin. By comparing the distribution of predictions with US labor market distributions, the authors find that GPT-2 reflects, and sometimes corrects, the skewed societal distributions found in US labour market data.

The paper's contributions are the following:
- A bias probing and statistical analysis protocol
- Analyzing more categories for intersectional bias: ethnicity, religion, sexuality, political affiliation and continental name origin
- As far as I am aware, a new comparison of model predicted distributions with real-world societal distributions
main_review: **Originality**

The question of examining bias in pre-trained language models is not new. However, the authors extend the use of prefix templates for conditional language generation (Sheng et al., 2019) by including more categories, and including a statistical analysis. The comparison of pre-trained langauge model predictions to real-world societal distributions is also a new contribution to the community.

I am chiefly concerned that the paper does not compare its proposed bias probing and statistical analysis protocol with existing methods (eg. embedding association tests, coreference resolution). While the authors do recognize this body of work exists and they cite them, I would have liked to see explanations of why their proposed protocol is better.

N.B. A comparison to sentiment classification in [21] is done and I agree with the authors, but is still insufficient to me.

- Can you evaluate your proposed bias detection protocol in relation to other methods (eg. embedding association tests)? Why do you think this is better?

**Quality**

The methods used are appropriate, and experimental results are well substantiated with comprehensive data and suitable illustrations. This paper is a complete work. Justifications for certain methodological choices (like 2019 labor data, proportional scaling, US-centricity) are astutely included and the authors are transparent with limitations of the paper.

- I am not convinced that one contribution of this work is to evaluate GPT-2 out-of-the-box. To my understanding, most other papers evaluating bias in large language models also do not tune hyperparameters before use, and the choice of model is often whichever was most salient at the time of experimentation.

**Clarity**

The paper is generally well written and structured. The provided appendix is also comprehensive.

- Line 99: missing link
- Line 166: should this be Pred(i, c)?
- Line 353: "going to made readily available"

**Significance**

Comparison with real-world societal distributions is an important contribution for the community, and will inform the debate on whether generative language models should reflect the real world or an ideal type world. In particular, the following results are useful:
- GPT-2 generated text indicates a greater clustering of women into fewer jobs
- Name origin has less of an effect on the occupation returned by GPT-2 than an explicit categorical intersection
- GPT-2 sometimes under-predicts the extent of occupational segregation, are interesting and useful to the community.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work presents an empirical study of GPT-2 regarding the biases in the generated texts. Based on the generated texts, this work compares the gender and ethnicity distribution with the US Labour Bureau data and found that the distributions in the generated distributions are skewed, which demonstrates the biases embedded in the GPT-2 model. 

main_review: - My first concern about this method is the text generation approach employed when collecting the 396K sentences. How to make sure the data generation process can reflect the intrinsic bias in the GPT-2? In other words, how do we know the template-based method did not misguide the model and then either amplify or reduce the bias in GPT-2?
- Regarding the selection of models, I agree GPT-2 is one of the popular models that has been used for text generation ever since it was released. However, it is unclear which observations are generic and which are only applied to GPT-2. I would like to argue a clear distinction between these two categories. 
- It is an interesting idea to compare with the US Labor Market Data. However, from the perspective of research, I am not how rigorous this method is. For example, how do we know that the generated texts describe real-world situations instead of fictional facts? Besides, even the underlying assumption of this work hold and the data from both sides are consistent, that does not mean the GPT-2 does not have gender bias --- the bias in language is more subtle than some statistics

Overall, I think this work needs more justification on model selection and methodology selection. 
limitations_and_societal_impact: Some limitations are addressed in this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tests occupational biases of GPT-2 (small) by generating many completions of "The X Y works as" (Y is either "man" or "woman", X is some ethnicity, religion, sexuality, or political affiliation word) and "Z works as a" where Z is a name associated with a gender and continent. The generations of GPT-2 are then compared against the US labor market (2019). Among other results, the paper finds that women are concentrated into a smaller number of professions, that some intersectional groups exacerbate this, and then generally the "X Y" prompt generates more skewed distributions than the "Z" prompt. Finally, the paper observes that the predicted probabilities roughly follow the US labor market, tending, in some cases, to "correct" for assymetries somewhat.

main_review: Strengths:

1. The setup is very carefully done. The authors deserve great praise for thinking through details (eg. the COVID detail) and being very up front about assumptions and their consequences. I wish all papers would do as well as this one -- it's truly great to see!

2. The breadth of topics covered is very good, and the focus on intersectional effects is nice to see.

3. The analyses are done well, motivated well, and explained well.

4. The results give some insights into the behavior of these models.
 
Weaknesses:

1. I find myself missing what I should take away from this paper. It is a very nice study, and a very good reporting on the results of a study, but I don't know what to make of it. It's not surprising that GPT-2 encodes occupational stereotypes (we've seen similar results in many other models at this point) and, at least from eyeballing table 4, the intersectional effects seem fairly minor -- these lists look pretty consistent across each group, per GPT-2. It seems almost like GPT-2 is ignoring the race/ethnicity characteristic and just focusing on gender. Similar, in tables 2 and 3, the numbers seem really similar -- which are actually significant/meaningful? In contrast, there seems to be at least somewhat more difference in the actual US jobs proportions. The key question I'd like to see answered in the author response is: What are the take-aways from this study that we don't already know from prior work?

2. Similarly, the consistent result that women occupy fewer jobs according to GPT-2 seems moderately interesting on its face, but it's not clear to me (a) why this is happening or (b) why it matters. Is there, for instance, work in social science, that would explain this phenomenon? Is it true in the real world? Why is this good/bad, and what does it tell us about GPT-2 that's interesting to know? As a counterpoint, I could imagine that this effect doesn't have anything to do with bias in the model, but rather with bias in the US Labor coding. For instance, it could be the case that occupations that are historically occupied by women get lumped together into one profession (eg "nurse" covering many different aspects of the nursing profession) while occupations that are historically occupied by men get fine grained distinctions (eg assistant professor, associate professor, full professor, emeritus professor, affiliate professor, etc.). Any claim about the spread of occupations seems like it would be substantially impacted by this sort of coding issue (see, eg, Bowker and Star's classic book "Sorting Things Out"). (Note that I'm not saying that this is true, just that it's plausible and would need to be accounted for.)



Less significant comments:

 - while the inclusion of intersectional groups other than race/ethnicity is nice to see, because of the labor data the evaluation is really restricted to gender and race/ethnicity, which is unfortunate.
 
 - how is sampling done? how does it match with what other people do typically? (eg nucleus sampling, etc.) 
 
 - The abstract states "downstream tasks" but generating text from a language model doesn't strike me as a "downstream task." I'm not sure what's meant here. This is also stated around l47.

 - line 36 "past month" -- which month is that?
 
 - In the data processing (l125), 10% and 20% of sentences are removed. Are these evenly distributed across categories or is there bias?
 
 - Why only one template?
 
 - Figure 2 is not readable in black and white

 - For some of the occupations that have inherant lexical gender (eg nun, waitress, etc.) it's not at all surprising to see gender skew. Why is this interesting?
 
 - The captions of the figures could be more informative -- for instance, figure 3 should state explicitly that this is from GPT-2.
 
 - The observations around line 224 could be from a defaulting effect. No one says "straight woman" then just say "woman" because straight is the default. This is actually a more general flaw with all of these prompt-based approaches, and the results here suggest it's meaningful.
 
 - around line 272, how does this interact with the data matching and filtering (namely where occupation names are matched)?
 
 - around line 302, this is a deeper question, but there is a big difference between "practicing a religion" and "being specifically called a Buddhist in text." That is, when people write text (which is used to train models like GPT-2) they have some communicative goal. They mention that someone is Buddhist if it is somewhat relevant for that communicative goal. Talking about someone's profession as a monk/nun would also plausibly be relevant to that communicative goal. So there's a big difference between "a person is Buddhist" and "a person is Buddhist and is explicitly mentioned as such in text." This of course confounds all results, but some discussion of this (see also Koller & Bender 2020) would help situate this work. Generally speaking I think the claim around religion is overstated.



limitations_and_societal_impact: The paper does an exemplary job of couching it's limitations. It briefly discusses societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on probing GPT-2, as a text generative model, for the potential bias in its generated content. The authors used a template-based data collection approach (feeding GPT2 with pre-designed sentence parts and asking the GPT2 model to complete the sentence, using the default parameter). Particularly the authors presented results on:
1. Bias found in occupations between women and men
2. Intersectional interactions (such as religion + gender) have more significant effects on bias then just gender alone (shown by a series of logistic regression analysis)
3. Comparison with occupational statistics from US Labour Bureau showed that GPT2 even pulled back the effect of bias, considering the imbalance in occupational distribution in the real world.

While I appreciate the authors efforts to investigate bias (esp considering intersectional marginalized community), I recommend "a weak accept" of the paper in its present form, due to the lack of clarity in its presentation and several issues (please refer to weaknesses in the main review section). That said, I’d be willing to increase scores upon reading the authors’ rebuttal.

main_review: Strengths:

1. The consideration of intersecting gender with other associations and the investigation of their effects are novel and extensively explored. 
2. The authors collected a large number of data points, using GPT2-small, the most commonly used version, and therefore the results are particularly informative to many practitioners.


Weaknesses:

1. Comparing the occupational statistics computed by GPT2 vs those by the United States is very interesting and informative. However, the presentation on the methodology and the subsequent discussion is confusing to me. Particularly from section 3.4, I am not sure what “adj.” in equation (1) means and why “adj. Pred” is appropriate as a scaling factor. Would appreciate it if the authors could clarify and make this section clearer. 
2. The analysis of intersection effects is interesting but I fail to see a clear presentation on statistical significance of these results. It may be clearer if the authors could specify p-values on some regressors and offer some discussions. From Table 3, I also do not believe that average pseudo-R2 is necessarily a meaningful measure for the individual factor.
3. The authors claim the contribution of “benchmarking the extent of bias relative to inherently skewed societal distributions of occupation associations”. However, I have some reservations as 1) the authors did not propose any quantitative measurement to the extent of occupation bias relative to real distributions in society; 2) the authors did not compare any models other than GPT2.
4. Several sections of the paper read confusing to me. There is a missing citation / reference in Line 99, section 3.1. The notation \hat{D}(c) from Line 165, section 3.4 is unreferenced.


limitations_and_societal_impact: The authors made great effort to acknowledge the limitations of their work.
review_title: null

Meta-review: This paper investigates associations between occupations and demographic attributes in text generated by GPT-2 from a narrow range of templated prompts, and shows that GPT-2's generations indeed tend to mirror observed trends in the US labor market, opening up the potential for it to reinforce existing inequities. 

Reviewers raised significant concerns about the novelty and significance of this work, but all agreed that the core claims are sound. During discussion, a consensus formed that it would be helpful to publish this work to help continue conversations about bias in self-supervised models at NeurIPS. 
Decision: Accept (Poster)