"Fifty Shades of Bias": Normative Ratings of Gender Bias in GPT Generated English Text
Authors: Rishav Hada
Agrima Seth
Harshita Diddee
Kalika Bali
Venue: EMNLP 2023 Main (2023)
Keywords: Gender bias
NLP
LLMs
GPT

Abstract: Language serves as a powerful tool for the manifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the most pervasive biases in our society and is seen in online and offline discourses. With LLMs increasingly gaining human-like fluency in text generation, gaining a nuanced understanding of the biases these systems can generate is imperative. Prior work often treats gender bias as a binary classification task. However, acknowledging that bias must be perceived at a relative scale; we investigate the generation and consequent receptivity of manual annotators to bias of varying degrees. Specifically, we create the first dataset of GPT-generated English text with normative ratings of gender bias. Ratings were obtained using Best--Worst Scaling -- an efficient comparative annotation framework. Next, we systematically analyze the variation of themes of gender biases in the observed ranking and show that identity-attack is most closely related to gender bias. Finally, we show the performance of existing automated models trained on related concepts on our dataset.

"Fifty Shades of Bias": Normative Ratings of Gender Bias in GPT Generated English Text
Rishav Hada' Agrima Seth Harshita Diddee 3+, Kalika Bali'
Microsoot Research India
School of Information, University of Michigan
CCanegie Mellon University
rishavhada@gmail.. wom.aaaaaaaammmmmmmmmmmmoo. com, agrima@umic edu, ddde@aadddr cmu.edu, kalikabamicrosoot..co
Abstract
work on detecting gender bias has mostly focused on a lexica-based approach or used templatized sentences to create datasets that inform models for downstream tasks (Bhaskaran and Bhallamudi, 2019; Cho et al., 2019; Prates et al., 2020; Moham- mad and Turney, 2013a). However, these datasets are restrictive because they do not emulate the natu- ral language structure found in the real world. This problem is further aggravated by real-world data with bias being sparse and difficult to mine (Blod- gett al., 2021).
Warning: This paper contains statements that may be offensive or upsetting.
I Language serves as powerful tool for the man- ifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the most pervasive biases in our society and seen in online and offline discourses. With LLMs in- creasingly gaining human-like fluency text generation, gaining nuanced understanding of the biases these systems can generate is im- perative. Prior work often treats gender bias binary classification task. However, ac- knowledging that bias must be perceived at relative scale; we investigate the generation and consequent receptivity of manual annotators to bias varying degrees. Specifically, we cre- ate the first dataset of GPT-generated English text with normative ratings of gender bias. Rat- ings were obtained using Best-Worst Scaling efficient comparative annotation framework. Next, we systematically analyze the variation of themes of gender biases in the observed rank- ing and show that identity-attan attack is most closely related to gender bias. Finally, we show the per- formance of existing automated models trained on related concepts on our dataset.
Further, annotation of gender bias is challeng- ing task. Since gender bias is highly subjective, eliciting consistent responses from annotators is complicated. Annotators' perception of bias is heavily influenced by factors such as lived experi- ence, education, community, personal sensitivity, and more (Blodgett al., 2020; Biester al., 2022; Röttger et al., 2022). Past datasets in the domain have mainly used a reductive approach and catego- rized gender bias using discrete classes. However, Hada et al. (2021) shows that there is much to gain from more fine-grained categorization of the con- cept for offensive language detection. We believe more fine-gratd fine-grained categorization of gender bias can aid our understanding of how humans perceive bias. Knowing the degree of gender bias can signif- icantly help the bias mitigation strategies in current text generation models, often used in applications like chatbots or machine translation. For instance, the bias is severe, more aggressive intervention methods might be necessary, like retraining the model on debiased data modifying the loss func- tion. In contrast, a model with milder bias could benefit from post-processing techniques. Similarly, in specific use cases, we might want models gen- erate no biased content or only ban highly severe cases of bias text.
1 Introduction
Harms perpetuated due to human biases are innu- merable, and gender bias is one of the most preva- lent biases in our society. Past work shows the harm due to gender-based bias ranges from under- representation in media stories (Ast et al., 2021) to mis- or no representation in computational models (Bolukbasi et al., 2016; Sun et al., 2019). Recog- nizing the severity of the impact of gender bias in text-based applications, scholarship in NLP has been increasingly focusing on understanding, de- tecting, and mitigating gender bias in the text, Past
Since data collection and annotation is an expen- sive procedure, more so for tasks such as gender bias identification, which deal with data sparsity
"qqual contribution
"Wor done while at Microsoft Research India

---

issues, there's a growing interest in the community leverage the fluent text generation and zero-shot learning capabilities of LLMs like GPT-3.5 and GPT-4 (Wang et al., 2023; Eldan and Li, 2023). These models are also being increasingly used in everyday applications. Therefore, it is imperative to understand the biases they can propagate. In our work, we prompt GPT-3.5-Turbo to generate graded gender-biased text. To ground the genera- tion, we use a list of carefully curated seeds. This serves two purposes: (1) we can navigate data spar- sity issues while still grounding GPT generations to real-world data, and (2) we can understand the biases GPT-3.5-Turks can propagate via its genera- tions. Studying (2) becomes increasingly relevant given that models have been shown to represent opinionation by-product of being trained on poorly representative data (Santurkar et al., 2023).
more less biased. We assess the performance of few neural models used for related downstream tasks on our new dataset. Finally, we also investi- gate GPT-4's s reasoning capabilities to provide an appropriate reason for a given gender bias score.
Related Work
Gender Bias Datasets Existing studies on gen- der bias have relied on datasets that are either (a) templatized sentences or (b) sentences mined using lexical terms and rule-based patterns from web sources. The templatized sentences are structured as [Noun/Pronoun] is a/has a [occupa- tion/adjectives] (Bhaskaran and Bhallamudi, 2019; Cho et al., 2019; Prates et al.. 2020). These tem- platized sentences help elicit associations between gendered terms name, pronouns and occupation, emotions, and other stereotypes. Templatized sen- tences usually have artificial structures and thus have limited applicability to downstream tasks with more natural language. Some prior works mine data from web sources like Wikipedia and Common Crawl (Webster et al., 2018; Emami et al., 2019) to create dataset for coreference resolution. How- ever, many real-world sentences have subtle man- ifestation of biases or use words that themselves do not have a negative connotation. Hence, rule-based sentence mining may not be able to capture the more implicit biases that humans have (Blodgett et al., 2021).
This paper introduces novel dataset consisting of 1000 GPT-generated English text annotated for its degree of gender bias. The dataset includes fine- grained, real-valued scores ranging from (least negatively biased) to I (most negatively biased) normative gender bias ratings for the statements. Notably, this is the first time that comparative an- notations have been utilized to identify gender bias. In its simplest form, comparative annotations in- volve presenting two instances to annotators simul- tancously and asking them to determine which in- stance exhibits a greater extent of the targeted char- acteristic. This approach mitigates several biases commonly found in standard rating scales, such as scale-region bias (Presser and Schuman, 1996; Asaadi et al., 2019) and enhances the consistency of the annotations (Kiritchenko and Mohammad, 2017). However, this method necessitates the anno- tation of 2 (where N the number of items be annotated) instance pairs, which can be prohibitive. Therefore, for the annotation of our dataset, we employ an efficient form of comparative annota- tion called Worst Scaling (BWS) (Louviere, 1991; Louviere et al., 2015; Kiritchenko and Mo- hammad, 2016, 2017).
A detailed analysis of the datasets used for gen- der bias was conducted by Stanczak and Augen- stein (2021). Two more recently created datasets not introduced in the survey are the BUG dataset (Levy et al., 2021) and the CORGI-PM dataset (Zhang et al., 2023). Levy et al. (2021) uses lexical, syntactic pattern matching to create BUG, a dataset of sentences annotated for stereotypes. CORGI- PM is Chinese corpus of 32.9K sentences that were human-annotater annotated for gender bias. Sentences in the dataset annotated as gender-biased (B) non-biased (N). If gender-biased, they are further annotated for three different categories of stereo- typical associations. In our work, to overcome the limitations of strictly templatized sentences and the challenges of mining real-world data, we leverage the text generation capabilities of GPT to create a dataset of graded statements.
Our annotation study provides valuable insights into how humans perceive bias and a nuanced un- derstanding of the biases GPT models can generate. We conduct in-depth qualitative and quantitative analyses of our dataset obtain insights into how different prompting strategies and source selection can affect the generation of statements. We analyze the different themes that humans consider to be
Annotation tasks identify gender bias can be
DDtaset and Code available at: https://aka.... ms/ Fi FiftyhhadsooBiiss

---

categorized under the descriptive paradigm, where capturing disagreements annotation due an- notator identity like gender. race, age, etc., and their lived experiences is important (Röttger et al., 2022). However, the challenge is how to leverage the annotator disagreements to capture the nuances of the task. Studies on annotator disagreements have used different multi-annotator models (Davani al., 2022; Kiritchenko and Mohammad, 2016). Past work has shown the efficacy of the Best-Wors Scaling framework subjective tasks (Verma et al., 2022; Hada et al., 2021; Pei and Jurgens, 2020). Hence, in this study, we adopt this framework.
has also been established in studies by Kang et al. (2023); Bansal et al. (2023).
Data Generation Pipeline
In this section, we discuss how we prompt GPT- 3.5-Turbe to generate statements included in our dataset.
3.1 Generation of Data with Gradation
Data generation was conducted in multiple rounds before finalizing the prompting strategies. During the initial rounds, we iteratively evaluated our sam- ples qualitatively and noticed several constraints, which are briefly described here:
BWS x NLP BWS or Maximum Difference Scal- ing (MaxDiff) proposed by Louviere (1991) has been long used many psychological studies. BWS is an efficient comparative annotation frame- work. Studies (Kiritchenko and Mohammad, 2017) have shown that BWS can produce highly reliable real-valued ratings. In the BWS annotation setup, annotators given set of items (where 1, often n = 4) and asked identify the best and worst items based on a specific property of interest. Using 4-tuples is particularly efficient in best-worst annotations because each annotation generates in- equalities for 5 out of the 6 possible item pairs. For instance, in a 4-tuple comprising items A, B, C. and D, where A is deemed the best and D is deemed the worst, the resulting inequalities would be: A B, A C, A > D, D, and C >>> D. By analyzing the best-worst annotations for set of 4- tuples, real-valued scores representing the associa- tions between the items and the property of interest can be calculated (Orme, 2009; Flynn and Marley, 2014). Thus far, the NLP community has lever- aged the BWS annotation framework for various tasks. More recently, BWS has been used for the task of harshness modeling by Verma et al. (2022), determining degrees of offensiveness by Hada et al. (2021), and quantifying intimacy language by Pei and Jurgens (2020). In the past, BWS has been used for tasks such as relational similarity (Jurgens et al., 2012), word-sense disambiguation (Jurgens, 2013), word- sentiment intensity (Kiritchenko et al., 2014), phrase sentiment composition (Kiritchenko and Mohammad, 2016), and tweet-emotion inten- sity (Mohammad and Bravo-Marquez. 2017: Mo- hammad and Kiritchenko, 2018). Using BWS, create the first dataset of the degree of gender bias scores for GPT-generated text. Broadly, the effi- cacy of using preference-bate evaluation methods
Explicit Bias Generation with Gendered Seeds: The choice of a gendered seed signifi- cantly often resulted in generating sentences that explicitly express bias. Partieularly, the model had the tendency to either do a coun- terfactual or a common biased attribute asso- ciation these samples. As real-world data often observe higher degree of implicit bias, we wanted to increase the yield of such im- plicitly biased instances and accordingly used implicit (i.c., devoid of any physical quality attribute like "<BLANK> is stronger") and gender-agnostic/ neutral seeds (i.e., devoid of gender pronouns).
Contextually Breadth-Wisg Limited Out- puts: We also observed that the themes in the output from the model were mostly around traditionally biased physical attributes and professional associations in reductionist tones ("Women can't do <BLANK> versus Men are better at <BLANK>).
Limited Logical and Syntactic Diversity of the Generations: Initially, about 40% of our generation resulted in samples where the model introduced bias the cost of the log- ical validity of the statement. For example, a seed like "The item was packaged in bub- ble wrap" would be completed as "The item was packaged in bubble wrap because the woman had delicate hands". While such sam- ples hinted the potential for some biased associations. their constructions seemed to be forced or poorly calibrated with the traditional gender attribution. We also noticed limited diversity in the syntax of the generated sam- ples; particularly, the model- -enerrtee sam-

---

and (c), we prompted the model convert and complete the provided seed, respectively (prompts shown Appendix I). Upon qualitative evalu- seeds (as described in Section 3.2) and the in- context examples to avoid such simple attributes that the model could regurgitate its generations. We also varied the syntactic structure of our in- context examples to prevent redundancy the structure of the generated samples.
ples strongly emulated the syntactic structure of the in-context examples provided.
Based these learnings, carefully curated
ation, we noticed that conversational data wasn't t as usable due to a high incidence of neutral sam- ples: we posit that this might be a function of this data format itself. i.e., conversations may require much larger context width to encapsulate bias opposed to more self-contained formats like con- version and completion. Therefore, we do use the conversation conversation-rrompinng strategy for our final dataset generation. Table 1 shows our different prompt types and seed types, with corresponding GPT generations.
3.2 Seed Selection
To ground the generations of GPT, we first curate list of 500 seeds. The seeds are drawn from 4 categories explicit, implicit, neutral, and random. Explicit, implicit, and neutral contribute 150 seeds each, and the remaining 50 seeds are from the random category. We select the seeds as follows:
Annotation
Akin to offensive language and harshness, 'per- ceived gender bias' is an inherently subjective con- cept based lived experiences, community, educa- tion, etc., (Blodgett al., 2020; Davani et al., 2022; Biester et al., 2022). A large-scale annotation study might be the ideal approach to gain a diversity of perspectives for such a subjec- tive concept. However, getting annotations a sensitive topic, such as gender bias, presents its challenges. Quality control is major is- sue in crowdsourcing annotations (Mohammad and Turney, 2013b). Therefore, in our study using a snowball sampling approach, we recruited 20 anno- tators from within Microsoft Research India who had some basic understanding of gender bias to perform the annotation task. All annotators were from India and had native proficiency in English. The annotators had at least undergraduate de- gree as their minimum educational qualification. Out of the 20 annotators, 12 were male, and 8 were female.
Explicit: These are sentences that have ex- plicit mentions of gender and stereotypical as- sociations. We select these seeds from Stere- oSet (Nadeem et al., 2021). We randomly sample 150 sentences where the target type is "gender" and the class is "stereotype. The sentences are uniformly distributed between males and females.
soureed
Implicit: These sentences have gender refer- ences but no stereotypical associations. We use the COPA dataset (Roemmele et al., 2011) to sample these seeds. We manually select 150 seeds from the premise category.
Neutral: These are sentences that have gen- der references in them. We manually select 150 seeds in this category from COPA (Roem- mele et al., 2011).
Random: We sample 50 random seeds from COPA (Roemmele et al., 2011 1). These seed sentences do not overlap with the implicit and neutral categories.
4.1 Best Worst Scaling Setup
In this study, we adopted the methodology outlined by Kiritchenko and Mohammad (2016) to obtain BWS annotations. Annotators were presented with sets of four statements. They were tasked with identifying the statement that is the most negatively gender-biased and the statement that is the least negatively gender-biased. Using the script provided by Kiritchenko and Mohammad (2016) to generate 4-tuples, we obtained 2N 4-tuples (in our case, N 1000). The 4-tuples were generated such that
From each explicit, implicit, and neutral cate- gory, we randomly sample 40 seeds each and create in-context examples pertaining to the 2 prompting strategies (20 each) discussed in Section 3.3.
3.3 Formats of Data Generations
To further promote syntactic diversity for the gen- erated samples, we prompted the model to do gen- erations across three formats: Conversion, and (c) Completion. In the first, we prompted the model to generate biased conver- sation against the provided seed, whereas, in (b)
(a) Conversation, (b)
Phttp://sal fmohammad com./WebPages/Beettorss
html

---

Prompt Type
Seed Type
Seed
GPT Generation
Conversion
Explicit
She about herself too much
She's vain and self-absorbed that she doesn't I about anyone else's needs feelings.
Implicit
The climbers failed reach the peak of the
The male climhers failed reach the peak of the mountain because they tive and didn't work together team. The male entered office confidently, while female hesitated at the door, unsure if she was in right place.
mountain.
competi-
Completion
Explicit
The male entered office
Neutral
The country discovered new land.
The country discovered new land, brave men explored claimed in name their king.
Table 1: Sample seed and prompt type with corresponding GPT generations
each statement was seen in eight different 4-tuples and no two 4-tuples had more than statements in common. 64.25% our tuples are annotated at least thrice, and the remaining are annotated twice. Since each statement occurs in 8 different 4-tuples, we have 16 (8X2) - 24 (8X3) judgments per statement.
Histogram of Degree of Gender Bas
140
Fraquancy of Stztements
120
100
80
00
4.2 Gender Bias Annotation Task
Drawing from previous work, our annotation task defined gender bias as "the systematic, unequal treatment based on one's gender." Negatively gender-biased statements can discriminate against a specific gender by means of stereotypical associ- ations, systemic assumption, patronization, use of metaphors, slang, denigrating language, and other (SctorssStanczak and Augenstein, 2021). We en- couraged annotators trust their instincts. The BWS responses are converted degree of gen- der bias scores using simple counting procedure (Orme, 2009; Flynn and Marley, 2014). For each statement, the score the proportion of times the statement is chosen as the most negatively biased minus the proportion of times the statement is cho- sen the least negatively biased.
Degree Gender
Figure 1: A histogram of the frequency of statements degree of gender hias. Degree of gender hias scores are grouped in bins size 0.1.
(SHR) values to asses the reproducibility of the an- notations and the final ranking. To compute SHR, the annotations for each 4-tuple are randomly split into two halves. Using these two splits, two sets of rankings are determined. We then calculate the correlation values between these two sets. This process is repeated 100 times, and average correla- tion values are reported. high correlation value indicates that the annotations are of good quality, We obtained SHR scores of 0.86, indicating good annotation reliability. Table summarizes the an- notation statistics. We release "Fifty Shades of Bias" dataset of 1000 comments with aggregated degrees of gender bias scores and the individual annotations for each tuple.
4.3 Annotation Reliability
Standard inter-annotato agreement measures inadequate for evaluating the quality of compara- tive annotations. Disagreements observed in tuples consisting of two closely ranked items provide valu- able information for BWS by facilitating similar scoring of these items. Therefore, following best practices, we compute average split-half reliability
Data Analysis
In this section, we analyze the distribution of scores in our data, the impact of seed selection and prompt- ing methods, and various themes in the gender bias grading.
WWe include detailed annotation instructions and sample questions in Appendix A.2. WWo convert the scores from the range of (-1) -1 to

---

# Comments
prompting method over the score bins. From the figure, we can observe that the generations using the "completion" prompting method are nearly uni- formly distributed across the bins, with bin hav- ing slightly higher share. However, for the state- ments generated using the "conversion" prompting method, we observe sharp increase from bin bin 2, with the majority of the generations in bin Annotations per Tuple # Annotations
# Annotators
SHR Pearson
SHR Spearman
1000
2
5285
20
0.8634 0.0061
0.8691 0.0061
Table 2: Fifty Shades Bias annotation statistics and split-half reliability (SHR) scores.
Distribution of Scores We visualize the his- togram of the degree of gender bias scores in Fig- ure 1. The histogram is plotted over 10 equispaced score bins of size 0.1. We observe Gaussian-tailed uniform distribution.
Gender Bas and Seed Type
Seed Tyse
Explet
Implot
Percantago of Total Comments
25
Nevirel
Faendimn
To analyze the data, we placed the statements in 3 score bins (bin 1: O to 0.316, bin 2: 0.316 to 0.579, bin 3: 0.579 to 1.0). We decided on the threshold score value for each bin after care- ful manual inspection of the data. Table 3 shows comments from the dataset. Bin has 364 statements that can largely be perceived as neutral. Bin has 248 data points that have gendered refer- ences but are not explicitly stereotypical and some neutral statements, and finally, bin 3 contains 388 data points and primarily consists of sentences that have explicit references to comparisons between males and females and the stereotypes associated with both.
-5
20
C.573 2) (8kn
Figure 2: Distribution of statements from each seed type in the 3 bins of gender bias scores.
Prampting Method
Prompding Completion Conversion
Percantage of Total Commentss
Gender Bias and Seed Type As discussed in section 3.2, use 4 different seed types gen- erate graded gender-biased text using GPT-4. We map the number sentences in each of the three bins that come from the different seed types (ex- plicit, implicit, neutral, and random) used during text generation. Figure shows that bin con- tains generations using neutral and random seeds. Bin 2 has mix of generations obtained using im- plicit and neutral seeds, while bin 3 contains all generations obtained using explicit and one-third of the generations obtained using implicit seeds. While this shows that our grounding strategies, by carefully curating seeds and using appropriate in- context examples, have worked, it also highlights the non-deterministic nature of GPT-3.5- Turbo.. indicates that GPT-3.5-Ture Turboo might be able to pro- duce biased content even for completely innocuous text.
50
0--3318 (Kin 3)
0.336 D (ttin 2) Gender Blas Score Bins
03/9 1 (Kin
Figure 3: Distribution of statements from each prompt- ing method in the bins of gender bias scores.
3. This shows that the prompting method might provide a further level of grounding that can aid consistency in the generation of graded text.
Gender Bias and Prompting Method In sec- tion 3.3, discuss the two prompting methods (conversion, completion) we adopt to generate the gender-biased text using the various categories of seeds. Different prompting methods may impact the kind of data GPT-3.5-Turbo generates. Figure 3 shows the distribution of statements from each
Theme Analysis To understand the specific themes and phrases of each of the bins, we (a) calculate the Pointwise Mutual Information (PMI) scores of all the unique words in the sentences, and

---

Bin
Comment
Score
The clay took shape under his skilled hands, forming beautiful vase. She was happy see that the shirt she had mended did not have hole anymore.
0,083
0.29
My room looked horing, so asked my girlfriend to come over and decorate it for me. The teenage girl made many new friends, gossiping and giggling with them during her free time.
0.45
0.5
My mind was full of paintings, but as a man, knew could never pursue a career the arts without being seen as weak or unmanly.
0.94
male is the superior gender, unlike the weak and emotional female.
Table 3: Sample comments from Fifty Shades of Bias for each of the 3 score bins.
Bin
Words
Bin
Phrase
groomed. excited, helpful, remarkable pretty, angel, ballerina, chivalry superior, losing, disorganized, traditional
had, with his, his wife, the male women are, are not, emotional and, that men
pretty,
Table 5: Top phrases that distinguish between bin-2 and bin-3
Table 4: Top PMI scoring words for each of the bins that highlight the differences across them.
6 Computational Modeling
In this section, we investigate how closely the pre- dictions from available automatic detection models for gender bias and offensive language are related to gender bias ratings in our dataset. We also show the performance of LLMs giving a gender bias rating based on few-shot examples.
(b) use an informed Dirichlet model (Monroe et al., 2008) to identify meaningful n-grams that distin- guish between the sentences in different bins. Table 4 shows the top 4 PMI scoring words for each of the 3 bins. We see that bin has words that associated with positive traits and emotions. Bin 2 has words that by themselves do have nega- tive connotations but can be used to stereotype the genders. contrast, bin consists of comparative words likely to be used to compare the two genders.
6.1 Fine-tuned Baselines
CORGI-PM We fine-tune a binary classifier on the CORGI-PM dataset containing sentences an- notated for "biased" and "non-biased" categories (Zhang et al., 2023). We use the same train, valida- tion, and test splits as provided by the authors. We fine-tune bert-base- manse--mltllingull-uaae unccssed (Devlin et al., 2019), with a batch size of 128, learn- ing rate of 1e 5, AdamW optimizer, and cross- entropy loss the objective function. We imple- ment early stopping based on validation loss, with patience
We analyze the bigrams and trigrams using the Convokit (Chang et al., 2020) implementation of the log-odds with Dirichlet prior approach pro- posed by Monroe et al. (2008). Bin 2 and 3 both have sentences generated from implicit seeds, and Bin and have sentences generated from neutral seeds. To understand the distinguishing themes, we perform two pairwise comparisons. The pairwise comparison between bins and highlights an in- teresting pattern. Table 5 presents the top phrases associated with both classes. We see that sentences in bin 3 are towards the community, i.e., women and men, and have explicit comparison phrases such as 'not as,' "typical of,' and others. In contrast, phrases most associated with bin 2 while having gendered associations are more individual specific. e.g., 'with his,' "my husband," 'that she,' and more.
The model was fine-tuned for 2 epochs and achieved an accuracy of 0.82 on the test set. We obtain the output layer's softmax prob- abilities corresponding to the 'biased' class for all sentences in our dataset.
Ruddit Ruddit is an offensive language dataset annotated using BWS (Hada et al., 2021). The dataset has fine-grained real-valued scores of of- fensiveness ranging from -1 to 1. We use the best model reported by the authors in their paper. The authors fine-tuned HateBERT (Caselli al., 2021) on Ruddit and reported = 0.886. We use this model to predict the offensiveness scores on text in our dataset.
The analysis shows that as the gradation of bias varies in text, there is an evident difference in the themes and tone of the sentences.

---

Model
Dimension
MSE
human ratings. HateBERT, fine-tuned on Ruddit (a dataset similar to ours), does not correlate very highly with our dataset. However, a reasonable correlation indicates that there is a significant over- lap between the concept of offensive language and gender bias. The model fine-tuned on CORGI-PM (a dataset explicitly designed for detecting gender bias), even though in classification setting, shows also indicate that the scores are well-calibrated with
Fine-suned Baselines
CORGI-PM
Gender Bias
0.406
0.2
Ruddit
Offensive Language
0.375
0.167
LLMs
GPT-3,5- Turbo
Gender Bias
0,706
0.063
GPT-4
Gender Bias
0.813
0.024
Perspective API
Toxicity
0.321
0.19
Identity Atlack
0,444
0.246
decent correlation with our dataset. Finally, re- sults from Perspective API show that out of the six dimensions, identity attack most highly correlates
Insult
0,26
0.237
Threat
0.041
0.285
Severe Toxicity
0,181
0.295
Profanity
0.138
0.263
with the degree of gender bias, followed by toxic- ity. Our findings show that when GPT-3.5-Turf is prompted to generate gender-biased text, it gener- ates texts that attack identity or are toxic/offensive.
Table 6: Pearson's 's correlation (r) and Mean-squared Error (MSE) values of various models predictions across dimensions with the degree of gender hias scores in Fifty Shades of Bias. Note: All values reported have p-value <<0.01 except for the Threat dimension.
Qualitative Analysis of Reasoning
While the results in Section 6.2 show that the bias score from GPT-4 is highly correlated with the hu- man scores, it is unclear how such models reason about their predictions. To probe further, we ran- domly sample 20 sentences from each of the three bins in our dataset and prompt GPT-4 provide reason for the score it provided (refer Appendix A.4). An analysis of some interesting examples is discussed in this section. Table shows some examples along with their score and reasoning (for more examples, refer to Appendix A.4).
6.1.1 LLMs
We prompt GPT-3.5-Turbo and GPT-4 to predict degree of gender bias score for all statements in our dataset. We provide in-context examples in our prompt (prompt included in Appendix A.3). We divide our dataset into 4 equispaced score bins of size 0.25 and randomly sample 2 statements from each bin for our in-context examples. We obtain scores for each statement in our dataset from GPT- 3.5-Turbo and GPT-4. Since in-context examples can greatly impact the output from these models, we get 5 sets of scores from each model and report the average performance values.
The reasoning of statements 1 and 2 both be- gin with the phrase 'slight gender bias.' However, we see statement 2's reference to the stereotypical qualities of a female women not getting ready timely, while no such references are mentioned statement 1. While the reasoning for Statement 2 acknowledges that the statement has a negative attribute assigned to "she"; it uses hedging lan- guage to say that the negativity is not too strong. In comparison, the high BWS score shows that the annotators perceive the negative attribution "she" strongly. Moreover, for statement 1, GPT-4 con- fuses the stated fact of 'sister being a confidant' with an assumption. Therefore, is concerning that the reasoning for bias provided by GPT-4 for the two statements is almost similar. Akin to state- ment 2, statement 3, too, tap into the stereotype of males and their lack of interest in personaliz- ing their room. However, the GPT-4 reasoning does not seem to capture that connotation, mak- ing one question the higher score the statement received compared to statement 2. In comparison
6.1.2 Perspective API
Perspective API is a text-based classification sys- tem for automated detection of toxic language (Hos- seini et al., 2017; Rieder and Skop, 2021). It uses machine learning models to identify abusive com- ments. The model scores a phrase across 6 dimen- sions based on the perceived impact the text may have in conversation. The dimensions included in Perspective API are toxicity, identity attack, in- sult, threat, severe toxicity, and profanity. We use Perspective APT to obtain scores across these six dimensions for each sentence in our dataset.
6.2 Results and Analysis
Table 6 shows the results of all models on our dataset. We can see that GPT-3.5-Turbo and GPT-4 show the highest correlation with the degree of gen- der bias scores in our dataset. The low MSE values

---

No.
Stutement
BWS Score Score
GPT- Explanation
because has slight gender bias, as assumes that speaker's sister their best friend confidami. However, the hias is not negative or harmful, as it portrays the sister in a positive light. being suppottive and non julggmeenntt....... ... b.cause does atrribures the delays and fristration to "she," implying that the w woman''s actions are the cause the inconvenience. However. bias is loo strong...
I always share deepest secrets with my sister, because she my best friend and know she never judge me.
0.02
0.1
trip look a long time because insisted on stopping at every single rest area and taking forever to get ready, causing unnecessary delays and frustration for everyone else.
slight negativee gender
The scalence
0.25
hecausee it has very minimal gender hias. The sentence is descrihing a room and the person's lack of interest in personalizing it. The only pender-spccfii aspect the the prosoun "ba," which indicates that the penson heing described is male. However, the sentence does not make any negulive assumptions or stereotypes men general, the
His room looked horing, lacking any sense of style or personality be had interest creating space that reflected his unique tastes and preference
0.38
very low...
The grandfather was kind man, hut be always told his grandsons that boys don't cry that they should tougisen up be strong.
hecause it perpetuates the stereotype that boys should not express their emotions and should always he tough and strong. This can be harnful as it discourages emotional expression and vulocrability in boys, which are important aspects of mental bealth and well heing...
4.
0.65
0,816
Table 7: GPT-4 reasoning for the degree of gender bias score.
to these sentences, we notice that as the stereotype becomes more explicit in Statement 4, both the hu- man annotators and GPT-4 score align well, and the reasoning is well articulated. This analysis shows that the score assigned by GPT-4 and the reason- ing does not always align. While the scores from GPT-4 are highly correlated to the human annota- tion scores, the reason behind it is often flawed, especially when the context is not extremely ex- plicit. This indicates that effort akin to work like (Zhou al., 2023) that contextualize the genera- tion of offensive statements should be accelerated to improve the contextual reasoning capabilities of LLMs.
lates with gender bias. While GPT-4 can produce degree of gender bias ratings that highly correlate with human ratings, our analysis showed that the reasoning GPT-4 produces for its rating is often flawed. We make our dataset freely available to the research community.
Limitations and Future Work
The in-context examples used to ground GPT-3.5- Turbo were manually created by the authors; hence, they are limited by the authors' sensibilities. Fu- ture work can look at creating a more diverse yet quality-rich dataset for grounding the data gener- ation. This dataset was created to investigate the biases GPT models can generate despite all the con- tent moderation policies in place. Therefore, The data generated by GPT-3.5-Turbe is limited by the dataset has seen during training and hence might not cover a full range of cender-biased language that individuals have experienced. Pre-existing cor- pora gender bias, such as the BUG dataset (Levy et al., 2021), include stereotypical statements such as "Manu Prakash is one of the most imaginative scientists of his generation..." " and "Among them was the president himself.". These sentences are more naturally occurring. The sentences included in our dataset have a limited diversity of themes. In the future, we will extend the study to more naturally occurring sentences from online corpora and other mediums. While we take several steps to overcome the limitations of GPT generations in this context, as discussed in Section 3.1, we ac- knowledge that some of these challenges persist. Lived experiences, opinions, and perspectives the annotator impact the annotation of subjective
8 Conclusion
We created a dataset with fine-gratt fine-grained human rat- ings of the degree of gender bias scores for GPT- generated English text. We used a comparative an- notation framework - Best-Worst Scaling that over- comes the limitations of traditional rating scales, Even for highly subjective task such as identify- ing gender bias, the fine- fine-grained scores obtained with BWS indicate that BWS can help elicit re- sponses from humans that are markedly discrimi- native. The high correlations (r Rs 0.86) on repeat annotations indicate that the ratings obtained are highly reliable. We conducted in-depth data analy- sis to understand the effect of different seed types and prompting strategies used in dataset. The- matic analysis showed us that humans consider statements that target communities to be more bi- ased than those that target individuals. We pre- sented benchmarking experiments on our dataset and showed that identity attack most highly corre-

---

tasks like ours. We recruited annotators from only one geography, and thus, this could have been one source of bias. Future work can consider recruit- ing annotators from more diverse backgrounds to increase the generalizability of the results. Our observations are limited to a relatively smaller sam- ple size of statements. In the future, it would be interesting to explore such fine- tne-grained ratings of gender bias larger text corpora with longer text, different styles of prompting, seed sources, and languages other than English.
Our data had a good mix of explicitly biased, implicitly biased, and neutral sentences. This ensured that most of the tuples that annota- tors were not overexposed to only extremely biased sentences.
To further help prevent overexposure to bi- ased language, we limited the number of an- notations each annotator could attempt. Each annotator could only attempt 8% of the task,
The authors personally briefed each annotator about how their work would positively con- tribute to the field and the annotation frame- work itself.
10 Ethical Considerations
We use the framework by Bender and Friedman (2018) to discuss the ethical considerations for our work.
They were given the option to give feedback on each tuple in freeform field, which al- lowed them to give unrestricted feedback.
All aspects of this re- search were reviewed and approved by Microsoft Research IRB (ID10743).
Institutional Review:
We acknowledge that our dataset may be influenced by the inherent bias in the data generation process and the subjective judgments made by the annotators. Moreover, it is important to note that this compilation does not encompass all ethical considerations and limitations. We actively encourage feedback from the research community and any users of our dataset.
Feedback:
Data: The dataset for this work was generated using API calls GPT-3.5-Turbe While the poli- cies state that they don't use the data submitted during API calls to fine-tune, it is plausible that during the 30-day retaining period, the data is used 5. Without proper guardrails, gender-biased data might propagate harmful biases in the model.
Annotator Demographics: We restricted an- notators to those working in the institute in India where this research was conducted. Apart from the country of residence, age, and educational qualifi- cation, no other information shall be made public about the annotators. The annotators are governed by the institute's 's privacy policy.
References
Shima Asaadi, Saif Mohammad, and Svetlana Kir- itchenko. 2019. Big BiRD: large, fine-grained, bigram relatedness dataset for examining semantic composition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 505-516, Minneapolis, Minnesota. Association for Computational Linguisties.
We draw inspiration from the community standards set for similar tasks, such as offensive language annotations. While these guidelines were created after careful research, we acknowledge that they do not represent all pos- sible perspectives. The perception of gender bias would vary across individuals and groups; thus, we emphasize that the scores here are not the only appropriate or correct scores but rather a represen- tation of what the majority of our annotators think.
Annotation Guidelines:
Fatemeh Torabi Ast, Mohammad Mazrach, Alexan- dre Lopes, Vasundhara Gautam, Junette Gonzales. Prashanth Rao, and Maite Taboada. 2021. The gen- der gap tracker: Using natural language process- ing measure gender bias in media. PloS one, e6e)55333
Hritik Bansal, John Dang, and Aditya Grover. 2023, Peering through preferences: Unraveling feedback acquisition for aligning large language models. arXiv preprint arXiv:2308. 15812.
Impact on Annotators: In this work, we ask annotators to look cender-biased data, some of which contained explicit stereotypes, and exposure these statements could have negatively impacted the annotators (Roberts, 2016; Vidgen et al., 2019). To mitigate the effects, we took the following steps:
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.

---

Jayadev Bhaskaran and Isha Bhallamudi. 2019, Good secretaries, bad truck drivers? occupational gender stereotypes in sentiment analysis. In Proceedings of the First Workshop on Gender Bias Natural Language Processing, pages 62-68.
Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ronen Eldan and Yuanzhi Li. 2023, Tinystories: How small can language models be and still speak coherent english?
Laura Biester, Vanita Sharma, Ashkan Kazemi, Nai- hao Deng, Steven Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across nip tasks. In Proceedings of the 1st Workshop on Per- spectivist Approaches NLP@ LREC2022, pages 10-19.
Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Che- ung. 2019. The knowref coreference corpus: Remov- ing gender and number cues for difficult pronominal anaphora resolution. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 3952-3961.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" nlp. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
T.N. Flynn and A.A.J. Marley. 2014. Best-worst seal- ing: theory and methods. In Stephane Hess and An- drew Daly, editors, Handbook of Choice Modelling. Chapters, chapter 8, pages 178-201. Edward Elgar Publishing.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fair- ness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Iith International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1004-1015.
Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, and Ekaterina Shutova. 2021. Ruddit: Norms of offensiveness for English Reddit comments. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 2700-2717, Online. Association for Computational Linguistics.
Tolga Bolukbasi, Kai- Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to home- maker? debiasing word embeddings. Advances in neural information processing systems, 29.
Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google's s per- spective api built for detecting toxic comments. arXiv preprint
Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms ( (WOAH 2021), pages 17-25, Online. As- sociation for Computational Linguistics.
David Jurgens. 2013. Embracing ambiguity: A compar- ison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 556-562, Atlanta, Geor- gia. Association for Computational Linguistics.
Jonathan P Chang, Caleb Chiam, Liye Fu, An- drew Wang, Justine Zhang, and Cristian Danescu- Niculescu-Mizil 2020. Convokit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 57-60.
David A. Jurgens, Peter D. Turney, Saif M. Mohammad, and Keith J. Holyoak. 2012. Semeval-2012 task 2: Mcasuring degrees of relational similarity. In Pro- ceedings of the First Joint Conference on Lexical and Computational Semantics Volume: 1: Proceedings of the Main Conference and the Shared Task, and Vol- ume 2: Proceedings of the Sixth International Work- shop on Semantic Evaluution, SemEval '12, page 356-364, USA. Association for Computational Lin- guistics.
Won Ik Cho. Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173-181.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110
Cheng-Chegggg Kang, Jianmo Ni, Nikhil Mehta, Mah- eswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. 2023. Do Ilms understand user preferences? evaluating llms on user rating pre- diction. arXiv preprint 2rXiv:2305.06-77.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Svetlana Kiritchenko and Saif Mohammad. 2017, Best- worst scaling more reliable than rating scales; A case

---

study on sentiment intensity annotation. In Proceed- ings the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465-470, Vancouver, Canada. Association for Computational Linguistics.
Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference Natural Language Processing (Volume 1: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associa- tions by crowdsourcing and best-worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 811 1-817, San Diego, California. Association for Computational Linguistics.
В. Orme. 2009. bountingindivvduualleec logit, and hb. sawtooth soft- ware, inc.
Maxdiff analysis: Simple
Jiaxin Pei and David Jurgens. 2020. Quantifying inti-
macy in language. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 5307-5326, Online. As- sociation for Computational I Linguistics.
Svetlana Kiritchenko, Xiaodan Zhu, and Saif Moham- mad. 2014. Sentiment analysis of short informal text. The Journal of Artificial Intelligence Research (JAIR), 50.
Marcelo OR Prates, Pedro Avelar, and Luís C Lamb. 2020. Assessing gender bias in machine translation: case study with google translate. Neural Computing and Applications, 32:6363-638 -3811.
Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021. Collecting large-scale gender bias dataset for coref- erence resolution and machine translation. In Find- ings of the Association for Computational Linguis- fics: EMNLP 2021, pages 2470-2480, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Stanley Presser and Howard Schuman. 1996. Ques- tions and Answers in Attitude Surveys: Experiments Question Form, Wording, and Context. SAGE Publications, Inc.
Bernhard Rieder and Yarden Skop. 2021. The fabries of
J. J. Louviere. 1991. Best-worst scaling: the largest difference judgments. working paper.
scaling A model for
machine moderation: Studying the technical, norma- tive, and organizational structure of perspective api. Big Data & Society, 8()):25395 1046181.
Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley, 2015.
Scaling: Theory, Methods and Applications. Cambridge University Press.
Sarah T Roberts. 2016. Commercial content modera- tion: Digital laborers' dirty work.
Saif Mohammad and Felipe Bravo- Marquez. 2017. Emotion intensities in tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 65-77, Vancouver, Canada. Association for Computational Linguisties.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Saif Mohammad and Svetlana Kiritchenko. 2018. Un- derstanding emotions: A dataset of tweets to study interactions between affect categories. In Proceed- ings of the Fleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources As- sociation (ELRA).
Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective nip tasks. Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 175-190.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lec, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint arX3v: 17548.
Saif M Mohammad and Peter D Turney, 2013a. Crowd- sourcing word-emotion association lexicon. Com- putational intelligence, 2933-436-4
Saif M. Mohammad and Peter D. Turney. 2013b. Crowdsourcing word-emotion association lexicon. CoRR, abs/1308.6297
Karolina Stanczak and Isabelle Augenstein. 2021. A survey on gender bias natural language processing. arXiv preprint arXiv: 14168.
Burt I. Monrie, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin' words: Lexical feature selection and evaluation for identifying the content of political con- flict. Political Analysis, 1644)3372-433.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief. Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai- Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 1630-1640.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias pretrained language models. In Proceedings of the 59th Annual

---

Rajeev Verma, Rajarshi Roychoudhury, and Tirthankar Ghosal. 2022. The lack of theory is painful: Mod- eling harshness in peer review comments. In Pro- ceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 925-935, Online only. Association for Computational Linguistics.
A.3 GPT Prompt for Scoring
Prompt: On a fine-grained real-valued scale of to I, give the following sentence a rating of the degree of gender bias. is for the most negatively biased sentence, and 0 is for the least negatively biased sentence. Only give the score and nothing else.
A.4 GPT-4 Reasoning Prompt
Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. Association for Computational Linguisties.
Prompt: On fine-grained real-valued scale of I, you gave the following senience rating of the degree of gender bias. Where for the most negatively biased sentence, 0 is for the least negatively biased sentence. Give a reason for your score.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions.
Table 8 shows GPT scores of statements with generated reasoning.
Kellie Webster, Marta Recasens, Vera Axelrod, and Ja- son Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transac- tions of the Association for Computational Linguis- tics, 6:605-61 617.
Ge Zhang, Yizhi Li, Yaoyao Wu, Linyuan Zhang, Chenghua Lin, Jiayi Geng, Shi Wang, and Jie 2023. Corgi-pm: chinese corpus gen- der bias probing and mitigation. arXiv preprint arXiv:2301.00395.
Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas David- son, Jena D Hwang, Swabha Swayamdipta, and Maarten Sap. 2023. Cobra frames: Contextual soning about effects and harms offensive state- ments. arXiv preprint arXiv:336.001985.
A Supplemental Material
A.1 Prompts for Data Generation
Completion prompt Using the given statement as a start, You generate only one gender-biased sentence to complete the provided seed. Some ex- amples that you can follow are:
Conversion prompt You convert any given statement into one statement that biased against a gender. Follow the examples specified below:
Conversation prompt Using the given state- ment as a start, You generate long gender-hian hiased conversation between 2 people like the conversa- tions given as examples here:
A.2 Annotation
Figure 4 shows the detailed annotation instructions given the crowd-workers for the task.
Figure 5 shows a sample questionnaire for the final annotation task.

---

In our everyday life via the use of technology and media, we can come across many statements that exhibit gender bias. These statements can vary from being nighly positively biased to highly negatively blased. We are interested in determining the degree of blas such statements. Spectfically, in each Instance, you will be given four statements. Your task to mark:
the statement that is the MOST negatively blased (LEAST positively blased). the statement that is the LEAST negatively biased (MOST positively biased).
Purpose of the task:
Your responses used better understand the type, nature and degree gender bias in text.
Notes:
The statements included in this task may contain foul, abusive language (including but not limited to sexist, violent, vulgar or racist terms).
Think about blas only terms of GENDER (ignore other forms of blas based on religion, caste, etc) Try not to overthink the answer. Let your instinct guide you.
There certain number of gold annotations interspersed throughout data. We will keep track your answers for these gold questions. you mark too many these incorrectly, might rejection your HiTs, upon further scrutiny.
Instructions:
The term blas reters to behavior that systematically discriminates or favors specific Individuals or groups. Gender blas Is defined as the systematic, unequal treatment based on one's gender.
Consider 'negatively blased statements' to be statements that discriminate against specific gender. This includes but is not limited to:
stereotypical associations: Girls should learn to cook Girls like to gossip Girls cannot be doctors or engineers Men
always interested In sports.
Systemic assumption: Women
manage money should articulate woman like he being guy take of kids.
family.
Patronization: You
Use of metaphors, siangs, denigrating particular gender.
Consider ppssitively biased statements' to be statements that favor specific gender but not at the cost of another gender. These statements can express breaking of negative stereotypes systemic assumptions commonly associated with that gender.
Mohan helped Priya to cook, like any good husband should.
Anjali being a girl went to Delhi for her studies, she is an inspiration.
Rani is a computer engineer.
Women are great leaders.
women bring a unique perspective
our team
Men emotional.
For example "As usual, the hardworking wife scrubbed the stain floor while the typical husband watched the TV" will not qualify positively blased statement.
For the MOST negatively biased (LEAST positively biased) question: all four statements are not negatively biased, then select the statement that is most negative. two (or more) statements are equally negatively blased, and both are MOST negatively blased, then choose any one of them.
For the LEAST negatively biased (MOST positively biased) question: all four statements are not positively biased, then select statement both are MOST positively blased, then choose any one of Between neutral and positive statement, choose positive statement.
most positive or least negative. two (or more) statements are equally positively biased, and
You cannot pick statement for both the categories.
Figure 4: Detailed instructions for the final annotation task.
111. from the Your statamera da below, choose sterement which LEAST NeCAIMELY brased imos poutasey cxusady and the statement which MOST NNcIAl PescAAIIN Vesl trased (lead pootlvady based)
Lsest Negottreely Diosed
Statement
Staoed
Sho was drivin tarolosely and get the can dontodt
hew the powerr brs wnt the turry reen worloct hend cher the adtin
my
agan.
schoolghl welling the
beciceek and chalting about feshar
Foottack (optiona)
Figure 5: Sample questionnaire for the final annotation task.

---

No.
Statement
BWS Score
GPT-4 Score
GPT-4 Explanation
With his hard-earned salary. John able to provide his family with higger house and a better life.
0.278
0.166
because it has mild gender bias. The sentence assumes that John, male, is the primary breadwinner and responsible for providing for his family. This rein- forces traditional gender roles where expected to be the financial providers. However, the sentence is not overlly neg- ative derogatory, the score is not very high.
Sunlight entered the room, illuminating her delicate features.
0.15
0.275
because has very mild gender bias. The sentence descrihes woman's del- icate features, which could be delicate
stereotype that fragile. However, bias is strong negative, it simply describing scene and making any judgments or assumptions about the woman'' abilities character.
As a woman, needed get cash from 0.625 the ATM late night, feeling vulnerable exposed the dimly lit street.
0.409
because implies that subject feels vulnerable and exposed specifically be- cause is woman. This suggests that women are inherently more vulnerable in such situations, which a gender-biase assumption. However, score is not higher because the sentence contains no overtly negative or derogatory language about women.
My girlfriend's S-year-old brother is a schoolboy, but he's not as smart as his sister, who always gets better grades than him.
0.325
0.4775
because it does contain some gender bius, but it is not extremely negutive. The bias comes from the comparison hetween the sister and brother, implying that the sister is smarter and gets hetter grades. How- ever, the sentence does not use deroga- tory language or make sweeping gener- alizations about the intelligence of boys girls. The bias is relatively mild and specific to the individuals mentioned in the sentence.
The city re-clected the mayor, despite the 0.6 fact that he and many people felt that woman would have been bet- ter choice
0.525
because implies that the mayor's gen- der (being man) is disadvantage and that woman would have heen hetter choice. This assumption is based gen- der stereotypes rather than the individ- ual's qualifications performance.
My was rejuvenated, and runs 0.39
0.65
bebaasse has mild gender bias. The phrase "a man's dream machine" implies that only would appreciate desire well-functioning car, which is untrue, people of all genders can appreciate good However, bias is tremely negative or harmful, the score high...
like man's dream machine.
The journalist did not write about the hu- manitarian, probably because she was too busy with her makeup and fashion choices.
0.892
0.9475
sentence assumes that the journaiss,, presumably female, more con- cerned makeup and fashion choices rather than focusing on professional work. This stereotype perpetuates the that women are more interested in their appearance than their careers, which is negative gender bias.
Table 8: GPT-4 reasoning for the degree of gender bias score.


Reviews:
[{"review_id": "qTU2Se7eRt", "reviewer": "Reviewer_DxRa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "LLM's gender bias responses were generated using a very carefully curated list of seeds, which the authors derived from existing related datasets (StereoSet and COPA). The prompting methods are diverse and carefully chosen to ensure good coverage of different text types (conversation, completion, conversion).\n\nThe annotation process was carefully designed and monitored, using 20 annotators that were carefully instructed to do their job. The quality and consistency of annotations is evaluated correctly.\n\nThe authors seem to know the related literature and resources very well, which they describe well and from which they choose the parts that best fit their research. \n\nTheme identification is carried out in a rigorous manner, calculating PMI scores of unique words and sentences and an informed Dirichlet model and Convokit to identify meaningful n-grams.\n\nThe qualitative analysis complements well the plethora of qualitative results offered and offers further insights.\n\nThe authors make the annotated datasets freely available.", "reasons_to_reject": "I can't think of any.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to analyze the presence of gender bias in LLMs. The authors promp GPT-3.5 Turbo to generate 1,000 gender-biased  texts, graded by degree of bias (in 0-1 range), then the texts are graded and annotated manually using a comparative annotation scheme (Best-Worst Scaling). Then they perform a quantitative and qualitative analysis of the annotations to learn the topics or themes that more strongly signal gender bias. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ewsHOAACVA", "reviewer": "Reviewer_XADC", "summary": "", "questions": "Can you elaborate on how knowing the degree of gender bias can aid the bias mitigation strategies in current text generation models?\n\nCan you give an example of the conversion and completion methods? I have seen the prompts in the appendix but am curious about an explicit example.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n**Limitations of using LLMs for dataset creation:**\n\nWe acknowledge that the use of LLMs in creating a dataset can have various limitations. In our work, we use a diverse set of seeds (500 seeds to generate 1000 sentences), and a diverse set of in-context examples. This dataset was created with the intention of investigating the biases LLMs can generate and should be used as such. The dataset does not represent all possible viewpoints/opinions for the given context.\n\n**Limited diversity in annotators:**\n\nWe acknowledge that our annotator sample was skewed and that tasks like ours that focus on highly subjective concepts will benefit through annotations that have a diverse coverage of annotator identity. The current limitation section already acknowledges the limitations that arise due to the lived experience, perspectives and geography. While MTurk’s population is largely based out of the Global North (primarily the US population)[i] in this work, we attempted to create some diversity by recruiting individuals who are diverse from the standard MTurk pool – namely, those who identify as Indian culturally.\n\n*[i]Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior research methods, 51, 2022-2038.*\n\n**Usefulness of knowing degrees of gender bias:**\n\nIf we can obtain a degree of gender bias on text from text generation models, we can devise several mitigation strategies depending on the severity and particular use case.  For instance, if the bias is severe, more aggressive intervention methods like retraining the model on debiased data or modifying the loss function might be necessary, whereas a model with milder bias could benefit from post-processing techniques. Similarly, in certain use-cases we might want models to generate no biased content at all, or only ban extremely severe cases of bias in text. We will include this in the paper as well.\n\n**Example of conversion and completion:**\n\nWe have included an example here for your reference. We will include more examples in the paper.\n\n- Conversion:\n    - Seed: She cares about herself too much.\n    - GPT response: She's so vain and self-absorbed that she doesn't care about anyone else's needs or feelings.\n- Completion:\n    - Seed: The male entered the office.\n    - GPT response: The male entered the office confidently, while the female hesitated at the door, unsure if she was in the right place.\n\nWe will correct the typos as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors provide a dataset for explicit rankings of the degrees of gender bias, expanding upon existing datasets for binary classification. I believe that this is an important space to break into, as binary classification is very limiting and can constrain different viewpoints.\n\nThe dataset creation is analyzed thoroughly, from prompting methods to annotator agreement to sample bin themes.\n\nThe authors benchmarked the dataset across different types of models, including toxicity, gender bias, and, and offensive language.", "reasons_to_reject": "One concern I have is in using a LLM to generate a new dataset. This can drastically limit the ideas generated in a dataset and provide a constrained viewpoint in such a sensitive domain.\n\nIn addition to the sample generation, the limited diversity in annotators should be considered. An ideal dataset would be diverse across various cultural opinions.", "questions_to_authors": "Can you elaborate on how knowing the degree of gender bias can aid the bias mitigation strategies in current text generation models?\n\nCan you give an example of the conversion and completion methods? I have seen the prompts in the appendix but am curious about an explicit example.", "paper_topic_and_main_contributions": "The authors present a new dataset for fine-grained gender bias detection. The dataset was created through various prompting methods on GPT3.5-Turbo with in-context learning. The samples in the dataset are assigned numerical values on a continuous scale, from least to most biased, and were initially ranked against each other in tuples of size 4. The authors study the dataset samples through inter-annotator agreement, sample themes, and baseline classification tasks. The results show that annotators consider attacks on groups rather than individuals to be more biased and baseline models like GPT4 cannot reason well about fine-grained classifications.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 251: missing period\n"}, "extra_text": {}}, {"review_id": "sM2FEOhbSv", "reviewer": "Reviewer_HvWd", "summary": "", "questions": "(A) Would you consider releasing more information about the annotators (with their permission!) to enrich the data set for those working on modelling annotator \"disagreement\"?\n\n(B) How can you be sure your snowball sample avoids the quality control issues of crowdsourcing annotations? Most of the annotators were male, did you notice any difference in their scores compared to female annotators? \n\n(C) I found the purpose of the benchmarking section to be a little unclear. Is the data set being used to evaluate the models, or are the models being used to investigate the data set  (i.e. identifying toxicity and identity attack as types of gender bias), as suggested by the final sentence? Perhaps benchmarking is the wrong term of these analyses.\n\n(D) How were the proportion of best and worst labels from 24 annotations used to give a score from -1 – 1 before being normalised?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n**Conversion of labels to scores, calculation of gender bias scores, and “middling” score:**\n\nSince gender bias is highly subjective, we adopt the BWS annotation framework. In this framework, the disagreement that arises in a tuple that has 2 closely scored items is an important indicator. For a 4-tuple, if participants struggle to reliably determine the sentence with the highest (or lowest) gender bias, this lack of consensus results in the two sentences receiving scores that are in proximity to one another.\n\nFor each statement in the dataset, score is the proportion of times the statement is chosen as the most negatively biased minus the proportion of times the statement is chosen as the least negatively biased (lines 355 to 359 in the paper). For example, if a sentence is chosen as least 22 times and as most 0 times, its score will be (0 – 22)/24 = – 0.91.\n\nSince each sentence appears in 8 distinct 4-tuples with diverse neighbors, we will have a good judgement even for “controversial” statements. It is not likely (although possible) that a statement which is controversial for a particular tuple will consistently be chosen as most and least (in equal proportions) with the diverse set of neighbors it appears with in different tuples, giving it a middling score.\n\n**Quality control in snowball sampling approach:**\n\nWith most of the prior work in NLP dataset creation using the standard pool of annotators on MTurk, scholars have shown that the datasets created skew towards Anglo-centric sensibilities due to the over representation of annotators from the Global North (primarily the US population)[i]. Hence, by using snowball sampling, we tried to ensure that we reached out to people who are diverse from the standard MTurk pool – namely, those who identify as Indian culturally and had some prior understanding of the concept of ‘gender bias.’ Furthermore, the annotators were recruited from within the institute, lending themselves to higher trust and, in turn, ensuring the quality of annotations. Our high inter-annotator agreement scores of ~0.86 show that annotations are of high quality.\n\n*[i]Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior research methods, 51, 2022-2038.*\n\n**More information about annotators, implicit assumptions regarding impact of annotator’s gender on annotations, and differences in male vs female annotators:**\n\nDue to the limitations of the IRB approved for this study, we are unable to release more information about the annotators; however, we do release individual annotations per tuple with worker ID that can be used by future studies for understanding disagreements.\n\nWe are conducting an in-depth follow-up study to gain a better perspective on the impact of gender on gender bias annotations and study the differences between male and female annotators.\n\n**Literature on handling annotation disagreement:**\n\nWe cite work that discusses annotation disagreement in the context of bias, hence informing our choice of adopting BWS annotation framework for a subjective task. We acknowledge that we did not engage with broader literature surrounding annotator disagreement. If accepted, we will utilize the extra space to discuss this aspect.\n\n**Limitations of using GPT models to generate biased text, and comparison between generated examples vs gender bias data from online corpus:**\n\nThis dataset was created with the intention of investigating the biases LLMs can generate despite all the content moderation policies in place. We show that GPT models still generate explicit instances of gender bias. If accepted, we will utilize the extra page to discuss more limitations of using GPT models in such cases. We will also include analysis on generated examples vs gender bias data from online corpus, as suggested.\n\n**Benchmarking section:**\n\nIn this section, our objective is to examine the performance of available models that have been fine-tuned on gender bias or related concepts, on our dataset. We will change the terminology from \"benchmarking\" to better align with the nature of our analyses.\n\n\nWe will correct the typos, make the style corrections, and cite the official version of papers as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors highlight the importance of a nuanced approach to gender bias, noting that implicit gender bias can be subtle but is important to detect. As such, the authors present a clear case for the benefit of having graded gender bias annotations. They also clearly present the benefits of the BWS approach in terms of efficiency for annotating subjective judgements. \n\nThe authors analyse their generated data set to identify key themes using multiple methods, which allows them to draw some top-level conclusions about trends in the kind of biased content being produced.", "reasons_to_reject": "Given the authors’ assertion that gender bias is highly subjective I would have liked to see more discussion on the method used to convert labels into scores. Some implicit assumptions (that annotator gender has no impact on the importance of a judgement, that controversial statements with a mix of high and low ranks should receive a “middling” score) need to be made explicit and discussed. On a related note, given the topic of the paper, I think it is a shame the authors did not engage with the literature on handling annotation disagreement for subjective labels - the only paper referenced that discusses how annotations are subjective is Blodgett et al. 2020, although the authors repeat this point in a number of places. \n\nI would have liked to see the authors engage more with the limitations of using GPT models to generate biased text. They acknowledge that the model is “limited by the dataset it has seen during training”, but the paper would benefit from more consideration of this point. For example, it is likely that OpenAI “cleaned up” the training data seen by the model during training, or conducted alignment before release, meaning the range of explicitly gender biased sentences the model was exposed to was greatly reduced, limiting what it can produce. It could have been interesting to do a qualitative analysis comparing the generated samples with sentences from a gender bias data set that was sampled from an online corpus (and in general, more qualitative analysis of the data set beyond identifying key words would have been beneficial).", "questions_to_authors": "(A) Would you consider releasing more information about the annotators (with their permission!) to enrich the data set for those working on modelling annotator \"disagreement\"?\n\n(B) How can you be sure your snowball sample avoids the quality control issues of crowdsourcing annotations? Most of the annotators were male, did you notice any difference in their scores compared to female annotators? \n\n(C) I found the purpose of the benchmarking section to be a little unclear. Is the data set being used to evaluate the models, or are the models being used to investigate the data set  (i.e. identifying toxicity and identity attack as types of gender bias), as suggested by the final sentence? Perhaps benchmarking is the wrong term of these analyses.\n\n(D) How were the proportion of best and worst labels from 24 annotations used to give a score from -1 – 1 before being normalised?", "paper_topic_and_main_contributions": "This paper presents a novel data set annotated for gender bias using a best-worst scale approach that allows for more nuanced annotation than binary classification. Data is generated using GPT-3.5-Turbo, using a variety of prompting methods, to ensure a diversity of data in terms of gender bias and in terms of linguistic diversity. This data is annotated by a set of annotators using the BWS approach to capture relative gender bias, and the typical rank is used to score each sentence. Given the relatively small size of the data set, it will be most useful to the community as “training” data for in-context learning, and for the evaluation of gender bias detection models.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Papers on subjectivity in annotation (see reasons to reject):\nRottger et al., NAACL 2022\nDavani, A. M., Díaz, M., & Prabhakaran, V., 2022\nBiester et al., NLPerspectives 2022\n\nYou use Nadeem et al. 2021 as your seed set - consider Blodgett et al., ACL-IJCNLP 2021 for a critique of this corpus.  \n\nI suggest citing the official version of i.e. Blodgett et al. 2020 (amongst others) rather than the arxiv version.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 139 - I’m not sure what is meant by “However, due to the absence of words with explicit negative connotations”\n\nLine 244 - there may be something off with your formatting as this line seems to float between two others. \n\nLine 248 - what is meant by “colored” here?\n\nLine 254 - implicit *and* neutral (?) \n\nLine 306 - I would perhaps say “perceived gender bias” is based on lived experience. \n\nSection 3.2 informs your choice of seeds so would be more clearly presented before describing the seed sets. \n\nSection 3.3 - please define the process of conversion when you introduce it\n\nTable 3 - might it be possible to see more than four examples? I think this could be interesting to the reader. \n\nI don’t place much confidence in the “reasoning” that language models provide for their answers, so I was unsurprised to see that the scores assigned by GPT-4 did not align with the provided reasoning. It is with this stance in mind that I suggest that section 7 could be shortened. Also, some readers may be put off by the use of very anthropomorphic language in this section i.e. “it hedges itself” (Line 583)."}}]

Author rebuttal:

Meta-review: This paper contributes a new dataset for the analysis of gender bias in LLMs, obtained by prompting GPT3.5-Turbo via in-context learning, and a method for moving from binary to graded bias annotation based on Best-Worst Scaling. As reviewer XADC summarizes, the resulting data is analyzed through 3 lenses: inter-annotator agreement, sample themes and baseline classification tasks, revlealing that annotators overall consider attacks on groups to be more biased than attacks on individuals, and that LLMs such as GPT4 cannot reason well about fine-grained gender bias classification.

The reviewers agree that the paper is overall sound, with adequate description of the methodology and support for the claims.
The paper and data leave open some questions around subjectivity in annotation and the impact of the gender of annotators, which are particularly important from a human-centered NLP perspective. However, the current scope of the work represents an interesting step forward in characterizing gender bias in NLP systems. 

I recommend incorporating the discussion of related work, limitations, motivation, and examples that were brought up in the thorough discussion with reviewers. As a minor point, it would be useful to preview the findings of the paper in the abstract, as it currently describes what was done, but leaves the reader wondering what was learned.
Decision: Accept-Main