Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions
Year: 2023

Abstract:
Caution: this paper contains potentially offensive or upsetting model outputs. Societal biases present in pre-trained large language models are a critical issue as these models have been shown to propagate biases in countless downstream applications, rendering them unfair towards specific groups of people. Since large-scale retraining of these models from scratch is both time and computeexpensive, a variety of approaches have been previously proposed that de-bias a pre-trained model. While the majority of current state-ofthe-art debiasing methods focus on changes to the training regime, in this paper, we propose data intervention strategies as a powerful yet simple technique to reduce gender bias in pretrained models. Specifically, we empirically show that by fine-tuning a pre-trained model on only 10 de-biased (intervened) training examples, the tendency to favor any gender is significantly reduced. Since our proposed method only needs a few training examples, our fewshot debiasing approach is highly feasible and practical. Through extensive experimentation, we show that our debiasing technique performs better than competitive state-of-the-art baselines with minimal loss in language modeling ability.

heading: 1 Introduction
text: Recently, there has been a surge of interest in pretrained large language models (LLM) in natural language processing (NLP). It has been shown that the pre-training + finetuning of a model drastically improves its performance on downstream tasks as the knowledge captured by the pre-training on a large corpus is transferred to the downstream application when finetuning the model. However, this also leads to societal biases like gender bias that were implicitly learned by the pre-trained models being transferred to crucial downstream applications like job recommendation engines (Zhao et al., 2019;
∗Equal Contribution
Barocas et al., 2017; Kurita et al., 2019). Analyzing and mitigating bias without requiring significant re-training or compute resources is crucial to the widespread adoption of LLMs in downstream applications.
Previous work (Nadeem et al., 2021), (Nangia et al., 2020a), (Cer et al., 2018) has attempted to quantify bias, and others such as Ravfogel et al. (2020) and Liang et al. (2021) have attempted to remove it algorithmically from the models. Closer to our work are data-manipulative techniques such as Zmigrod et al. (2019) and Maudslay et al. (2019) that modify the dataset and further fine-tune the model. In this paper, we propose simple data intervention strategies and show that they can mitigate gender bias in pre-trained models with the help of few-shot fine-tuning. Moreover, taking inspiration from Schick et al. (2021), we find that by utilizing a biased pre-trained LLM for mining for most gender-biased samples in a dataset, our methods can mitigate gender bias with very few training samples. Finally, we perform an extensive evaluation of our debiasing technique on two recent bias benchmarks (Nadeem et al., 2021) and show that our method outperforms three existing state-of-the-art techniques and performs comparably to the other two. Our main contributions are the following:
• We propose simple data intervention techniques that can be used to reduce gender bias in a pre-trained LLM with few training examples (few-shot), thus making human-in-theloop bias mitigation strategies feasible.
• We introduce a novel data sampling technique that utilises LLMs to mine for the most biased samples from a dataset and can benefit existing state-of-the-art debiasing methods. When used for debiasing a model, these few samples serve as exemplars and induce large reductions in gender bias.
340

heading: 2 Related Work
text: In recent years, there has been growing concern about the bias/stereotypical discriminatory behavior by NLP models, particularly concerning gender. Several studies have investigated the presence of gender bias in various NLP tasks and proposed methods for mitigating it.
One line of research has focused on analyzing the extent of gender bias in pre-trained language models such as BERT and GPT-2. These studies have found that these models exhibit a significant amount of gender bias in their word embeddings for BERT (Jentzsch and Turan, 2022) and for GPT2 (Kirk et al., 2021) and are prone to making stereotypical gender-based predictions (e.g., assuming that a doctor is male and a nurse is female). A standard evaluation metric used in this line of research is Stereotype metrics such as StereoSet (Nadeem et al., 2021), which evaluates the model’s ability to predict gender stereotypes and CrowS pairs (Nangia et al., 2020b) which measure whether a model generally prefers more stereotypical sentences. A similar line of work is gender bias tests proposed in BIG-bench (Srivastava et al., 2022). The tests assess the language model’s gender biases, stereotypes, and ability to infer gender information. It evaluates gender bias and stereotype between male
and female, and gender minority bias and stereotype between majority and minority. It also examines the model’s language modeling performance, which can be affected during de-biasing.
Another line of research has proposed methods for debiasing these models. These methods can be broadly categorized into two groups: data-based and algorithm-based. Data-based methods aim to reduce bias by removing or altering biased words from the training set. In contrast, algorithm-based methods aim to modify the model’s architecture or training procedure to reduce bias. One popular databased method is "uncertainty sampling" (Lewis and Gale, 1994), where the model is trained on the instances that it is most uncertain about, which can help to reduce bias by forcing the model to learn from a diverse set of examples. A popular algorithmbased method is "Adversarial Debiasing" proposed by Zhang et al. (2018), which fine-tunes the model using an adversarial loss to make it less sensitive to sensitive attributes such as gender. OSCar proposed by Dev et al. (2021), is another algorithm based method that utilizes the idea of disentangling "problematic concepts" like occupation and gender relationship instead of removing them altogether. MABEL (He et al., 2022) has both algorithm and data-based components, as it first augments the training data by swapping gender words and then applies a contrastive learning objective and alignment via entailment pairs. Their data augmentation strategy is similar in spirit to the data intervention techniques we propose, however our analysis does not require training auxiliary models and uses significantly lesser data.
Data-based methods include the "Equalization" technique proposed by Bolukbasi et al. (2016), which aims to equalize the representation of genderspecific words in the embedding space, the "Counterfactual Data Augmentation" (CDA) method proposed by Zimmermann and Hoffmann (2022), which generates counterfactual examples to improve the model’s robustness to bias, and "NameBased Counterfactual Data Substitution" proposed by Maudslay et al. (2019) which reduces gender bias by replacing gender-informative names in the dataset with gender-neutral names. Our proposed method is also a data-based method, which aims to effectively reduce gender bias by taking inspiration from different techniques such as uncertainty sampling and name-based counterfactual data substitution (Maudslay et al., 2019).

heading: 3 Probing Bias in Large Language Models
text: Pre-trained LLMs are biased towards different genders, as seen in a simple mask-fill experiment using BERT. (Here, and in the rest of the paper, we assume a binary treatment of gender for simplicity.) The task is then to mask out the gender-related nouns and pronouns (such as he, she, her, woman, etc.) and get BERT to predict the masked words for the affected sequences in the dataset. Here, we consider a fixed list of gender-specific words curated from previous work (Lu et al., 2018; Zmigrod et al., 2019) and neutral words list1. We finally compute the "total confidence difference" as the sum of differences in the model’s prediction confidence for each gender-word pair (such as confidence of predicting he − she, man − woman, etc.). Formally, we define total confidence difference as |∑Ni=0(f(x (i) female) − f(x (i) male))| where f(x) represent the confidence of model’s prediction, N is the total number of tokens in the dataset and x is the tokenized gender word. The higher this number, the more biased the model is concluded to be. We compute the metric at token level and ensure that each of the gender word gets tokenized into exactly one token by initially extending the tokenizer with our gender word list. The top 3 biased gender-word pairs in StereoSet are shown in Table 1. Intuitively, our technique for gauging bias in LLMs is sensitive to the fixed word list used to represent the sensitive attributes (here, gender). In Table 2, we show the number of words covered by the word list used for both WikiText-2 and StereoSet datasets.

heading: 4 Data Interventions
text: In order to reduce gender bias in pre-trained models, we carefully select diverse and hard-biased examples and then replace gender words with more neu-
tral or equality-focused phrases. This is achieved by using a wordlist to find gender terms in sentences and then segregating words as name and non-name words.
We call our initial approach naive-masking as it does not require a word list for mapping gender words to gender-neutral words. Instead, it replaces all gender words with the fixed word "person." In our next approach, neutral-masking, we swap words in a slightly more semantically accurate manner. In this, we use a word-pair list that goes from gender words to gender-neutral words. With both approaches, we intend to introduce new words in a model’s vocabulary to make it more likely to choose a more neutral word in gender-biased sentences.
In our final approach, we exploit the existing vocabulary of the model and try to balance the confidence of prediction on opposite-gender words by using phrases instead. Thus, we call our final approach random-phrase-masking as we instead substitute words with phrases that reflect the equality of gender. This approach not only reduces gender bias but also preserves the original meaning of the sentence in most cases. In our approach, we chose the phrases and order of gender words at random with equal probability.
Intervention Input word Converted word
naive-masking he person she person boy person
neutral-masking he they her their
schoolgirl schoolkid
random-phrase-masking he he or she she she and he boy either girl or boy
Additionally, we hypothesize that the choice of the dataset for fine-tuning is also essential. We choose two datasets: the WikiText-2 (Merity et al., 2017) dataset, which has implicit gender bias since its sources from Wikipedia articles, and the StereoSet dataset (Nadeem et al., 2021), which has explicit/more gender bias as it has been designed to evaluate gender bias. WikiText-22 has 600 train articles and roughly 2M tokens while StereoSet3 (dev) has 2123 samples out of which we only consider 800 samples which are not unrelated. Naturally, our data intervention method should work better on a dataset with training examples with gender bias while being devoid of meaningful gender associations like "She needs a gynecologist," where the gender of the person is important. By testing our method on both datasets, we can understand the sensitivity of our approach to the quality of training samples used.

heading: 5 Bias Evaluation Metrics
text: We focus on evaluating the bias of a model while also measuring its language modeling capability. The ideal model would not just be one with the least bias but also one which does not compromise its language modeling performance. The dual estimation of bias and performance of a model was proposed in the StereoSet benchmark (Nadeem et al., 2021), with the Language Modeling Score (LMS) measuring the percentage of times a meaningful token is predicted for the mask as opposed to a meaningless token, the Stereotype Score (SS) measuring the percentage of times the model predicted a stereotypical word as compared to an anti-stereotypical word, and an idealized CAT score (ICAT) combining the LMS and SS score into a single metric. An ideal model has an ICAT score of 100, while the worst biased model has an ICAT score of 0. We additionally evaluate the CrowS-Pairs benchmark (Nangia et al., 2020a), which captures data with greater diversity in both the stereotypes expressed and the structure of sentences (50 is ideal). However, we note that the Crow-S benchmark is much more limited compared to StereoSet (Nadeem et al., 2021) in terms of both the volume and variety of linguistic phenomenon relating to gender bias it covers.
2An English language dataset (Creative Commons Attribution-ShareAlike License).
3An English language dataset available at bias-bench (Creative Commons Attribution-ShareAlike 4.0 International Public License)

heading: 6 Experiments
text: We compare our proposed interventions with five baselines, 4 of which are state-of-the-art methods and the original pre-trained model. Our first baseline is the application of dropouts to neural networks, Dropout proposed by (Webster et al., 2020). Next, we consider an algorithmic de-biasing technique INLP technique proposed by (Ravfogel et al., 2020). Then, we consider a sentence embedding de-biasing approach SentenceDebias (Liang et al., 2020). Finally, we consider a data-based approach CDA (Zmigrod et al., 2019) that is closest to our work. For a fairer comparison, we run the baselines with the same size (100) of the training set as our method. For all of our experiments, we consider the “bert-base-uncased” pre-trained model available from HuggingFace. For fine-tuning our model, we select a varying number of most-biased training samples (10, 50, and 100) from the WikiText-2 and StereoSet (we only use the dev set) datasets, as discussed in section 4. We also compare this to a random selection of data points as an ablation study. On the selected dataset, we apply our interventions and obtain the modified dataset, which is then used to fine-tune our pre-trained model using masked language modeling (MLM) loss. The key point is that we only fine-tune the model on the gender words conditioned on the remaining text, significantly reducing the fine-tuning time. We perform ablations on various types of interventions as discussed in Table 7. The model is trained for 30 epochs, with a learning rate of 0.001 and AdamW optimizer. We ran all of our experiments on NVIDIA Tesla T4 GPU on Google Colab for roughly 48 hours. For all experiments, we report the numbers as the mean and standard deviations (6) of 3 different runs. Our experiment code can be found here.4

heading: 7 Results
text: Table 4 shows the StereoSet and Crow-S scores for our baselines and our best-performing interventions on the WikiText-2 Dataset. In the StereoSet benchmark, we observe that random-phrase-masking obtains lower SS than all other baselines. On the Crow-S benchmark, random-phrase-masking does better than thre of the baselines except SentenceDebias which achieves slightly better scores. While random-phrase-masking results in lower SS scores than neutral-masking, it also obtained
4https://github.com/himansh005/data_debias
very low LMS scores. We attribute this performance degradation to the blunt substitution of phrases that our method uses, which might lead to odd-sounding sentences. In the CrowS benchmarks, we see similar behavior and find that random-phrase-masking does better than neutral-masking. Since we believe that our method is sensitive to the choice of the dataset, we also present results on the StereoSet (dev) dataset 6. In Figure 2, we perform a qualitative analysis of our proposed approach and find that random-phrase-masking is able to flip the predictions on fill-mask tasks for stereotypical sentences.

heading: 8 Conclusion
text: In this paper, we show that simple data interventions on limited training data effectively reduce gender bias in LLMs. We also show that a biased pretrained LLM can be used to mine the most effective de-biasing training examples. Evaluation of our methods on state-of-the-art bias benchmarks empirically suggests that our methods effectively reduce gender bias. Given that our methods can work in a few-shot manner and do not require any auxiliary model training, we hope that our work benefits further research in the domain of human-in-the-loop bias mitigation techniques by making the creation of bias mitigation datasets feasible.

heading: 9 Limitations
text: Our proposed method has the following main limitations which we believe are important directions for future work to address:
1. Gender dependency: Our approach does not account for sentences that only make sense for a single gender. For example, sentences like "She needs to see a gynecologist" would not be captured by our method. This is a common problem encountered by most debiasing algorithms as it is difficult to distinguish these.
2. Finite wordlist: The wordlist does not contain all gender-based words as the language con-
tinues to evolve. We believe that future works could employ better approaches that can automatically mine gender words relevant to a dataset.
3. Blunt substitution: The phrase substitution method is an improvement over direct word substitution, but there are still plenty of instances where the new sentence might be semantically incorrect. This does not have any major implication on inference as we are only doing few-shot learning, but it should not be extended to the entire dataset.
4. Binary gender: The method only focuses on the male and female gender. It does not consider non-binary or gender-neutral pronouns such as "ze/hir." This can be solved by using an updated wordlist, but the authors could not come across one at the time of writing.
5. Downstream analyses: While our work proposes methods that show reduced gender bias as per a set of metrics, the work in no way claims to reduce gender bias in general, especially on downstream tasks. However, we strongly believe that this technique holds potential to reduce gender bias on downstream tasks as well since we adopt a regular finetuning approach and focus mainly on better data interventions. Moreover, recent research has shown that fine-tuning-based debiasing approaches do not damage a model’s internal representations to a critical extent (Meade et al., 2022).
Overall, these limitations suggest that our approach may not be suitable for use in contexts where gender-specific or non-binary language is prevalent, and the underlying wordlist should be frequently updated.

heading: 10 Ethics Statement
text: This study was conducted in accordance with ethical principles and guidelines. The study was designed to provide beneficial knowledge and not harm any group or individual. We recognize that the wordlist we use might not represent all contexts of gender bias and that our debiasing method does not cover all contexts of occurrences of gender bias. However, we made sure to consider the ethical implications of our methodologies and the results of our analysis. The authors have tried to ensure the method
does not amplify any other inherent bias but also acknowledge that our approach may have limitations. We take responsibility for any ethical concerns that may arise as a result of our research.

heading: Acknowledgments
text: This material is based upon work partially supported by the National Science Foundation (Awards #1722822 and #1750439) and National Institutes of Health (Awards #R01MH125740, #R01MH096951, and #U01MH116925). PPL is partially supported by a Facebook PhD Fellowship and a Carnegie Mellon University’s Center for Machine Learning and Health Fellowship. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF, NIH, Facebook, or CMLH, and no official endorsement should be inferred. Additionally, we express our appreciation to the anonymous reviewers for their insightful suggestions, which greatly improved our work. Furthermore, we would like to acknowledge the contributions of our colleagues, Atishay Jain and Praneetha Vaddamanu, who played a significant role in the development of this research.

heading: A Appendix
text: A.1 Dataset Bias Analysis
To gauge the feasibility of using a wordlist based intervention approach, we first analyze our datasets for occurrences of gender words. As shown in the word cloud 4, gender pronouns are the mostfrequent word in our datasets. Moreover, as per Figure 1, "she," "he," and "her" are the top three most frequently occurring words in our dataset. This suggests that we can definitely detect gender words in our corpus and apply our interventions.
A.2 Sensitivity to Choice of Dataset
To understand the effectiveness of our proposed data-interventions, we study apply our methods to two datasets under varying number of training samples (10, 50 and 100) and selection strategies (most biased first and random) as per Table 6. Our methods obtain better results on StereoSet (dev) dataset. One reason this could happen is due to the fact that StereoSet has explicit gender bias, thus it would be less likely for a sentence like "She needs a gynaecologist" to appear on it. Because our interventions perform blunt substitutions, this sentence might become incorrect due to our method - "Either he or she needs a gynaecologist".
A.3 Sensitivity to Number of Training Samples and Sampling Strategy
As per Figure 5, When we vary the number of training samples, we observe that the difference in performance is not huge when we transition from 10 to 100 samples, thus suggesting that our method
is capable of few-shot fine-tuning. Moreover, sampling the most biased data points helps our methods achieve better performance consistently, as shown in Figure 5 and Table 6. Table ?? shows some top three most gender biased entries found in the StereoSet dataset.
A.4 Ablations of interventions We study the effects of choosing different ways of replacement for name and non-name words. In addition to our three interventions proposed previously, we also experimented with a couple of others. In female-first-random-phrase-masking, we always keep the female gendered word before a male word. We wanted to understand if the order of gender words encountered by a model renders any effect on the debiasing. In Table 7, we see that it does not perform any better than random-phrase-masking. Then, we also try fixing the phrases from random-phrase-masking, thus making it fixed-phrase-masking. We obtain 4 variants of this method corresponding to the following four phrases:
1. both [1] and [2]
2. [1] and [2]
3. [1] or [2]
4. either [1] or [2]
Here, [1] and [2] are substituted with opposite gender words. As we observe in Table 7, fixed-phrase-masking-3 obtains the lowest StereoSet Gender SS score out of all our intervention methods. Similarily, naive-masking obtains the lowest Crow-S pair score.
ACL 2023 Responsible NLP Checklist
A For every submission: 3 A1. Did you describe the limitations of your work?
9
3 A2. Did you discuss any potential risks of your work? 10
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets used are open-source.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data is a general vocabulary. Hence, none of the used data contains any personal information.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The dataset pages have all the needed information
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 7
C 3 Did you run computational experiments? 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 6
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Our proposed method has the following main limitations which we believe are important directions for future work to address:
1. Gender dependency: Our approach does not account for sentences that only make sense for a single gender. For example, sentences like "She needs to see a gynecologist" would not be captured by our method. This is a common problem encountered by most debiasing algorithms as it is difficult to distinguish these. 2. Finite wordlist: The wordlist does not contain all gender-based words as the language con-
tinues to evolve. We believe that future works could employ better approaches that can automatically mine gender words relevant to a dataset. 3. Blunt substitution: The phrase substitution method is an improvement over direct word substitution, but there are still plenty of instances where the new sentence might be semantically incorrect. This does not have any major implication on inference as we are only doing few-shot learning, but it should not be extended to the entire dataset. 4. Binary gender: The method only focuses on the male and female gender. It does not consider non-binary or gender-neutral pronouns such as "ze/hir. " This can be solved by using an updated wordlist, but the authors could not come across one at the time of writing. 5. Downstream analyses: While our work proposes methods that show reduced gender bias as per a set of metrics, the work in no way claims to reduce gender bias in general, especially on downstream tasks. However, we strongly believe that this technique holds potential to reduce gender bias on downstream tasks as well since we adopt a regular finetuning approach and focus mainly on better data interventions. Moreover, recent research has shown that fine-tuning-based debiasing approaches do not damage a model’s internal representations to a critical extent (Meade et al. , 2022). Overall, these limitations suggest that our approach may not be suitable for use in contexts where gender-specific or non-binary language is prevalent, and the underlying wordlist should be frequently updated