A Parameter-Efficient Multi-Objective Approach to Mitigate Stereotypical Bias in Language Models (2024)

Abstract: Pre-trained language models have shown impressive abilities of understanding and generating natural languages. However, they typically inherit undesired human-like bias and stereotypes from training data, which raises concerns about putting these models into use in real-world scenarios. Although prior research has proposed to reduce bias using different fairness objectives, they usually fail to capture different representations of bias and, therefore, struggle with fully debiasing models. In this work, we introduce a multi-objective probability alignment approach to overcome current challenges by incorporating multiple debiasing losses to locate and penalize bias in different forms. Compared to existing methods, our proposed method can more effectively and comprehensively reduce stereotypical bias, and maintains the language ability of pre-trained models at the same time. Besides, we adopt prefix-tuning to optimize fairness objectives, and results show that it can achieve better bias removal than full fine-tuning while requiring much fewer computational resources. Our code and data are available at https://github.com/Ewanwong/debias_NLG.

heading: 1 Introduction
text: Language models (LMs) pre-trained on large-scale self-supervised datasets have shown impressive capacities in various natural language processing (NLP) tasks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; Liu et al., 2019; Lan et al., 2020). In particular, pre-trained generative LMs, e.g., GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), OPT (Zhang et al., 2022) and GPT-4 (OpenAI, 2023), have gained great attention from both academic communities and non-expert users, due to their remarkable instruction-following and zero-shot task adaptation abilities (Brown et al., 2020; OpenAI, 2023; Wei et al., 2022).
Despite their remarkable achievements and great
practical values, potential ethical risks cannot be neglected. Since these pre-trained LMs are mostly trained on online datasets, their training data is likely to contain undesired patterns including toxic speech and social biases (Zhao et al., 2019; Tan and Celis, 2019). Numerous experiments have revealed that LMs trained in these datasets also demonstrate similar social biases, raising concerns that they could amplify biases and discrimination against disadvantaged demographics (Zhao et al., 2019; May et al., 2019; Tan and Celis, 2019; Bommasani et al., 2020; Guo and Caliskan, 2021; Kurita et al., 2019; Brown et al., 2020; Sheng et al., 2019; Yeo and Chen, 2020). Recently, several methods for reducing stereotypical biases have been proposed (Barikeri et al., 2021; Bommasani et al., 2020; Kaneko and Bollegala, 2021). However, most methods neglect the fact that bias can be represented in various forms in LMs. For example, LMs can violate equal social group associations by predicting different occupations for male and female genders, or violate equal neutral neutral associations by believing that criminals are more likely to be people of color (Gallegos et al., 2023). In addition, biased LMs generate sentences containing higher-level disparity, such as sentiment (Huang et al., 2020) and regard (Sheng et al., 2019) for different demographics, demonstrating global bias (Liang et al., 2021). As a result, methods targeting only one specific form of bias can lead to incomplete bias removal and unsatisfactory debiasing performance.
Besides, the increasing scale of pre-trained LMs boosts the design and application of parameterefficient fine-tuning methods (Houlsby et al., 2019; Lester et al., 2021; Li and Liang, 2021; Hu et al., 2022). Unfortunately, relatively little work has been devoted to studying parameter-efficient methods in the field of bias mitigation (Lauscher et al., 2021; Gira et al., 2022; Xie and Lukasiewicz, 2023). In this work, we also aim to further explore lightweight debiasing techniques using parameter-
1
efficient fine-tuning methods. The main contribution of this work includes:
1. We refine and integrate existing probabilistic alignment debiasing approaches to simultaneously address multiple forms of bias representation, employing a parameter-efficient prefix-tuning technique for implementation.
2. We empirically demonstrate the effectiveness of our method on diverse intrinsic and extrinsic bias evaluation benchmarks and compared it with existing debiasing techniques.
3. We thoroughly analyze our parameter-efficient debiasing framework and show that it can achieve better bias mitigation performance and parameter efficiency than full fine-tuning. Additionally, our method is effective in reducing bias in large LMs.

heading: 2 Bias Statement
text: In this work, we mainly address stereotypical bias, with binary gender bias as an example1. We define stereotypical bias as an overgeneralized belief about a particular group of people that can hurt target groups (Nadeem et al., 2021). "Women are bad drivers" and "Asians are good at math", for instance, are gender and racial stereotypical biases. Generative LMs can also contain such bias. For example, "doctor" can receive a higher probability when conditioned on "he worked as a [BLANK]" than "she worked as a [BLANK]" (Liang et al., 2021). Unlike discrimination, stereotypical bias is more implicit and thus can cause both representational and allocational harms (Blodgett et al., 2020) to target groups without them being aware of it. As is commonly seen in our society, boys and girls are encouraged to engage in different activities and expected to possess different characteristics during their childhood, and those gender-related expectations might affect their future academic success and career choices (Olsson and Martiny, 2018). Since people are increasingly turning to LLMs for advice giving or decision making, reducing stereotypical bias in LLMs is of practical relevance.

heading: 3 Related Work
text: Bias in NLP systems Stereotypical bias can manifest itself in different forms in LMs (Gallegos et al.,
1We recognize that gender is non-binary and in Section 4.2 we formulate our training objective in a way that can handle non-binary gender bias as well.
2023). Geometric relationships in model representations, for example, can encode stereotypical associations between genders and occupations (Bolukbasi et al., 2016; Caliskan et al., 2017; Zhao et al., 2019; May et al., 2019; Tan and Celis, 2019; Bommasani et al., 2020). Bias is also indicated by various divergence of probabilities from LMs. Kurita et al. (2019) and Brown et al. (2020) observed different probabilities predicted by both masked LMs and generative LMs for male and female genders given stereotypical attributes; Liang et al. (2021) identified local bias as different next token probability distributions conditioned on same contexts with only social group swapped; Barikeri et al. (2021) additionally considered difference in probabilities assigned to whole sentence pairs which are minimally different in social groups, which corresponds to global bias defined in Liang et al. (2021). Bias can also be observed as disparity in model generation (Sheng et al., 2019; Yeo and Chen, 2020) and performance in downstream tasks, such as toxicity detection (Sap et al., 2022) and coreference resolution (Kurita et al., 2019). In this work, we mainly mitigate bias reflected by divergent probability distributions predicted by LMs.
Mitigating bias in pre-trained LMs While many studies aimed to train fair LMs from scratch by constructing fairer datasets (Zhao et al., 2019; Zmigrod et al., 2019), it can be computationally expensive and not always feasible in practice. As a result, much effort has been put into mitigating bias from pre-trained LMs via debiasing finetuning. Kaneko and Bollegala (2021) extended projection-based methods from static word embeddings (Bolukbasi et al., 2016) and fine-tuned models to output orthogonal contextualized representations for gendered and stereotypical words. However, Gonen and Goldberg (2019) argued that projection-based methods did not completely capture and remove bias. Other experiments involved introducing fairness regularization operating on probability level. Qian et al. (2019) and Garimella et al. (2021) proposed equalizing losses to assign similar probabilities to male and female words, and Guo et al. (2022) aligned the distributions of neutral words given the same prompts with different demographic groups. However, as bias can have different notions and forms in LMs (Kaneko and Bollegala, 2019; Gallegos et al., 2023), failure of existing studies to address multiple forms of bias can lead to suboptimal debiasing results, especially
when they are evaluated on different benchmarks. To overcome these limitations, in this work we will explore simultaneously mitigating multiple bias representations in LMs. Barikeri et al.’s (2021) work is the most similar to ours in adopting multiple fairness objectives, but their method focused on bias as geometrical relations between words, while ours more explicitly aligns the model predictions for different demographics.
Parameter-efficient fine-tuning As pre-trained LMs are growing ever larger (Brown et al., 2020; Zhang et al., 2022; Raffel et al., 2020; OpenAI, 2023), fine-tuning the whole model is also becoming impractical due to high computational costs. Consequently, tuning only a small proportion of model parameters, namely parameter-efficient finetuning, is gaining popularity. Houlsby et al. (2019) proposed adapter tuning that inserted and tuned adapter modules on downstream tasks with other model parameters fixed. Lester et al. (2021); Li and Liang (2021) experimented with training continuous prompts to adapt LMs to new domains. Low-rank adaptation (LoRA) represented parameter updates using low-rank matrices and efficiently updated them during fine-tuning (Hu et al., 2022). Empirical results showed these parameter-efficient fine-tuning methods can obtain competitive performance compared to full fine-tuning while requiring much fewer computational resources.
Parameter-efficient fine-tuning approaches have also been applied to debias pre-trained LMs. Lauscher et al. (2021) adopted an adapter module to update pre-trained LMs on a fair dataset. Sheng et al. (2020) learned discrete prompts to in-
duce or reduce gender bias in generative LMs. Gira et al. (2022) experimented with only fine-tuning a small proportion of model parameters on genderfair datasets. ADEPT (Yang et al., 2023) applied prefix-tuning to debias BERT with a manifold learning loss. Guo et al. (2022) and Li et al. (2023) both searched prefixes to construct adversarial training data. Recently, Xie and Lukasiewicz (2023) empirically evaluated various parameter-efficient debiasing methods and showed promising results. In this work, we focus on prefix-tuning (Li and Liang, 2021) to demonstrate the efficacy of parameterefficient fine-tuning for multi-objective debiasing, as it avoids the inference overhead and complex hyperparameter selection present in adapter tuning and LoRA.

heading: 4 Debiasing LMs by Multi-Objective Probability Alignment
text: In this section, we present our method which simultaneously mitigates different forms of gender bias in LMs via probability alignment. Besides, our method updates LMs using prefix-tuning, which leads to more efficient model debiasing. An overview of our pipeline is shown in Figure 1.

heading: 4.1 Task Formulation
text: Following previous work (Caliskan et al., 2017; Guo et al., 2022), we first define the target and attribute words: Target words are paired words related to demographic groups that can define a bias direction (e.g., female-male, she-he), and attribute words are gender-neutral words yet containing stereotypical associations with certain target groups
(e.g., “programmer”, “technology”). In the following parts of the work, we use a set of m-tuples C = {(c(1)1 , c (2) 1 , ..., c (m) 1 ), (c (1) 2 , c (2) 2 , ..., c (m) 2 ), ...} to denote target words and W = {w1, w2, ...} to denote attribute words. A set of sentences containing at least one target word in C and one attribute word in W can then be collected from natural language datasets. After applying counterfactual data augmentation (Zhao et al., 2019; Zmigrod et al., 2019) by replacing target words with their opposite gender counterparts, we can obtain our training dataset S = {(s(1)1 , s (2) 1 , ..., s (m) 1 ), (s (1) 2 , s (2) 2 , ..., s (m) 2 ), ...}. As we address binary gender bias in this work, m = 2 and C and S are sets of two-tuples. We omit subscripts of C and S when there is no ambiguity.
In this work, we mainly explore debiasing generative LMs using prefix-tuning, that is, learning a task-specific continuous prompt to steer model generation without varying pre-trained parameters. Assume that we have a pre-trained decoder-only model parameterized by ϕ. In prefix-tuning, we introduce a small set of trainable parameters Pθ that we call the "prefix". They are essential a set of key and value pairs of each token in an imaginary continuous prompt, which can affect generation when following tokens attend to it. During training, ϕ is frozen and only Pθ is optimized against designed objectives. For further details please refer to Li and Liang (2021). We also adopt the re-parameterization method in their work for stable training.
With these concepts defined, the task can be formulated as: given attribute words W , target words C, dataset S and a pre-trained generative model LMϕ, train a prefix Pθ that mitigates different forms of stereotypical gender bias in the model.

heading: 4.2 Debiasing Objectives
text: After collecting training data S, we can then debias LMϕ by learning Pθ. In our proposed method, we introduce multiple fairness objectives, corresponding to different types of bias we aim to reduce. Specifically, Pθ is learned by minimizing the following debiasing losses:
Language Modeling Loss As previous work pointed out, bias in pre-trained models can be attributed to the selection and amplification biases in imbalanced training data (Zhao et al., 2019; Tan and Celis, 2019; Shah et al., 2020), thus tuning the model on counterfactually augmented data can
mitigate stereotypical associations and reduce bias (Zhao et al., 2019; Zmigrod et al., 2019). Here we optimize the prefix matrix to minimize language modeling loss (LLM ), which is the negative loglikelihood (NLL) loss on S:
LLM = − 1
|X| log(Pϕ(X|Pθ))
= − 1|X|
|X|∑
i
log(Pϕ(Xi|X<i;Pθ))
Neutralization Loss To further dissociate target and attribute concepts, we then introduce neutralization loss (Lneu) to inform models where to find the bias. Neutralization loss is intended to achieve equal social group association that a neutral word should be equally likely given its context regardless of social groups (Gallegos et al., 2023). We borrow the approach from Auto-Debias (Guo et al., 2022) to penalize Jenson-Shannon divergence (JSD) between predicted next token distributions conditioned on paired contexts:
Lneu = JSD(p1, p2)
= 1
2
∑
i∈{1,2} KLD(pi||
p1 + p2 2 )
where p1 and p2 are normalized probability distributions over W given original and counterfactually augmented contexts in S. Unlike Auto-Debias which minimized JSD on non-sensible prompts, we apply neutralization loss to natural language sentences, believing this can better maintain the language ability of models.
Equalizing Loss Another type of fairness we aim to achieve is equal neutral association, namely target words in the same tuple should be equally likely in a neutral context. Qian et al. (2019) proposed to penalize the predicted probability difference by
Leq = 1 |C| ∑|C| i |log p(c (1) i )
p(c (2) i ) |. However, it can be easily observed that this loss is too coarse-grained in that it should not penalize positions where gendered information is present in the context. For example, in the sentence “The little girl is actually a famous [actress/actor].”, equalizing the probabilities of “actress” and “actor” hurts the language modeling capacity. Therefore, some modifications are made to the equalizing loss in our approach:
Firstly, we introduce a simple yet effective vocabulary-based data selection process: we only
penalize equalizing loss when there is at least one attribute word and no target word ahead of the current position. By filtering out positions with gendered context, we ensure that we are not penalizing reasonable predictions from the model, and keeping at least one attribute word in the context can better dissociate target and attribute concepts.
Secondly, instead of using the loss from Qian et al. (2019), we re-formulate equalizing loss as the KL-divergence between predicted probability distribution within a target word pair and a uniform distribution. This loss, denoted as Leq_tok, is shown below:
Leq_tok = 1
|C|
|C|∑
i
KLD(q||pi)
where pi is the normalized probability distribution in (c(1)i , c (2) i ) and q is a binary uniform distribution q(c (1) i ) = q(c (2) i ) = 1 2 , encoding our prior belief that both binary genders should be equally likely given the same context.
The advantages of KLD over the original equalizing loss are two-fold: firstly, measuring KLD can be easily extended to multi-class debiasing tasks by replacing the target distribution q with an nclass uniform distribution. Secondly, it allows us the flexibility of introducing desired target distributions other than uniform distributions.
Finally, Liang et al. (2021) categorized bias in LMs into local bias and global bias, and our tokenlevel equalizing loss Leq_tok can only capture local bias. However, some stereotypical bias is not represented by single tokens, but spans multiple words or phrases. To mitigate such global bias, we introduce sequence-level equalizing loss Leq_seq to penalize differences in probabilities assigned to sentence pairs in S. For the same reasons described above, Leq_seq is also defined as KLD between normalized probability distribution within each sentence pair and a uniform distribution.
Leq_seq = KLD(q||p)
where p is the normalized probability distribution in a sentence pair (s(1), s(2)) in S and q is a binary uniform distribution q(s(1)) = q(s(2)) = 12 .
Combining all loss functions described above, we set our final training objective as a weighted sum of these losses, hoping this multi-objective approach can more comprehensively address different
forms of bias in LMs:
L = α1LLM + α2Lneu + α3Leq_tok + α4Leq_seq
We then train a prefix matrix Pθ to minimize the overall loss L on the training data S.

heading: 5 Experiments
text: We evaluate our approach’s performance of mitigating stereotypical bias in a GPT-2 small model on multiple benchmarks and compare its performance to various existing debiasing methods.
Benchmark methods Benchmark methods we consider fall into the following categories depending on which stages they are applied to:
• Pre-training: CDA (Zhao et al., 2019; Zmigrod et al., 2019; Lu et al., 2020) is a commonly used data augmentation method that augments the original biased dataset with synthetic gender-swapped sentences for fairer model pre-training. Dropout dissociates attributes and targets by increasing dropout rate in model pre-training (Webster et al., 2020).
• Fine-tuning: Here we extend the concept of fine-tuning to include both full fine-tuning and parameter-efficient fine-tuning. ContextDebias (Kaneko and Bollegala, 2021) is a projection-based full fine-tuning method that encourages models to encode attribute and target words orthogonally to each other. Controllable-Bias (Sheng et al., 2020) mitigates bias by learning a discrete prompt that reduces negative regards for both genders.
• Post-hoc: Iterative null-space projection (INLP) (Ravfogel et al., 2020) trains a set of linear classifiers to predict genders from embeddings and then projects embeddings to the null-space of learned classifiers. Self-Debias (Schick et al., 2021) adjusts next token probabilities at each step according to model’s prediction to what extent the next token is biased.
As baselines, we also report the performance of vanilla GPT-2 and GPT-2 with randomly initialized prefix.
Dataset In our experiment, we collected sentences with at least one attribute and one target word from the News-Commentary V15 dataset and
obtained our training data of 13995 sentence pair after counterfactual data augmentation. As for benchmark methods, we follow settings from Meade et al. (2022) that pre-training methods (CDA and Dropout) use Wikipedia-10 dump for continued pre-training and INLP uses Wikipedia-2.5 dump to learn linear classifiers.
Bias Word List We use the same target word list as in Zhao et al. (2018b) and combine word lists in Kaneko and Bollegala (2019) and the SemBias dataset Zhao et al. (2018b) as attribute word list. In the end, we have a target word list C of 222 pairs and an attribute word list W of 209 words. The two lists are provided in Appendix A. For a fair comparison, CDA, INLP and Context-Debias are also trained using the same bias word lists.
Evaluation Metrics We adopt various different metrics to comprehensively evaluate the performance of our approach and benchmark methods.
• CrowS-Pairs: CrowS-Pairs (Nangia et al., 2020) consists of pairs of minimally distant sentences, with one sentence expressing stereotype while the other being antistereotypical. Stereotypical bias of a model is evaluated as the frequency that it assigns higher probability to stereotypical sentences than anti-stereotypical ones. Ideally, a fullydebiased model should have a score of 50.
• StereoSet: Each example in StereoSet (Nadeem et al., 2021) contains a context and three options: stereotype, anti-stereotype and unrelated. Stereotype score (SS) is computed similarly to CrowS-Pairs as how often model prefers stereotypical options. Besides, language modeling score (LMS) measures how often related options (stereotype or anti-stereotype) rank higher than unrelated options. Finally, idealized context association test (ICAT) score combines SS and LMS. Higher ICAT indicates a better balance between bias reduction and language modeling ability preservation. In our experiment we use both intra- and intersentence subsets of StereoSet, which are fill-in-the blank and next sentence prediction tasks respectively.
• Perplexity: In addition to LMS in StereoSet, we also measure models’ perplexity on 10% of WikiText-2 dataset to reflect their language modeling ability. Lower perplexity indicates
the language ability of pre-trained models is better maintained after debiasing.
• Regard: As the principal application of generative LMs is to produce natural language texts, we also study bias in generation by comparing regard polarity distributions of samples generated by models. We generate 50 samples based on every one of the ten context templates from Sheng et al.’s (2020) work for each gender. Then the regard of all 1000 samples is predicted by a pre-trained classifier to determine whether the distributions for male and female samples are different. To quantitatively measure the effects of debiasing techniques, we compute regard difference and regard shift as the absolute difference between male and female distributions and between a debiased model and vanilla GPT-2 distributions. Higher regard difference implies bias and higher regard shift means a debiasing technique disturbs inherent distribution of pretrained LMs. We provide all context templates we use in Appendix B.
For CrowS-Pairs, perplexity and intrasentence task of StereoSet, we adopt the implementation from Meade et al. (2022)2. We implement intersentence task by ourselves. The pre-trained classifier used in regard experiments is from Sheng et al. (2020)3.
Experiment Setting Following the work of Li and Liang (2021), we train a prefix of length 10 with prefix projection dimension of 800. As GPT-2 works on sub-word level, we only use target pairs and attribute words that can be represented as a single token when computing neutralization and token level equalizing losses. Based on our observations, the counts of stereotypically masculine and feminine words remain roughly equivalent after the filtering process. For convenient selection of hyperparameters, the coefficient α1 for language modeling loss is fixed to be 1, and we find the combination of α2 = 50, α3 = 200, α4 = 250 results in the highest ICAT score on StereoSet validation set after 5 epochs of training. For a fair comparison, CDA, Dropout and Context-Debias checkpoints are also selected according to StereoSet validation set. Further information about experimental details can be found in Appendix C.
2https://github.com/McGillNLP/biasbench 3https://github.com/ewsheng/controllablenlgbiases

heading: 5.1 Automatic Evaluation
text: Results of automatic evaluation are presented in Table 1 and Table 2. For each metric, we mark the best and equivalently good results (i.e., no statistically significant difference from the best score) in bold. Significant improvements over vanilla model are also marked in the tables.4
StereoSet In StereoSet, our method achieves consistently strong performance across different settings (see Table 1): our model shows the lowest degree of bias in the intrasentence task and remains competitive in the intersentence task. It also preserves satisfactory language modeling ability, falling only slightly behind pre-training methods in LMS. As a result, our approach demonstrates the best balance between bias reduction and language ability preservation with significantly higher ICAT scores than vanilla GPT-2 in both tasks, and it exceeds most benchmark methods by a remarkable margin. In comparison, post-hoc approaches (INLP and Self-Debias) generally lead to fairer predictions, yet dramatically hurt model’s language ability. Pre-training and projection-based fine-tuning methods (CDA, Dropout and Context-Debias), on the other hand, obtain decent LMS, whereas they do not guarantee to effectively remove bias. This marks the importance of utilizing more informative and explicit fairness objectives in bias mitigation. Controllable-Bias shows unstable results in two tasks, likely because its training objectives are not directly related to demographic parity.
4We choose different statistical tests for each metric: for LMS and SS we conduct a McNemar test, for perplexity and ICAT score we adopt a paired T-test with bootstrapping, and for regard experiments we run the generation process 5 times with different random seeds and apply a paired T-test.
CrowS-Pairs In Table 2, similar results can be seen on CrowS-Pairs. Post-hoc methods effectively remove bias from vanilla GPT-2. In contrast, CDA and Dropout demonstrate trivial or negative debiasing effects. The observation that Context-Debias achieves lower degree of bias on CrowS-Pairs but not on StereoSet indicates projection-based methods do not generalize to different forms of bias when evaluated on diverse benchmarks. Our model again sees the best debiasing performance, followed by Controllable-Bias. Both methods have produced close-to-zero bias in this metric.
Perplexity All debiasing techniques lead to significantly worse perplexity than vanilla GPT-2. Self-Debias and Controllable-Bias obtain the lowest perplexity among all debiased models, despite the fact that neither method involves modeling human language as optimization objective. Pretraining methods and Context-Debias remain competitive. INLP and our method perform the worst, followed by adding random prefixes. This can be partly explained by discrepancy in domains of training and evaluation data: our debiased model is trained to minimize LLM in news domain, which is different from Wikipedia used for perplexity measurement. Besides, incorporating multiple debiasing losses could impose additional constraints on model training, thereby impairing the language ability. To determine whether the PPL loss is due to domain discrepancy or worse language ability induced by our method, we use human evaluation for a more accurate assessment of the language ability of debiased LMs.
Regard As regard reflects the language polarity towards and social perceptions of a demo-
graphic group, we see a low regard difference as better stereotype reduction in generation. According to results calculated from 1000 examples, our model achieves the lowest regard difference score of merely 0.052. Controllable-Bias, which is trained to align regard polarity using the same set of context templates, also performs strongly in this metric. Both systems significantly reduce the regard difference compared to default generation. Dropout shows only minor improvement, while CDA, Context-Debias and Self-Debias lead to more bias. We also report regard shift i.e., how much the regard distributions of debiased models are different from that of vanilla GPT-2. Our system is by far the worst in regard shift. By manual inspection, we assume this to be another result of overfitting to the training data from news domain: our model frequently generates politics and science related content which are preferred by the regard classifier. Consequently, our model is dramatically more likely to produce sentences with positive regard than the vanilla model. More details and examples can be found in Appendix D.

heading: 5.2 Human Evaluation
text: While automated metrics can quantitatively reflect the degree of bias in models, they may fail to capture more deeply underlying stereotypes, thus human perception is needed for a more accurate evaluation. Following prior work of Liang et al. (2021), we ask annotators to score sentences generated by each model in three dimensions: 1) clarity: coherence and grammatical correctness, 2) content: whether sentences are factually consistent with real
world, and 3) fairness: whether sentences contain discrimination or gender-related stereotypical associations. Each metric is evaluated on a 1-5 scale and each annotators sees 10 pairs of sentences from each model. To better balance between workload and the amount of examples being read, we ask annotators to only provide final scores for systems rather than for each sentence. The questionnaire for human evaluation can be found in Appendix E. A Fleiss’ κ score of 0.055 indicates slight interannotator agreement.
The human evaluation results in Table 3 further confirm the success of our proposed method. Our debiased model slightly underperforms compared to vanilla GPT-2 and Dropout in clarity, with a score comparable to CDA. Meanwhile, it significantly improves the fairness score to 4.50, surpassing other models by a substantial margin. Additionally, the content scores exhibit very small variance, indicating that different debiasing approaches do not significantly disturb factual knowledge in pretrained LMs. These findings suggest that our model can generate coherent and factually accurate sentences while substantially reducing the likelihood of biased and stereotypical outputs.

heading: 5.3 Ablation Study
text: To further demonstrate the effectiveness of our multi-objective probability alignment debiasing method, we run an ablation experiment to study the effect of each fairness objective. Starting from a vanilla model, we add one loss function to the final model at a time and report the performance on StereoSet. The coefficients for each model are re-selected based on the validation set.
As shown in Table 4, the addition of each loss function leads to better SS and ICAT scores compared to the previous model, with the only exception of Lneu in intersentence task. This drop in
performance is then remedied by equalizing losses, especially Leq_seq, which is in accordance with our expectation that Leq_seq can effectively capture and reduce global bias. However, when we remove Lneu from the full system, it leads to worse results (77.602 ICAT score in the intrasentence and 75.207 ICAT score under intersentence settings), which means that Lneu is also indispensable to the success of our final model. Besides, LLM induces worse intersentence LMS due to the fact that our training data consists of only single sentences, and the score increases when other losses are incorporated. The ablation study demonstrates that optimizing multiple fairness objectives simultaneously results in better bias removal.

heading: 5.4 Comparison to Full Fine-Tuning
text: We also compare the performance and parameter efficiency of our prefix-tuning model to a full finetuning setting. We adopt the same debiasing objectives to update all parameters in a GPT-2 small model. The combination of α2 = 200, α3 = 150, α4 = 200 yields the best performance of full finetuning in the validation set and its results are reported.
Table 5 and Table 6 contain our results. It can be seen that full fine-tuning model makes less biased decisions than vanilla GPT-2, but underperforms prefix-tuning on all bias benchmarks, especially StereoSet intrasentence subset. Besides, full finetuning is more likely to overfit training data, giving rise to its high perplexity. Our findings that full finetuning does not lead to better debiasing and can obtain worse perplexity than parameter-efficient methods align with the Xie and Lukasiewicz’s (2023) results. In addition, our prefix-tuning approach only
needs to train as little as approximately 12.36% of parameters compared to full fine-tuning.

heading: 5.5 Effect on Downstream Task
text: To investigate how bias mitigation can affect knowledge transfer of pre-trained LMs, we adapt debiased models to perform downstream tasks. In particular, to better understand the impacts of both debiasing on fine-tuning and fine-tuning on debiasing, we conduct our experiments on a coreference resolution dataset WinoBias (Zhao et al., 2018a), where we can simultaneously evaluate models’ downstream task performance and degree of bias.
Following the practice in Xie and Lukasiewicz (2023), we adapt coreference resolution to a generation task by appending the question "{Pronoun} refers to the {Candidate}" after each example, where {Pronoun} is the expression for which we hope to find the corresponding entity. In the example of "The developer argued with the designer because she did not like the design.", the question will then be "She refers to the {Candidate}." The candidate between developer and designer with higher probability assigned by the model is seen as the model prediction. Specifically, WinoBias provides pairs of examples that differ only in the gender of pronouns, therefore the performance difference between pro- and anti-stereotype subsets can indicate whether models make decisions based on semantic and syntactic knowledge or simply according to stereotypical associations. We report the pro-stereotype, anti-stereotype and average
F1 scores and their differences. We choose only the more challenging Type-1 examples in WinoBias, as models have already achieved nearly perfect performance on Type-2 subset and the results are not informative. Here we fine-tune a CDAdebiased model and a full fine-tuning model trained against our debiasing objective, and prefix-tune our proposed prefix-tuning system on the WinoBias dataset for 20 epochs. The results of vanilla GPT-2 fine-tuning and prefix-tuning are also reported.
In Table 7, CDA and full fine-tuning systems can achieve comparable performance to the finetuned vanilla model, which shows bias mitigation does not necessarily lead to forgetting of knowledge in pre-trained LMs. Similarly, our prefixtuning debiased model outperforms the vanilla model with prefix-tuning. As for bias mitigation, models trained against our proposed training objective (full fine-tuning and prefix-tuning) achieve a competitive debiasing performance even after finetuning on downstream datasets (Diff=-0.21 & 0). Therefore, we conclude that the debiasing effects of our proposed method can still be effectively maintained after downstream fine-tuning, and it does not hurt performance on these tasks.

heading: 5.6 Application to Large Language Models
text: We additionally verify whether our method can be applied to large pre-trained LMs, where parameterefficient fine-tuning methods are particularly necessary. To this end, we test our debiasing technique on two large LMs: GPT-2 XL (Radford et al., 2019) and Llama-2-7b (Touvron et al., 2023). Both models, like GPT-2 small, are auto-regressive models with a decoder-only structure, trained on a causal language modeling task. They consist of approximately 1.5 billion and 7 billion parameters, making them about 12 and 56 times larger than GPT-2 small, respectively. Given the high resource and time costs of training these large models, we adopted the same hyperparameters used in the GPT-2 small experiments without further hyperparameter tuning. We trained GPT-2 XL and
Llama-2-7b for 9 and 3 epochs, respectively, using different random seeds. The results are shown in Table 8.
The StereoSet results of GPT-2 XL and Llama2-7b reveal that large LMs, despite their enhanced expressiveness, exhibit increased bias and stereotypes and perform worse than GPT-2 small. This underscores the critical need for bias evaluation and mitigation in large LMs before deployment. Our results further show that debiased GPT-2 XL and Llama-2-7b produce significantly fewer biased predictions in both intra- and intersentence tasks compared to vanilla models. While debiasing results in a drop in language modeling scores, both debiased models achieve higher ICAT scores in all settings. These findings confirm the efficacy of our debiasing method in mitigating stereotypical bias in large LMs.
Notably, Llama-2-7b shows high variance in performance across runs with different random seeds and epochs, with the worst models performing nearly at random. This issue can be mitigated by training for more epochs, and we believe that tuning hyperparameters on the validation set can enhance both performance and robustness.

heading: 5.7 Error Analysis
text: Despite improved debiasing performance, our model still commits certain errors. Therefore, we manually analyzed 100 sentences generated in the regard experiment by our model and report types and numbers of errors we observe.
According to manual inspections, the model is almost entirely free of occupational stereotypes, associating all types of professions with both genders with equal chance. However, we discover a distribution shift of the predicted occupations, as our
model mentions jobs relevant to politics, academia, and scientific research in 24 sentences, compared to only 11 times in vanilla model generation. This observation suggests that our model may suffer from an overfitting to news domain in training data and therefore a lower degree of diversity during generation. While being successful with superficial bias, our debiasing technique still struggles to fully remove certain implicit stereotypes. As shown in examples from Table 9, sentences starting with a female mention sometimes talk about their husbands, while wives are much less mentioned when the subject is a male. Besides, females are occasionally depicted as a weak figure prone to assaults, which is not observed in the cases of males. These implicit stereotypical biases cannot be simply attributed to certain tokens and are instead rooted in the narrative manners, therefore extra information regarding stereotypes beyond lexical level is needed. For example, (Stahl et al., 2022) targeted unequal narrative patterns that women are usually portrayed as passive and powerless by introducing agency and power analyses. In addition, debiased models may generate repetitive and less coherent sentences (9 times), and can introduce more genderrelated coreference errors (12 times), which happen 8 and 7 times respectively in vanilla GPT-2. For example, our model wrongly refers to a man using "her" and associates "actress" with a male.

heading: 6 Conclusion
text: Driven by concern about fairness issues in existing NLP systems, this work introduces a lightweight multi-objective probability alignment method to mitigate different forms of stereotypical bias in pretrained generative language models. By incorporating several newly adapted debiasing losses, our method achieves excellent bias reduction results in both automated and human evaluation. At the same time, it largely preserves language modeling ability
of pre-trained models and therefore obtains better balance between language ability and debiasing effect over existing methods. Besides, its prefixtuning framework leads to remarkably high parameter efficiency and better fits the ever-larger model size in today’s NLP community. Further analyses confirm multi-objective fairness optimization is crucial for comprehensive removal of stereotypical bias, and the competitive debiasing performance can be maintained in downstream tasks.

heading: 7 Limitations
text: There are several limitations we need to acknowledge in this study. Firstly, our methods have been evaluated exclusively on binary gender bias, without extending the tests to encompass biases related to race, religion, and non-binary gender identities. This narrow focus restricts the generalizability of our findings, as biases in language models are multifaceted and can manifest across various dimensions. Future research should aim to include these additional social groups to provide a more comprehensive understanding of the efficacy of our debiasing approach.
Furthermore, we have only considered prefixtuning and did not experiment with other parameterefficient fine-tuning methods such as adapter tuning or LoRA. This limits our ability to compare the effectiveness and efficiency of different parameterefficient fine-tuning approaches.

heading: 8 Acknowledgements
text: This work was funded by the DFG project GRK 2853 "Neuroexplicit Models of Language, Vision, and Action" (project number 471607914). We are grateful to the anonymous reviewers and area chairs for their exceptionally detailed and helpful feedback.

heading: A Bias Word Lists
text: Our attribute word list W and target word lists C are provided in the following tables (Table 10 and Table 11).

heading: B Regard Context Templates
text: The context templates we use for regard experiment are listed in Table 12. During the experiments, 100 samples are generated conditioned on the templates for each gender and evaluated by a pre-trained regard classifier.

heading: C Experimental Details
text: A learning rate of 5e− 5 with 500 linear warmup steps and a batch size of 16 are used during prefix training. To prevent numerical instability, all logits
calculated by the model are first divided by a coefficient β before Softmax function in neutralization and equalizing losses. We choose β to be 8 in our experiment. We run a grid search for α2, α3 and α4 in the range of [50, 250] with an interval of 50 and train each model for 5 epochs. The checkpoint that maximizes average intra- and intersentence ICAT score on StereoSet validation set are selected. For regard experiments, we sample 50 sentences for each context template with k=40, temperature=0.7 and a maximum length of 40, generating 1000 sentences in total. Experiments are run on an NVIDIA A100 GPU and training 5 epochs takes approximately 25 minutes.

heading: D Regard Experiment Results
text: Looking closer at the shifted distributions (Figure 2), we notice that debiasing methods can affect model generation differently. CDA, Dropout, Context-Debias and Self-Debias maintain the original distribution overall, with more examples receiving positive regard. INLP substantially reduces the ratio of positive outputs and the number of negative outputs increases remarkably, which can limit its application in practice. Controllable-Bias and Random prefix system have most generated sentences with a neutral regard, likely due to their inability to produce fluent sentences. Our model generates positive sentences in about 60 percent of all cases, dramatically more frequently than any other benchmark system, and the ratio of neutral examples is much lower. One possible reason for the positive shift is that our method generates academia and politics related content at high frequency (e.g., mention of occupations such as researcher and senator, see Table 13). These sentences are usually favored and labelled as positive by regard classifier even without other positive descriptions. This spurious correlation in the regard classifier can be a limitation of our automated regard experiments.

heading: E Human Evaluation Guideline
text: Here, we provide a more detailed explanation of each metric and evaluation criterion used in our human assessment.
• Clarity: coherence and grammatical correctness of generated sentences. Any incoherence or low readability or grammatical error should lead to a worse score. Note that please do not deduct points only because a sentence is not finished.
Attribute Words aerobics, adventurer, apparel, aggressive, assistant, tycoon, baker, warrior, bathing, ambitious, beautiful, trucker, beauty, welder, blonde, strong, bookkeeper, terrorist, ca, soldier, cashier, astronomer, chatty, sniper, cheerleader, skipper, cheerleading, banker, clerk, baseball, cocktail, sergeant, cooking, bodyguard, counselor, boss, crafting, boxer, cute, cabbie, dancer, captain, educator, cardiologist, emotional, carpenter, flirt, ceo, flirtatious, chairperson, flower, chancellor, gossip, chef, graceful, colonel, hairdresser, commander, hairdryer, conductor, homemaker, police, hooker, custodian, housekeeper, dentist, housekeepers, detective, housework, diplomat, hula, doctor, indoor, driving, jealousy, drummer, jewelry, economist, kawaii, electrician, laundering, engineer, librarian, engineering, librarians, entrepreneur, lotion, financier, lovely, firefighter, marvelous, footballer, mirror, gambler, moisturizer, gamer, nanny, gangster, neat, geek, nurse, geeks, nursery, gentle, nurses, guitarist, nurturing, industrialist, parenting, inventor, passive, investigator, pink, laborer, pretty, lawyer, receptionist, leader, ribbon, lieutenant, romance, lifeguard, romantic, magistrate, secretary, manager, selfie, marshal, server, mathematician, sew, mechanic, sewing, muscle, shopping, muscular, smoothie, owner, soft, philosopher, softball, physicist, stylist, pilot, submissive, plumber, sweet, politician, tailor, president, tall, professor, teacher, programmer, thin, rugby, violinist, sailor, waiter, science, weak, scientist, yoga, sculptor, hysterical, blue, makeup, football, executive, management, professional, corporation, salary, office, business, career, home, parents, children, family, cousins, marriage, wedding, relatives, math, algebra, geometry, calculus, equations, computation, numbers, addition, poetry, art, dance, literature, novel, symphony, drama, sculpture, science, technology, physics, chemistry, Einstein, NASA, experiment, astronomy, Shakespeare

heading: Future Work
text: Future research should aim to include these additional social groups to provide a more comprehensive understanding of the efficacy of our debiasing approach. Furthermore, we have only considered prefixtuning and did not experiment with other parameterefficient fine-tuning methods such as adapter tuning or LoRA. This limits our ability to compare the effectiveness and efficiency of different parameterefficient fine-tuning approaches