BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization (2024)

Abstract: Large Language Models (LLMs) have become pivotal in advancing natural language processing, yet their potential to perpetuate biases poses significant concerns. This paper introduces a new framework employing Direct Preference Optimization (DPO) to mitigate gender, racial, and religious biases in LLM-generated English text. By developing a loss function that favors less biased over biased completions, our approach cultivates a preference for respectful and non-discriminatory language in LLMs. We also contribute a manually designed dataset for training LLMs to recognize and correct biases. This dataset encompasses a diverse range of prompts paired with both biased and unbiased completions. Implementing this approach on the Microsoft Phi-2 model, we demonstrate substantial reductions in biased outputs as our model outperforms the baseline model on almost all bias benchmarks. Our model also achieves better performance compared to other open-source models on most benchmarks. By reducing biases in the language generated by the model, our study marks a significant step towards developing more ethical and socially responsible LLMs. We publicly release BiasDPO dataset on HuggingFace.1

heading: 1 Introduction
text: Even though Large Language Models (LLMs) have shown remarkable capabilities in complex language tasks, they are not without their flaws. One of the main concerns with LLMs is the presence of biases in their generated text, reflecting prejudices present in their training data. These biases can be in several forms, including racial, gender, and religious biases.
Efforts have been directed towards applying different methodologies for aligning LLMs with human preferences and values. One of the most
1The dataset is available at https://huggingface.co/ datasets/ahmedallam/BiasDPO.
popular approaches used is Reinforcement Learning from Human Feedback (RLHF), which trains LLMs to generate responses that are more likely to be rated highly by human evaluators (Ouyang et al., 2022). However, RLHF faces several challenges, such as mode collapse, training instability, as well as requiring a separate reward model which adds complexity to the training process (Casper et al., 2023).
Recently, Direct Preference Optimization (DPO) has emerged as a promising approach for training LLMs to follow certain preferences. DPO works by training the model to maximize the log probability of preferred tokens and minimize the log probability of dispreferred tokens given a certain prompt from the dataset (Rafailov et al., 2023). By directly optimizing the model to favor certain tokens over others, DPO can help the model generate more preferred and high-quality responses, without the need of reinforcement learning.
In this paper, we present a new framework for leveraging Direct Preference Optimization to reduce gender, race, and religious biases in the text generated by LLMs. Our approach trains the LLM by using a loss function that maximizes the log probability of tokens in completions that are considered less biased, non-harmful, and respectful, and minimizes the log probability of tokens in completions that are biased, harmful, or offensive. This approach gives the model a preference for generating less biased and more respectful language, leading to a reduction in bias in the language generated.
We also present a new dataset to be used for training LLMs using our approach. The dataset consists of a diverse set of prompts and corresponding biased and unbiased completions, covering a wide range of topics and contexts. For each prompt, a biased completion is provided that contains biased, harmful, or offensive content, and a completion that is less biased, more respectful, and non-harmful.
71
By applying our training approach using our dataset to the recently released Microsoft Phi-2 model, results indicate that our approach reduces bias in the language generated by the LLM when tested both quantitatively and qualitatively. Specifically, the model trained with our approach achieves a higher accuracy on all bias benchmarks compared to the baseline model. The model also outperforms other similarly sized open-source models on most benchmarks. The results of the qualitative analysis show that the responses generated by the model after applying BiasDPO are more neutral, less biased, and respectful compared to the responses generated by the baseline model, which also proves the effectiveness of our approach in reducing bias in language models.

heading: 2 Background and Related Work

heading: 2.1 Bias in LLMs
text: Recent studies have highlighted the presence of biases in LLMs, and the potential impacts of these biases on society. Navigli et al. (2023) define social biases in LLMs as prejudices, stereotypes, and discriminatory attitudes against a group of people. These biases can be in several forms including gender, race, social class, disability, nationality, and religion. The study also tests the presence of these biases in several LLMs, and finds that they exhibit biases that reflect the biases present in their training data. In addition, many studies have proposed different approaches to evaluate and quantify biases in LLMs. Parrish et al. (2022) introduce the Bias Benchmark for Question Answering (BBQ) to evaluate the biases present in language models in the context of question answering. The BBQ benchmark consists of a set of multiple-choice questions designed to uncover different types of biases. The BOLD benchmark introduced by Dhamala et al. (2021) is designed to assess the extent of bias in language models when generating text without specific prompts.

heading: 2.2 Mitigating Bias in LLMs
text: Several approaches for mitigating bias in LLMs have been proposed in recent studies. One approach is to use prompt engineering to guide the model towards generating less biased and respectful responses. Gallegos et al. (2024) introduce a self-debiasing approach that uses prompts to ask the model to identify any implicit biases or stereotypes before answering a question, in a zero-shot
setting. Other approaches use few-shot learning and chain-of-thought reasoning to remove bias from generated language (Dwivedi et al., 2023; Huang et al., 2024). Both approaches have shown promising results in reducing bias and can be used as a solution to mitigate bias in LLMs without the need for additional training. However, these approaches may struggle to generalize and scale to different types of biases and contexts, and may require a large amount of human supervision. Moreover, these approaches should be considered as complementary to other approaches that train the model to be inherently less biased.
A popular approach for training LLMs to be less biased is Reinforcement Learning from Human Feedback (RLHF). RLHF works by training a reward model on human evaluations of the language model’s outputs, and then fine-tuning the language model through Proximal Policy Optimization (PPO) to generate responses that are more likely to be rated highly by human evaluators (Ouyang et al., 2022). RLHF has been shown to be effective in aligning language models with human preferences and reducing bias in the language generated by the model. However, RLHF faces several challenges, including reward hacking, training instability, and mode collapse, which can limit its effectiveness in reducing bias in LLMs (Casper et al., 2023). Moreover, the need for a seperate reward model to provide feedback to the model can be considered as a limitation of RLHF, as it requires additional resources and training time.

heading: 2.3 Direct Preference Optimization
text: Direct Preference Optimization (DPO) is a recent approach that has been proposed as an alternative to RLHF for training LLMs to follow certain preferences. DPO works by training the model to maximize the log probability of preferred tokens and minimize the log probability of dispreferred tokens given a certain prompt from the dataset (Rafailov et al., 2023). By directly optimizing the model to favor certain tokens over others, DPO can help the model generate more preferred and high-quality responses, without the need of reinforcement learning. It avoids the need for a separate reward model to provide feedback to the model, as it directly optimizes the model using a closed-form expression, which can make it more efficient and less prone to reward hacking and training instability compared to RLHF. Rafailov et al. (2023) demonstrate the effectiveness of Direct Preference Optimization (DPO)
in training language models to follow specific human preferences through various experiments. For example, in the controlled sentiment generation task, they fine-tuned a model to generate IMDB reviews with a more positive sentiment. This task required the model to generate text continuations that maintained a positive tone when given a prefix from a movie review. Their results showed that DPO performs as well as or better than existing methods such as Proximal Policy Optimization (PPO) in aligning the model’s outputs with human preferences. This demonstrates that DPO can effectively train language models to adhere to specific preferences, addressing some of the limitations associated with RLHF.

heading: 3 Approach

heading: 3.1 Framework
text: Our approach in mitigating language bias uses the Direct Preference Optimization (DPO) method (Rafailov et al., 2023) by training the model using a defined loss function that encourages the model to prefer less biased, respectful, and nonharmful completions over biased or offensive completions. Specifically, for a language model πθ, given a prompt x and two completions yw and yl, where yw is the less biased completion and yl is the biased completion from a dataset D, the debiasing loss function LDPO is defined as follows:
LDPO(πθ;πref) = −E(x,yw,yl)∼D[ log σ ( β log
πθ(yw | x) πref(yw | x) −β log πθ(yl | x) πref(yl | x)
)] .
(1) Where πref is the reference frozen version of the model. The reference model is used in order to prevent the model from deviating too much from the original distribution of the data using the Kullback–Leibler divergence term. The hyperparameter β controls the amount of divergence of the model from the reference model. This training loss penalizes the model for generating biased completions and rewards it for generating less biased completions, leading to a preference for generating nonharmful and respectful language. This approach is more effcient than RLHF as it directly optimizes the model using a closed-form expression, and does not require a separate reward model to provide feedback to the model.

heading: 3.2 The Dataset
text: As our approach requires a dataset consisting of prompts x and their corresponding biased and less biased completions yw and yl, we introduce a new dataset created for this purpose. The dataset contains more than 1,100 entries covering biases related to gender, race, and religion, as well as their intersections, across a wide range of topics and contexts. Table 1 shows some examples of prompts and their corresponding biased and unbiased completions for each type of bias.
In creating the dataset, we first identified scenarios and contexts prone to eliciting biased responses from language models, such as discussions around gender roles, racial stereotypes, and religious beliefs. We crafted prompts designed to provoke biased completions. For each prompt, we generated a biased completion that reflects stereotypes or prejudices. Then, we created unbiased completions for each prompt. These unbiased completions were designed to neutralize the biases present in their biased counterparts, emphasizing respectful and inclusive language. The aim was to ensure that the unbiased completions not only counteract the biases but also promote equitable discourse.
To expand the dataset, we used GPT-4 to generate additional samples. The generated samples were carefully reviewed and refined to maintain quality, ensuring they were consistent with the manually created ones.

heading: 3.3 Implementation Details
text: As the original DPO loss function has been shown to have limitations in practice, such as being prone to overfitting to the preferences dataset (Azar et al., 2023), we experiment with different recently proposed variations of the DPO loss function.
Specifically, we experiment our approach with Identity Preference Optimization (IPO) (Azar et al., 2023), which adds a regularization term to the DPO loss function in order to prevent overfitting. IPO does this by controlling the gap between the loglikelihood ratios of the prefered and disprefered completions for both the model and the reference model. The IPO loss function is defined as follows:
LIPO(πθ;πref) = −E(x,yw,yl)∼D[( log
( πθ(yw|x)πref(yl|x) πθ(yl|x)πref(yw|x) ) − β −1 2 )2] (2)
Additionally, Sequence Likelihood Calibration
(SLiC) is another variation that adds a rank calibration term and cross-entropy loss term to the loss function, which has been shown to reduce overfitting as well (Zhao et al., 2023). The SLiC loss function is defined as follows:
LSLiC(πθ) = max(0, δ − log πθ(yw|x) + log πθ(yl|x))− β log πθ(yref |x) (3)
Where δ is a hyperparameter for the margin of the ranking loss.
Moreover, Kahneman-Tversky Optimization (KTO) is another variation that directly maximizes the utility of generations using a model of human utility based on Kahneman & Tversky’s prospect theory (Ethayarajh et al., 2024). Unlike DPO, KTO only requires a binary signal of whether an output is desirable or undesirable, making it more practical for many real-world applications. The KTO loss function is defined as follows:
LKTO(πθ, πref) = E(x,y)∼D [w(y) (1− vKTO(x, y;β))]
(4)
Where vKTO(x, y;β) = σ(rKTO(x, y)− zref) for
desirable outputs and vKTO(x, y;β) = σ(zref − rKTO(x, y)) for undesirable outputs. The term zref represents the reference reward, and rKTO(x, y) = β log πθ(y|x)πref(y|x) . The weighting function w(y) is used to differentiate between desirable and undesirable outputs.
Intuitively, KTO forces the model to learn exactly what makes an output desirable by increasing the reward without increasing the KL divergence term, which serves as a regularization factor.
We incorporate each of these variations of the loss function into the implementation of our approach, and compare how they affect the performance of the model in reducing bias in the language generated given the same dataset and hyperparameters.

heading: 4 Experiments

heading: 4.1 Experimental Design
text: To apply and test our approach, we use Microsoft Phi-2 as the base model to be trained. Phi-2 is a recently released 2.7B parameter open-source LLM that demonstrates state-of-the-art performance on a wide range of language tasks compared to other
models in its size range. Phi-2 is trained following the “Textbooks Are All You Need” approach (Li et al., 2023), which allows it to achieve high performance on tasks such as common sense, language understanding, and logical reasoning. However, one of its limitations is that it has some degree of bias in its language generation as it is not trained using RLHF or any other bias mitigation approach. The model is intentionally left open-source to allow the research community to experiment with it and develop new approaches to reduce its bias and toxicity, making it an ideal candidate to apply the BiasDPO approach to.
We train the Phi-2 model using the BiasDPO approach with our dataset described earlier. We experiment with different variations of the DPO loss function, including IPO, SLiC, and KTO, to study their impact on the performance of the model. We also experiment with different values of the hyperparameter β. The model is trained for 5 epochs using the Adam optimizer with a learning rate of 1e-6, and a batch size of 4 on an 8 V100 GPUs server.

heading: 4.2 Bias Benchmarks
text: In order to measure the degree of bias in the language generated by the Phi-2 model before and after applying our approach and also compare it to other models, we use a set of widely used bias benchmarks.
The BBQ (Bias Benchmark for Question An-
swering) (Parrish et al., 2022) evaluates biases in language models through questions designed to uncover gender, race, religion, and intersectional biases. The BOLD (Bias in Open-Ended Language Generation) Benchmark (Dhamala et al., 2021) assesses bias in language models during open-ended text generation, covering a wide range of scenarios likely to elicit biased responses. RealToxicityPrompts Benchmark (Gehman et al., 2020) evaluates the propensity of language models to generate harmful or toxic content in response to specific prompts. TruthfulQA Benchmark (Lin et al., 2022) tests the accuracy and honesty of language models with questions that reveal common pitfalls in human misconceptions and false beliefs.
We run each benchmark using the HELM framework (Liang et al., 2023), which is a widely used framework for evaluating LLMs on a wide range of language tasks. We compare the performance of different open-source models including Gemma2B (Team et al., 2024), StableLM-3B (Tow et al.), as well as Mistral-7B (Jiang et al., 2023). We rerun all the benchmarks using the same settings to ensure a fair comparison between the models.

heading: 4.3 Benchmark Results
text: We test the performance of our approach when applied to the Phi-2 model with the different variations of the DPO loss function and the hyperparameter β against the BBQ benchmark to measure their effect on the model’s performance in reducing bias
compared to the baseline original Phi-2 model. The results are shown in Figure 1. The results show that the IPO variation of the DPO loss function with a β value of 0.01 achieves the highest accuracy on the BBQ benchmark, with an accuracy of 0.65, compared to the baseline accuracy of 0.5. In general, results show that lower values of β tend to perform better than higher values in reducing bias in the language generated by the model.
The best performing variation of the model is then further evaluated on the other benchmarks. The results are shown in Table 2. The results show that the Phi-2 model with BiasDPO outperforms the baseline Phi-2 model on all benchmarks, achieving a higher accuracy on the BBQ and TruthfulQA, and lower toxicity score on BOLD and RealToxicityPrompts.
Moreover, the Phi-2 model with BiasDPO also outperforms other similarly sized open-source models, including Gemma-2B, StableLM-3B, and Mistral-7B, on most benchmarks. Specifically, our model achieves the lowest toxicity score on the RealToxicityPrompts benchmark, as it has a score of 0.11, compared to the baseline score of 0.17, and 0.14 for Mistral-7B, the second lowest score. Our model also achieves the highest accuracy on the TruthfulQA benchmark, with a score of 0.45, compared to the baseline score of 0.42.
On the BBQ benchmark, our model achieves the highest accuracy on the gender, and race categories by a large margin, with scores of 0.68 and 0.87, respectively, compared to the baseline scores of 0.6
and 0.77, and the second highest accuracy on the religion, only behind Mistral-7B. On the BOLD benchmark, our model achieves the lowest score on the gender bias category, while being less performant on race and religion categories compared to Gemma-2B that achieves the best scores. Overall, the evaluation results shows that our model is better in terms of being less biased and toxic compared to the baseline model and other open-source models on nearly all benchmarks, indicating that our approach is effective in reducing bias in the language generated by the model, leading to more fair, respectful, and less biased responses.

heading: 4.4 Qualitative Analysis
text: To further evaluate the effectiveness of our approach in reducing bias in the language generated by the model, we conduct a qualitative analysis of the model’s outputs before and after applying our approach by asking both models to respond according to its opinion and beliefs to a set of prompts designed to induce bias. We provide examples of prompts and their corresponding responses before and after applying BiasDPO in Table 3. The examples show that the responses generated by the Phi-2 model after applying BiasDPO are more neutral, less biased, and respectful compared to the responses generated by the baseline model, which contains relatively more biased content that amplifies stereotypes and prejudices. Specifically, in the first example, we try to assess the model’s bias in the intersection of gender and race by asking it
to compare the mathematical abilities of African women, to Asian men. The baseline model responds by stating that Asian men have a higher average IQ, and better logical reasoning than African women, which is a biased and harmful statement. On the other hand, the model trained with our approach responds by stating that mathematical abilities are not determined by gender or race, and that both can excel in mathematics with the right support and opportunities, which is a more neutral and respectful response. Overall, the differences in the responses in the qualitative analysis illustrate the effectiveness of our approach in mitigating bias in the language generated by the model.

heading: 5 Conclusion
text: In this paper, we introduced BiasDPO, a new approach designed to reduce bias in language models through Direct Preference Optimization. We applied the BiasDPO approach to the recently released Microsoft Phi-2 model and evaluated its performance on a set of widely used bias benchmarks. The results show that the BiasDPO approach is effective in reducing bias in the language generated by the model, achieving higher accuracy on the BBQ and TruthfulQA benchmarks, and lower toxicity scores on the BOLD and RealToxicityPrompts benchmarks. The qualitative analysis further con-
firms the effectiveness of the BiasDPO approach in reducing bias in the language generated by the model, resulting in more fair, respectful, and less biased responses. The BiasDPO approach has the potential to have a significant positive impact on society by reducing bias and toxicity in language models, leading to more fair, respectful, and inclusive language generation.

heading: 6 Limitations
text: While the BiasDPO approach shows promising results in reducing bias in the language generated by LLMs, there are several limitations and challenges that need to be addressed in future work. One of the main limitations of the BiasDPO approach is that it requires a large amount of labeled data to train the model effectively. The dataset used in this study was manually crafted and then augmented with synthetic data, and may not cover all possible biases and scenarios. Future work should focus on developing more comprehensive datasets that cover a wider range of biases and contexts to improve the generalizability of the model.
Additionally, in this study, we tested our approach on the Phi-2 model, which is a 2.7B parameter model, which is relatively small compared to other state-of-the-art models. Future work should focus on testing this approach on larger models, to
evaluate its effectiveness in reducing bias in larger models with more parameters.

heading: Acknowledgements
text: This research was conducted with the support of The Tomorrow’s Leaders Gender Scholars Program (TLS). This program is a joint effort between the U.S. Department of State, Bureau of Near Eastern Affairs (NEA/AC) and the AUC. The views and opinions expressed in this research are those of the participants and do not necessarily reflect the official policy or position of the American University in Cairo, Tomorrow’s Leaders Program, or the U.S. Department of State. In addition, I would like to thank my mentors, Dr. Hala Kamal and Dr. Mascha Kurpicz-Briki for their guidance and feedback throughout this research project.

heading: Future Work
text: While the BiasDPO approach shows promising results in reducing bias in the language generated by LLMs, there are several limitations and challenges that need to be addressed in future work. One of the main limitations of the BiasDPO approach is that it requires a large amount of labeled data to train the model effectively. The dataset used in this study was manually crafted and then augmented with synthetic data, and may not cover all possible biases and scenarios. Future work should focus on developing more comprehensive datasets that cover a wider range of biases and contexts to improve the generalizability of the model. Additionally, in this study, we tested our approach on the Phi-2 model, which is a 2. 7B parameter model, which is relatively small compared to other state-of-the-art models. Future work should focus on testing this approach on larger models, to
evaluate its effectiveness in reducing bias in larger models with more parameters