Debiasing Large Language Models with Structured Knowledge (2024)

Abstract: Due to biases inherently present in data for pretraining, current pre-trained Large Language Models (LLMs) also ubiquitously manifest the same phenomena. Since the bias influences the output from the LLMs across various tasks, the widespread deployment of the LLMs is hampered. We propose a simple method that utilizes structured knowledge to alleviate this issue, aiming to reduce the bias embedded within the LLMs and ensuring they have an encompassing perspective when used in applications. Experimental results indicated that our method has good debiasing ability when applied to existing both autoregressive and masked language models. Additionally, it could ensure that the performances of LLMs on downstream tasks remain uncompromised. Our method outperforms state-of-the-art (SOTA) baselines in the debiasing ability. Importantly, our method obviates the need for training from scratch, thus offering enhanced scalability and costeffectiveness.1

heading: 1 Introduction
text: There have recently been rapid developments in natural language processing (NLP) with the emergence of pre-trained Large Language Models (LLMs). Fine-tuning these models can significantly improve their performance in downstream tasks because, during the fine-tuning process, the knowledge acquired during pre-training on large corpora can be awakened and effectively applied to the downstream tasks. However, biases present in LLMs (e.g., gender and occupation biases) can be a serious problem because of being propagated to the downstream tasks. Therefore, their analysis and mitigation in LLMs have become a critical issue.
Analysis of the text generated by LLMs has shown that the bias exists at the word-level (Sheng
1Our code is at https://github.com/KGDebias/ KGDebias.
et al., 2019; Nozza et al., 2021). The first phenomenon entails that when given a context, LLMs prefer generating words of a certain category with a higher probability. For example, when the input context is “The CEO believes that”, the distribution of output probabilities shows a higher likelihood of the next word being “he” rather than “she”. However, if the word “CEO” in the context is replaced with “nurse”, the bias leads to a higher probability of generating “she” over “he” (Hewitt et al., 2023). The second phenomenon is that when generating text related to a particular noun, LLMs tend to prefer generating content with a specific attribute. For example, as observed in our experiments (see Table 1), when LLMs generate text related to “laborer” and “CEO” separately, the text related to “laborer” exhibits a neutral sentiment, whereas the text related to “CEO” possesses a higher positive sentiment.
It is commonly believed that these biases are caused by the biases inherent in the pre-training data (Brunet et al., 2019; Dev et al., 2020; Papakyriakopoulos et al., 2020). We assume the heightened association between “CEO” and the positive sentiment is attributed to the co-occurrence of “CEO” with many positive connotations. Consequently, LLMs incorporate positive features into the representation of “CEO”. During the inference process, LLMs tend to reproduce and magnify the bias inherent in the pre-training data (Kurita et al., 2019; Sheng et al., 2019), which causes a rise in the prob-
10274
ability of generating positive content, resulting in the production of biased text.
Previous research has sought to alleviate the bias by controlling the pre-training data (Touvron et al., 2023a) or by adjusting embeddings (Hewitt et al., 2023). However, these approaches require models to be pre-trained from scratch, which significantly increases the training cost. To address this issue, we propose a simple method characterized by diminished computation cost. It mitigates the inherent bias in LLMs by incorporating structured knowledge during a second phase of pre-training. The structured knowledge contains hypernyms for a word as one of the relationships between words. Since the hypernyms tend to cover broader or more general concepts, even when a word causes a bias for surrounding words, the second pre-training by incorporating the information of its hypernyms can enhance its representation to take into account the hypernyms, thus decreasing the bias caused by the word. For example, from the knowledge piece “CEO is-a employee”, we can construct a sentence “a CEO is an employee.”, which signifies that a “CEO” belongs to the superordinate concept of “employee”. By training with these sentences for the hypernym information, we can incorporate the representation for “employee” into that for “CEO”, that makes LLMs generate content more related to “employee” and reduce the focus on positive content related to “CEO”, achieving the goal of debiasing. Since our method is implemented only with a second phase of pre-training and needs no pre-training from scratch, it reduces the cost of training compared to the previous methods.
Our experiments demonstrated the efficacy of our method in debiasing across various LLMs while preserving their generalization ability. Specifically, our results for autoregressive and masked language models indicated that our method can lower bias. Concurrently, models with our method did not exhibit significant performance degradation in downstream tasks, affirming the preservation of their generalization ability. Furthermore, comparing to strong debiasing baselines, our method yielded superior scores and exhibited enhanced bias control.

heading: 2 Related Work

heading: 2.1 Bias in LLMs
text: While the majority of tasks have been accomplished using the LLMs (Devlin et al., 2019; Rad-
ford and Wu, 2019; Touvron et al., 2023a,b) with the advancement of LLMs, an increasing number of researchers have discovered bias embedded within them. Several previous studies have demonstrated the presence of the bias in word embeddings (Bolukbasi et al., 2016; Brunet et al., 2019; Papakyriakopoulos et al., 2020). When applying the LLMs to downstream tasks, such as automatic summarization or web search, the bias can lead to significant harm. Jentzsch and Turan (2022) and Kirk et al. (2021) identified obverse gender bias in BERT and GPT2, where their prediction often reinforces gender-based stereotypes, e.g., doctors are assumed to be male and nurses to be female. Standard benchmarks utilized to evaluate the bias consist of various kinds of stereotypical and antistereotypical sentence pairs. CrowSPairs (Nangia et al., 2020) is a commonly used dataset to measure whether a model generally prefers stereotypical sentences.

heading: 2.2 Debiasing Methods
text: Researchers have proposed various methods to mitigate the bias in NLP models. Park et al. (2018); Zhao et al. (2019); Garg et al. (2019); Touvron et al. (2023a) mitigated the bias by changing the pre-training data or the underlying word embeddings, and then by retraining the model. Bordia and Bowman (2019) mitigated the bias in a word level and also required the retraining. The Counterfactual Data Augmentation (CDA) method, proposed by Zimmermann and Hoffmann (2022), and the Data Interventions (DI) method, proposed by Thakur et al. (2023), both used designed examples to mitigate bias in the models. Liang et al. (2020) proposed the SENT-DEBIAS method, which captures the bias subspace of sentence representations by using templates. However, all these methods still suffer from high training cost or unsatisfactory bias control.

heading: 3 Proposed Method
text: As shown in Figure 1, we propose the utilization of structured knowledge to mitigate the bias within LLMs. Our method consists of two main steps: first acquiring textual data for second pre-training from the structured knowledge, and then pre-training the pre-trained LLM using the acquired data, before fine-tuning it for specific downstream tasks.

heading: 3.1 Acquiring Training Textual Data from Structured Knowledge
text: Previous research (Brunet et al., 2019; Dev et al., 2020; Shaikh et al., 2023; Touvron et al., 2023a,b) has suggested that the types of bias in LLMs are linked to human-related attributes such as gender, occupation, religion, and so on. Hence, in our method, we utilize structured knowledge related to humans for debiasing.
We first collect human-related nouns, such as “CEO” and “artist”. Then, we obtain a set of structured knowledge pieces for humans, where a structured knowledge piece is between two nouns within the ‘is-a’ relation, by utilizing the collected nouns; for example, “human-related noun is-a X” and “Y is-a human-related noun”, where X and Y are a hypernym and hyponym of the noun, respectively. Since we cannot incorporate the acquired structured knowledge directly into LLMs, to use the structured knowledge in LLMs, by adhering to human grammatical convention, we transform it into sentences, such that “a human-related noun is a X”. This sentence construction method has been commonly used in previous work (Bosselut et al., 2019; Guan et al., 2020).

heading: 3.2 Training LLMs on Acquired Textual Data
text: Next, we pre-train the pre-trained LLMs on these acquired textual data to incorporate the structured knowledge into the LLMs. We have contemplated the following two aspects:
1. Higher flexibility: We aspire for a knowledge incorporation method to be plug-andplay. That is, even if a new knowledge base is available in the future, the training would
necessitate only adjustment in light of the new knowledge base.
2. Lower training cost: We aim that the training should not start from scratch, thereby diminishing energy consumption throughout the training process, with the broader goal of reducing carbon emission.
Here, in order to make the training process consistent with the features of the LLMs themselves, for different LLMs, we adopt whichever training method was utilized in their pre-training to incorporate the structured knowledge into the LLMs. The details of the objectives are listed in Appendix A.

heading: 4 Experiments

heading: 4.1 Knowledge Databases
text: To obtain human-related nouns, we used WordNet (Miller, 1995), a large lexical database in English. Within the version v3.1 of WordNet, there exists a category “noun.person”, which contains various human-related nouns. After the processing, we procured 6,904 nouns related to humans, for example, forager and runner. In Appendix B, we show the details of the processing steps.
Hyponym Relation Hypernym
Subsequently, we utilized ConceptNet (Speer et al., 2017) to obtain human-related structured knowledge pieces. Table 2 showcases examples of the knowledge pieces derived from ConceptNet.
Finally, we converted the obtained structured knowledge pieces into sentences. The number of the obtained sentences for second pre-training is 33,224 in total.

heading: 4.2 LLMs

heading: 4.2.1 Autoregressive Language Models
text: In the experiments for autoregressive language models, we utilized GPT2, GPT-Neo (Black et al., 2021), and LLaMAs (Touvron et al., 2023a,b) as base LLMs. GPT2 is a general model that can be applied to a variety of tasks. GPT-Neo is an instantiation of models similar to GPT2, applying Mesh TensorFlow to facilitate distributed processing support. LLaMA2 is one of the pre-trained LLMs trained only on publicly available datasets. It shows a competitive performance with existing state-of-the-art (SOTA) LLMs.
The number of parameters of the models is shown in Table 3. The learning rate for GPT2 was set to 2e-5, the optimizer was Adam (Kingma and Ba, 2015). The learning rate for GPT-Neo was set to 2e-5, the optimizer was Adam. The experiments with LLaMA2 used deepspeed.2 The learning rate was set to 2e-4, the optimizer was Adam. The models with GPT2 and GPT-Neo were trained on an A6000 server, and the models with LLaMAs were trained on an A100 server.
We also show the reported scores for three large autoregressive language models to compare with: LLaMA 65B (Touvron et al., 2023a), OPT 175B (Zhang et al., 2022), an open-sourced language model with 175 billion parameters, and
2https://www.deepspeed.ai/
GPT3 (Brown et al., 2020), a non-public language model with 175 billion parameters.

heading: 4.2.2 Masked Language Models
text: In the experiments for masked language models, we utilized BERT and RoBERTa (Liu et al., 2019) as base LLMs. BERT is a pre-trained language model that uses a deep bidirectional transformer architecture, that can capture contextual information from both left and right contexts in all layers. RoBERTa is a pre-trained language model that was improved upon BERT through several modifications to the training procedure.
The number of parameters of the models are listed in Table 3. The learning rate for BERT was set to 4e-5, the optimizer was Adam. The learning rate for RoBERTa was 4e-5, the optimizer was Adam. All experiments were done on an A6000 server.

heading: 4.3 Baselines
text: We selected several methods for mitigating bias in the models, including the SOTA model, as baselines for comparison. SENT-DEBIAS: A sentence embedding debiasing approach proposed by Liang et al. (2020). DI (Thakur et al., 2023): A data-based approach that utilizes few-shot data to mitigate gender bias. They claimed it is a SOTA model. Backpack (Hewitt et al., 2023): A baseline applying multiple non-contextual sense vectors and representing a word with the sense vectors for debiasing. It is the current SOTA model.
Following the prior work (Hewitt et al., 2023; Thakur et al., 2023), we used a small version of GPT2 as the base model for training and comparing against Backpack, and used a base version of BERT as the base model for training and comparing with the remaining baselines, for fair comparison. To augment the analysis, we also introduced GPT-Neo 1.3B and the large version of RoBERTa for training and comparison purposes.
Further, we also constructed two strong baselines for comparison. One is the model pre-trained in the same way as in our model on the dataset from Wikitext-2 (Merity et al., 2017) of the same size as our dataset constructed from structured knowledge. Since Wikitext-2 consists of formally written Wikipedia articles, it has been said to contain less explicit bias (Thakur et al., 2023), while it does not consist of structured knowledge, thus making it a suitably strong baseline. We call this baseline
Wikitext-tuning. The second model was pre-trained using sentences generated from LLaMA2. We employed a prompt to make LLaMA2 generate sentences that describe the relationship between the extracted nouns and their hypernyms. In contrast to Wikitext2, these newly generated sentences were not utilized during the pre-training, containing the information of the extracted nouns, but might contain bias. This baseline is termed Gen-tuning. The specifics of the training data for Gen-tuning are presented in Appendix C.3
Additionally, we introduced a variant of our method. In this variant, instead of using the isa relation, we used the synonym relation to obtain structured knowledge for the extracted nouns. Then, we transformed it into sentences using a template, such that “a human-related noun is similar to
3As these sentences originate from LLaMA2, they were not introduced in the experiments for LLaMA2.
a X”, where X is a synonym of the human-related noun. We call this variant Synonym-KG.

heading: 4.4 Datasets
text: CrowSPairs (Nangia et al., 2020) is a common dataset for evaluating bias in LLMs. It contains 1,508 instance pairs in nine categories: Race/Color, Gender, Socioeconomic Status, Nationality, Religion, Age, Sexual Orientation, Physical Appearance, and Disability. Each instance pair consists of a stereotypical and anti-stereotypical sentence.
BOLD (Dhamala et al., 2021) is a dataset that contains 23,679 instances in five domains: Religion, Profession, Gender, Race, and Policy. Every instance is a sentence extracted from Wikipedia, and the first six to nine words are extracted from the sentence as a prompt. Since the first four categories in the BOLD dataset, i.e., Religion, Profession, Gender, and Race, are related to human, we selected the prompts in these four categories for
evaluation.

heading: 4.5 Evaluation Metrics for Debiasing
text: Bias Score We need to evaluate bias on CrowSPairs. Following Touvron et al. (2023a), we calculated the perplexity for both sentences in each pair in a zero-shot setting to measure the model preference for the stereotypical sentence. The CrowSPairs score represents the percentage of instance pairs in which the stereotypical sentence has lower perplexity than the anti-stereotypical sentence in the total number of instance pairs. Since it is not easy to understand that the scores closer to 50 indicate less bias, we instead utilized the Bias score, which is defined as equal to |CrowSPairs score − 50|, to replace the CrowSPairs score. A lower score represents lower bias.
Regard We further evaluated bias in the experiments for the autoregressive language models by the Regard metric (Sheng et al., 2019). This metric
evaluates the bias by introducing the concept of regards (positive, neutral, and negative), which is similar to sentiment. When inputting a text, this metric computes three scores for the text. We first used prompts in the BOLD dataset to generate continuous texts with the model. Then, each generated text gets three scores for the corresponding regard from the Regard metric. Finally, we calculated the average score for each regard as the Regard score. Since we need to evaluate only whether a generated text shows bias (positive or negative) or not, we summed the Regard scores for positive and negative regards as a new category, polarity. A lower score for polarity indicates lower bias and a higher score for neutral indicates lower bias.

heading: 4.6 Downstream Tasks
text: To evaluate whether the performance of the models on downstream tasks might degrade with applying our method for debiasing, we tested eight downstream tasks, spanning four different aspects for testing.
• For testing the Common Sense Reasoning ability, we utilized WinoGrande and PIQA (Physical Interaction: Question Answering) under a zero-shot setting. In both tasks, when given a question, a model is required to determine the correct answer from the list of answer candidates.4
• For testing the Sentiment Analysis ability, we utilized SST (Stanford Sentiment Treebank) under fine-tuning. Models need to generate the correct sentiment polarity (negative or positive) for the input text.
• For testing the Natural Language Inference ability, we utilized MultiNLI (Multi-Genre
4The software used for the these two tasks is from https://github.com/EleutherAI/lm-evaluation-harness (Gao et al., 2021).
Natural Language Inference), RTE (Recognizing Textual Entailment), QNLI (Questionanswering Natural Language Inference), and WNLI (Winograd Natural Language Inference) under fine-tuning. MultiNLI and RTE require models to generate the entailment relationship (entailment or contradiction) between two sentences. QNLI requires models to discern whether a question and a text contain the correct answer. WNLI requires models to discern if a text whose pronouns were replaced with their referents is entailed by the original text.
• For testing the Paraphrase Identification ability, we utilized QQP (Quora Question Pairs) under fine-tuning. In this task, models need to discern whether two questions have the same meaning.

heading: 5 Results

heading: 5.1 Debiasing
text: Table 4 presents the experimental results of various models on the CrowSPairs dataset. Our models exhibit lower bias in the majority of bias categories, that indicates the effectiveness of our method. Notably, LLaMA, only with 7 billion parameters, showcases lower bias than other LLMs with more parameters.
In contrast, the strong baseline Wikitext-tuning shows only a limited ability to mitigate bias, even
though its pre-training data contain text with less bias. While Gen-tuning contains the hypernym information, its capacity for debiasing is lower than our method. It might be because the generated sentences for training it do not necessarily provide correct structured knowledge and contain bias. For instance, the actually generated sentence for “CEO”, “The CEO serving as the highest-level leader and managers, supervisors, and individual employees reporting to them.”, contains bias related to its superiority. This might prevent the models from accurately enhancing its representation.
The variant Synonym-KG, which used synonym information, also showed the effectiveness in mitigating bias. This might highlight the influence of structured knowledge, which contributes to the bias mitigation. However, its Bias scores are still higher than our method. This might indicate that the hypernym information is more useful for debiasing than the synonym information.
However, we found that our method shows a weaker control in some categories, e.g., Age, Sexual Orientation, and Physical Appearance, than the others (e.g., Race/Color and Gender). To investigate the reason, we calculated the ratio of knowledge for the categories, as shown in Table 8.5 The weaker control is caused by the lack of the
5We trained a classification model to classify sentences obtained from the structured knowledge into one of the nine bias categories in the CrowSPairs dataset. More details are in Appendix F.
structured knowledge related to these categories. This indicates that more structured knowledge is required when mitigating specific kinds of bias.
Table 5 shows the Regard scores for autoregressive language models. It is evident that models trained with structured knowledge exhibit lower polarized regard scores across various categories than the original models. This demonstrates that our method enables the models to use more general concepts from the hypernyms of a word when generating text, thus preventing excessive bias towards specific polarized content related to the word and favoring the generation of neutral content. The higher scores for the neutral regard further support this point.
As shown in Table 6, the Bias scores of our model are lower than other debiasing methods, especially in cases of larger-scale models with more bias, indicating that our method outperforms the previous methods in effective debiasing.
In order to conduct a more thorough comparison, our approach was also evaluated alongside baseline methods using the StereoSet dataset (Nadeem
et al., 2021), a natural English dataset designed for assessing stereotypical bias. The experimental results, presented in Appendix E, illustrate that our method effectively mitigates bias while it slightly decreases the language modeling performance in the Autoregressive Language Models.

heading: 5.2 Downstream Tasks
text: Table 7 shows the results for the models on downstream tasks. Our models exhibit close performances to the original models across various downstream tasks. This demonstrates that using our method can ensure preservation of the generalization ability of the original models. Since the hypernyms of a word contain more general concepts for the word, after training with our method, although the representation of the word is adjusted, it still keeps the basic information. Thus, it can ensure that the trained models maintain their generalization abilities acquired during pre-training without leading to dramatic degradation in the performance of the downstream tasks.

heading: 5.3 Case Study
text: Figure 2 shows the conditional probability distribution among words after inputting a specific prompt “When the X walked into the room,” to both GPT2-Neo trained with our method and the original GPT2-Neo. We used different nouns, “nurse” and “CEO”, to replace “X”. The gap of the probabilities between “she” and “he” is smaller after applying our method than the original model in both cases. This clearly shows the effect of our method in mitigating gender-related bias for these cases.

heading: 6 Conclusion
text: We proposed a simple method that utilizes structured knowledge to mitigate the issue of bias within LLMs. Our method trains the LLMs by incorporating information of hypernyms into the representations for the words to mitigate the bias. Experimental results from both autoregressive and masked language models demonstrated that our method effectively controls the inherent bias in LLMs without compromising the performance in downstream tasks. Comparative studies with other debiasing techniques showed that our method achieves a better debiasing performance. Since our method does not require models to be trained from scratch, it boasts the advantages of low cost and scalability.

heading: 7 Limitations
text: Since we used only limited human-related nouns to extract knowledge from the existing knowledge base, the knowledge is not comprehensive. This causes a weaker control in certain bias categories. Developing a method for acquiring more comprehensive structured knowledge or data augmentation will be the focus of future research.

heading: A Objectives for LLMs
text: A.1 Autoregressive Language Models For autoregressive language models, such as GPT2 (Radford and Wu, 2019), we utilize the nexttoken prediction objective to train the model. The objective is calculated as follows:
L = − |s|∑
t=1
logP (st|s<t), (1)
where s is a sequence of sentences constructed from the obtained structured knowledge pieces. t denotes the t-th token in the sequence.
A.2 Masked Language Models For masked language models, such as BERT (Devlin et al., 2019), we utilize the masked token prediction objective to train the model. The objective is calculated as follows:
L = − ∑
ŝ∈m(s) logP (ŝ|s\m(s)), (2)
where s is a sequence of sentences constructed from the obtained structured knowledge pieces, m(s) denotes the masked tokens in s, and s\m(s) denotes the remaining tokens in s.

heading: B Extracting Human-related Nouns from WordNet
text: We processed the nouns retrieved from the “noun.person” category in the WordNet as follows:
1. We retrieved all the nouns belonging to the “noun.person” category in WordNet.
2. Since we found many person names, denoted with initial capitalization, in those nouns, we extracted the person names and their definitions in WordNet and replaced the names with the definitions, for example, “William Shakespeare” with ”dramatist”.
3. We also found compound nouns consisting of multiple words. For those compound nouns, we retained only the last head noun, for example, from the noun “nationalist leader”, we extracted only “leader”.
4. Finally, we removed any duplicated nouns.

heading: C Training Data for Gen-tuning
text: For every extracted noun, we formulated a prompt, “Use a sentence to describe the relation between X and Y.” with X representing the noun and Y representing its hypernym. Subsequently, we input this prompt into the LLaMA2-7B model to obtain responses, which serve as our training data. Excluding instances where the model failed to generate a response, the total number of sentences is 33,105. Three examples of generated sentences, along with the corresponding noun-hypernym pairs, are presented in Table 9.

heading: D CrowSPairs Scores
text: In Autoregressive Language Models, the perplexity is calculated by evaluating the probability of generating each token sequentially based on the preceding context. In Masked Language Models, we calculated the perplexity by evaluating the likelihood of predicting masked tokens within the given context. Table 10 shows the original CrowSPairs scores for different models in Table 4. Table 11 shows the original CrowSPairs scores in Table 6.

heading: E Results on the StereoSet Dataset
text: StereoSet (Nadeem et al., 2021) comprises 16,995 instances distributed across four domains: gender, profession, race, and religion. Within each instance, a stereotypical, a anti-stereotypical and a meaningless sentences are included. The SS score is employed to indicate the percentage of instances in which the stereotypical sentence exhibits lower perplexity than the anti-stereotypical sentence in the total number of instances in StereoSet. The LMS score is employed to indicate the percentage of instances in which the stereotypical sentence has lower perplexity than the meaningless sentence in the total number of instances in StereoSet. The experimental results are presented in Table 12. A SS score closer to 50 indicates a reduced level of bias. A higher LMS score indicates better language modeling performance.

heading: F Classification Model
text: We trained the classification model based on the large version of RoBERTa. We used the stereotypical sentences in the CrowSPairs dataset to train this model. The nine labels are the same as the bias categories in the CrowSPairs dataset. The learning rate was set to 1e-5. We used Adam as the optimizer.

heading: Future Work
text: Developing a method for acquiring more comprehensive structured knowledge or data augmentation will be the focus of future research