Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination (2024)

Abstract: Instruction-following language models often show undesirable biases. These undesirable biases may be accelerated in the real-world usage of language models, where a wide range of instructions is used through zero-shot example prompting. To solve this problem, we first define the bias neuron, which significantly affects biased outputs, and prove its existence empirically. Furthermore, we propose a novel and practical bias mitigation method, CRISPR, to eliminate bias neurons of language models in instruction-following settings. CRISPR automatically determines biased outputs and categorizes neurons that affect the biased outputs as bias neurons using an explainability method. Experimental results demonstrate the effectiveness of our method in mitigating biases under zero-shot instruction-following settings without losing the model’s task performance and existing knowledge. The experimental results reveal the generalizability of our method as it shows robustness under various instructions and datasets. Surprisingly, our method can mitigate the bias in language models by eliminating only a few neurons (at least three).

heading: 1 Introduction
text: Instruction-following language models perform various tasks using instruction-based prompts (Chung et al., 2022; Touvron et al., 2023; Taori et al., 2023; OpenAI, 2023). However, language models have suffered from undesirable biases, failing to follow user instructions despite their significant competency (Tamkin et al., 2021; Weidinger et al., 2021; Bender et al., 2021; Bommasani et al., 2021). Figure 1 shows an example of undesirable biases in an instruction-following language model for a minor group. In this case, a language model assigns a high probability to a minor group label for a negative question despite the ambiguity of
∗Work done during an internship at LG AI Research.
the given context. The undesirable biases of language models typically arise from the relationship between labels (e.g., "poor people") and tokens (e.g., "drugs") within data instances (Zhao et al., 2021; Fei et al., 2023).
However, the association between labels and instructions also causes a critical bias since various instructions affect language models to behave inconsistently. Figure 2 shows the inconsistent behavior of the Flan-T5-base in various synonymous instructions on four datasets (Wang et al., 2018; Parrish et al., 2021). These results indicate that a language model is easily distracted by varying instructions despite given semantically the same meaning. These phenomena suggest that language models exhibit significant cognitive biases in understanding
9061
instructions, and these are some of the most critical biases to mitigate when using instruction-following language models.
To mitigate biases in language models, Zhao et al. (2021); Fei et al. (2023) have investigated label biases in few-shot in-context learning settings. Specifically, they have regarded the imbalanced probability distribution that occurred by inputting content-free texts (e.g., "N/A" or random tokens) as biases, and degraded the original output probability of each input instance by the output probability of the content-free texts. However, they have only aimed to mitigate biases in few-shot in-context learning settings, not considering the zero-shot instruction prompting. Since utilizing a language model in only the instruction prompting without few-shot examples is an efficient and practical usage scenario of language models, they have significant limitations in the scope of application.
To resolve these issues, in this paper, we first define the bias neuron, which significantly affects biased outputs in language models. Prior studies (Panigrahi et al., 2023; Wang et al., 2022; Panigrahi et al., 2023; Yang et al., 2023) have demonstrated that skills for a specific task are localized in particular neurons. From these findings, we hypothesize
the presence of the bias neuron and empirically validate its existence. Furthermore, we propose a novel bias mitigation method called CRISPR, which stands for CalibRating InStruction Bias via Bias Neuron PRuning. We utilize an attribution (Yang et al., 2023), an explainability method for language models, to quantify the bias of each neuron at three aspects (i.g., token, instance, and instruction). Specifically, we first compute the bias influence of each token in inferring a predefined biased output, and aggregate the token score to an instance and instruction scores to derive the final bias influence (bias attribution) of each neuron. Furthermore, defining biased outputs manually for each data instance is time-consuming; thus, we propose an automatic identification method of biased outputs for computing the bias attribution using the confusion score of a language model.
We demonstrate our method in various social bias and natural language understanding benchmarks and dramatically outperform other baselines under varying zero-shot instruction settings. Surprisingly, we reveal that only a few bias neurons (at least three) cause the bias, proving our method’s practicality. In addition, we show that mitigating bias for a particular task does not adversely affect the existing knowledge of language models for solving other tasks. We also note that bias neurons identified for a specific dataset also function as biases in other analogous datasets, revealing that the bias knowledge is transferred to datasets from correlative domains. CRISPR is an efficient bias mitigation method since it needs only a few data samples (e.g., ten samples) to quantify the bias score for the whole neurons. CRISPR also enables language models to adapt flexibly by eliminating some existing bias neurons without any training process.

heading: 2 Related Works

heading: 2.1 Bias Mitigation
text: Despite demonstrating significant efficacy in various natural language understanding tasks, Language Models (LLMs) have been noted to exhibit undesirable biases (Ravfogel et al., 2019; Braverman et al., 2020; Liu et al., 2021; Lu et al., 2021; Bommasani et al., 2021; Work; Sorensen et al., 2022; Kim et al., 2023; Koh et al., 2024). Therefore, existing studies have attempted to solve the bias problems. For instance, Zhao et al. (2021); Fei et al. (2023) have mitigated biases found in few-shot in-
context learning settings by utilizing outputs probability obtained from content-free texts (e.g. "N/A" or random tokens). They have argued that an imbalanced output probability for the content-free texts corresponds to unintended label biases. Specifically, they have shifted the original output probability of each input instance by dividing it through the output probability of each class obtained from content-free texts. However, existing studies have limitations in that they have only aimed to mitigate biases in the few-shot in-context learning setting. Since more realistic LLM usage settings are based on the zero-shot instruction-following mechanism, it is essential to consider these settings to suggest a practical bias mitigation method. Furthermore, the existing studies have only tackled the simple classification problem as a target to mitigate biases. To enhance the scalability of bias mitigation methods, they should demonstrate effectiveness in real-world natural language understanding tasks that involve handling diverse and inconsistent label options.

heading: 2.2 Skill Neurons Detection
text: Despite the impressive performance of language models, it is challenging to precisely illuminate the role of each parameter in models during the execution of a specific task. Existing studies have sought to detect important skill neurons in performing a specific task (Panigrahi et al., 2023; Wang et al., 2022; Yang et al., 2023).
Panigrahi et al. (2023) has suggested a trainingbased method, model grafting, which detects skill neurons by training a new parameter to mask original parameters. Although it has effectively detected skill neurons, it has required an additional training process for masking parameters as many as the number of model parameters.
Wang et al. (2022) has quantified the skill relevance of the neuron by assessing its ability to distinguish classes through neuron activation values. Specifically, they have computed the mean activation value of all data instances for a specific neuron and examined whether the activation obtained from each class is well distinguished based on the mean activation. However, applying this method to language modeling tasks poses challenges since it requires overwhelming computation to examine all word-piece combinations to determine the distinguishing ability of neurons.
Yang et al. (2023) has detected skill neurons by utilizing the attribution technique (Shrikumar et al., 2016), an explainability method that derives the
importance of each feature when solving a specific task. Yang et al. (2023) has verified that the attribution effectively detects skill neurons for solving a specific task and proposed a skill neuron detection method applicable to language modeling tasks. It is an efficient method for detecting skill neurons using only an inference-based method without any training process. Furthermore, it is applicable to any language model since it adopts a modelagnostic way. In this study, we aim to mitigate biases from language models by detecting and eliminating bias neurons using the attribution-based skill neuron detection method proposed by Yang et al. (2023).

heading: 3 Methods
text: In this section, we describe the process of quantifying and mitigating the bias of instruction-following language models. Specifically, we compute attribution scores of each neuron for inferring automatically defined biased output. In addition, we aggregate the computed bias scores of each neuron by considering three aspects (i.g., token, instance, and instruction) to quantify biases in instructionfollowing settings. Finally, we categorize bias neurons using the bias score and eliminate them using a pruning method to mitigate biases in language models.

heading: 3.1 Quantifying Skill Relevance
text: We utilize an attribution method (Shrikumar et al., 2016) to extract the importance of neurons from the pre-trained language models. It is usually used to derive the importance of the input features (i.g., pixel, token) for performing a specific task, but Yang et al. (2023) expands the attribution formula to the importance of intermediate neurons in language models. Formally, suppose we have a function P : Rd → [0, 1]m that represents a language model. The contribution of an i-th neuron for representation h to the prediction of an output text y using an instruction ι ∈ I and a text input x for P is defined as follows:
A (ι,x,y) i (h) = hi × ∂P(y|ι, x) ∂hi
(1)
where ∂P(y|ι, x)/∂hi is the gradient of P(y|ι, x) with respect to hi. I means an instruction set.

heading: 3.2 Bias Neurons Detection
text: Quantifying Bias. In this section, we describe how to compute the bias attribution for determining
bias neurons. Suppose we have an undesirable biased text ŷ; then the importance of each neuron for the output text ŷ can be computed using the attribution formula, A(ι,x,ŷ)i (h) = hi × ∂P(ŷ|x)/∂hi. However, A(ι,x,ŷ)i (h) includes skill knowledge in addition to biased knowledge since estimating the biased text also contains the knowledge of language modeling, such as understanding instruction knowledge. Therefore, we should disentangle the skill knowledge to compute the clean bias attribution B
(ι,x) i (h) as follows:
B (ι,x) i (h) = A (ι,x,ŷ) i (h)− Ã (ι,x,y) i (h) (2)
where Ã(ι,x,y)i (h) means the attribution score for the golden label text y, where negative values of A
(ι,x,y) i (h) are converted to zero values. Since the negative values of the attribution score are undesirable negative importance for a specific task, it is reasonable to exclude that information.
Automatic Identification of Biased Labels. We should determine a biased text to compute the bias attribution for each input instance. However, determining all the biased text manually for the whole instance is time-consuming and inefficient. For example, the BBQ-SES dataset, a socioeconomic status bias dataset, contains various different text labels for the protected group, such as poor people, low-income people, the truck driver, etc. Thus, if we consider the realistic application of our method, then we have to determine the biased text automatically. Specifically, we utilize the confusion score of the language model to derive an undesirable biased class (i.e., text) for each instance as follows:
ŷj = argmax c
P(c|ι, xj)
where c ∈ {c′|c′ ∈ C ∩ c′ ̸= y} (3)
where c and C mean a class and the class set of the dataset, respectively.

heading: 3.3 Aggregation of Bias Scores
text: Token Aggregation. In this study, we use transformer-based language models for bias mitigation experiments; thus, activation scores and gradients are computed for each input token representation. Therefore, if an input text xj includes K tokens, we have K attribution scores for each neuron; thus, we should aggregate attributions for tokens as follows:
B (ι,xj) i (h) = max
k B
(ι,xj ,tk) i (h) (4)
where tk ∈ xj means each token of an input text. B
(ι,xj ,tk) i (h) is the attribution score computed for
each token tk.
Instance Aggregation. Also, there are multiple instances for each task; thus, we should aggregate attributions for instances as follows:
B (ι,D) i (h) =
N∑
j
α(ι,xj)B (ι,xj) i (h)
α(ι,xj) = P(ŷj |ι, xj) (5)
where D and N mean a specific dataset and the number of instances in the dataset, respectively. The more confusing a data instance is, the more information it contains about bias; thus, we use its confusion score as a weight α.
Although B(ι,D)i (h) can be computed using the entire dataset, we report the experimental results of computing it using only a significantly small amount of data (i.e., only twenty data samples) to ensure the efficiency of our method.
Instruction Aggregation. We also aim to mitigate the bias that occurs from the association between instructions and labels. Although it is important to mitigate bias within an instruction (intra-instruction bias), reducing the understanding gap between synonymous instructions (interinstruction bias) is also essential. Therefore, we calculate the mean attribution for all instructions to get the bias neuron score considering the information of all instructions as follows:
B (I,D) i (h) = 1
M
I∑
ι
B (ι,D) i (h) (6)
where M means the number of instructions. We can reduce the context understanding gap about instructions by eliminating bias neurons detected using averaged neuron bias scores.

heading: 3.4 Biased Knowledge Mitigation
text: This section describes how to eliminate the detected bias neurons using a structured pruning method. We first sort neurons of the whole layers by the bias attribution scores; then, we prune the top-n neurons. Suppose that a weight matrix W ∈ Rd×l is a linear matrix multiplication parameter, and then the matrix after pruning is denoted as W̃ = (Wij)1≤i≤d
j /∈M ,
where M is the set of bias neuron indices about the W . If the bias term b ∈ Rl is added to the operation for an affine transformation, the bias term can also be pruned by performing the b̃ = (bi)i/∈M operation similarly. The bias-mitigated parameters are used to compute the new representation by performing the transformation operation hW̃ or hW̃ + b̃. Notice that this method is model-agnostic since all neural network models consist of linear transformation layers. For example, transformer variants have self-attention, cross-attention, and feed-forward network (FFN) modules, all of which include linear matrix multiplication operations.

heading: 4 Experiments

heading: 4.1 Experimental setup
text: Datasets. We conduct experiments on three social bias question answering (SBQA) (Parrish et al., 2021) and three natural language understanding (NLU) (Wang et al., 2018) datasets. Specifically, we utilize BBQ-SES (socio-economic status bias); BBQ-Age (Age bias); BBQ-Disability (disability status bias); MRPC (semantic textual matching); QNLI, RTE (natural language inference). BBQ datasets are QA datasets and contain inconsistent multiple candidate labels. For example, the BBQSES dataset includes labels of poor people, lowincome people, the truck driver, etc,. for minor groups, and this vast label space makes bias mitiga-
tion more challenging. Since BBQ datasets contain only the test set, we split them as 10% for a development set and 90% for a test set, and we compute bias attribution by sampling twenty instances from the development set. Our reported BBQ datasets performances (§4.2) for all baselines are the results of our test set split.
Implementation details. We select the instruction-following language models, Flan-T51 (Chung et al., 2022) and T-Zero2(Sanh et al., 2021), as a backbone model in our study. We use ten instructions for all datasets, which are acquired using ChatGPT (OpenAI, 2023) by paraphrasing an instruction template of each dataset. We evaluate bias mitigated models using the whole ten instructions and report the mean accuracy of them. All instructions for each dataset are shown in Appendix D.
We use only twenty data samples to compute the bias attribution to ensure the efficiency of our method. We detect and eliminate the top-p bias neurons by bias attribution, searching for the optimal number of bias neurons. Specifically, we investigate the varying neuron pruning rates p ∈ [0, 0.01] for the whole layers (i.e., self-attention, crossattention, and FFN) and early stop by measur-
1https://huggingface.co/docs/transformers/ model_doc/flan-t5
2https://huggingface.co/bigscience/T0pp
ing whether the task performance increases or decreases. We select only a small portion of a train set (≤ 10%) for evaluating the best number of bias neurons considering efficiency. For the BBQ datasets, we use our development set to determine the best number of bias neurons. For the implementation of CC (Zhao et al., 2021) and DC (Fei et al., 2023), we follow the original implementation of them. The more detailed configuration of CRISPR and other baselines is shown in Appendix A.

heading: 4.2 Bias neurons exist
text: We evaluate the bias mitigation performance of our method and other baselines for the instructionfollowing prompt setting. Table 1 shows the mean accuracy of various methods for the six datasets. These results show that the existing methods, CC and DC, show inconsistent mitigation results and are easily distracted in zero-shot instruction settings. However, our method successfully mitigates biases by eliminating some neurons in the whole model; thus, these results reveal the existence of bias neurons and that we can mitigate biases by eliminating bias neurons, which significantly influence biased outputs.

heading: 4.3 Gaps in understanding instructions are alleviated after bias neuron elimination
text: Instruction-following language models tend to derive inconsistent outcomes when presented with synonymous but different textual instructions. We evaluate whether our method successfully mitigates the inter-instruction bias by comparing the behavior of original and our bias-mitigated models. Specifically, we measure the standard deviation of accuracy for ten synonymous instructions about each model and compare them. Table 2 shows the experimental results of the inter-instruction bias mit-
igation. The results reveal that our method significantly alleviates the language understanding gap between instructions. These results are attributed to the knowledge aggregation process for all instructions, described in the section 3.3. Since the bias is quantified by considering all instructions, the overall ability to understand instructions increases.

heading: 4.4 How many bias neurons are eliminated?
text: This section describes how many bias neurons are eliminated to mitigate the bias of language models. Table 3 shows the number of neurons eliminated from each model and each dataset.
Surprisingly, bias is attributed to a significantly small number of neurons (e.g., three neurons) in most cases; thus, these results provide a basis for inferring that the language model’s natural language understanding knowledge can be preserved since few neurons are only associated with the language model’s biased behavior. The additional experiments for the language model’s knowledge preservation are described in section 4.6.
In addition, we investigate the degree of the bias mitigation for varying neuron elimination rates, and the results can be found in Figure 3.

heading: 4.5 How many data samples are needed to quantify bias?
text: Our method can precisely quantify the bias with only a few data samples. Figure 4 shows the bias mitigation results for varying numbers of data samples when computing the bias attribution.
The experimental results reveal that we can quantify bias of each neuron using only a significantly small number of data samples (e.g., ten data samples).

heading: 4.6 Skill knowledge is preserved after bias mitigation
text: Skill Knowledge Preservation. Even if bias neurons are eliminated for a specific task, natural lan-
guage understanding knowledge and skill knowledge of other tasks should be preserved from the language models. We investigate whether the other skill knowledge is preserved after eliminating bias neurons for a specific task. Specifically, we detect and eliminate bias neurons for a specific task (source) and measure the performance degradation for other tasks (target). Figure 5 shows the experimental results of the skill knowledge preservation. These results demonstrate that natural language understanding knowledge and skill knowledge of other tasks are preserved.
Bias Knowledge Transfer. Surprisingly, Figure 5 also conveys that the detected bias neurons for a specific dataset function as a bias in other analogous datasets. In the case of bias neurons derived from the BBQ-SES dataset, if we eliminate those bias neurons from the language model, the performance for the other datasets also significantly increases. Similarly, the performance of the NLU datasets (i.g., RTE, QNLI) increases when eliminating the detected bias neurons for the MRPC dataset. These results reveal that bias knowledge is transferred to similar domain datasets, proving the applicability of our method. We recommend determining the bias neuron elimination rate by measuring and evaluating mean accuracy to obtain the optimal results for all datasets to use.
More detailed experimental results for skill
knowledge preservation about six datasets are shown in the appendix C.

heading: 4.7 Qualitative analysis
text: We also qualitatively analyze our method on BBQSES and MRPC datasets. The qualitative analysis results are shown in Table 4. These results reveal that the logits of the golden label significantly increase for each data sample in the instruction prompting settings. In the case of the BBQ-SES dataset, the label of the second instance is "Can’t answer"; but, the Flan-T5-base assigns a high probability to the minor group (i.g., The Cashier) for a negative question. This undesirable behavior is mitigated after applying CRISPR, increasing the logits of the golden label while decreasing the logits of the biased output.

heading: 4.8 Ablation studies
text: In this section, we perform ablation experiments over each CRISPR method to better understand their relative importance. Max Token Agg means the aggregation method for the token attribution, described in the section 3.3. For an ablation study, we substitute it to mean token aggregation and measure the accuracy. Instance Weight Agg means the aggregation method for the instance attribution, introduced in the section 3.3. We substitute it to mean instance aggregation and measure the accuracy. Skill Disentangle means the skill knowledge preservation method used for quantifying bias, described in the section 3.2. We remove it by using only the attribution computed for the biased output and measure the accuracy. Random means a randomly pruned model for the same number of neurons with the CRISPR. We conduct the ablation studies for the Flan-T5-base, and the results for
ablation studies are shown in Table 5.
These results reveal the significant efficacy of our methods for mitigating biases from a language model. Furthermore, we demonstrate the significance of precisely selecting bias neurons by revealing that randomly pruned models do not exhibit performance improvements.

heading: 5 Conclusion
text: In this study, we define the bias neuron and prove its existence empirically. Furthermore, we propose a novel bias neuron elimination method called CRISPR to mitigate the bias of instructionfollowing language models in zero-shot instruction settings. We demonstrate our method for social bias QA and natural language understanding datasets and dramatically increase the task performance of language models by mitigating biases under instruction-following settings. Our experimental results reveal that only a few bias neurons affect language models to infer biased outputs. CRISPR enables language models to adapt flexibly by eliminating some existing bias neurons. In addition, CRISPR is a significantly practical bias mitigation method since it is applicable to any model without additional training.

heading: 6 Limitations
text: We introduce the concept of bias neuron and prove its existence. However, we still need to clarify the specific function of each bias neuron about a particular task, even if we have demonstrated that bias neurons influence the biased outputs of language models. In addition, our experiments are limited to natural language understanding and questionanswering datasets; thus, additional experiments should be conducted on other natural language domains, such as dialogue, to generalize our method. These aspects of investigation are left to future works.

heading: 7 Ethical Considerations
text: Each dataset has labels assigned according to a predefined policy, and our method defines and mitigates bias based on these predetermined labels. Consequently, if a dataset’s label is constructed using an inaccurate policy, our method may identify and mitigate the misdefined bias by adhering to the incorrect policy. Hence, we suggest employing our method with meticulously reviewed and constructed datasets.

heading: Acknowledgements
text: This work was supported by LG AI Research. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics & RS-2021II211343, Artificial Intelligence Graduate School Program (Seoul National University) & RS-2021II212068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)]. This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS2024-00348233). K. Jung is with ASRI, Seoul National University, Korea. The Institute of Engineering Research at Seoul National University provided research facilities for this work.

heading: B How many data samples are required to quantify bias score?
text: We compute bias scores by using varying number of data samples. Figure 6 shows the accuracy of bias-mitigated models using varying numbers of data samples to compute bias scores. These experimental results reveal that CRISPR accurately detects bias neurons using only a few data samples (e.g., ten samples), proving the efficiency of our method.

heading: C Skill Knowledge Preservation and Bias Knowledge Transfer
text: Skill Knowledge Preservation We aim to mitigate biases from language models in instructionfollowing settings. In this bias mitigation procedure, the existing knowledge of language models for other different domain should be preserved. Figure 7 shows the experiments for skill knowledge preservation. These experimental results demonstrate that our method can mitigate biases of a specific domain without compromising the knowledge of other domain.
Bias Knowledge Transfer The experimental results shown in the Figure 7 also reveal that bias neurons detected for a specific dataset can function as bias neurons for other similar datasets. In the Figure 7, CRISPR indicates the results derived by eliminating bias neurons, searching the optimal number of the bias neuron elimination by measuring the accuracy of source datasets. CRISPR† shows the results searching the optimal number of the bias neuron elimination by measuring the accuracy of target evaluation datasets. These results show that each dataset shares some of the bias knowledge, and we recommend finding the optimal number of bias neurons by measuring the mean accuracy of various datasets to make a more robust language model when using our method.
D Instruction Details
This section describes detailed instruction settings used in our experiments. The instruction templates of each dataset are shown in Figure 8
Also, Table 6 shows the ten instruction texts used in our experiments for each dataset.

heading: E Where do biases come from?
text: This section describes an additional analysis of the source of the biases. Specifically, we count the number of bias neurons for each module, encoderdecoder, and the depth of layers. Figure 9 shows the experimental results for illuminating where biases
come from. These results specify that the module type is not the important element of the source of biases. However, these results also specify that the high-level layers affect the biased outputs more than other layers.

heading: Future Work
text: These aspects of investigation are left to future works