UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation
Authors: Hanzhang Zhou
Zijian Feng
Zixiao Zhu
Junlang Qian
Kezhi Mao
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM Bias
In-Context Learning
Attention and FFN Manipulation
Prompt Brittleness
Primary area: natural_language_processing
TL;DR: Unveiling internal mechanisms that lead to LLM bias and mitigate such bias by manipulation of LLM internal structures.

Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various tasks using the in-context learning (ICL) paradigm. However, their effectiveness is often compromised by inherent bias, leading to prompt brittleness—sensitivity to design settings such as example selection, order, and prompt formatting. Previous studies have addressed LLM bias through external adjustment of model outputs, but the internal mechanisms that lead to such bias remain unexplored. Our work delves into these mechanisms, particularly investigating how feedforward neural networks (FFNs) and attention heads result in the bias of LLMs. By Interpreting the contribution of individual FFN vectors and attention heads, we identify the biased LLM components that skew LLMs' prediction toward specific labels. To mitigate these biases, we introduce UniBias, an inference-only method that effectively identifies and eliminates biased FFN vectors and attention heads. Extensive experiments across 12 NLP datasets demonstrate that UniBias significantly enhances ICL performance and alleviates prompt brittleness of LLMs.

UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation
Hanzhang Zhou 1.2 Zijian Feng Zixiao Zhu Junlann Qian', Kezhi Mao 1.2
*Nanyang Technological University "Singapore-ETH Centre (hanzhang001, feng0119, zixia0001, junlang001)0e ntu edu.sg ekzmao@ntu, edu.
Abstract
Large language models (LLMs) have demonstrated impressive capabilities in various tasks using the in-context learning (ICL) paradigm. However, their ef- fectiveness is often compromised by inherent bias, leading to prompt brittle- ness ness-Sensitivity to design settings such as example selection, order, and prompt formatting. Previous studies have addressed LLM bias through external adjust- ment of model outputs, but the internal mechanisms that lead to such bias remain unexplored. Our work delves into these mechanisms, particularly investigating how feedforward neural networks (FFNs) and attention heads result in the bias of LLMs. By Interpreting the contribution of individual FFN vectors and attention heads, we identify the biased LLM components that skew LLMs' prediction toward specific labels. To mitigate these biases, we introduce UniBias, an inference-only method that effectively identifies and eliminates biased FFN vectors and attention heads. Extensive experiments across 12 NLP datasets demonstrate that UniBias significantly enhances ICL performance and alleviates prompt brittleness of LLMs. The code is available at httttss//guuuuu..om//zzooouuuuuu
Introduction
Large language models (LLMs) have shown exceptional capabilities in various natural language processing (NLP) tasks, employing the in-context learning (ICL) paradigm. This paradigm conditions LLMs on a context prompt comprising of few example-label pairs [Brown et al.) 2020 Wei et al..
2022
Dong et al. 2023 Zhou et al.) 2024].
Despite their impressive performance, LLMs prone prompt brittleness, characterized by high sensitivity to the choice [Zhao et al., 2021] and order [Lu et al. 2022) of examples, and prompt formatting [Min al. 2022], demonstrated in Figure] 1 Such prompt brittleness found to be arise from the bias in LLMS towards predicting certain answers [Zhao et al. (202]). The presence of the LLM bias undermines the robustness and adaptability of LLMs in diverse applications.
Extensive research has focused on identifying factors that lead to LLM bias and strategies for mitigation. For instance, vanilla label bias (Fei et al.) 2023] and recency bias (Zhao al. 2021 demonstrate the LLM's inherent non-contextual preference for certain labels and contextual preference
for specific positions, respectively. Additionally, several calibration methods (Fei et al.) 2023 Han
et al. 2023 Zhao et al. 2021] are proposed to counteract the bias by adjusting decision boundaries of model output probabilities. However, these approaches derived from external observations or adjustments of LLMM outputs, leaving the internal mechanisms within LLMs that cause such bias
poorly understood.
In this work, we investigate the internal mechanism of LLM bias, specifically how feedforward neural networks (FFNs) and attention heads contribute to such bias. Building findings in mechanistic
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Example Selection
Prompt Formatting
Exampie Order
90
90
Accurecy (%)
90
Accurecy (%
85
Accuracy 0%)
80
85
75
80
70
ICL
UniBias
ICL UniBias
ICL
UniBias
Figure 1: illustrates the prompt brittleness of ICL and the effectiveness of our method in mitigating this issue. Experiments are conducted in one-shot setting, using SST2 [Socher et al. [0033] dataset for experiments example selection and prompt formatting and AGnews [Zhang al. 2015] dataset for example order experiment due to more diverse combination of orders.
interpretability [Elhage et al. 2021 Dar et al. 2023], we assess the contribution of individual attention heads and FFN vectors label predictions in LLMs. By identifying FFN vectors and attention heads that convey biased influences towards label prediction, we reveal the internal mechanisms behind several key bias factors, including vanilla label bias (Fei et al. 2023] recency bias [Zhao et al., 2021]. and selection bias [Zheng et al., 2023). For instance, our analys sis of FFN vectors without input context demonstrates that their cumulative impact biases the LLM towards specific labels. indicating a non-contextual preference for certain labels, i.e., vanilla label bias. We elaborate on the background of mechanistic interpretability in Section 2.1 and present our findings on the internal mechanisms of LLM biases in next section.
Given findings that various bias factors stem from the biased behaviors of attention heads and FEN vectors, we are prompted to ask: Can we identify the biased components of LLMs and mitigate their detrimental impact label prediction? Motivated by this intuition, propose UniBias, inference-only method designed to identify and eliminate biased FFN vectors and attention heads in LLMs. Specifically, we begin by projecting each FFN vector and attention head into the vocabulary space to interpret the information conveyed by their outputs. We then detect biased components based on three criteria we defined: the relatedness criterion, the bias criterion, and the low variance criterion. After identification, we mitigate their impact by masking these biased components. Extensive experimental results demonstrate that LLMs, from which biased components have been removed, consistently outperform their original counterparts by a significant margin. Further, as illustrated in Figure T our method significantly improves both the performance and robustness of ICL with perturbations of various design settings.
The contributions of work summarized follows:
In contrast to existing works based on external adjustments of LLM outputs, we mitigate LLM bias through manipulation of LLM internal structure. This novel perspective potentially offers new direction for the field. Moreover, our method demonstrate an effective way to manipulate internal structures of LLMss.
We conduct a thorough investigation the internal mechanisms underlying biases in LLMs, revealing the inner causes of these biases.
Extensive experiments across 12 NLP datasets demonstrate that, by removing the biased compo- nents, our UniBias method significantly enhances ICL performance and achieve state-of-the-ar results. Additionally, effectively addresses the issue of prompt brittleness.
Internal Mechanisms Causing the Bias of LLMs
This section reveals the internal mechanisms within LLMs that lead to various bias factors.
FFN vector refers to the value vector in the second weight matrix of the FFN layer. We elaborate on this in Section 2.1

---

2.1 Background
The theoretical background of this work is based on research on mechanistic interpretability (Elhage al., 2021 Wang et al. 2022 Geva et al.) 2021]. which aims explain the internal processes in language models (LMs), facilitating the interpretation of the contributions of individual model components to the final prediction.
We are focusing on decoder-only LMs in this paper. They are composed by sequence of transformer layers, each composed of multi-head self-attention layer and an feedforward neural network layer. The background knowledge for interpreting the contribution of each FFN vector and attention head to the models' prediction are demonstrated as follows.
The Residual Stream We interpret Transformers following the view of residual stream (Elhage et al. 2021 Dar al., 2023]. Due to the residdual connection of Transformers, each layer takes a hidden state as input, and adds information obtained by its attention layer and FFN layer to the hidden state through residual connection. In this sence, the hidden state is a residual stream passed along layers, and each attention layer and FFN layer contribute to the final prediction by adding information to the residual stream.
Attention Heads Following Elhage et al. 2021 Dar [023], the output of each attention layer of LM can be computed as the sum of all its attentton heads. Specifically, for /-th layer, the input is X Nxd, and the attention layer parameterized by four matrices W2 W W W E Rdxd The columns of each projection matrix and the of the output matrix be split into H parts: w W w IRdx & and Wo E R xd where H is the number of attention heads. We then find that:
Au' Ax')x Concat [1* x'xx NeW.".AA20 x A x W wo = (x
"X'"""("'"""""" M & the attention mask. Therefore, the Va/H
where A softmax
output of an attention layer is equivalent to computing attention heads independently, multiplying each by its own output matrix, and adding them into the residual stream of the LM.
FFN In line with Geva et al. [2021] 2022], transformer FFN layers can be cast as linear combination of vectors. Specifically, for an input vector x E I FFN parameter matrices K' V Rd the FFN output can be derived as:
TFN((x)) -(x'K' V (_(--K/)
Σ mfv
where / is the activation function, i is the index of the vector. Then, the FFN layer can be viewed as linear combination vectors: the multiplication of x and the key vector k; produces the coefficient mf that weights the corresponding value vector V..
Logit Lens The logit lens [Nostaleebraist 2020) a technique that directly decode hidden states
into the vocabulary space using the unembedding matrix of the LLM for interpretation. This approach has been validated in various studies as an efficient method for interpreting the weight matrix or
hidden states of LLMs [Dar et al. 2023 Hanna et al.) 2023 Feng et al. 2024 Yu et al. 2023 Geva al. 202T].
In summary, each attention layer and FFN layer contribute to the final prediction by adding their output hidden states the residual stream. These outputs be viewed the sum of their respective attention heads and FFN vectors. Each attention head or FFN vector's output can be interpreted through the logit lens.
2.2 Internal Mechanisms of Bias Factors
We delve into the mechanisms behind several bias factors, analyzing the contributions of attention heads and FFN vectors to the biased predictions in LLMs. We explore vanilla label bias, position bias, and selection bias using the Llama-2 7B model [Touvron al. 2023

---

Vanilla Label Bias The vanilla label bias Fei et al.
Label Logits
2023]. also known common token bias [Zha
1609
2021]. J. is the inherent, uncontextual preference of
model towards predicting certain label names. Given the contextual nature of attention layers, our
1200
503
investigation focuses on the FFN layers, where we
400
203
identified corresponding uncontextual preference. Specifically, by projecting the FFN value vectors into the vocabulary space, compute the logits for ious label names for each FFN vector. Utilizing the residual stream insight, we then aggregate these log- its for all FFN vectors whose label logits rank within the top 10 over the vocabulary, reflecting uncontex- tual influences of FFN vectors that are effective in label prediction. This process yields what term uncontextual accumulated FFN logits, revealing the intrinsic bias of the LLM towards predicting label names without the influence of input.
Regetie Poxliee
Red Cood
D sepprttte sttd
Label Prediction Frequence
0.8
0.0 Nogatve: / PosDve
Badl Good
Disappenntas Saesffid
Figure 2: Unveiling vanilla label bias by un-
contextual accumulated FFN logits.
Figure 2 illustrates the accumulated uncontextual FFN logits across different label names in the sentiment analysis task, alongside their corresponding zero-shot prediction frequencies on the SST-2 dataset. For example, the label name 'positive' exhibits higher uncontextual accumulated FFN logits compared to "negative," leading to a higher frequency of 'positive' predictions. Additionally, when comparing the labels 'oodd' and "bad", the difference in their uncontextual accumulated FFN logits more pronounced than that between 'positive' and 'negative," resulting in a larger discrepancy in prediction frequency. Conversely, the accumulated logits for the labels 'satisfied' and 'disappointed" show a reverse trend relative to 'positive' and 'negative', which results in a corresponding reverse trend in their prediction frequency ratios.
Recency Bias Recency bias refers to the tendency of LLMs to favor the label of the example at the end of the prompt (Zhao al., 2021
(Raversed Sequence)
06
Attennge Werte
0.0
RTC
By examining the behavior of atten- tion heads within LLMs, observe that specific heads consistently pri- oritize the example the end of the prompt, providing an internal per- spective on the origin of recency bias.
0.3
0.3
26
let
2nd
2nd
Attention
Attention
Attention Head
0.20
0.18
We identify the biased attention head using the method introduced in Section 3 We compare the be- haviors of a biased attention head (layer 16, head 29) and an unbiased attention head (layer 16, head 19) in terms the attention weight as- signed to examples at different positions and the label logits of the corresponding attention head's output. Specifically, we use the SST-2 dataset, including one positive and one negative example in the prompt, and test with 40 samples, evenly split between positive and negative examples. More experimental details are provided in Appendix[A]
go14
0.2
0.0
Poolive
Eegidiee
Restive
Megative
Postive
Negative
Predktizo
Red
Ground buth Label
Figure The internal mechanism of the recency bias.
Experimental results in Figure B reveal that the biased attention head (layer 16, head 29) consistently assigns significantly larger attention weights to the final example, irrespective of the ground truth labels of the test samples. This bias persists even when the sequence of examples is reversed, as shown in the second subfigure, indicating a biased preference of this attention head for the last example in the prompt. Furthermore, the biased attention weight assignment leads to biased logits, as shown the third subfigure. contrast, the unbiased attention head (layer 16, head 19) assigns very close averaged attention weights to both examples in the prompt. Interestingly, observe that this unbiased head generally assigns larger weights to the example whose label matches the ground

---

truth label of the test sample, resulting in 35 out of 40 samples being correctly classified based on this pattern by this single attention head. The preference shown by specific attention heads for the example at the end of the prompt reveals the internal mechanism of recency bias.
Selection Bias The selection bias refers that LLMs prefer to select spe- cific option ID (like "Option A")
Accumslated FRN Labet Logits
Attenciosn Head (74,99)
Altention Hoad (24,29
0.5
answers for multiple choice ques-
tions [Zheng et al. 2023]. We have identified both FFN vectors and at- tention heads that consistently favor a specific option regardless of the ground truth label of the test sample, revealing the internal mechanism of selection bias.
2000
1300X
Oytion Ayriion
Optiin Recitoss
Figure 4: The internal mechanism of the selection bias.
We evaluate the Llama-2 7B model on the ARC dataset, which contains four options (A, B, C. D). We use a zero-shot setting to avoid the influence of position bias from multiple examples. More details are provided in Appendix A Experimental results are illustrated in Figure 4 Firstly, we observe that the LLM exhibits a vanilla label bias favoring option "A", as shown in the first subfigure. Additionally, identify biased attention head that demonstrates position bias consistently favoring the first option regardless of the ground truth labels of the test samples (second subfigure) or changes in the sequence of options (third subfigure). Since option A is usually the first option, these two biases both lead to the L.M's preference for option A.
3 Methodology
In the previous section, we unveil that various bias factors are stem from the biased behaviors of attention heads and FFN vectors. Naturally, we pose the question: Can we identify the biased components of LLMs and mitigate their impact on label prediction? Therefore, we propose our UniBias method to Unveil and mitigate LLMs' label Bias through internal attention and FFN manipulation. Notably, our method is proposed for decoder-only LLMs.
3.1 Biased FFN Vectors Identification
Identifying biased FFN vectors in LLMs hinges on whether the contribution of each FFN vector independent and interpretable. As discussed in Section 2.1 the output of an FFN layer can be cast as linear combination of FFN vectors. Each FFN vector contributes to the final prediction by adding information encoded its value vector, v weighted by its corresponding coefficient, mf.
This information within v can be interpreted through the logit lens, enabling us to interpret it as distribution of logits across the vocabulary space.
How to identify an FFN vector as biased? we assess whether it consistently introduces biased preference towards specific labels into the residual stream, regardless of variations in the test samples. Such consistent biases can skew the LLM's s predictions. We introduce the following criteria to detect biased components in LLMs, which are also applicable for identifying biased attention heads:
Relatedness Criterion: The information introduced by the FFN vector (or attention head) should closely relate to label prediction.
Biased Criterion: The information contributed to the residual stream by the FFN vector (or attention head) exhibits a biased distribution, favoring certain labels over others.
Low Variance Criterion: The label prediction information added by the FFN vector (or attention head) to the residual stream is almost identical across set of test samples with different labels, i.e., exhibits very small variance.
The third criterion is key to identifying biased FFN vectors (or attention heads), as consistently low variance indicates that the FFN vector is not adequately responsive to varying inputs. Combined with the second criterion, this suggests a bias towards certain predictions regardless of the input's contextual differences.
5

---

To examine these criteria, we interpret the information contributed by each FFN vector, i.e., mv. For simplicity, we omit the layer number l and FFN index Since the FFN value vector fixed, changes in the FFN coefficient m across different samples reflect the change in information brought by the FFN vector. We interpret this information by projecting each FFN value vector into the vocabulary space and analyzing the logit distribution over label tokens, termed label logits.
Specifically, given an FFN value vector v E I&d the unembedding matrix E E R label token mapping matrix L E Ig- where each row is a one-hot vector indicating the token id of the first
token of each label name, the label logits g (k) [gb corresponding to the FFN value vector v of k-th sample can be obtained by:
(4) _0[*) -6-1 I (where i the class number)
= v. E.
We use p unlabeled samples from the task to assess the three criteria we defined. The coefficients and label logits of an FFN vector for these samples are denoted as m = [mo: m....., mp- 1| and
G = [g( g()) ..... g ((")).....(_"") g(p-1yT E IRxx respectively. An FFN vector is considered biased if it meets the following conditions, each corresponding to one of the three criteria we defined:
Sum (Gr..)
Sum
g
(k)
93
thEEN
(1)
P
P
P
km
Bias (Gr..)
1
Σ
p-1
Bias
8
(kk)
p-
c-1
(0)²)
-m(g())
> thREN
(2)
P K=D
p
kmu
cc
- (m² ((m)))
CV (m) (m)
thFFN
(3)
[mm)
p EE= me
where (1) = 2 -ー (K) 9j (mm) E mg. The thresholds PhPEN Upp prx are
set by grid search, which elaborated in Section
The first equation corresponds to the relatedness criterion, measured by the sum of label logits. A higher sum indicates that the information introduced by the FFN vector more relevant to label prediction. The second equation relates to the bias criterion, quantified by the deviation of the average logit for each label from the overall average logit across all labels. Ideally, for set of test samples with different labels, the average logits for each label should be relatively balanced. A greater deviation from each label's average compared to the overall average across all labels indicates more biased distribution. The third equation addresses the low variance criterion, measured by the coefficient of variation (CV) of the FFN vector coefficients across different samples. The CV, calculated as the standard deviation normalized by the mean, indicates whether the label prediction information added by the FFN vector remains almost the same across different samples.
3.2 Biased Attention Heads Identification
The identification of biased attention heads closely resembles the process of identifying biased FFN vectors. As discussed in Section 2. each attention head's 's contribution to the final prediction is independent and interpretable. Therefore, we project the output hidden states of each attention head into the vocabulary space to interpret the information they contribute.
To identify biased attention heads, we use the same three criteria introduced for identifying biased FFN vectors. To apply these criteria, we project the output hidden states from each attention head into the vocabulary space and analyze their label logits as the information contributes to label prediction. The output from each attention head consists of hidden states generated for every token in the sequence. For our analysis, we specifically use the hidden state of the last token preceding the prediction of label names, interpreting it as the most direct contribution of the attention head to the prediction, given the autoregressive nature of LLMss.
Specifically, to obtain the label logits for an attention head, consider the output hidden states H IR Nxd of this head, the unembedding matrix E E idd and the label token mapping matrix E R N Xxd, Given the token position Plabel E {0, I,..., N 1}. which indicates the index of the
first token of the predicted label names, the label logits (b) = [a] aa ......c (+)) of the attention head for the k-th sample are derived by:
(k) (k)
a (R) H (amma-1). 1), E E.

---

we employ the same p unlabeled samples from the task to assess the criteria for identifying baised
attention head. The label logits for these samples are formed as A = [a a . a 1 E
Rmxxe An attention head is considered biased if it meets the following conditions:
Sum (Ak..)
=
Sum (a
1
> the
p
P
Bias (Ak.))
1
p-1
Σ
Bias a
p-1
Σ
(1)
=l(aa)))) > thac
p k-o
pc k=0
((A.s)
Σ My 1-0
CV (A:,) wy (((a.))
< th th
M(A.)) where wy = E ) ((A.))= A,, - ()
The functions of the first two criteria are identical to those for biased FFN vector identification. The third function is the weighted sum of the coefficient variance of each label across test samples. The thresholds for biased attention head identification are also derived by grid search.
3.3 Biased FFN Vectors and Attention Heads Manipulation
After identifying the biased components of the LLM, we eliminate their influence by masking these biased FFN vectors and attention heads. Specifically, create masks for the attention heads in each attention layer and reset the coefficient of the biased FFN vector and biased attention head mask.
3.4 Grid Searching
Specifically, utilize small subset of training data as support set, with 20 samples for each class. We then grid search all combinations of threshold values and select the combination that results in the most balanced distribution of average label logits. Specifically, let T represents the set of threshold combinations, and P(t) denote the average label logits for a threshold combination t E T, we aim to find the combination t" that minimizes the bias of label logits: t" arg minter Bias (Pt))..
is noteworthy that although there are multiple combinations of thresholds, they usually result in a few set of different biased components. For example, for a grid search of thresholds of FFN vectors with 80 combinations, it only result in 4 different sets of biased FFN vectors that need to be examined with the support set on the SST-2 dataset. Additionally, during the inference stage of evaluating test samples, the computation time of the UniBias method is completely identical to that of the original LLMs.
Additionally, the support set can be replaced with unlabeled samples, using approximately twice the number of unlabeled samples compared to labeled ones. For further details, please see Appendix F
4 Experiments
In this section, we aims to investigate a few research questions (RQ). RQ After eliminating biased components from LLMs, does the ICL performance improve compared to the original LLM? Additionally, how does our UniBias method compare to existing calibration methods? RQ 2: Given that ICL suffers from prompt brittleness, can our UniBias method contribute to more robust ICL performance? RQ3 3: Are there any observable patterns of biased FFN vectors and attention heads within and across tasks? RQ 4: What is the performance of LLMs after climinating only the biased FFN vectors and only the biased attention heads, respectively? RQ 5: What is the impact of support set size on the performance of the UniBias method?
4.1 Experimental Setup
We evaluate our UniBias method on 12 diverse natural language processing datasets across various tasks, including sentiment analysis, topic classification, natural language inference, reasoning, and word disambiguation. Statisties and details about the datasets can be found in Table in Appendix.
Datasets
7

---

Table 1: Comparison of one-shot ICL performance for different methods across datasets using Llama- 2 7b and Llama-2 13b models. The mean and standard deviation are reported for five repetitions with different ICL examples.
Daraset ICT.
Llama-2 DXC
Llama-2 2 13b D3C
Mathod
CC
PC
UniBias
ICT.
CC
PC
UniBias
SST-2
87.22603
92.244,.3
94,15
93.90,.5
94,54
93.90..7
95.25
95.376.70
94,561.71
95.466,52
MNLI
23.83
53.361.16
52. 52
45.385.01
54.97us4
62,43
63.89kst
61 61.86123
57.477.53
64.652.7
WIC
50.000.1
52.192.00
52.40h.
S7.1124
L3.71
54,481.9
50,66315
49,72a.30
55.67
57.93
COPA
37.60
67.80
60.40g
67.80
69.002.4
67.501o.40
35.20
71 71.00gm0
76.80
83.20
CR
91.54 90
92.130.4
92.61n44
91.97a3s
92.61an
91.011.00
92.13
92.23n.75
91.65as4
92.344.34
AGNews
85,59)
83.54
89.08ku
86.81
88.29 24
89, 89.14
88.231.14
89,34.n1
86.03
88, 68.44
MR
89.37
91.771.42
92.35
91.39,65
19
90.10..im
93.20
93.00 33.000.s2
92, 92.80
92.23
RTE
66.21
64.333,58
65.49
62.599.71
67.65&44
70,10
71.99k.0
66.21
75.312.00
78.233.13
SST-5
46.97a.57
51.36
51.921.77
$5.41.31
53.79
51,03
47.20 1.80
48 18.98
53.63a.9
$1.80,00
TREC
72.92;242
76,443.21
77. 77.16294
74,92,78
80.800
74.701210
3,.80
80.50kga7
81.85
81 61.2556,6
ARC
51.90000
53.10,
53.00hee
40.40
53.10
66.54,35
64.33
64.88u.9
59.47
66.810.3
MMLU
25
43.720,97
43.57
34.12341
44.83
53.53
5,.844,57
51 51.81
45.50).6
53.55 o05
Avg.
67.07
68.49
68.70
66.81
70.46
72.54
73.06
72.08
72.56
75.51
80
COPA
100
SST-2
50
MMLU
Method
KL
70
95
45
CC
DC
Accuracy
UniBias
60
90
40
50
85
35
40
0-shot
1-shot
2-shot
4-shot
80
0-shot
1-shot
2-shot
4-shot
30
0-shot
1-shot
2-shot
4-shot
Figure 5: The performance comparison under different numbers of ICL shots using Llama-2-7b.
Baselines In addition to the standard ICL, we compare our proposed UniBias with state-of-thes thet LLM debiasing and calibration baselines, including Contextual Calibration (CC) (Zhao et al. 2021] J, Domain-Context Calibration (DC) [Fei et al. 20023.. and Prototypical Calibration (PC) [Han et al.,
2023]. We reproduce all basclines strictly follows the authors' instructions and recommendations to ensure fair comparison.
Models and implementation details We evaluate our method using a range of LLMs, including Llama-2 7b, Llama-2 13b [Touvron et al.) al.] (023), GPT-J [Wang and Komatsuzaki] [021] and GPT2- XL [Radford et al. 2019]. For all experiments, unless stated otherwise, we use 1-shot ICL setting. i.e. one example per class, and repeat five times under different random seeds. We use k 20 sampes per class as the support set to obtain all threshold values by grid searching, as mentioned in the method section. The prompt template and more implementation details are specified in Appendix A
4.2 Main Experiments
Table I presents the performance of various datasets and model sizes under the 1-shot setting. Our proposed UniBias method consistently achieves the highest accuracies most cases. In terms of overall average accuracy, UniBias improves upon the standard ICL by substantial margin of 3.39% and exceeds the state-of-the-ar (SOTA) DC by 1.76% using Llama-2 7b. With Llama-2 13b, UniBias surpasses the standard ICL and the SOTA CC by 2.97% and 2.45%, respectively. Figure 5 further illustrates the results under zero-shot and various few-shot settings for COPA, SST2, and MMLU. Additionally, to demonstrate the effectiveness of our method across different large language models, we present results for GPT-J and GPT2-XL in Figure 7 of Appendix C Our proposed UniBias consistently surpasses other baselines in all scenarios, underscoring its effectiveness.
In response to RQ 1, UniBias not only enhances the performance of original LLMs but also out- performs existing methods. We attribute this success to its internal analysis and bias mitigation techniques, which leverage FFNs and attentions, unlike other methods that rely solely on external observations.
8

---

Blased Components of AGNows
Common Blased Components of ABC and NKLU
Common Based AHs Across Different Tasks
fressnng vount
MNLU
Delaneta
101t0 6
mo 20000
INOT (000
5013
CA
Figure 6: Analysis of biased attention heads (AHs) and FFN vectors (FFNs). The frequency count of biased LLM components across five repeat experiments with different example selections is reported.
Table 2: Experiments on eliminating common biased components. Attention heads that are frequently identified as biased are removed from the original Llama-2 7b model.
SST2
MMLU
COPA
RTE
MR
Tree
Avg.
ICL
87 87.22
41.732.15
67.602.30
66.21
89.37
72.92
70.84
Unibias
94.54m.2
44.83024
69.002.74
67.656.44
92.19uu
80.80
74.84
Eliminating Common
94.32 0.60
44.20
68.00
67.374.60
92.430.09
77.604.75
73.98
Biased Components
4.3 Alleviating Prompt Brittleness
Existing studies have found that LLMs are prone to prompt brittleness, with various factors such as the selection and order of examples, as well as the prompt formatting. To address RQ 2, we simulate these brittle scenarios by choosing different demonstration samples, using different prompt formats, and changing the example order to observe variations in LLM performance.
Figure presents Llama-2 7b's performance both with and without UniBias. Without UniBias, the standard ICL's performance varies significantly, ranging from 8% 26%, demonstrating its instability. After applying UniBias, the accuracy remains consistently high and stable, with variations consistently less than 4% under perturbations of various design settings. We provide further theoretical analysis on why UniBias can mitigate prompt brittleness and address various bias factors in Appendix
4.4 Biased Components Analysis and Common Biased Components Elimination
In response to RQ3, we present the frequency counts of identified biased attention heads (AHs) and FFNs under repeated experiments in Figure A large frequency count for an LLM component indicates a higher repeat of being identified as biased in the corresponding dataset. The first subfigure displays the biased components for various example selections, revealing several commonly biased LLM components across different prompts within single dataset. The second subfigure highlights the common biased components across different datasets (ARC and MMLU) for the reasoning task, indicating that different datasets with similar tasks could share common biased L.M components. The third subfigure demonstrates the presence common biased L.M components across different tasks.
Experimental results suggest an interesting future direction: we may identify global biased compo- nents that would mitigate bias across multiple tasks and diverse prompt design settings. We conduct an preliminary experiment explore the potential of eliminating common biased components. Specif- ically, we eliminate attention heads that are frequently identified as biased and apply this setting to diverse tasks, rather than handling each task individually. Experimental results in Table Tableeddemonstraee that although not as effective as our full Unibias method, eliminating common biased components outperforms the vanilla ICI. by a large margin. Experiment details are Appendix D
4.5 Ablations
We conduct ablation studies to analyze the impact exclusively eliminating biased AHs or FFNs to address RQ 4. Table presents the results of removing only biased FFN vectors (FFN-only) and only

---

Table 3: Performance comparison of only removing biased FFN vectors (FFN-onlyy) only removing biased attention heads (attention-only), our Unibias method, and the ICL of original LLM.
Method
SST-2
MNLI
WiC
COPA
CR
AGNews
MR
RTB
SST-5
TREC
ARC
MML.U
ICL
87.22
53.83
50.00
67.60
91.54
85.59
89.37
66.21
46.97
72.92
51.90
41.73
FFN-only
94.17
54.59
92.57
85.52
91.78
67.33
47.09
73.04
51.92
42.62
Allention-only
94.22
52.83
52.76
68.50
91.49
86.25
92.61
66.55
52.68
80.68
53,00
44.67
UniBius
94.54
54,97
53.71
69.00
92.61
88.29
92.19
67.65
53.79
80.80
53,10
44.83
biased attention heads (attention-onlyy. Both FFN-only and attention-oni only methods outperform the standard ICI., demonstrating their effectiveness. When combined as UniBias, the method achieves the best results across most datasets, indicating that the two approaches are complementary.
Additionally, further conduct experiments to investigate the impact of support set size (RQ 5), which is detailed in Appendix
5 Related Work
Bias in LLMs: It is well recognized that LLMs are unstable under various ICL design settings, and this instability arises from biases in LLMs toward predicting certain answers [Zhao et al. 202T Lu et al.) [022]. To understand these biases, existing studies have identified various bias factors, including recency bias, majority label bias, common token bias (Zhao et al.) 2021]. and domain label bias [Fei et al. 2023] in classification tasks. More recently, selection bias, which consistently favors specific options in multiple-choice questions, has also been identified (Zheng cct al. 2023 Wang et al., 2023b]. To address these biases, several calibration methods have been proposed, including contextual calibration (Zhao et al. 2021]. domain-contes calibration (Fei et al.) 2023], and prototypical calibration (Han et al. 2023 However, these identified bias factors and calibration methods are derived from external observations adjustments of LLM outputs, leaving the underlying mechanisms within LLMs that cause such biases poorly understood.
Prompt Brittleness: Regarding prompt brittleness, it is demonstrated in the literature that this instability of prompt arises from LLMs' inherent bias towards predicting certain answers (Zha et al.
2021]. Therefore, current research efforts address the prompt brittleness by mitigating LLMs' bias towards labels [Fei et al. 2023] HHnn et al. 2023] /hao et al. 2021).
Mechanistic interpretability [Elhage et al. 2021 Wang al. 2022] aims to explain the internal processes in language models, facilitating the interpretation of the contributions of individual model components the final prediction. Our work builds on the
Mechanistic Interpretability:
understanding of the residual stream [Elhage et al. (202T). the logit lens [Nostalgebraist 2020). and the interpretation of LLM components in the vocabulary space [Dar al. 2023 Geva et 2021
Conclusion
In this work, we have deepened the understanding of biases in LLMs by unveiling the internal mechanisms that contribute to various bias factors. Building on this understanding, we proposed our UniBias method to mitigate these biases by identifying and eliminating biased FFN vectors and attention heads, demonstrating an effective way to manipulate the internal structures of LLMs. Extensive experiments show that our UniBias method achieves state-of-the the-art performance across 12 NLP datasets and different ICL settings. Additionally, our method successfully alleviates prompt brittleness and enhances the robustness of ICL.
Acknowledgments
The authors would like to thank Edmond Lo, Lihui Chen, Xiyu Zhang, and the anonymous reviewers for their constructive comments and suggestions. The research was conducted the Future Resilient Systems at the Singapore-ETHh Centre, which was established collaboratively between ETH Zurich and the National Research Foundation Singapore. This research is supported by the National Research Foundation Singapore (NRF) under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
10

---

References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint XrXiv: 1803:880300000 003.05557, 2018.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177-190. Springer, 2005.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers, Jordan Boyd-Grad Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16124-16170, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18663/11/20333acl
http:tttss://acaaneeee .og. 2023. acl acl-long. 893
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Srefield-Doddss Danny Hemandezz, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. hutp::/transformer.iicciiit puuprrrrrrrccuussuu/22222222 /frameewrk/indees html.
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating label biases for in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the s1 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 14014-14031, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/v 1tt33//11222..em acl-ooo.....--o URL
wrgg/023..coo. acl long. 783
Zijian Feng, Hanzhang Zhou, ZIXIAO ZHU, Junlang Qian, and Kezhi Mao. Unveiling and manip- ulating prompt influence in large language models. In The Twelfth International Conference on Learning Representations, 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 10.18653/1120221.cmm .ew.lp-enpppppee.ee main.446. URL https://aclantthll.o org 20211 emnlp- main. 446
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.1866/11120222emmm https://aclanthollogy www.oo2.//22....... emnlp-main.3
URL
Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. Prototypical calibration for few-shot learning of language models. In The Eleventh International Conference on Learning Representa- tions, 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater- than?? Interpreting mathematical abilities in a pre-trained language model. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neu- ral Information Processing Systems, volume 36, pages 76033-76060. Curran Associates, Inc., 2023. URL https: htattss://proceedinngg wwwpeeewddnnnnn...is...e. neurips. www.iis..crpppprppeeee wwwpreeppppppppeeeeeeeeeeeee..ee # Perprieeeepppppee-eoooeeeeeeeeee filees/paper/222//Eiee
11

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference Knowledge discovery and data mining, pages 168-177, 2004.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 10.18663/1120222accHooor
/ttpt:tts://aclntttooo wwwpoolloooogggg//goooooo wtgg2202.acl-1ongg acl- 556
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1 11048-1 1064, Abu Dhabi, United Arab Emirates, December 2022, Association for Computational Linguistics. doi: 10. 1865vv/3v 2022.emnlp-main.775 7599. URL htttps://acllntthoo
org/ emnlp-main 759
Nostalgebraist. Interpreting gpt: the logit lens, 2020. URI WtRl:http:///ww/ wwwpsswrong.coo/ooooeo dei.eawwwwaaaaeeeeee///eerrreree the---ttttttt------------ logit-
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Kevin Knight, Hwee Tou Ng, and Kemal Oflazer, editors, Proceedings of the 43rd Anmual Meeting of the Association for Computational Linguistics (ACL '05), pages 115- 124, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. doi: 10.3115/1219840. 1219855. URL /tps
org/P05- 1015
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computaatinnal Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1267-1273, Minncapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 18.13/53/v 1/N19/1119911111 128. /tttt:t////aaaantee
org /N19 1128
Alec Radford, Jeffrey Wu, Rewon Child, David I Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011 AAAI Spring Symposium Series, 201 1.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL http::////alaantooog..oooo
1170
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.092288 2023.
Ellen M Voorhees and Dawn M Tice. Building question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development information retrieval, pages 200-207, 2000.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: billion parameter autoregressive language model, 2021.
12

---

Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference Learning Representations, 2022.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors. Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 9840-9855, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.18655/w1/2203..mmmil hwtp::/...22....2212m.m www.emmmii.mmmi... URL attps://aclanthann
org.oog//// 2023. 2tt.2222....mmm22mm emnlp-main 609
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305 2eei2220557.29 17926, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24882 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1 1101. URL
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9924-9959, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v hatp:23..mmllpmmaa 615. URL htttp:://aclanthlogy)
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Cortes, Lawrence, D. Lee, M. Sugiyama, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: htapoe///proceedingg newriiiinnnnnnuuussseeeeeereiiee neurips. cc/ #50 Paper pdf
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings Machine Learning Research, pages 12697-12706. PMLR, 18-24 Jul 2021. htt.httts://prrceediii mlr. wtwps//139// zhwo2ic. html
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023.
Hanzhang Zhou, Junlang Qian, Zijian Feng, Lu Hui, Zixiao Zhu, and Kezhi Mao. Lims learn task heuristics from demonstrations: A heuristic-driven prompting strategy for document-level event argument extraction. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11972-11990 2024.
13

---

A Experimental Details
A.1 Datasets
We evaluate our Unibias method using 12 diverse natural language processing datasets across tasks such as sentiment analysis, topic classification, reasoning, natural language inference, and word disambiguation, as presented in Table 4 In our experiments, we utilize k (where l = 0, 1, 2, 4) training samples per class as prompt examples for k-shot ICL. For testing, we randomly select 2000 samples for MMLU and 3000 samples for MNLI and MR, while employing the original testing sets for other datasets. Detailed dataset statisties are available in Table
Table 4: Detailed Dataset information
Dataset
# Classes
Testing Size
Sentiment classification
SST2 Socher et al.
2013
2
872
SST-5
SScher
2013
5
2210
MR
Pang and Lec
2005
2
3000
CR
Hu and Liu| [2004]
2
376
Topic classification
AGNews (Zhang et al. 2015
7600
TREC Voorhees and Tice 2000
500 6
Natural language inference
MNLI [Williams al. 2018
3
3000
RTE (Dagan et al. 2005
277 2
Reasoning
ARC-Challenge Clark et al.
2018
4
1170
MMLU Hendryykk al. 2020
2000
COPA Roemmele et al.
201T
2
100
Word disambiguation
WiC [Pilehvar and Camaach-Coolladee
2019
2
638
A.2 Implementation Details
Experiments on internal mechanisms of biased factors: All experiments are conducted on Llama- 7b model. For the vanilla label bias experiment, we projecting all FFN value vectors into the vocabulary space and sum the label logits for all FFN vectors whose label logits rank within the top 10 over the vocabulary to calculate uncontextual accumulated FFN logits. We change different set of label words in prompt derive the label prediction frequency of different label pairs. For the recency bias experiment, based on findings in [Wang et al. 2023a]. instead of the summed attention weights over the whole example, we adopt the of attention weights on label words of the example, e.g. "Answer: positive" as the effective attention weight on each example. For the selection bias experiment,
zeroshot ARC dataset prompts in Table & and use 12 samples for each class. The attention weight is also summed on label words instead of the whole option.
Baselines: We reproduce all baselines using the publicly available code released by the authors to ensure fair comparison. For the PC method, instead of using test samples as the original work, we employ 200 training samples per class as the estimate set for parameter estimation using the EM algorithm. This adjustment is made to reflect real-world scenarios where test samples are not readily available. Additionally, the number of samples used by the PC method is significantly larger than that used by our UniBias method.
Unibias:
In our method, all threshold values are determined through grid searching as described in the methodology section. Specifically, we use 20 samples per class as the support set for grid searching in all experiments. For each repetition of the experiment, the support set is randomly selected based on different random seeds. Additionally, to manipulate biased FFN vectors and attention heads, we create masks for the attention heads of all attention layers and adjust the FFN coefficient values and attention head masks using the hook operation. Additionally, we conduct the experiment on four A5000 GPUs.
14

---

GPT-J
GPT2-XL
100
100
Method
Accuracy (%)
80
80
70
70
50
50
SST-2
WIC
COPA
MR
RTE
Ave.
SST-2
WIC
COPA
MR
RTE
Ave.
Dataset
Dataset
Figure 7: Performance comparison of our UniBias method against baseline methods using GPT-J and GPT2-XL models.
B Limitation and Future Work
In this work, we provide a novel insight into the internal mechanisms behind the bias of LLMs. As pioneering effort in mitigating LLM bias through manipulation of the model's internal structures, our approach relies on grid searching with a small set of labeled training samples. Future research could focus reducing this reliance, potentially improving the efficiency and applicability of our method.
There are many interesting avenues for future research. For instance, instead of identifying biased components for each ICI. prompt, future work could explore the identification of global biased components that mitigate bias across multiple tasks and diverse prompt design settings. Additionally, the biased FFN vectors and attention heads we identify could potentially serve as sensors for guiding effective prompt generation. We expect that this internal perspective on LLM bias will inspire more innovative applications in both bias mitigation methods and prompt engincering.
C Evaluation on More LLMs
Figure 7 demonstrates the performance of various methods across multiple datasets when applied to GPT-J and GPT2-XI. models. For both models, our UniBias method consistently outperforms the baseline methods including vanilla ICI,, CC, DC and PC. Notably, the improvement the GPT2-XI. model is substantial, demonstrating over an over 20% increase accuracy on SST-2 dataset compared to vanilla ICL.
D Eliminating Common Biased Components
Table 5: List of common biased attention heads climinated. Indexing Starts from 0.
(19, 10) (19, 14)
(16, 29) 219,2) (25, 21) (16, 11) (18,31) (18, 1)
We explore the potential of eliminating common biased components and apply it to diverse tasks, rather than addressing each task individually. We conduct additional experiments on multiple tasks to assess the effectiveness of directly elinimate these components. Experimental results in Table indicate that although not as effective as our full Unibias method, it outperforms the vanilla ICL by large margin. Notably, eliminating common biased components represents cost-free gain in performance, involves only the direct masking of biased components identified our work and applicable to diverse tasks. The attention heads that masked are listed in Table
15

---

SST-2
SST-2
96
UniDies
UniBies with unlebeled data
94
Accuracy (50
94
Unibios
Accuracy
92
92
90
90
88
88
86
10
40
86
20
40
60
80
100
COPA
COPA
70
UniBles
UniBias unlebeled data Org Unibias
69
Accuracy (%)
69
Accuracy
68
68
ICL
67
67
66
66
65
10
40
65
Support set size (per class)
80
Unlabeled samples (x class number)
Figure 8: Performance of Unibias under different
Figure 9: Performance of Unibias using unla- beled samples support set. It is compared against standard ICL and the original Unibias.
support set.
E Impact of Support Set Size
Our proposed UniBias method employs a small support set for grid searching. To analyze its effect, we vary the size of the support set. Figure 7 illustrates Unibias's performance with support set sizes ranging from 5 to samples. The results indicate that the performance stabilizes when the support set contains 20 or more samples per class. Notably, for the SST2 dataset, even with much fewer support samples, Unibias significantly outperforms the standard ICI.
F
Using Unlabeled Samples for Support Set
To Address the potential challenge in accessing labeled samples, we further explore the alternative of using unlabeled samples during grid search. In our method, labeled samples are used to ensure each class is represented proportionally in the grid scarch, without direct of the specific label information. Therefore, for balanced datasets, it equally effective to employ a slight larger pool of unlabeled samples.
Our experimental findings, illustrated in Figure of the rebuttal PDF, indicate that approximately 40 x #Classes unlabeled samples achieves performance comparable to that obtained with labeled samples.
G Additional Analysis
We further analyze why mitigating model's bias towards labels can alleviate prompt brittleness in our method. Due to the inherent bias of LLMs, different prompts can lead to varying biases towards labels. For example, due to recency bias, placing negative sentiment analysis sample at the end of a prompt can make LLMs tend to predict negative", incorrectly classifying positive samples and thus degrading ICL performance. Various bias factors lead to different direction and extend of bias, resulting in different changes in ICL performance and leading to the prompt brittleness. our UniBias method effectively mitigates various potential biases inherent in LLMs by addressing their root causes internally from LLMs. By doing so, it minimizes the introduction of bias towards labels regardless of the difference in prompts, leading more stable and accurate ICL performance across different prompt configurations.
contrast,
Additionally, our UniBias method seeks to address broad range of factors that lead to LLM bias, extending beyond those discussed in Section 2 Given the significant variability prompts, models, and data corpuses, numerous unanticipated bias factors may emerge. Our approach is designed to
16

---

tackle these diverse bias factors comprehensively. This is feasible because biased behaviors observed externally in LLMs originate from their internal network (FFN) vectors and attention heads, which house nearly all L.LM parameters. By directly identifying and mitigating biases within these FFN vectors and attention heads, UniBias offers foundational strategy to counteract various forms of bias.
specifically, the feedforward neural
H
Prompt Templates
The prompt templates used in this work are provided below, We generate few-shot ICL templates
follow the template styles in (Han et al. 2023 Fei et al. 2023 as illustrated in Table
Table 6: Prompt templates for all k-shot ICL experiments.
Dataset
Template
Label Space
SST-2
Review: {sentence)
negative / positive
CR
Sentiment: (label)
MR
MNLI
Premise: {premise)
yes maybe//
Hypothesis: [hypothesis)
Answer: (label)
ARC
Question: {question]
A/B/C/D
MMLU
(options)
Answer: (label)
SST-5
Review: {sentence)
terrible bad / okay good great
Sentiment: (label)
AGNews
Article: {passage]
world sports / business technology & science
Answer: (label)
TREC
Question: {sentence]
abbreviation entity / description person location / number
Answer Type: (label)
CОРА
Premise: (premise]
1 1/2
Choicel: (choicel)
Choice2: (choice2)
Answer: (label)
RTE
Premise: {sentencel)
yes no
Hypothesis: [sentence2) Answer: (label)
WIC
Sentence1: (sentencel) Sentence2: (sentence2) Word: (word)
false / true
Answer: (label)
17

---

Table 7: Templates of different prompt formatting used the prompt brittleness experiment for SST-2.
ID Template
Label Space
(Sentence) Sentiment: (Label)
Review:
Positive Negative
Input: (Sentence) Prediction: |Label]
Positive Negative
3 Review: (Sentence)
good / bad
Sentiment: [Label]
[Sentence] It was [Label]
good bad
5 Review: (Sentence)
Yes / No
Positive Review: (Label)
(Sentence) My overall feeling was that the movie was (Label)
good bad
7 Review: (Sentence)
Positive Negative
Question: Is the sentiment of the above review Positive or Negative?
Answer: (Label}
8 My review for last night's film: (Sentence) The crities agreed that this movie was (Label)
good bad
Table 8: Prompt templates for the 0-shot experiments.
Dataset
Template
Label Set
SST-2
(sentence) Sentiment: (label)
Review:
negative / positive
CОРА
Premise: [premise) ]
1/2
Choicel: (choicel
Choice2: (choice2) }
Answer: (label)
MMLU
Question: (question)
A/B/C/D
(options)
Answer: (label)
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe the main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We will release our code upon acceptance to facilitate easy reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwtc//ggui////eeeeceoocc CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: p-value less than 0.01 is derived on our main experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp:: http::::////iis...ee cwwtaripscccppppppe cttppubicc//thhccccciidee..e.
Answer:
[Yes]
Justification:
We believe our research conform NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
They are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer:[NA
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ttIIW0yB1E", "reviewer": "Reviewer_BhZE", "summary": "The paper explores the influence of FFNs and attention heads in LLMs on biases, resulting in model predictions that exhibit favoritism towards specific labels. The authors propose UniBias, an inference-only technique designed to detect and mitigate biased components within LLMs by analyzing and manipulating FFN vectors and attention heads.  Experiment results on 12 diverse NLP datasets show that UniBias significantly improves the in-context learning performance of LLMs and reduces their sensitivity to prompt design.", "questions": "- Is this analytical method compatible with LLMs that have undergone SFT? If so, how would the fine-tuning process potentially influence the identification and mitigation of biases within the model's internal mechanisms?\n- Can the grid search process be automated or optimized to minimize the need for manual parameter tuning?", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces UniBias, a novel method designed to mitigate bias in LLMs by manipulating internal model components, specifically FFN vectors and attention heads. This approach represents a unique contribution to bias reduction within LLMs.\n- UniBias demonstrates significant improvements over standard ICL and state-of-the-art calibration methods, indicating its potential to become a leading technique in bias mitigation for LLMs.\n- The authors conduct an in-depth exploration of the internal mechanisms underlying biases in LLMs, providing insights that delve beyond superficial observations to uncover the root causes of these biases.", "weaknesses": "- While the method shows promising results, it is crucial to assess how well these findings generalize across different types of LLMs and datasets (e.g., machine translation tasks) beyond those tested. This paper only conducts experiments on Llama and classification tasks.\n- The reliance on grid search with a limited number of labeled training samples for identifying biased components may limit scalability and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BhZE,\n\nWe deeply appreciate your thorough review and valuable feedback. Below is a summary of our answers (**A**) to the weaknesses (**W**) and questions (**Q**) you raised in the review.\n\n---\n**[W1]**: Assess how well the findings generalize across different types of LLMs and datasets.\n\n**[A1]**: We greatly value your insightful suggestion and we have conducted additional experiments to validate our method.\n\n**Model Generalization**: Following your suggestion, we have conducted additional experiments to evaluate more LLMs, including *GPT-J (6B)* and *GPT2-XL (1.5B)*. The experimental results are detailed in *Table 1* of the rebuttal PDF. Together with the *Llama2-7B* and *Llama2-13B* models evaluated in our paper, we rigorously evaluate our method across four popular LLMs of different sizes and demonstrated the effective of our method across these models. \n\n**Task Coverage**: Our work is currently focus on classification tasks, consistent with the prevailing LLM bias mitigation literature. Our work evaluates extensive array of tasks compared to the exsiting studies. To provide context, current works typically focus either on mitigating bias in general classification tasks [1-2] or exclusively address bias in multiple-choice questions (MCQs) in reasoning tasks [3-4]. In contrast, our study covers 12 datasets spanning 5 distinct tasks, including both general classification tasks and MCQs in reasoning tasks. This breadth not only aligns with but surpasses the scope of tasks examined in the literature. Moving forward, we are enthusiastic about adapting our method to a wider variety of tasks as you suggested.\n\nWe appreciate your insights, which significantly strengthened our experimental validation.\n\n---\n**[W2]**: Grid search with a limited number of labeled samples may limit scalability and efficiency\n\n**[A2]**: Thank you for raising this concern, which prompted us to further enhance our method. Although using a small set of labeled samples is typically manageable, we recognize the potential for improvement. In response, we conduct experiments to address your concern from three aspects.\n\n**Optimization of Grid Search**: To streamline the grid search process, we now provide an alternative to perform grid search on a single dataset to obtain a set of threshold values, which are then applied universally across other datasets. This alternative eliminates the need for repeated grid searches across each dataset, significantly enhancing efficiency and scalability of the method. \n\nOur experimental results in *Table 2* of the rebuttal PDF confirm that using a fixed set of thresholds maintains good performance, slightly below the original UniBias method but still superior to vanilla ICL and other baseline methods. This experiment also illustrate the robustness of our method in terms of threshold selection.\n\n**Replacing Labeled Samples with Unlabeled Ones**: To Address the potential challenge in accessing labeled samples, we further explore the alternative of using unlabeled samples during grid search. In our method, labeled samples are used to ensure each class is represented proportionally in the grid search, without direct use of the specific label information. Therefore, for balanced datasets, it is equally effective to employ a larger pool of unlabeled samples. \n\nOur experimental findings, illustrated in *Figure 1* of the rebuttal PDF, indicate that approximately $40 \\times \\text{number of classes}$ unlabeled samples can achieve performance comparable to that obtained with labeled samples.\n\n**Exploration of Eliminating Common Biased Components**: In response to your feedback, we have delved deeper into the potential of eliminating common biased components and apply it to all tasks, rather than handling each task individually. In Section 4.4 of our paper, we discuss the presence of shared biased LLM components across different tasks. Inspired by this finding, we conduct additional experiments to directly elinimate these common biased components and evaluate on different tasks. \n\nExperimental results in *Table 3* of the rebuttal PDF indicate that although not as effective as our full Unibias method, it outperforms the vanilla ICL by a large margin. Notably, this is a free gain in performance as it involves merely applying a direct masking to the biased components identified in our work and is applicable to diverse tasks.\n\nThe enhancements suggested by your feedback have markedly improved the efficiency and scalability of our approach. Furthermore, the third experiment indicates the potential of our work in stimulating future bias mitigation methods in a novel perspective of LLM inner structure manipulation.\n\n---\n**[Q1]**: Compatibility with SFT\n\n**[A3]**: Thank you for the interesting question. We would like to respond in two aspects: the performance of our method on fully fine-tuned models and the impact of SFT on biased components.\n\n**Compatibility with SFT**: UniBias is fully compatible with LLMs that have undergone SFT. We applied UniBias to Llama2-7b fine-tuned on the SST2 and MNLI datasets, with following experimental results:\n\n||SST2|MNLI \n|--|--|--|\n|ICL after SFT|94.19|64.78\n|Unibias|**94.72**|**65.88**\n\n**Impact of SFT on Biased LLM Components**: Algorithmically, the processes for identifying and mitigating biases in both SFT and non-SFT models are identical. However, we observed a reduction in the coefficients of biased FFN vectors and the magnitudes of label logits of biased attention heads post-SFT in some cases, suggesting that the SFT process may suppress the biased components in LLMs.\n\n---\n**[Q2]**: Can grid search process be automated?\n\n**[A4]**: Yes, the grid search process in our method is automated. We will release our code to facilitate reproduction and further exploration.\n\n---\nWe are encouraged to find that these experiments and enhancements, prompted by your valuable insights, have substantially strengthened our paper. We look forward to hearing your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzN82IfR6r", "reviewer": "Reviewer_LH5i", "summary": "This paper seek to address prediction bias in ICL through intervening feedforward vectors and attention heads.", "questions": "See weaknesses.", "limitations": "This paper does not have a limitation section.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "* This paper studies a critical problem of in ICL.\n* The proposed method is well-motivated.", "weaknesses": "The major problem with the proposed method is that it requires approximately 20 labeled instances per class. This leads to the following concerns:\n* The comparison between some baselines and the proposed method is unfair. The proposed method requires labeled instances for searching thresholds, while baselines like DC require no labeled instances.\n* The generalizability of the proposed method is also questionable. Can we search thresholds once and apply them to all datasets? If not, it is impractical for real-world application. If we have that many labeled instances and access to the model parameters for the target task and dataset, we can always use them to finetune the model to achieve better performance.\n* Moreover, the cost will be intractable when the number of classes increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer LH5i,\n\nWe deeply appreciate your thorough review and valuable feedback. We appreciate your acknowledgment of the significance of the problem we investigated and motivation behind our approach. Below is a summary of our answers (**A**) to the weaknesses (**W**) you raised.\n\n---\n**[W1]**: Grid search with 20 labeled samples per class can lead to concerns in fairness in comparison with baselines, generalizability of the method, and increased cost when the number of classes is large.\n\n**[A1]**: Thank you for raising this important concern! It prompted us to further enhance our method. \n\nInitially, we choose a limited number of labeled samples for grid search because it is typically manageable given the small size of labeled data. Additionally, it is only needed for the initial set up to identify biased LLM components. During inference stage, the computation cost of UniBias is completely identical to that of the vanilla LLMs. We believe this is a reasonable trade-off for the substantial improvements in ICL performance and the novel perspective Unibias offers on mitigating bias through LLM inner components.\n \nHowever, we understand the concern that using labeled samples can be challenging when labeled data are scarce or the number of classes is large. Therefore, we have optimized our method in light of your valuable feedback:\n\n**Replacing Labeled Samples with Unlabeled Ones**: To Address the potential challenge in accessing labeled samples and ensure a fair comparison with baselines, we further explore the alternative of using unlabeled samples during grid search. In our method, labeled samples are used to ensure each class is represented proportionally in the grid search, without direct use of the specific label information. Therefore, for balanced datasets, it is equally effective to employ a slight larger pool of unlabeled samples.\n\nOur experimental findings, illustrated in *Figure 1* of the rebuttal PDF, indicate that approximately $40 \\times \\text{number of classes}$ unlabeled samples achieves performance comparable to that obtained with labeled samples. This number is significantly lower than the quantity required by the PC baseline, which requires at least  $200 \\times \\text{number of classes}$ unlabeled samples.\n\n**Optimization of Grid Search**: Inspired by your feedback, we streamline the grid search process. We now provide an alternative to perform grid search on a single dataset to obtain a set of threshold values, which are then applied universally across other datasets. This alternative eliminates the need for repeated grid searches across each dataset, significantly enhancing efficiency and scalability of the method. By directly adopting the fixed threshold, it also addesses the scalability issues associated with an increasing number of classes.\n\nOur experimental results in *Table 2* of the rebuttal PDF confirm that using a fixed set of thresholds maintains good performance, slightly below the original UniBias method but still superior to vanilla ICL and other baseline methods. This also illustrate the robustness of our method in terms of threshold selection.\n\n**Exploration of Eliminating Common Biased Components**: In response to your feedback, we have delved deeper into the potential of eliminating common biased components and apply it to all tasks, rather than handling each task individually. In Section 4.4 of our paper, we discuss the presence of shared biased LLM components across different tasks. Inspired by this finding, we conduct additional experiments to directly elinimate these common biased components and evaluate on different tasks. \n\nExperimental results in *Table 4* of the rebuttal PDF indicate that although not as effective as our full Unibias method, it outperforms the vanilla ICL by a large margin. Notably, this is a free gain in performance as it involves merely applying a direct masking to the biased components identified in our work and is applicable to diverse tasks.\n\nFinally, following your suggestion, we evaluate the performance of fine-tuning LLMs using 20 labeled samples per class and find that their performance is significantly lower than that of UniBias. This results further validates the effectiveness of our method.\n\n| |SST2 |MNLI|Trec\n|--|--|--|--|\n| ICL | 87.22 | 53.83 |72.92|\n| ICL after SFT |88.64 | 53.95  |73.2|\n| UniBias | **94.54** |**54.97**|**80.80**|\n\nThe enhancements suggested by your feedback have markedly improved the efficiency and scalability of our approach. Furthermore, the third experiment underscores the potential of our work in stimulating diverse bias mitigation methods in a novel perspective of LLM inner structure manipulation.\n\n---\nDuring rebuttal, we also **extend the evaluation of UniBias** to include *GPT-J* and *GPT2-XL*, which are detailed in *Table 1* of the new PDF.\n\n---\nWe would like to further highlight the contributions of our work.\n* **A New Insight for LLM Bias Mitigation**: Unlike existing works based on *external* adjustments of LLM outputs, we mitigate LLM bias through manipulation of LLM *internal* structure. This novel perspective could potentially stimulate future research on LLM bias mitigation from inner structure manipulation, offering a new direction for the field.\n* **Unveil Internal Mechanisms of LLM Bias**: We conduct a thorough exploration of the internal mechanisms underlying biases in LLMs, providing deep insights that go beyond superficial observations, revealing the inner causes of these biases.\n* **Extensive Evaluation**: UniBias is evaluate across 12 datasets and 4 LLMs, consistently demonstrating its effectiveness. \n* **Addressing Prompt Brittleness**: Our experimental results show that our method effectively mitigates prompt brittleness, a critical issue in ICL.\n\n---\nWe are encouraged to find that these experiments and enhancements, prompted by your valuable insights, have substantially strengthened our paper. We look forward to hearing your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YOWoT39Wru", "reviewer": "Reviewer_8E5c", "summary": "This paper aims to address prompt brittleness in in-context learning methods introduced by seemingly inconsequential changes such as vanilla label bias, recency bias, and selection bias. They use the logit lens technique, where  the linear function is applied from the final layer of a decoder-only transformer (AKA the unembedding matrix) on the intermediate layers to interpret the attention activations and FFN embeddings. They detect biased activations and FFN vectors by their relatedness, residual bias, and low variance. Applying their method results in unchanged or improved performance compared to the baseline.", "questions": "- Do you have any insights on why UniBias improves performance on certain tasks but not others? \n\n- For vanilla label bias, how do you choose the alternative label names? \n\n- It is interesting that the unembedding matrix can also be applied to the attention hidden state. Did you notice any discrepancies in the logit distribution between the FFN and the attention head of the same layer? \n\n- The sections and overall flow of Sections 2 and 3 may be organized more clearly   \n\n  - Sections 3.3 and 3.4 seem very small (3 and 11 lines respectively) and could be combined with other sections.  \n\n  - It is unclear how the criterion they introduce (relatedness, bias, low variance) in lines 195-201 relate to the three bias conditions introduced in line 218  \n\n  - In general, Sections 2 and 3 seem related to each other but the writing does not tie them together clearly (eg. lines 202-205 seem to be describing the importance of low variance to identifying the biases in section 2.2, but it is not stated very clearly)  \n\n- There are further minor writing issues that could be revised, including various minor typos.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel application of mechanistic interpretability to mitigate biases in in-context learning and demonstrates that specific attention heads and FFNs in the decoder can be isolated as responsible for prompt brittleness. \n\n- The methodology and findings are insightful and bring consistent performance improvements. \n\n- The experiments are extensive, covering various datasets and settings, providing strong evidence for the effectiveness of the proposed methods to support the conclusions. \n\n- The paper is in general well-written, with clear explanations of the methodology and results.", "weaknesses": "- The paper mentions some tasks could share biased components (line 318-321) – based on their experiments, I think it would have been possible to identify these components in this work, and I would have liked to see this result.  \n\n- The analysis of their results could be expanded further. In table 2 the performance of FFN-only, attention-only, and their full UniBias method is inconsistent – sometimes FFN-only performs the best, sometimes attention-only, with no pattern on dataset task.  \n\n- It would be interesting to see how bias exists across different layers of the model. Analyzing the distribution and impact of biased components in each layer may provide deeper insights into how biases are propagated and amplified throughout the model.  \n\n- Some of the mathematical formulations, particularly those related to the identification of biased components, could be more detailed to enhance reproducibility and understanding. The authors could provide more thorough mathematical derivations and examples illustrating how the bias criteria are identified. This would make the methodology clearer and easier to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8E5c,\n\nWe deeply appreciate your thorough review and valuable feedback. Below is a summary of our answers (**A**) to the weaknesses (**W**) and questions (**Q**) you raised.\n\n---\n**[W1]**: Show shared biased components across tasks.\n\n**[A1]**: We greatly value your suggestion! Inspired by your insights, we not only list the common biased components, but also explore eliminating common biased components for bias mitigation.\n\n**Listing of Common Biased Components**:  The shared biased attention heads with their frequency of occurrence (>=3) are detailed in *Table 3* of the rebuttal PDF. The shared biased FFN vectors are not discussed because they are associated with label words, which typically vary across tasks.\n\n**Eliminating Common Biased Components**: Inspired by your feedback, we explore the potential of eliminating common biased components and apply it to all tasks, rather than handling each task individually. We conduct additional experiments on multiple tasks to assess the effectiveness of directly elinimate these components. Experimental results in *Table 4* of the new PDF indicate that although not as effective as our full Unibias method, it outperforms the vanilla ICL by a large margin. Notably, this is a free gain in performance as it involves merely the direct masking of the biased components identified in our work and is applicable to diverse tasks.\n\n---\n**[W2]**: Result analysis could be expanded for Table 2.\n\n**[A2]**: We deeply appreciate your suggestion and provide a detailed analysis here.\n\nAs an ablation experiment, we find that both FFN-only and attention-only methods outperform the standard ICL, demonstrating their effectiveness. However, as you suggested, there is no consistent pattern on when these two ablation methods perform better. The removal of biased FFN vectors predominantly addresses uncontextual biases associated with label words, while the removal of biased attention heads targets contextual biases, such as recency bias. Therefore, the relative effectiveness of these methods depends on the dominance of either type of bias in the given task and the specifics of the prompt examples, which can vary unpredictably. \n\nDespite these variations, it is notable that our comprehensive UniBias method consistently yields the best performance, underscoring the effectiveness and robustness of UniBias.\n\n---\n**[W3]**: It would be interesting to see how bias exists across different layers of the model.\n\n**[A3]**: Thanks for your insightful suggestion! We visualize the identified biased attention heads in *Figure 2* of the new PDF and provide analysis below. Biased FFN vectors are not visualized due to the high dimensionality of FFN layers.\n\nIn *Figure 2*, the intensity of color corresponds to the frequency with which an attention head is identified as biased across 12 datasets, each repeated 5 times. The visualization reveals that biased attention heads mostly occur in the middle layers. This observation aligns with the established understanding of LLM architecture: early layers tend to encode shallow and discrete concepts  such as lexical features, while higher layers are responsible for high-level semantics and complex reasoning. Therefore, discrete concepts at early layers are less likely to directly contribute to the bias. In contrast, middle layers, where simple reasoning begins to form are more likely to introduce biases (e.g., \"A sentence is always positive rgardless of its content\"). The higher layers are less likely to produce this kind of bias associated with simpler, shallow reasoning.\n\n---\n**[W4]**: Some mathematical formulations can be more detailed.\n\n**[A4]**: We appreciate the valuable suggestion! The formulations for identifying biased components involve measuring the sum of label logits, the bias of label logits, and the coefficient variance of FFN vector coefficients or attention head label logits. They are corresponding to the relatedness, biased, and low variance criterions. Based on your feedback, we will include more comprehensive mathematical derivations and examples in the revised paper to clearly illustrate how these bias criteria are identified and applied.\n\n---\n*We apologize for the brevity of the following responses due to space limitations.*\n\n---\n\n**[Q1]**: Unibias performance\n\n**[A5]**: Thank you for this important question. To address it, it's essential to consider that a more appropriate comparison for UniBias would be between standard ICL and UniBias, as this comparison contrasts ICL based on vanilla LLMs with LLMs after the elimination of biased components. In this case, our UniBias consistently improve performance across all 12 datasets we evaluated. On the other hand, calibration methods are under different methodology and perform better on a limited number of datasets. This may be attributed to the specific characteristics of each dataset.\n\n---\n**[Q2]**: Alternative label names for vanilla label bias\n\n**[A6]**: They are generated by GPT-4.\n\n---\n**[Q3]**: Discrepancies in logit distribution between FFN and attention head\n\n**[A7]**: Thank you for the insightful question. Generally, they are consistent across the middle and higher layers. However, we observe  discrepancies in some individual components.\n\n---\n**[Q4]**: Organization of Sections 2 and 3.\n\n**[A8]**: Thank you for the detailed suggestions! We will revise Sections 2 and 3 carefully. This includes merging Sections 3.3 and 3.4, enhancing coherence between the criteria and conditions, and strengthening connections between 3.3 and 3.4. We will also ensure a thorough review for typos and writing issues as suggested in **Q5**.\n\n---\nWe wish to highlight that we also **extend the evaluation of UniBias** to include *GPT-J* and *GPT2-XL*, which are detailed in *Table 1* of the new PDF.\n\n---\nWe are encouraged to find that these experiments and enhancements, prompted by your valuable insights, have substantially strengthened our paper. We look forward to hearing your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kGib0ItXVD", "reviewer": "Reviewer_CnNd", "summary": "This paper addresses internal biases in LLMs that cause prompt brittleness. Unlike previous external adjustment methods, it examines the roles of MLPs and attention heads in creating these biases. The proposed UniBias method identifies and masks these biased components, thereby enhancing the model's in-context learning ability. Experiments across several NLP tasks demonstrate that UniBias effectively improves ICL performance.", "questions": "NA", "limitations": "Limitations: Yes; Broader impacts: NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This paper explores the internal contributions of FFN vectors and attention heads to LLM bias, a relatively unexplored area.\n* The proposed method is straightforward and does not introduce any additional inference time cost, unlike post-calibration methods.", "weaknesses": "The main weakness of this paper is that there seems to be some disconnect between the problem it aims to solve (the prompt sensitivity of large language models, or LLMs) and two of the objectives of the preliminary analysis (how internal components generate common token bias and selection bias towards labels).\n\nSpecifically, the challenge the paper proposes to address is the sensitivity of LLMs to input prompts (example selection, format, and order), and this issue is also illustrated in Figure 1. However, two of the targets of the exploration section are the biases present when LLMs use labels from inputs.\n\nThe authors do not explicitly state why suppressing the model's bias toward labels can simultaneously alleviate the model's bias toward prompts. For example, intuitively, suppressing the label's preference for option A does not necessarily suggest an improvement in the model's bias towards a certain format or example.\n\nGiven this lack of clarity, and since the criteria for detecting biased components also focus on labels, it is unclear why the proposed method can reduce prompt brittleness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CnNd,\n\nWe deeply appreciate your thorough review and valuable feedback. Below is a summary of our answers (**A**) to the weaknesses (**W**) you raised.\n\n---\n**[W1]**:  There seems to be some disconnect between the problem it aims to solve and two of the objectives of the preliminary analysis.\n\n**[A1]**: Thank you for raising this important issue! We apologize for the lack of clarity in linking these aspects in our paper.\n\nThe connection between prompt sensitivity of LLMs and their bias towards labels is well-established in the LLM bias mitigation literature and we mentioned this in Line 21 of our paper. However, thanks to your insightful feedback, we realize the need to more thoroughly strengthen and elucidate this connection to enhance understanding of our method's effectiveness. A detailed analysis is provided below:\n\n**The Connection Established by Literature**: Firstly, the literature demonstrates that the prompt sensitivity arises from LLMs' inherent bias towards predicting certain answers. Therefore, the prompt brittleness can be addressed by mitigating LLMs' bias towards labels. For example, in the classic study of \"Calibrate Before Use\" [1] it explicitly states that:\n> We demonstrate that this instability of prompt arises from the bias of language models towards predicting certain answers.\n\nThey detailed this analysis in Section 4 of their paper. Building on this foundation, following studies address the bias of LLMs toward to address prompt brittleness. For example, a domain-label bias is identified and mitigated to alleviate prompt brittleness in [2].\n\n**Additional Analysis on Our Method**: We further analyze why mitigating model's bias towards labels can alleviate prompt brittleness in our method. Due to the inherent bias of LLMs, different prompts can lead to varying biases towards labels. For example, due to recency bias, placing a negative sentiment analysis sample at the end of a prompt can make LLMs tend to predict 'negative', incorrectly classifying positive samples and thus degrading ICL performance. Various bias factors lead to different direction and extend of bias, resulting in different changes in ICL performance and leading to the prompt brittleness. In contrast, our UniBias method effectively mitigates various potential biases inherent in LLMs. By doing so, it minimizes the introduction of bias towards labels regardless of the difference in prompts, leading to more stable and accurate ICL performance across different prompt configurations. \n\n**Experimental Validations**: We further provide experimental results to demonstrate how mitigating bias towards labels can alleviate prompt brittleness in ICL. We analyze the average prediction probabilities of 'negative' and 'positive' across all test samples in various prompt formats on the SST2 dataset. The ICL performance corresponding to these formats is depicted in Figure 1 of our paper. As shown in the table below, the average prediction probabilities for each format are represented as $(P_{\\text{ave}}(\\text{neg}), P_{\\text{ave}}(\\text{pos}))$. Our results reveal that different prompt formats introduce varying biases, leading to different levels of label bias which, in turn, can destabilize ICL performance. Conversely, our UniBias method produces balanced probabilities across labels and significantly mitigates label bias, resulting in more stable and accurate ICL performance and reducing prompt brittleness.\n\n|Method|Format 1|Format 2| Format 3|Format 4|Format 5|\n|--|--|--|--|--|--|\n| Vanilla ICL| (0.34, 0.55) | (0.35, 0.61) | (0.31, 0.40) | (0.33, 0.39)|(0.33, 0.55)|\n| UniBias | (0.49, 0.49) | (0.47, 0.48) | (0.35, 0.35) |(0.40, 0.39) |(0.46,0.48)|\n\nWe appreciate your constructive comments, which have significantly enhanced the clarity of our paper. We will make sure to incorporate the above discussions into the revised manuscript.\n\n---\nMoreover, we would like to highlight other enhancements made during the rebuttal process:\n* **Expanded LLM Evaluation**: We conduct additional experiments to evaluate UniBias on more LLMs, including *GPT-J (6B)* and *GPT2-XL (1.5B)*. Detailed results can be found in *Table 1* of the attached one-page PDF.\n* **Exploration of Common Biased Components Across Tasks**: We identify and eliminate the common biased components within LLMs and evaluate its performance on multiple tasks. Results are detailed in *Table 4* of the new PDF. This experiment demonstrates the potential of our work in stimulating diverse bias mitigation methods in a novel perspective of LLM inner structure manipulation.\n* **Enhancement in Grid Search Process**: We optimize the grid search process used in our method by providing alternatives of implementing one-time grid search and using unlabeled samples in place of labeled ones. The results of these optimizations are depicted in *Table 2* and *Figure 1* of the rebuttal PDF.\n* **In-depth Component Analysis**: We further visualize and analyze the distribution of identified biased attention heads, as shown in *Figure 2* in the PDF.\n\n---\nWe are encouraged to find that these additioanl analysis and experiments, prompted by your valuable insights, have substantially strengthened our paper. We look forward to hearing your feedback!\n\n---\n**References**:\n\n[1] Calibrate before use: Improving few-shot performance of language models. ICML 2021.\n\n[2] Mitigating Label Biases for In-context Learning. ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to express our gratitude for the time and effort you've dedicated to reviewing our paper. 

We are deeply grateful for your recognition of our work: 
* The proposed UniBias method is novel (*Reviewer BhZE, 8E5c*), well motivated (*Reviewer LH5i*), straightforward and does not introduce any additional inference time cost (*Reviewer CnNd*), and has potential to become a leading technique in LLM bias mitigation (*Reviewer BhZE*).
* The exploration on internal mechanisms of underlying biases in LLMs is a relatively unexplored area (*Reviewer CnNd*) and provides insights that uncover the root causes of these biases (*Reviewer BhZE*).
* The methodologies and findings are insightful (*Reviewer 8E5c*).
* Experiments are extensive (*Reviewer 8E5c*), demonstrating significant improvements over standard ICL and SOTA calibration methods (*Reviewer BhZE*).
* A critical problem in ICL is investigated in this paper (*Reviewer LH5i*) and the work represents a unique contribution to bias reduction within LLMs (*Reviewer BhZE*).
* The paper is well-written (*Reviewer 8E5c*).

---
In response to your valuable feedback, we have carefully responded to each comment and wish to outline the main changes as follows:
* **Expanded LLM Evaluation**: We conduct additional experiments to evaluate UniBias on more LLMs, including *GPT-J (6B)* and *GPT2-XL (1.5B)*. Detailed results can be found in *Table 1* of the attached one-page PDF.
* **Exploration of Common Biased Components Across Tasks**: We identify and eliminate the common biased components within LLMs and evaluate its performance on multiple tasks. Results are detailed in *Table 4* of the attached PDF. This experiment demonstrates the potential of our work in stimulating diverse bias mitigation methods in a novel perspective of LLM inner structure manipulation.
* **Enhancement in Grid Search Process**: We optimize the grid search process used in our method by providing alternatives of implementing one-time grid search and using unlabeled samples in place of labeled ones. The results of these optimizations are depicted in *Table 2* and *Figure 1* of the rebuttal PDF.
* **In-Depth Analysis**: We further analyze why mitigating LLMs' bias towards labels can alleviate prompt brittleness in our method. We also visualize and analyze the distribution of identified biased attention heads, as shown in *Figure 2* in the PDF.

---
Moreover, We would like to further highlight the contributions of our work.
* **A New Insight for LLM Bias Mitigation**: Unlike existing works based on *external* adjustments of LLM outputs, we mitigate LLM bias through manipulation of LLM *internal* structure. This novel perspective could potentially stimulate future research on LLM bias mitigation from inner structure manipulation, offering a new direction for the field.
* **Unveil Internal Mechanisms of LLM Bias**: We conduct a thorough exploration of the internal mechanisms underlying biases in LLMs, providing deep insights that go beyond superficial observations, revealing the inner causes of these biases.
* **Extensive Evaluation**: UniBias is evaluate across 12 datasets and 4 LLMs, consistently demonstrating its effectiveness. 
* **Addressing Prompt Brittleness**: ur experimental results show that our method effectively mitigates prompt brittleness, a critical issue in ICL.
---

In light of your valuable feedback, we find our work has been significantly enhanced during the rebuttal. Again, we would like thank you for your dedication throughout the review process.

Details of our rebuttal are outlined in our individual responses and the attached rebuttal PDF. Please do not hesitate to contact us for any further suggestions or discussions.

With Gratitude,

Authors of Paper 12087

Meta-review: This paper analyzes the effect of internal components in transformers (e.g., FFs) on the resulting model biases. It also presents a method for mitigating these biases. Reviewers found the problem important, the contribution unique, the method novel and of great potential, and the experiments extensive. Concerns regarding experiments and the link between the claims and the experiments are mostly resolved.
Decision: Accept (poster)