UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation

Authors: hanzhang  zhou
zijian  feng
zixiao  zhu
junlang  qian
kezhi  mao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:49:25

Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various tasks using the in-context learning (ICL) paradigm. However, their effectiveness is often compromised by inherent bias, leading to prompt brittleness—sensitivity to design settings such as example selection, order, and prompt formatting. Previous studies have addressed LLM bias through external adjustment of model outputs, but the internal mechanisms that lead to such bias remain unexplored. Our work delves into these mechanisms, particularly investigating how feedforward neural networks (FFNs) and attention heads result in the bias of LLMs. By Interpreting the contribution of individual FFN vectors and attention heads, we identify the biased LLM components that skew LLMs' prediction toward specific labels. To mitigate these biases, we introduce UniBias, an inference-only method that effectively identifies and eliminates biased FFN vectors and attention heads. Extensive experiments across 12 NLP datasets demonstrate that UniBias significantly enhances ICL performance and alleviates prompt brittleness of LLMs.

Main paper: 
 introduction: 
 Large language models (LLMs) have shown exceptional capabilities in various natural language processing (NLP) tasks, employing the in-context learning (ICL) paradigm. This paradigm conditions LLMs on a context prompt comprising of a few example-label pairs [Brown et al., 2020 , Wei et al., 2022 , Dong et al., 2023 , Zhou et al., 2024] .Despite their impressive performance, LLMs are prone to prompt brittleness, characterized by high sensitivity to the choice [Zhao et al., 2021] and order [Lu et al., 2022] of examples, and prompt formatting [Min et al., 2022] , as demonstrated in Figure 1 . Such prompt brittleness is found to be arise from the bias in LLMs towards predicting certain answers [Zhao et al., 2021] . The presence of the LLM bias undermines the robustness and adaptability of LLMs in diverse applications.Extensive research has focused on identifying factors that lead to LLM bias and strategies for mitigation. For instance, vanilla label bias [Fei et al., 2023] and recency bias [Zhao et al., 2021] demonstrate the LLM's inherent non-contextual preference for certain labels and contextual preference for specific positions, respectively. Additionally, several calibration methods [Fei et al., 2023 , Han et al., 2023 , Zhao et al., 2021] are proposed to counteract the bias by adjusting decision boundaries of model output probabilities. However, these approaches are derived from external observations or adjustments of LLM outputs, leaving the internal mechanisms within LLMs that cause such bias poorly understood.In this work, we investigate the internal mechanism of LLM bias, specifically how feedforward neural networks (FFNs) and attention heads contribute to such bias. Building on findings in mechanistic Figure 1 : illustrates the prompt brittleness of ICL and the effectiveness of our method in mitigating this issue. Experiments are conducted in one-shot setting, using SST2 [Socher et al., 2013 ] dataset for experiments on example selection and prompt formatting and AGnews [Zhang et al., 2015] dataset for example order experiment due to more diverse combination of orders.interpretability [Elhage et al., 2021 , Dar et al., 2023] , we assess the contribution of individual attention heads and FFN vectors1 to label predictions in LLMs. By identifying FFN vectors and attention heads that convey biased influences towards label prediction, we reveal the internal mechanisms behind several key bias factors, including vanilla label bias [Fei et al., 2023] , recency bias [Zhao et al., 2021] , and selection bias [Zheng et al., 2023] . For instance, our analysis of FFN vectors without input context demonstrates that their cumulative impact biases the LLM towards specific labels, indicating a non-contextual preference for certain labels, i.e., vanilla label bias. We elaborate on the background of mechanistic interpretability in Section 2.1 and present our findings on the internal mechanisms of LLM biases in next section.Given our findings that various bias factors stem from the biased behaviors of attention heads and FFN vectors, we are prompted to ask: Can we identify the biased components of LLMs and mitigate their detrimental impact on label prediction? Motivated by this intuition, we propose UniBias, an inference-only method designed to identify and eliminate biased FFN vectors and attention heads in LLMs. Specifically, we begin by projecting each FFN vector and attention head into the vocabulary space to interpret the information conveyed by their outputs. We then detect biased components based on three criteria we defined: the relatedness criterion, the bias criterion, and the low variance criterion. After identification, we mitigate their impact by masking these biased components. Extensive experimental results demonstrate that LLMs, from which biased components have been removed, consistently outperform their original counterparts by a significant margin. Further, as illustrated in Figure 1 , our method significantly improves both the performance and robustness of ICL with perturbations of various design settings.The contributions of our work are summarized as follows:• In contrast to existing works based on external adjustments of LLM outputs, we mitigate LLM bias through manipulation of LLM internal structure. This novel perspective potentially offers a new direction for the field. Moreover, our method demonstrate an effective way to manipulate internal structures of LLMs.• We conduct a thorough investigation of the internal mechanisms underlying biases in LLMs, revealing the inner causes of these biases.• Extensive experiments across 12 NLP datasets demonstrate that, by removing the biased components, our UniBias method significantly enhances ICL performance and achieve state-of-the-art results. Additionally, it effectively addresses the issue of prompt brittleness. 
internal mechanisms causing the bias of llms: 
 This section reveals the internal mechanisms within LLMs that lead to various bias factors.The theoretical background of this work is based on research on mechanistic interpretability [Elhage et al., 2021 , Wang et al., 2022 , Geva et al., 2021] , which aims to explain the internal processes in language models (LMs), facilitating the interpretation of the contributions of individual model components to the final prediction.We are focusing on decoder-only LMs in this paper. They are composed by a sequence of transformer layers, each composed of a multi-head self-attention layer and an feedforward neural network layer.The background knowledge for interpreting the contribution of each FFN vector and attention head to the models' prediction are demonstrated as follows.The Residual Stream We interpret Transformers following the view of residual stream [Elhage et al., 2021 , Dar et al., 2023] . Due to the residdual connection of Transformers, each layer takes a hidden state as input, and adds information obtained by its attention layer and FFN layer to the hidden state through residual connection. In this sence, the hidden state is a residual stream passed along layers, and each attention layer and FFN layer contribute to the final prediction by adding information to the residual stream.Attention Heads Following Elhage et al. [2021] , Dar et al. [2023] , the output of each attention layer of LM can be computed as the sum of all its attention heads. Specifically, for l-th layer, the input is X l ∈ R N ×d , and the attention layer is parameterized by four matricesW l Q , W l K , W l V , W l O ∈ R d×d .The columns of each projection matrix and the rows of the output matrix can be split into H parts:W ℓ,j Q , W ℓ,j K , W ℓ,j V ∈ R d× d H and W ℓ,j O ∈ R d H ×d, where H is the number of attention heads. We then find that:Att ℓ (X ℓ ) = Concat A ℓ,1 X ℓ W ℓ,1 V , A ℓ,2 X ℓ W ℓ,2 V , . . . , A ℓ,H X ℓ W ℓ,H V W ℓ O = H j=1 A ℓ,j (X ℓ W ℓ,j V )W ℓ,j O where A ℓ,j = softmax (X ℓ W ℓ,j Q )(X ℓ W ℓ,j K ) T √ d/H+ M ℓ,j , M ℓ,j is the attention mask. Therefore, the output of an attention layer is equivalent to computing attention heads independently, multiplying each by its own output matrix, and adding them into the residual stream of the LM.FFN In line with Geva et al. [2021 Geva et al. [ , 2022]] , transformer FFN layers can be cast as linear combination of vectors. Specifically, for an input vector x ℓ ∈ R d , FFN parameter matrices K ℓ , V ℓ ∈ R dm×d , the FFN output can be derived as:FFN ℓ (x ℓ ) = f (x ℓ K ℓ T )V ℓ = dm i=1 f (x ℓ • k ℓ i )v ℓ i = dm i=1 m ℓ i v ℓ iwhere f is the activation function, i is the index of the vector. Then, the FFN layer can be viewed as a linear combination of vectors: the multiplication of x ℓ and the key vector k i produces the coefficient m ℓ i that weights the corresponding value vector v i . Logit Lens The logit lens [Nostalgebraist, 2020] is a technique that directly decode hidden states into the vocabulary space using the unembedding matrix of the LLM for interpretation. This approach has been validated in various studies as an efficient method for interpreting the weight matrix or hidden states of LLMs [Dar et al., 2023 , Hanna et al., 2023 , Feng et al., 2024 , Yu et al., 2023 , Geva et al., 2021] .In summary, each attention layer and FFN layer contribute to the final prediction by adding their output hidden states to the residual stream. These outputs can be viewed as the sum of their respective attention heads and FFN vectors. Each attention head or FFN vector's output can be interpreted through the logit lens.We delve into the mechanisms behind several bias factors, analyzing the contributions of attention heads and FFN vectors to the biased predictions in LLMs. We explore vanilla label bias, position bias, and selection bias using the Llama-2 7B model [Touvron et al., 2023] . Vanilla Label Bias The vanilla label bias [Fei et al., 2023] , also known as common token bias [Zhao et al., 2021] , is the inherent, uncontextual preference of the model towards predicting certain label names. Given the contextual nature of attention layers, our investigation focuses on the FFN layers, where we identified a corresponding uncontextual preference. Specifically, by projecting the FFN value vectors into the vocabulary space, we compute the logits for various label names for each FFN vector. Utilizing the residual stream insight, we then aggregate these logits for all FFN vectors whose label logits rank within the top 10 over the vocabulary, reflecting uncontextual influences of FFN vectors that are effective in label prediction. This process yields what we term uncontextual accumulated FFN logits, revealing the intrinsic bias of the LLM towards predicting label names without the influence of input.Figure 2 illustrates the accumulated uncontextual FFN logits across different label names in the sentiment analysis task, alongside their corresponding zero-shot prediction frequencies on the SST-2 dataset. For example, the label name 'positive' exhibits higher uncontextual accumulated FFN logits compared to 'negative,' leading to a higher frequency of 'positive' predictions. Additionally, when comparing the labels 'good' and 'bad', the difference in their uncontextual accumulated FFN logits is more pronounced than that between 'positive' and 'negative,' resulting in a larger discrepancy in prediction frequency. Conversely, the accumulated logits for the labels 'satisfied' and 'disappointed' show a reverse trend relative to 'positive' and 'negative', which results in a corresponding reverse trend in their prediction frequency ratios. Recency Bias Recency bias refers to the tendency of LLMs to favor the label of the example at the end of the prompt [Zhao et al., 2021] . By examining the behavior of attention heads within LLMs, we observe that specific heads consistently prioritize the example at the end of the prompt, providing an internal perspective on the origin of recency bias. We identify the biased attention head using the method introduced in Section 3. We compare the behaviors of a biased attention head (layer 16, head 29) and an unbiased attention head (layer 16, head 19) in terms of the attention weight assigned to examples at different positions and the label logits of the corresponding attention head's output. Specifically, we use the SST-2 dataset, including one positive and one negative example in the prompt, and test with 40 samples, evenly split between positive and negative examples. More experimental details are provided in Appendix A.Experimental results in Figure 3 reveal that the biased attention head (layer 16, head 29) consistently assigns significantly larger attention weights to the final example, irrespective of the ground truth labels of the test samples. This bias persists even when the sequence of examples is reversed, as shown in the second subfigure, indicating a biased preference of this attention head for the last example in the prompt. Furthermore, the biased attention weight assignment leads to biased logits, as shown in the third subfigure. In contrast, the unbiased attention head (layer 16, head 19) assigns very close averaged attention weights to both examples in the prompt. Interestingly, we observe that this unbiased head generally assigns larger weights to the example whose label matches the ground truth label of the test sample, resulting in 35 out of 40 samples being correctly classified based on this pattern by this single attention head. The preference shown by specific attention heads for the example at the end of the prompt reveals the internal mechanism of recency bias. Selection Bias The selection bias refers that LLMs prefer to select specific option ID (like "Option A") as answers for multiple choice questions [Zheng et al., 2023] . We have identified both FFN vectors and attention heads that consistently favor a specific option regardless of the ground truth label of the test sample, revealing the internal mechanism of selection bias. We evaluate the Llama-2 7B model on the ARC dataset, which contains four options (A, B, C, D). We use a zero-shot setting to avoid the influence of position bias from multiple examples. More details are provided in Appendix A. Experimental results are illustrated in Figure 4 . Firstly, we observe that the LLM exhibits a vanilla label bias favoring option "A", as shown in the first subfigure. Additionally, we identify a biased attention head that demonstrates a position bias consistently favoring the first option regardless of the ground truth labels of the test samples (second subfigure) or changes in the sequence of options (third subfigure). Since option A is usually the first option, these two biases both lead to the LLM's preference for option A. 
methodology: 
 In the previous section, we unveil that various bias factors are stem from the biased behaviors of attention heads and FFN vectors. Naturally, we pose the question: Can we identify the biased components of LLMs and mitigate their impact on label prediction? Therefore, we propose our UniBias method to Unveil and mitigate LLMs' label Bias through internal attention and FFN manipulation. Notably, our method is proposed for decoder-only LLMs.Identifying biased FFN vectors in LLMs hinges on whether the contribution of each FFN vector is independent and interpretable. As discussed in Section 2.1, the output of an FFN layer can be cast as a linear combination of FFN vectors. Each FFN vector contributes to the final prediction by adding information encoded in its value vector, v ℓ i , weighted by its corresponding coefficient, m ℓ i . This information within v ℓ i can be interpreted through the logit lens, enabling us to interpret it as a distribution of logits across the vocabulary space. How to identify an FFN vector as biased? we assess whether it consistently introduces a biased preference towards specific labels into the residual stream, regardless of variations in the test samples. Such consistent biases can skew the LLM's predictions. We introduce the following criteria to detect biased components in LLMs, which are also applicable for identifying biased attention heads:• Relatedness Criterion: The information introduced by the FFN vector (or attention head) should closely relate to label prediction.• Biased Criterion: The information contributed to the residual stream by the FFN vector (or attention head) exhibits a biased distribution, favoring certain labels over others.• Low Variance Criterion: The label prediction information added by the FFN vector (or attention head) to the residual stream is almost identical across a set of test samples with different labels, i.e., exhibits very small variance.The third criterion is key to identifying biased FFN vectors (or attention heads), as consistently low variance indicates that the FFN vector is not adequately responsive to varying inputs. Combined with the second criterion, this suggests a bias towards certain predictions regardless of the input's contextual differences.To examine these criteria, we interpret the information contributed by each FFN vector, i.e., mv.For simplicity, we omit the layer number ℓ and FFN index i. Since the FFN value vector v is fixed, changes in the FFN coefficient m across different samples reflect the change in information brought by the FFN vector. We interpret this information by projecting each FFN value vector into the vocabulary space and analyzing the logit distribution over label tokens, termed label logits.Specifically, given an FFN value vector v ∈ R d , the unembedding matrix E ∈ R d×de , a label token mapping matrix L ∈ R N ×de , where each row is a one-hot vector indicating the token id of the first token of each label name, the label logitsEQUATIONc-1 ] ⊤ (where c is the class number) corresponding to the FFN value vector v of k-th sample can be obtained by:g = v • E • L ⊤We use p unlabeled samples from the task to assess the three criteria we defined. The coefficients and label logits of an FFN vector for these samples are denoted as m = [m 0 , m 1 , . . . , m p-1 ] and G = [g (0) , g (1) , . . . , g (p-1) ] ⊤ ∈ R p×c , respectively. An FFN vector is considered biased if it meets the following conditions, each corresponding to one of the three criteria we defined:EQUATION(3)where µ(g(k) ) = 1 c c-1 j=0 g (k) j , µ(m) = 1 p p-1 k=0 m k . The thresholds th 1 F F N , th 2 F F N , th 3 F F N are set by grid search, which is elaborated in Section 3.4The first equation corresponds to the relatedness criterion, measured by the sum of label logits. A higher sum indicates that the information introduced by the FFN vector is more relevant to label prediction. The second equation relates to the bias criterion, quantified by the deviation of the average logit for each label from the overall average logit across all labels. Ideally, for a set of test samples with different labels, the average logits for each label should be relatively balanced. A greater deviation from each label's average compared to the overall average across all labels indicates a more biased distribution. The third equation addresses the low variance criterion, measured by the coefficient of variation (CV) of the FFN vector coefficients across different samples. The CV, calculated as the standard deviation normalized by the mean, indicates whether the label prediction information added by the FFN vector remains almost the same across different samples.The identification of biased attention heads closely resembles the process of identifying biased FFN vectors. As discussed in Section 2.1, each attention head's contribution to the final prediction is independent and interpretable. Therefore, we project the output hidden states of each attention head into the vocabulary space to interpret the information they contribute.To identify biased attention heads, we use the same three criteria introduced for identifying biased FFN vectors. To apply these criteria, we project the output hidden states from each attention head into the vocabulary space and analyze their label logits as the information contributes to label prediction. The output from each attention head consists of hidden states generated for every token in the sequence. For our analysis, we specifically use the hidden state of the last token preceding the prediction of label names, interpreting it as the most direct contribution of the attention head to the prediction, given the autoregressive nature of LLMs. Specifically, to obtain the label logits for an attention head, consider the output hidden states H ∈ R N ×d of this head, the unembedding matrix E ∈ R d×de , and the label token mapping matrix L ∈ R N ×de . Given the token position p label ∈ {0, 1, . . . , N -1}, which indicates the index of the first token of the predicted label names, the label logits a(k) = [a (k) 1 , a (k) 2 , . . . , a (k)c ] ⊤ of the attention head for the k-th sample are derived by:a (k) = H (plabel-1),: • E • L ⊤ .we employ the same p unlabeled samples from the task to assess the criteria for identifying baised attention head. The label logits for these samples are formed as A = [a (0) , a (2) , . . . , a (m-1) ] ⊤ ∈ R m×c . An attention head is considered biased if it meets the following conditions:                         1 p p-1 k=0 Sum (A k,: ) = 1 p p-1 k=0 Sum a (k) = 1 p p-1 k=0 c j=1 a (k) j > th 1 Att 1 p p-1 k=0 Bias (A k,: ) = 1 p p-1 k=0 Bias a (k) = 1 p 1 c p-1 k=0 c-1 j=0 a (k) j -µ(a (k) ) > th 2 Att c-1 j=0 wj • CV (A:,j) = wj • σ(A:,j) µ(A:,j) < th 3 Att where w j = c-1 j=0 µ(A:,j ) µ(A:,j ) , µ(A :,j ) = 1 p p-1 k=0 A i,j , σ(A :,j ) = 1 p p-1 k=0 (A i,j -µ(A :,j )) 2. The functions of the first two criteria are identical to those for biased FFN vector identification. The third function is the weighted sum of the coefficient variance of each label across test samples. The thresholds for biased attention head identification are also derived by grid search.After identifying the biased components of the LLM, we eliminate their influence by masking these biased FFN vectors and attention heads. Specifically, we create masks for the attention heads in each attention layer and reset the coefficient of the biased FFN vector and biased attention head mask.Specifically, we utilize a small subset of training data as a support set, with 20 samples for each class. We then grid search all combinations of threshold values and select the combination that results in the most balanced distribution of average label logits. Specifically, let T represents the set of threshold combinations, and P (t) denote the average label logits for a threshold combination t ∈ T, we aim to find the combination t * that minimizes the bias of label logits: t * = arg min t∈T Bias(P(t)).It is noteworthy that although there are multiple combinations of thresholds, they usually result in a few set of different biased components. For example, for a grid search of thresholds of FFN vectors with 80 combinations, it only result in 4 different sets of biased FFN vectors that need to be examined with the support set on the SST-2 dataset. Additionally, during the inference stage of evaluating test samples, the computation time of the UniBias method is completely identical to that of the original LLMs.Additionally, the support set can be replaced with unlabeled samples, using approximately twice the number of unlabeled samples compared to labeled ones. For further details, please see Appendix F. 
experiments: 
 In this section, we aims to investigate a few research questions (RQ). RQ 1: After eliminating biased components from LLMs, does the ICL performance improve compared to the original LLM? Additionally, how does our UniBias method compare to existing calibration methods? RQ 2: Given that ICL suffers from prompt brittleness, can our UniBias method contribute to more robust ICL performance? RQ 3: Are there any observable patterns of biased FFN vectors and attention heads within and across tasks? RQ 4: What is the performance of LLMs after eliminating only the biased FFN vectors and only the biased attention heads, respectively? RQ 5: What is the impact of support set size on the performance of the UniBias method?Datasets We evaluate our UniBias method on 12 diverse natural language processing datasets across various tasks, including sentiment analysis, topic classification, natural language inference, reasoning, and word disambiguation. Statistics and details about the datasets can be found in Table 4 in Appendix. Models and implementation details We evaluate our method using a range of LLMs, including Llama-2 7b, Llama-2 13b [Touvron et al., 2023] , GPT-J [Wang and Komatsuzaki, 2021 ] and GPT2-XL [Radford et al., 2019] . For all experiments, unless stated otherwise, we use 1-shot ICL setting, i.e. one example per class, and repeat five times under different random seeds. We use k = 20 sampes per class as the support set to obtain all threshold values by grid searching, as mentioned in the method section. The prompt template and more implementation details are specified in Appendix A.Table 1 presents the performance of various datasets and model sizes under the 1-shot setting. Our proposed UniBias method consistently achieves the highest accuracies in most cases. In terms of overall average accuracy, UniBias improves upon the standard ICL by a substantial margin of 3.39% and exceeds the state-of-the-art (SOTA) DC by 1.76% using Llama-2 7b. With Llama-2 13b, UniBias surpasses the standard ICL and the SOTA CC by 2.97% and 2.45%, respectively. Figure 5 further illustrates the results under zero-shot and various few-shot settings for COPA, SST2, and MMLU. Additionally, to demonstrate the effectiveness of our method across different large language models, we present results for GPT-J and GPT2-XL in Figure 7 of Appendix C. Our proposed UniBias consistently surpasses other baselines in all scenarios, underscoring its effectiveness.In response to RQ 1, UniBias not only enhances the performance of original LLMs but also outperforms existing methods. We attribute this success to its internal analysis and bias mitigation techniques, which leverage FFNs and attentions, unlike other methods that rely solely on external observations. Existing studies have found that LLMs are prone to prompt brittleness, with various factors such as the selection and order of examples, as well as the prompt formatting. To address RQ 2, we simulate these brittle scenarios by choosing different demonstration samples, using different prompt formats, and changing the example order to observe variations in LLM performance.Figure 1 presents Llama-2 7b's performance both with and without UniBias. Without UniBias, the standard ICL's performance varies significantly, ranging from 8% to 26%, demonstrating its instability. After applying UniBias, the accuracy remains consistently high and stable, with variations consistently less than 4% under perturbations of various design settings. We provide further theoretical analysis on why UniBias can mitigate prompt brittleness and address various bias factors in Appendix G.In response to RQ3, we present the frequency counts of identified biased attention heads (AHs) and FFNs under repeated experiments in Figure 6 . A large frequency count for an LLM component indicates a higher repeat of being identified as biased in the corresponding dataset. The first subfigure displays the biased components for various example selections, revealing several commonly biased LLM components across different prompts within a single dataset. The second subfigure highlights the common biased components across different datasets (ARC and MMLU) for the reasoning task, indicating that different datasets with similar tasks could share common biased LLM components. The third subfigure demonstrates the presence of common biased LLM components across different tasks.Experimental results suggest an interesting future direction: we may identify global biased components that would mitigate bias across multiple tasks and diverse prompt design settings. We conduct an preliminary experiment to explore the potential of eliminating common biased components. Specifically, we eliminate attention heads that are frequently identified as biased and apply this setting to diverse tasks, rather than handling each task individually. Experimental results in Table 2 demonstrate that although not as effective as our full Unibias method, eliminating common biased components outperforms the vanilla ICL by a large margin. Experiment details are in Appendix D.We conduct ablation studies to analyze the impact of exclusively eliminating biased AHs or FFNs to address RQ 4. Table 3 presents the results of removing only biased FFN vectors (FFN-only) and only biased attention heads (attention-only). Both FFN-only and attention-only methods outperform the standard ICL, demonstrating their effectiveness. When combined as UniBias, the method achieves the best results across most datasets, indicating that the two approaches are complementary.Additionally, we further conduct experiments to investigate the impact of support set size (RQ 5), which is detailed in Appendix E. 
related work: 
 Bias in LLMs: It is well recognized that LLMs are unstable under various ICL design settings, and this instability arises from biases in LLMs toward predicting certain answers [Zhao et al., 2021 , Lu et al., 2022] . To understand these biases, existing studies have identified various bias factors, including recency bias, majority label bias, common token bias [Zhao et al., 2021] , and domain label bias [Fei et al., 2023] in classification tasks. More recently, selection bias, which consistently favors specific options in multiple-choice questions, has also been identified [Zheng et al., 2023 , Wang et al., 2023b] . To address these biases, several calibration methods have been proposed, including contextual calibration [Zhao et al., 2021] , domain-context calibration [Fei et al., 2023] , and prototypical calibration [Han et al., 2023] . However, these identified bias factors and calibration methods are derived from external observations or adjustments of LLM outputs, leaving the underlying mechanisms within LLMs that cause such biases poorly understood.Prompt Brittleness: Regarding prompt brittleness, it is demonstrated in the literature that this instability of prompt arises from LLMs' inherent bias towards predicting certain answers [Zhao et al., 2021] . Therefore, current research efforts address the prompt brittleness by mitigating LLMs' bias towards labels [Fei et al., 2023 , Han et al., 2023 , Zhao et al., 2021] . Mechanistic Interpretability: Mechanistic interpretability [Elhage et al., 2021 , Wang et al., 2022] aims to explain the internal processes in language models, facilitating the interpretation of the contributions of individual model components to the final prediction. Our work builds on the understanding of the residual stream [Elhage et al., 2021] , the logit lens [Nostalgebraist, 2020] , and the interpretation of LLM components in the vocabulary space [Dar et al., 2023 , Geva et al., 2021] . 
conclusion: 
 In this work, we have deepened the understanding of biases in LLMs by unveiling the internal mechanisms that contribute to various bias factors. Building on this understanding, we proposed our UniBias method to mitigate these biases by identifying and eliminating biased FFN vectors and attention heads, demonstrating an effective way to manipulate the internal structures of LLMs.Extensive experiments show that our UniBias method achieves state-of-the-art performance across 12 NLP datasets and different ICL settings. Additionally, our method successfully alleviates prompt brittleness and enhances the robustness of ICL. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: They are properly credited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer:[NA] Justification: Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We evaluate our Unibias method using 12 diverse natural language processing datasets across tasks such as sentiment analysis, topic classification, reasoning, natural language inference, and word disambiguation, as presented in [Socher et al., 2013] 2 872 SST-5 [Socher et al., 2013] 5 2210 MR [Pang and Lee, 2005] 2 3000 CR [Hu and Liu, 2004] 2 376 Topic classification AGNews [Zhang et al., 2015] 4 7600 TREC [Voorhees and Tice, 2000] 6 500 Natural language inference MNLI [Williams et al., 2018] 3 3000 RTE [Dagan et al., 2005] 2 277 Reasoning ARC-Challenge [Clark et al., 2018] 4 1170 MMLU [Hendrycks et al., 2020] 4 2000 COPA [Roemmele et al., 2011] 2 100 Word disambiguation WiC [Pilehvar and Camacho-Collados, 2019] 2 638Experiments on internal mechanisms of biased factors: All experiments are conducted on Llama-2 7b model. For the vanilla label bias experiment, we projecting all FFN value vectors into the vocabulary space and sum the label logits for all FFN vectors whose label logits rank within the top 10 over the vocabulary to calculate uncontextual accumulated FFN logits. We change different set of label words in prompt to derive the label prediction frequency of different label pairs. For the recency bias experiment, based on findings in [Wang et al., 2023a] , instead of the summed attention weights over the whole example, we adopt the sum of attention weights on label words of the example, e.g. "Answer: positive" as the effective attention weight on each example. For the selection bias experiment, we use zeroshot ARC dataset prompts in Table 8 , and we use 12 samples for each class. The attention weight is also summed on label words instead of the whole option.Baselines: We reproduce all baselines using the publicly available code released by the authors to ensure a fair comparison. For the PC method, instead of using test samples as in the original work, we employ 200 training samples per class as the estimate set for parameter estimation using the EM algorithm. This adjustment is made to reflect real-world scenarios where test samples are not readily available. Additionally, the number of samples used by the PC method is significantly larger than that used by our UniBias method.Unibias: In our method, all threshold values are determined through grid searching as described in the methodology section. Specifically, we use 20 samples per class as the support set for grid searching in all experiments. For each repetition of the experiment, the support set is randomly selected based on different random seeds. Additionally, to manipulate biased FFN vectors and attention heads, we create masks for the attention heads of all attention layers and adjust the FFN coefficient values and attention head masks using the hook operation. Additionally, we conduct the experiment on four A5000 GPUs.Figure 7 : Performance comparison of our UniBias method against baseline methods using GPT-J and GPT2-XL models.In this work, we provide a novel insight into the internal mechanisms behind the bias of LLMs. As a pioneering effort in mitigating LLM bias through manipulation of the model's internal structures, our approach relies on grid searching with a small set of labeled training samples. Future research could focus on reducing this reliance, potentially improving the efficiency and applicability of our method.There are many interesting avenues for future research. For instance, instead of identifying biased components for each ICL prompt, future work could explore the identification of global biased components that mitigate bias across multiple tasks and diverse prompt design settings. Additionally, the biased FFN vectors and attention heads we identify could potentially serve as sensors for guiding effective prompt generation. We expect that this internal perspective on LLM bias will inspire more innovative applications in both bias mitigation methods and prompt engineering.Figure 7 demonstrates the performance of various methods across multiple datasets when applied to GPT-J and GPT2-XL models. For both models, our UniBias method consistently outperforms the baseline methods including vanilla ICL, CC, DC and PC. Notably, the improvement on the GPT2-XL model is substantial, demonstrating over an over 20% increase in accuracy on SST-2 dataset compared to vanilla ICL. We explore the potential of eliminating common biased components and apply it to diverse tasks, rather than addressing each task individually. We conduct additional experiments on multiple tasks to assess the effectiveness of directly elinimate these components. Experimental results in Table 2 indicate that although not as effective as our full Unibias method, it outperforms the vanilla ICL by a large margin. Notably, eliminating common biased components represents cost-free gain in performance, as it involves only the direct masking of biased components identified in our work and is applicable to diverse tasks. The attention heads that are masked are listed in Table 5 . Our proposed UniBias method employs a small support set for grid searching. To analyze its effect, we vary the size of the support set. Figure 7 illustrates Unibias's performance with support set sizes ranging from 5 to 50 samples. The results indicate that the performance stabilizes when the support set contains 20 or more samples per class. Notably, for the SST2 dataset, even with much fewer support samples, Unibias significantly outperforms the standard ICL.To Address the potential challenge in accessing labeled samples, we further explore the alternative of using unlabeled samples during grid search. In our method, labeled samples are used to ensure each class is represented proportionally in the grid search, without direct use of the specific label information. Therefore, for balanced datasets, it is equally effective to employ a slight larger pool of unlabeled samples.Our experimental findings, illustrated in Figure 9 of the rebuttal PDF, indicate that approximately 40 × #Classes unlabeled samples achieves performance comparable to that obtained with labeled samples.We further analyze why mitigating model's bias towards labels can alleviate prompt brittleness in our method. Due to the inherent bias of LLMs, different prompts can lead to varying biases towards labels. For example, due to recency bias, placing a negative sentiment analysis sample at the end of a prompt can make LLMs tend to predict 'negative', incorrectly classifying positive samples and thus degrading ICL performance. Various bias factors lead to different direction and extend of bias, resulting in different changes in ICL performance and leading to the prompt brittleness. In contrast, our UniBias method effectively mitigates various potential biases inherent in LLMs by addressing their root causes internally from LLMs. By doing so, it minimizes the introduction of bias towards labels regardless of the difference in prompts, leading to more stable and accurate ICL performance across different prompt configurations.Additionally, our UniBias method seeks to address a broad range of factors that lead to LLM bias, extending beyond those discussed in Section 2. Given the significant variability in prompts, models, and data corpuses, numerous unanticipated bias factors may emerge. Our approach is designed to tackle these diverse bias factors comprehensively. This is feasible because biased behaviors observed externally in LLMs originate from their internal components-specifically, the feedforward neural network (FFN) vectors and attention heads, which house nearly all LLM parameters. By directly identifying and mitigating biases within these FFN vectors and attention heads, UniBias offers a foundational strategy to counteract various forms of bias.The prompt templates used in this work are provided below. We generate few-shot ICL templates follow the template styles in [Han al., 2023 , Fei et al., 2023] , as illustrated in Table 6 . Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).FFN vector refers to the value vector in the second weight matrix of the FFN layer. We elaborate on this in Section2.1 


Decision: This paper analyzes the effect of internal components in transformers (e.g., FFs) on the resulting model biases. It also presents a method for mitigating these biases. Reviewers found the problem important, the contribution unique, the method novel and of great potential, and the experiments extensive. Concerns regarding experiments and the link between the claims and the experiments are mostly resolved.

Reviews: review: paper_summary: paper_summary: The paper explores the influence of FFNs and attention heads in LLMs on biases, resulting in model predictions that exhibit favoritism towards specific labels. The authors propose UniBias, an inference-only technique designed to detect and mitigate biased components within LLMs by analyzing and manipulating FFN vectors and attention heads.  Experiment results on 12 diverse NLP datasets show that UniBias significantly improves the in-context learning performance of LLMs and reduces their sensitivity to prompt design.
strength_weakness: strength_weakness: - While the method shows promising results, it is crucial to assess how well these findings generalize across different types of LLMs and datasets (e.g., machine translation tasks) beyond those tested. This paper only conducts experiments on Llama and classification tasks.
- The reliance on grid search with a limited number of labeled training samples for identifying biased components may limit scalability and efficiency.
questions: questions: - Is this analytical method compatible with LLMs that have undergone SFT? If so, how would the fine-tuning process potentially influence the identification and mitigation of biases within the model's internal mechanisms?
- Can the grid search process be automated or optimized to minimize the need for manual parameter tuning?
limitations: limitations: None
paper_summary: paper_summary: This paper seek to address prediction bias in ICL through intervening feedforward vectors and attention heads.
strength_weakness: strength_weakness: The major problem with the proposed method is that it requires approximately 20 labeled instances per class. This leads to the following concerns:
* The comparison between some baselines and the proposed method is unfair. The proposed method requires labeled instances for searching thresholds, while baselines like DC require no labeled instances.
* The generalizability of the proposed method is also questionable. Can we search thresholds once and apply them to all datasets? If not, it is impractical for real-world application. If we have that many labeled instances and access to the model parameters for the target task and dataset, we can always use them to finetune the model to achieve better performance.
* Moreover, the cost will be intractable when the number of classes increase.
questions: questions: See weaknesses.
limitations: limitations: This paper does not have a limitation section.
paper_summary: paper_summary: This paper aims to address prompt brittleness in in-context learning methods introduced by seemingly inconsequential changes such as vanilla label bias, recency bias, and selection bias. They use the logit lens technique, where  the linear function is applied from the final layer of a decoder-only transformer (AKA the unembedding matrix) on the intermediate layers to interpret the attention activations and FFN embeddings. They detect biased activations and FFN vectors by their relatedness, residual bias, and low variance. Applying their method results in unchanged or improved performance compared to the baseline.
strength_weakness: strength_weakness: - The paper mentions some tasks could share biased components (line 318-321) – based on their experiments, I think it would have been possible to identify these components in this work, and I would have liked to see this result.  

- The analysis of their results could be expanded further. In table 2 the performance of FFN-only, attention-only, and their full UniBias method is inconsistent – sometimes FFN-only performs the best, sometimes attention-only, with no pattern on dataset task.  

- It would be interesting to see how bias exists across different layers of the model. Analyzing the distribution and impact of biased components in each layer may provide deeper insights into how biases are propagated and amplified throughout the model.  

- Some of the mathematical formulations, particularly those related to the identification of biased components, could be more detailed to enhance reproducibility and understanding. The authors could provide more thorough mathematical derivations and examples illustrating how the bias criteria are identified. This would make the methodology clearer and easier to understand.
questions: questions: - Do you have any insights on why UniBias improves performance on certain tasks but not others? 

- For vanilla label bias, how do you choose the alternative label names? 

- It is interesting that the unembedding matrix can also be applied to the attention hidden state. Did you notice any discrepancies in the logit distribution between the FFN and the attention head of the same layer? 

- The sections and overall flow of Sections 2 and 3 may be organized more clearly   

  - Sections 3.3 and 3.4 seem very small (3 and 11 lines respectively) and could be combined with other sections.  

  - It is unclear how the criterion they introduce (relatedness, bias, low variance) in lines 195-201 relate to the three bias conditions introduced in line 218  

  - In general, Sections 2 and 3 seem related to each other but the writing does not tie them together clearly (eg. lines 202-205 seem to be describing the importance of low variance to identifying the biases in section 2.2, but it is not stated very clearly)  

- There are further minor writing issues that could be revised, including various minor typos.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper addresses internal biases in LLMs that cause prompt brittleness. Unlike previous external adjustment methods, it examines the roles of MLPs and attention heads in creating these biases. The proposed UniBias method identifies and masks these biased components, thereby enhancing the model's in-context learning ability. Experiments across several NLP tasks demonstrate that UniBias effectively improves ICL performance.
strength_weakness: strength_weakness: The main weakness of this paper is that there seems to be some disconnect between the problem it aims to solve (the prompt sensitivity of large language models, or LLMs) and two of the objectives of the preliminary analysis (how internal components generate common token bias and selection bias towards labels).

Specifically, the challenge the paper proposes to address is the sensitivity of LLMs to input prompts (example selection, format, and order), and this issue is also illustrated in Figure 1. However, two of the targets of the exploration section are the biases present when LLMs use labels from inputs.

The authors do not explicitly state why suppressing the model's bias toward labels can simultaneously alleviate the model's bias toward prompts. For example, intuitively, suppressing the label's preference for option A does not necessarily suggest an improvement in the model's bias towards a certain format or example.

Given this lack of clarity, and since the criteria for detecting biased components also focus on labels, it is unclear why the proposed method can reduce prompt brittleness.
questions: questions: NA
limitations: limitations: Limitations: Yes; Broader impacts: NA

Comments: title: Looking Forward to Your Valuable Response
Response to Reviewer CnNd
Response to Reviewer CnNd
References
comment: Dear Reviewers,

We greatly appreciate your thorough review, insightful feedback, and recognition of our work. In response to your valuable comments, we have diligently provided detailed explanations and conducted additional experiments to address each point raised. We are eagerly looking forward to hearing your valuable feedback!

We would also like to take this opportunity to highlight the contributions of our work:
- **A New Direction for LLM Bias Mitigation**: Unlike existing works based on external adjustments of LLM outputs, we mitigate LLM bias through manipulation of LLM internal structure. This novel perspective could potentially stimulate future research on LLM bias mitigation from an internal perspective, offering a new direction for the field.
- **Addressing an Important Issue of Prompt Brittleness**: Our experimental results show that our method effectively mitigates prompt brittleness, a critical issue in ICL.
- **Unveiling Internal Mechanisms of LLM Bias**: We conduct a thorough exploration of the internal mechanisms underlying biases in LLMs, providing deep insights that go beyond superficial observations, revealing the inner mechanisms of these biases.
- **Extensive Evaluation**: UniBias is evaluated across 12 datasets and 4 LLMs, consistently demonstrating its effectiveness.

Thank you again for your time and efforts dedicated to reviewing our paper. We welcome any further questions and are very happy to discuss them.

With Gratitude,

Authors of Paper 12087
Dear Reviewer CnNd,

Thank you for your follow-up and for clarifying your concerns regarding prompt format bias. We are pleased to the common ground we share and appologize for any misunderstanding. Your insights are invaluable in refining the depth and clarity of our work!

We appreciate the questions you provided, as they are very helpful in facilitating our discussion. We would like to respond by addressing these questions.

---
Firstly, our method can mitigate prompt format bias, which is supported by experimental evidence in our paper. We evaluate the ICL performance of both the vanilla LLM and the LLM employing our UniBias method under various prompt formats. These experimental results are depicted in the middle figure of Figure 1 in the paper, with different prompt templates used in this experiment detailed in Table 5 of the Appendix. The prompt formats we evaluated include many of the formats discussed in [1], such as differences in separators (': ' vs ' '), spacing ('\n' vs ' '), casing (Positive/positive—with 'positive' used in prompts in Table 4 of the Appendix), label names (Positive/good/Yes), question formats, and the overall format of the prompts.

From Figure 1 of the paper, we can observe that these variations in prompt formats indeed lead to significant fluctuations in ICL performance of the vanilla LLM, as suggest by you and [1]. In contrast, despite these challenging prompt format variations, our UniBias method consistently achieve much more accurate and stable ICL performance across all tested formats.

---
Next, we address the important question of why UniBias effectively mitigate prompt format bias:

- As concluded in [2] and analyzed our rebuttal, prompt instability arises from the LLMs' bias towards predicting certain answers. Different prompt formats can induce biases of varying directions and extents, leading to different effects of bias, and consequently, fluctuations in ICL performance. This is also evidenced by the table provided in the rebuttal, which shows average prediction probabilities for the five prompt formats evaluated in the prompt formatting experiment in Figure 1 of the paper.
- On the other hand, recalling our analysis on mechanistic interpretability of LLMs (detailed in Section 2.1 of the paper): each attention layer  FFN layer contribute to the final prediction by adding their output hidden states to the residual stream. These outputs can be viewed as the sum of their respective attention heads and FFN vectors.

Integrating these two insights, we can learn that the bias of the overall LLM introduced by prompt format bias can be attributed to biased behaviors internally in either the FFN vectors or the attention heads—components where nearly all LLM parameters reside. Therefore, by directly identifying and mitigating biases within these FFN vectors and attention heads, UniBias provides a foundational approach to address prompt format bias and its resultant fluctuations in ICL performance. Because of the same rationale, UniBias is not only effective against the three types of bias discussed in our paper but is also capable of mitigating prompt format bias, example selection bias (as analyzed in Figure 1), and other potential bias factors. This reflects our objective, as discussed in this discussion thread, of designing a method capable of universally addressing all bias factors that contribute to LLM bias.

Finally, we want to emphasize that the your insightful question is crutial in demonstrating why Unibias works. We intend to include a detailed subsection in of our revised manusprict to thoroghly discuss this issue.

Thank you again for your thoughtful engagement, which is instrumental in refining our presentation and deepening our analysis. We truly appreciate these inspiring discussions.

---
[1] "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting." ICLR 2024.

[2] Calibrate before use: Improving few-shot performance of language models. ICML 2021.
Thank you for your further clarification. I agree that addressing as many types of bias as possible is important, and consequently, debiasing the models internally might be the most fundamental approach. I also agree that it's impossible to analyze every type of bias. However, I believe there may be some misunderstanding about my concerns.

My remaining concern is specifically about the prompt format bias that I mentioned in my previous comment, which neither your provided literature nor your additional response has addressed. I believe this bias is quite significant but not covered by the three biases discussed in your paper, as mainly evaluated in [1]. By saying this bias is significant, I mean that models are also highly sensitive to prompt formatting, not that there is too much room for prompt formatting to discuss in detail.

Therefore, when the claim states "effectively mitigating the models' prompt sensitivity," I assume this type of bias is also addressed. I think the following questions might help you understand my concerns and the discussion I expect to see in the paper more clearly:

1. Does the prompt sensitivity issue that the paper claims to mitigate cover the prompt format bias?

2. If so, since this bias is not discussed, what is your speculation as to why the proposed method can mitigate this? Or are you unsure whether the method indeed mitigates this type of bias since you haven't analyzed it in the paper, and thus the method is not explicitly addressing this?

3. If not, I think it's necessary to discuss more about the scope of prompt sensitivity, explicitly stating what is not addressed by the proposed method.
Dear Reviewer CnNd,

Thank you for your prompt response and for recognizing the discussions presented in our rebuttal! We greatly appreciate your insightful feedback regarding the broad spectrum of prompt variations that can lead to fluctuations in model performance. We are glad to further discuss this important issue.

We completely agree with your insight about the significant room for variation across different parts of a prompt. Actually, this insight has been a key motivation for the development of our UniBias method. We believe that merely identifying and addressing a limited number of specific bias factors based on superficial observations is not optimal or entirely effective for mitigating LLM bias. Indeed, due to the significant room in variations in prompts/models/data corpus, it is nearly impossible to identify and address all bias factors from just analyzing external effects of prompt variations on model performance. Therefore, the key challenge and limitation of current bias mitigation methods lie in designing a method that can universally address all potential factors contributing to LLM bias and fluctuations ICL performance.

This challenge motivates the rationale for our UniBias method. UniBias is designed to address all potential biases, whether arising from prompt variations, data corpus discrepancies, or model variations. This is feasible because, **fundamentally, all these superficial variations lead to biased behaviours internally in either the FFN vectors or the attention heads**—components where nearly all LLM parameters reside. By directly identifying and mitigating biases within these FFN vectors and attention heads, UniBias provides a foundational approach to address all forms of bias, representing a poineering work in achieve this objective and a significant advancement in LLM bias mitigation.

The three types of bias factors discussed in the paper are intended to demonstrate the internal mechanisms of these biases observed superficially. Given the vast potential for variations in prompt-induced biases, as you noted, it would be impractical to discuss the internal mechanisms for every possible bias factor.

Thank you for highlighting this crucial point, which not only validates a very important strength of our method but also aligns perfectly with our motivation for developing it. We intend to include a discussion on this issue in our revised manuscript.
Thank you for your comprehensive response. I believe these discussions and explanations indeed serve as excellent supplements to clarity. However, before I finalize my decision, I feel there is still one point that hasn't been covered by these discussions. Even after we've determined the label names, examples, and their order, there's still significant room for variation in the other parts of the prompt beyond these elements. For instance, the way queries and labels are connected, the method of separating examples, the overall format of the input, and so on. These variations, beyond what has already been discussed, can also lead to substantial fluctuations in model performance [1]. However, this situation cannot be explained by any of the three types of bias discussed in this work.

[1] Sclar, Melanie, et al. "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting." The Twelfth International Conference on Learning Representations.
The references in the rebuttal are as follows:

[1] Mitigating Label Biases for In-context Learning. ACL 2023.

[2] Prototypical Calibration for Few-shot Learning of Language Models. ICLR 2023

[3] Large Language Models Are Not Robust Multiple-Choice Selectors. ICLR 2023

[4] Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. Findings of NAACL 2024.