Mitigating Selection Bias with Node Pruning and Auxiliary Options

## Abstract

Abstract Large language models (LLMs) often show unwarranted preference for certain choice options when responding to multiple-choice questions, posing significant reliability concerns in LLM-automated systems.
To mitigate this selection bias problem, previous solutions utilized debiasing methods to adjust the model’s input and/or output.
Our work, in contrast, investigates the model’s internal representation of the selection bias.
Specifically, we introduce a novel debiasing approach, Bias Node Pruning (BNP), which eliminates the linear layer parameters that contribute to the bias.
Furthermore, we present Auxiliary Option Injection (AOI), a simple yet effective input modification technique for debiasing, which is compatible even with black-box LLMs.
To provide a more systematic evaluation of selection bias, we review existing metrics and introduce Choice Kullback-Leibler Divergence (CKLD), which addresses the insensitivity of the commonly used metrics to imbalance in choice labels. Experiments show that our methods are robust and adaptable across various datasets when applied to three LLMs.

### 1 Introduction

The advent of large language models (LLMs) has revolutionized artificial intelligence applications, particularly in the domain of natural language processing.
These models have demonstrated outstanding performance across a variety of use cases, including chatbots, machine translation, text generation, data annotation, etc.
Their ability to answer questions with high precision has opened up new avenues for automated systems.

Figure: Figure 1: We propose BNP and AOI to reduce selection bias for white-box and black-box models. The CKLD metric is also proposed to encourage a more standardized evaluation of the bias.
Refer to caption: /html/2409.18857/assets/C_Figures/img/intro2.png

Despite their remarkable abilities, LLMs suffer from the selection bias problem that often occurs in answering multiple-choice questions (MCQs).
When selecting the answer for an MCQ, many LLMs prefer the choices in a given position (e.g., the last choice), or with a specific choice symbol (e.g., (A) or (3)) (Zheng et al., ; Wei et al., ; Pezeshkpour & Hruschka, ).
This phenomenon degrades model performance.

Many previous works have attempted to explain this phenomenon and/or propose diverse ways to mitigate selection bias.
While there are a few works focused on either modifying the input format (Li et al., ; Robinson et al., ) or calibrating the output probabilities (Zheng et al., ; Reif & Schwartz, ; Wei et al., ), to the best of our knowledge, no embedding or parameter-level investigation has been performed.
Because selection bias originates from internal parameter-level computations, it is crucial to explore how the LLM embeddings contribute to the bias in their output responses.

Understanding the internal representation of selection bias can help us combat it.
By scrutinizing the interaction between the internal representation and the LLM parameters, we develop a novel approach to debias the model.
Specifically, we propose Bias Node Pruning (BNP), which eliminates nodes in the final linear layer that contribute to selection bias.
By dropping as few as 32 out of 4096 nodes in the final layer, we can significantly reduce selection bias and improve question-answering performance.
In addition, we find that introducing an “I don’t know” option in the input reduces bias and enhances task performance.
This Auxiliary Option Injection (AOI) technique is a simple method that can be applied to even black-box scenarios.

Although mitigating selection bias is an important task, even quantifying the extent of selection bias is in itself a difficult problem.
Previous research has adopted several bias evaluation metrics, such as the Standard Deviation of Recalls (RStd) (Zheng et al., )) and the Relative Standard Deviation (RSD) (Reif & Schwartz, )).
However, these metrics are insensitive to imbalance of choices, which can lead them to incorrectly indicate selection bias when none exists.
To address this concern, we propose the Choice Kullback-Leibler Divergence (CKLD), a novel bias evaluation metric that is sensitive to the imbalance.
Figure [1](#S1.F1) depicts our contributions to the overall pipeline.

We conducted experiments and analyses to evaluate the debiasing performance of our methods, adopting the proposed CKLD metric.
We validate the efficacy of our approach on widely used public benchmark datasets with various LLMs.
Results show that our method generally improves debiasing and task performance, and can be utilized together with other baseline methods (e.g., Chain-of-Thought, In-Context Learning, or Decoding by Contrasting Layers).

Our contributions are four-fold. In this work, we:

- •
Propose Bias Node Pruning (BNP), a novel debiasing approach that removes parameters from the final linear layer that contribute to selection bias.
- •
Introduce Auxiliary Option Injection (AOI), which is a simple prompting tactic for MCQ answering. Along with BNP, our debiasing methods improve accuracy by upto 24.9$\%$.
- •
Review existing metrics to systematically evaluate selection bias, and introduce Choice Kullback-Leibler Divergence (CKLD) to address their weakness with imbalanced labels.
- •
Underscore the broad applicability of our approach to various baselines and also demonstrate that our AOI method can debias black-box large language models.

### 2 Selection Bias in LLMs

Although LLMs are most often used for text generation, some tasks involve responding to multiple-choice questions (MCQs).
For example, LLMs are increasingly used to annotate data samples, a task that requires selecting the best choice from several options.
When responding to MCQs, however, LLMs suffer from selection bias, which is the model’s inclination to prefer a choice option bound with a specific symbol or located in a certain order.
In this section, we formally define selection bias (§ [2.1](#S2.SS1)) and discuss when and where the signs of selection bias are observed (§ [2.2](#S2.SS2)).

#### 2.1 Selection Bias Problem

Figure: Figure 2: Comparison of the original and voting accuracy evaluated using different LLMs.
Refer to caption: /html/2409.18857/assets/C_Figures/img/voting2.png

Selection bias refers to a model’s tendency to select options in a given position or with a given symbol among MCQ choices, regardless of the correctness of its choice.
This includes the model’s a priori preference of a certain choice symbol, and its inclination to favor the choice presented at a specific ordering position (Zheng et al., ).
We take a holistic view of this selection bias and formally define it as the discrepancy between the model response to the original choice ordering and the expected model response over all the choice orderings.
That is, a model has no selection bias when reordering the MCQ answer choices leads to no change in the model predictions.

###### Empirical demonstration.

Motivated by this definition, Figure [2](#S2.F2) shows the existence of selection bias on four LLMs.
The lighter bars show each model’s accuracy on the ARC-Challenge dataset (Clark et al., ).
The darker bars, on the other hand, show the accuracy of the answers retrieved by majority voting across all possible choice permutations, which can be interpreted as the expected output across all choice orderings.
If the model is free of selection bias, voting will always output the same choice as the original question, rendering the same accuracy in all cases.
If the model entails selection bias, on the other hand, its response to the original question may deviate from the expected response, leading to a bigger gap between the voting accuracy and original accuracy.
In the figure, selection bias exists with all four models and is greatest with Llama-3.

#### 2.2 Motivating Analyses

Figure: Figure 3: (a) Choice frequency tends to have a sharper distribution when the model’s response is incorrect. (b) In Llama-3, selection bias is predominantly observed to be in the final output layer of the decoder. Other model figures are in Appendix [D](#A4).
Refer to caption: /html/2409.18857/assets/C_Figures/img/bias_heatmap3.png

While selection bias is a prevalent problem in querying the large language model (LLM), it is important to properly capture when and where the bias occurs.
Here, we provide two simple analyses that motivate the design of our debiasing methods.

###### Selection bias is amplified when the model is incorrect.

Figure [3](#S2.F3)(a) shows the frequency of choices of the four response options on the ARC-Challenge dataset (Clark et al., ) using Llama-3-8B-Instruct (Meta, ) and Bloomz-7b1 (Muennighoff et al., ). We manipulated the test dataset to include all possible orderings of the MCQ choices. Thus, the bars should be at 0.25.
However, the models prefer answer choices ‘D’ and ‘A’, respectively. These preferences are stronger for incorrect responses than correct ones, suggesting that the models’ uncertainty or ignorance amplifies the selection bias.

###### Selection bias stems from the final decoder layers.

To identify the source of selection bias, we investigate the difference between the correct and incorrect sample embeddings.
Specifically, we explore the discrepancies within a single sample by permuting the sequence of choices in the question.
The difference between the embeddings within the choice-permuted set removes the sample-specific semantic information while the pure effect of the selection bias remains in the difference.

Accordingly, we first retrieve the intermediate embeddings of an LLM by computing the $t$-th token embedding from the $\ell$-th decoder layer as $\mathbf{z}_{\ell,t}=f_{\ell}(\mathbf{x}_{\mathcal{A}})_{t}$,
where $f_{\ell}$ is the LLM decoder up to the $\ell$-th layer and $\mathbf{x}_{\mathcal{A}}$ is the input with answer choices $\mathcal{A}$.
For brevity of notation, let $\mathbf{z}\in\mathbb{R}^{d}$ be the embedding from an arbitrary layer and token location.
Then, we quantify the selection bias by computing the embedding difference between the correct and incorrect questions within the permutations of $\mathcal{A}$.
That is, the bias vector $\mathbf{b}$ for a sample $\mathbf{x}$ is defined as

$$ $\mathbf{b}_{\mathbf{x}}=\frac{1}{n_{-}}\sum_{i=1}^{n_{-}}\mathbf{z}_{-}^{(i)}\,-\,\frac{1}{n_{+}}\sum_{i=1}^{n_{+}}\mathbf{z}_{+}^{(i)},$ (1) $$

where $\mathbf{z}_{-}$ is the embedding vector of the choice-permuted questions that the model answered incorrectly, and $\mathbf{z}_{+}$ is from the correctly answered questions.
Also, $n_{-}$ and $n_{+}$ correspond to the number of incorrect and correct questions, respectively.
To balance the number of correct and incorrect samples, we use the vector sets $\{\mathbf{z}_{-},\mathbf{z}_{+}\}$ only when $1\leq n_{+}/n_{-}\leq 2$.
Then, we average the bias vectors across the samples in data subset $\mathcal{X}$ to define the average bias vector

$$ $\mathbf{b}=\frac{1}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}\mathbf{b}_{\mathbf{x}},$ (2) $$

where we use a subset size $|\mathcal{X}|$ of 32 in this work.
Refer to Figure [4](#S3.F4)(a) for visual aid.

We use the L2 norm of the average bias vector retrieved from different layers and tokens as a proxy for the magnitude of selection bias.
Figure [3](#S2.F3)(b) shows the norm value from each location as a heatmap, where the x-axis lists the layer indices, and the y-axis shows the last 50 token embeddings of the inputs.
Interestingly, the magnitude of the bias vector is prominent only in the final layer, motivating us to focus on the interaction of the average bias vector with the linear output head.

### 3 Methods

Motivated by our findings that the selection bias is (1) amplified when the model response is incorrect, and (2) represented in the final decoder layer, we introduce two methods for debiasing the model predictions: Bias Node Pruning (BNP) and Auxiliary Option Injection (AOI).
As the names suggest, BNP drops nodes in the final output layer that contribute to the selection bias, and AOI utilizes an auxiliary “I don’t know” option to eliminate bias induced by ignorance.

Figure: Figure 4: Bias Node Pruning with Auxiliary Option Injection. (a) The bias vector $\mathbf{b}_{\mathbf{x}}$ is computed for each sample using its choice-permuted embeddings (equation [1](#S2.E1)). The bias vectors are averaged across a small subset of training data to retrieve the average bias vector, $\mathbf{b}$ (equation [2](#S2.E2)). Then, $\mathbf{b}$ is used to select nodes to prune in $\bm{W}$, where $\bigotimes$ refers to the operation in equation [4](#S3.E4). (b) The pruned $\tilde{\bm{W}}$ is used to retrieve answers for the test questions, along with our Auxiliary Option Injection technique that injects the “I don’t know” option in the inputs (§ [3.2](#S3.SS2)). Our debiasing approaches may correct potentially erroneous responses retrieved with $\bm{W}$ and without AOI, as in (c).
Refer to caption: /html/2409.18857/assets/C_Figures/img/mainfig2.png

#### 3.1 Bias Node Pruning

As shown in § [2.2](#S2.SS2), the average bias vector $\mathbf{b}\in\mathbb{R}^{d}$ is most prominent in the final layer, and the selection bias materializes in the final output projection parameters, $\bm{W}\in\mathbb{R}^{d\times|\mathcal{V}|}$, where $\mathcal{V}$ is the vocabulary set.
To mitigate the selection bias problem induced by the linear layer, we prune the parameters in $\bm{W}$ that contribute to the bias.
In choosing which parameters to prune, we gain intuition by approximating a biased model, $\mathcal{F}$, as

$$ $\mathcal{F}(\mathbf{x}_{\mathcal{A}})\approx(\mathcal{D}(\mathbf{x}_{\mathcal{A}})+\mathbf{b})\cdot\bm{W},$ (3) $$

where $\mathcal{D}$ is a conceptual LLM decoder with zero selection bias, and $\mathbf{b}$ is the average bias vector defined in equation [2](#S2.E2).
Then, $\mathbf{b}\cdot\bm{W}$ is the factor that contributes to the selection bias, and removing the parameters in $\bm{W}$ that has the most active interaction with $\mathbf{b}$ will reduce selection bias.
Accordingly, we choose the top-$k$ rows in $\bm{W}$ with respect to

$$ $\mathcal{K}=\underset{i\in[1,d]}{\text{Top-}k}\;\bigg{(}\sum_{j=1}^{|\mathcal{V}|}\;\mathbf{b}_{i}\times\bm{W}_{ij}\bigg{)},$ (4) $$

where $|\mathcal{V}|$ is the vocabulary size of the output.
Then, we use the index in $\mathcal{K}$ to zero out the corresponding rows (i.e., nodes) in $\bm{W}$.
Bias Node Pruning (BNP) is a one-time process with the average bias vector $\mathbf{b}$ being pre-computed, and the pruned weight $\tilde{\bm{W}}$ is applied to all test samples as $f(\mathbf{x}_{\mathcal{A}})\cdot\tilde{\bm{W}}$ where $f$ is the LLM decoder.
Refer to Appendix [B](#A2) for complexity analysis.
Another design choice would deduct $\mathbf{b}$ from the decoder output embedding; however, we observed more stable performance by pruning the parameters in $\bm{W}$.

#### 3.2 Auxiliary Option Injection

Because selection bias is more likely when a model is incorrect, we hypothesized that providing an “I don’t know” (IDK) option would reduce selection bias.
The auxiliary option $o_{\text{aux}}$ is applied as

$$ $\mathcal{A}:=\mathcal{A}\cup\{o_{\text{aux}}\}$ (5) $$

$$ $\hat{\mathbf{a}}=\operatorname*{arg\,max}_{a\in\mathcal{A}\backslash o_{\text{aux}}}\;P(\hat{\mathbf{y}}=a\,|\,\mathbf{x}_{\mathcal{A}}),$ (6) $$

where $\mathcal{A}$ is the set of answer choices, and $\mathbf{x}_{\mathcal{A}}$ is the input question with choices $\mathcal{A}$.
How we retrieve the probability for each choice $a$ will be later discussed in the implementation details in § [5](#S5) and Appendix [A.2](#A1.SS2).
Further analyses on AOI will be provided in § [6.2](#S6.SS2).

### 4 Evaluation

There is no consensus in the literature on how to measure selection bias.
Here, we first review two selection bias metrics, Standard Deviation of Recalls (RStd) and Relative Standard Deviation (RSD), which evaluate the consistency of performance across choices.
By scrutinizing their limitations, we propose Choice Kullback-Leibler Divergence (CKLD), which is a novel distribution-based bias metric.

Definition 1. (Standard Deviation of Recalls) is the standard deviation of the class-wise recall:

$$ $\text{RStd}=\sqrt{\frac{1}{k}\sum_{i=1}^{k}(r_{i}-\bar{r})^{2}},$ (7) $$

where $k$ is the number of choices, $r_{i}$ is the recall of the $i$-th class, and $\bar{r}$ is the arithmetic mean of $r_{i}$ values (Zheng et al., ).

Definition 2. (Relative Standard Deviation) is the class-wise accuracy standard deviation normalized by the overall accuracy:

$$ $\text{RSD}=\frac{\sqrt{\frac{1}{k}\sum_{i=1}^{k}(s_{i}-\bar{s})^{2}}}{\bar{s}},$ (8) $$

where $k$ is the number of choices, $s_{i}$ is the accuracy of the $i$-th class, and $\bar{s}$ is the mean accuracy averaged across classes (Croce et al., ; Reif & Schwartz, ).

We empirically show how these performance-based metrics, RStd and RSD, behave across different data characteristics.
We constructed synthetic 4-way MCQ datasets by varying the choice selection ratio under different ground-truth ratios.
For instance, in the third column of Figure [5](#S4.F5), labeled “‘A’ Label Ratio = 0.55”, answer choice ‘A’ is the correct choice in 55$\%$ of the samples and the rest are labeled ‘B’, ‘C’, or ‘D’ 15$\%$ of the time, respectively.
To simulate realistic predictions, we have the model render correct predictions half of the time, and predict with respect to the choice selection ratio (i.e., ‘A’ selection rate) for the other half.
For example, if ‘A’ Selection Rate is 0.4, each choice will be sampled with respect to $P(A)=0.4$ and $P(B)=P(C)=P(D)=0.2$ half of the time, and will predict the correct answer for the other half.
With this set up, the selection bias metrics should be lowest at the ‘A’ Label Ratio, shown with a vertical dashed line in Figure [5](#S4.F5).

In contrast, the minimum points of RStd and RSD are not in the expected locations (Figure [5](#S4.F5)).
Both metrics are insensitive to the ground-truth ratios. (RSD is lowest when the ‘A’ Selection Rate is $\frac{1}{\text{\# Choices}}=\frac{1}{4}$ regardless of the ‘A’ Label Ratio.) These results highlight the inability of RStd and RSD to measure selection bias in datasets with skewed distributions of the correct label.
Therefore, we propose Choice Kullback-Leibler Divergence (CKLD), a distribution-based metric sensitive to data distribution and imbalance of choice labels.

Definition 3. (Choice Kullback-Leibler Divergence) is the KL divergence between the ratio of each predicted choice and the ratio of each ground truth choice label:

$$ $\text{CKLD}=\sum_{i=1}^{k}\,p_{i}\log{\frac{p_{i}}{q_{i}}},$ (9) $$

where $k$ is the number of choices, $p_{i}$ is the ratio of ground truth label choices, and $q_{i}$ is the ratio of each predicted choice label.

Figure: Figure 5: Empirical analyses of selection bias metrics. The metrics are tested on a 4-way classification task using synthetic data with varying levels of label ratios (outer x axis) and selection rates (inner x axis). We randomly generate 3000 samples and run 100 times to retrieve the mean and standard deviation of the metrics. The corresponding ‘A’ Ratios are denoted with dashed lines.
Refer to caption: /html/2409.18857/assets/C_Figures/img/metrics3.png

CKLD is minimized when the predictions match the ground-truth ratio without bias towards certain choices (bottom row of Figure [5](#S4.F5); proof in Appendix [C](#A3)).
However, CKLD does not account for the model performance in downstream tasks. Hence, it is important to refer to multiple metrics for a robust assessment.
In this work, we leverage both RSD and our CKLD metrics to evaluate selection bias.
We chose RSD because the groundtruth ratios of the benchmark datasets are close to uniform, and we can expect RSD to be minimized when the predictions are uniform.

### 5 Experiments

In this section, we evaluate our Bias Node Pruning (BNP) and Auxiliary Option Injection (AOI) in various settings.
We demonstrate the effect of our methods in § [5.1](#S5.SS1) and show that AOI can debias black-box models in § [5.2](#S5.SS2).

###### Datasets and Models.

We evaluate our method on three multiple-choice question answering data test sets, ARC-Challenge (Clark et al., ), MMLU-Redux (Gema et al., ), and CommonsenseQA (Talmor et al., ).
To retrieve the average bias vectors (equation [2](#S2.E2)), a separate set of out-of-bag samples is used.
Further dataset details are provided in Appendix [A.1](#A1.SS1).
For the models, we mainly evaluate our approach on Llama-3-8B-Instruct (Meta, ), Mistral-7B-Instruct-v0.2 (Jiang et al., ), and Bloomz-7b1 (Muennighoff et al., ).

###### Implementation Details.

As discussed in Section § [4](#S4), we employ RSD and CKLD to measure selection bias and assess the debiasing performance of our approach.
We use Accuracy and the weighted F1 score for question answering performance evaluation.
In predicting the answers from LLMs, we follow previous works (Zheng et al., ): we select the choice symbol (e.g., A, B, C, D) with the highest probability.
For BNP, we prune 32 nodes for Llama-3 and Mistral, and 128 nodes for Bloomz.
Because we modify the inference step, the entire process is not stochastic.
More detailed explanation and further implementation details are provided in Appendix [A.2](#A1.SS2).

**Table 1: Bias Node Pruning (BNP) and Auxiliary Option Injection (AOI) are tested on three datasets with Llama-3, Bloomz, and Mistral. The best performances are in bold.**
|  | ARC-Challenge | MMLU-Redux | CSQA |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ |
| Llama-3 | 52.3 | 54.1 | 0.562 | 0.494 | 41.8 | 46.7 | 1.021 | 0.589 | 65.4 | 66.2 | 0.261 | 0.095 |
| Llama-3 + BNP | 56.7 | 57.0 | 0.434 | 0.302 | 43.1 | 47.2 | 0.965 | 0.501 | 66.6 | 66.8 | 0.218 | 0.074 |
| Llama-3 + AOI | 60.7 | 61.0 | 0.364 | 0.231 | 47.3 | 49.9 | 0.807 | 0.321 | 67.4 | 67.8 | 0.211 | 0.065 |
| Llama-3 + BNP + AOI | 65.3 | 65.1 | 0.262 | 0.124 | 48.3 | 50.5 | 0.531 | 0.288 | 68.1 | 68.2 | 0.174 | 0.049 |
| Bloomz | 43.9 | 44.2 | 0.461 | 0.283 | 28.0 | 32.8 | 1.003 | 0.661 | 58.5 | 57.2 | 0.215 | 0.136 |
| Bloomz + BNP | 46.8 | 47.0 | 0.352 | 0.191 | 31.0 | 33.0 | 0.537 | 0.326 | 61.4 | 60.9 | 0.178 | 0.083 |
| Bloomz + AOI | 48.9 | 48.5 | 0.590 | 0.147 | 29.5 | 32.7 | 0.808 | 0.456 | 64.2 | 63.6 | 0.134 | 0.060 |
| Bloomz + BNP + AOI | 48.8 | 48.9 | 0.208 | 0.088 | 32.0 | 33.3 | 0.672 | 0.205 | 64.9 | 64.9 | 0.159 | 0.052 |
| Mistral | 67.4 | 67.6 | 0.156 | 0.040 | 46.4 | 47.6 | 0.366 | 0.186 | 63.6 | 63.9 | 0.184 | 0.042 |
| Mistral + BNP | 67.2 | 67.3 | 0.157 | 0.040 | 46.4 | 47.6 | 0.366 | 0.186 | 63.7 | 64.0 | 0.180 | 0.041 |
| Mistral + AOI | 69.8 | 69.9 | 0.108 | 0.019 | 48.6 | 49.3 | 0.308 | 0.139 | 66.8 | 66.8 | 0.101 | 0.016 |
| Mistral + BNP + AOI | 69.5 | 69.5 | 0.108 | 0.019 | 48.6 | 49.3 | 0.309 | 0.140 | 66.8 | 66.8 | 0.099 | 0.016 |

**Table 2: Comparison with Baselines. Ours (BNP + AOI) is compared and applied to baseline methods. Best performances are in bold, and values denoted with * are Ours with only BNP. Note that Bloomz + DoLa performed poorly and was meaningless to compare with baselines.**
|  | ARC-Challenge | MMLU-Redux | CSQA |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ |
| Llama-3 | 52.3 | 54.1 | 0.562 | 0.494 | 41.8 | 46.7 | 1.021 | 0.589 | 65.4 | 66.2 | 0.261 | 0.095 |
| Llama-3 + Ours | 65.3 | 65.1 | 0.262 | 0.124 | 48.3 | 50.5 | 0.531 | 0.288 | 68.1 | 68.2 | 0.174 | 0.049 |
| Llama-3 + CoT | 66.2 | 66.3 | 0.178 | 0.050 | 50.2 | 51.0 | 0.641 | 0.124 | 65.3 | 65.7 | 0.161 | 0.025 |
| Llama-3 + CoT + Ours | 69.2 | 69.5 | 0.156 | 0.024 | 50.4 | 51.1 | 0.281 | 0.095 | 65.9 | 66.0 | 0.123 | 0.012 |
| Llama-3 + ICL | 62.2 | 61.7 | 0.292 | 0.169 | 42.6 | 46.4 | 0.735 | 0.486 | 69.0 | 69.0 | 0.116 | 0.026 |
| Llama-3 + ICL + Ours | 70.0 | 70.0 | 0.167 | 0.054 | 46.9 | 49.2 | 0.526 | 0.280 | 69.5 | 69.3 | 0.124 | 0.037 |
| Llama-3 + DoLa | 51.1 | 52.8 | 0.578 | 0.524 | 41.5 | 46.3 | 1.033 | 0.581 | 65.1 | 65.6 | 0.244 | 0.087 |
| Llama-3 + DoLa + Ours | 64.1 | 63.7 | 0.271 | 0.139 | 47.6 | 49.8 | 0.545 | 0.292 | 66.7 | 66.7 | 0.178 | 0.052 |
| Bloomz | 43.9 | 44.2 | 0.461 | 0.283 | 28.0 | 32.8 | 1.003 | 0.661 | 58.5 | 57.2 | 0.215 | 0.136 |
| Bloomz + Ours | 48.8 | 48.9 | 0.208 | 0.088 | 32.0 | 33.3 | 0.672 | 0.205 | 64.9 | 64.9 | 0.159 | 0.052 |
| Bloomz + CoT | 47.5 | 47.2 | 0.169 | 0.070 | 30.7 | 32.2 | 0.445 | 0.162 | 62.7 | 62.6 | 0.093 | 0.020 |
| Bloomz + CoT + Ours | 50.2 | 50.1 | 0.058 | 0.013 | 34.3 | 34.7 | 0.215 | 0.019 | 62.8* | 62.8* | 0.104* | 0.020* |
| Bloomz + ICL | 39.9 | 42.2 | 0.534 | 0.298 | 30.4 | 32.0 | 0.566 | 0.272 | 50.3 | 52.1 | 0.434 | 0.239 |
| Bloomz + ICL + Ours | 42.8* | 45.2* | 0.433* | 0.249* | 30.7* | 31.1* | 0.310* | 0.135* | 55.5 | 57.3 | 0.365 | 0.167 |
| Mistral | 67.4 | 67.6 | 0.156 | 0.040 | 46.4 | 47.6 | 0.366 | 0.186 | 63.6 | 63.9 | 0.184 | 0.042 |
| Mistral + Ours | 69.5 | 69.5 | 0.108 | 0.019 | 48.6 | 49.3 | 0.309 | 0.140 | 66.8 | 66.8 | 0.099 | 0.016 |
| Mistral + CoT | 66.6 | 66.5 | 0.510 | 0.021 | 50.3 | 50.5 | 0.551 | 0.063 | 63.2 | 63.4 | 0.476 | 0.025 |
| Mistral + CoT + Ours | 66.9 | 66.8 | 0.071 | 0.014 | 50.6 | 50.7 | 0.527 | 0.032 | 64.5 | 64.5 | 0.127 | 0.021 |
| Mistral + ICL | 65.7 | 66.0 | 0.183 | 0.054 | 43.1 | 44.5 | 0.410 | 0.253 | 61.7 | 61.7 | 0.167 | 0.046 |
| Mistral + ICL + Ours | 65.7 | 65.7 | 0.127 | 0.032 | 44.6 | 45.8 | 0.382 | 0.203 | 63.4 | 63.5 | 0.118 | 0.026 |
| Mistral + DoLa | 67.4 | 67.5 | 0.155 | 0.040 | 46.4 | 47.6 | 0.363 | 0.184 | 63.6 | 63.9 | 0.184 | 0.042 |
| Mistral + DoLa + Ours | 69.4 | 69.4 | 0.106 | 0.019 | 48.7 | 49.4 | 0.305 | 0.135 | 66.8 | 66.9 | 0.098 | 0.015 |

#### 5.1 Main Experiments

###### BNP + AOI consistently improves base model performance by reducing selection bias.

Table [1](#S5.T1) shows the performance of our methods with three LLMs and MCQ datasets.
For all models and data sets, BNP and/or AOI increased accuracy and F1 score and decreased RSD and CKLD.
It is especially noteworthy that Llama-3’s accuracy on ARC-Challenge improves from 52.3$\%$ to 65.3$\%$ when both BNP and AOI are applied; an outstanding 24.9$\%$ increase.

###### Our method can be applied together with other debiasing and decoding methods.

For further insight, we compare our methods with other debiasing and decoding approaches: Chain-of-Thought (CoT; Wei et al. ()), In-Context Learning (ICL; Brown et al. ()), and Decoding by Contrasting Layers (DoLa; Chuang et al. ()).
For CoT, we follow the implementation of OpenAI Evals () by first prompting with “Let’s think step by step”, and then using the generated explanation to regenerate the final prediction.
In the case of ICL, we take one question from the training set to retrieve $N!$ choice-permuted questions, where $N$ is the number of choices.
Then, we randomly select three questions from the choice-permuted pool and create demonstrative examples from them, where the LLM agent always answers the choice-permuted questions correctly.
Concrete prompt formats and details are provided in Appendix [A.3](#A1.SS3). These baseline methods can be used along with our debiasing methods.
Both question answering and debiasing improve when our methods are applied together (Table [2](#S5.T2)), even achieving the best performance in collaboration with appropriate baselines.
Note that the values denoted with ‘$*$’ are measured only when our BNP is applied because AOI did not fare well in those cases.

#### 5.2 Black-box Settings

Several state-of-the-art models are black-box and their parameters are not open to the public.
In these cases, BNP is not feasible, leaving AOI as the only available technique for debiasing the model, using text outputs for prediction.
For this reason, we devise a comparative experiment where only AOI is applied to the models.
For white-box models Llama-3, Bloomz, and Mistral, we compute the Jaccard similarity between each choice option and the generated text to select the choice with the highest similarity score, instead of the probability-based answer selection method used in our main experiments.
This approach simulates a black-box setting with the white-box models.
Moreover, we extend our experiment to Claude-3 Haiku and Sonnet models (Anthropic, ), which are closed-source black-box models.
In Table [3](#S5.T3), AOI generally improves black-box model performance (accuracy and F1) and reduces selection bias (RDS and CKLD).

**Table 3: Applying AOI to black-box settings. For Llama-3, Bloomz, and Mistral, we assume that we do not have access to the parameters nor the probability outputs, identical to black-box models.**
|  | ARC-Challenge | MMLU-Redux | CSQA |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ |
| Llama-3 | 65.7 | 65.8 | 0.086 | 0.007 | 51.9 | 52.2 | 0.184 | 0.034 | 69.9 | 69.8 | 0.051 | 0.003 |
| Llama-3 + AOI | 66.9 | 66.9 | 0.076 | 0.007 | 52.6 | 53.0 | 0.177 | 0.033 | 71.3 | 71.2 | 0.030 | 0.003 |
| Bloomz | 41.9 | 42.6 | 0.703 | 0.208 | 27.6 | 31.0 | 1.102 | 0.523 | 55.9 | 55.3 | 0.252 | 0.142 |
| Bloomz + AOI | 44.7 | 45.0 | 0.305 | 0.155 | 29.4 | 31.8 | 0.972 | 0.413 | 59.2 | 58.2 | 0.180 | 0.105 |
| Mistral | 55.2 | 55.2 | 0.140 | 0.036 | 47.4 | 47.6 | 0.216 | 0.069 | 54.6 | 54.8 | 0.155 | 0.031 |
| Mistral + AOI | 59.0 | 59.0 | 0.117 | 0.020 | 48.5 | 48.8 | 0.217 | 0.069 | 62.8 | 62.8 | 0.082 | 0.013 |
| Claude-3-Haiku | 65.3 | 65.0 | 0.095 | 0.024 | 52.1 | 52.0 | 0.057 | 0.008 | 36.4 | 37.3 | 0.587 | 0.331 |
| Claude-3-Haiku + AOI | 71.4 | 71.5 | 0.087 | 0.004 | 51.7 | 51.7 | 0.052 | 0.004 | 47.0 | 47.9 | 0.302 | 0.023 |
| Claude-3-Sonnet | 86.9 | 86.9 | 0.034 | 0.001 | 60.6 | 60.7 | 0.133 | 0.024 | 71.0 | 70.8 | 0.072 | 0.015 |
| Claude-3-Sonnet + AOI | 87.6 | 87.6 | 0.027 | 0.001 | 60.3 | 60.4 | 0.111 | 0.019 | 73.1 | 72.7 | 0.057 | 0.022 |

### 6 Analyses

In this section, we provide in-depth analyses on the mechanism and efficacy of our methods: Bias Node Pruning (§ [6.1](#S6.SS1)) and Auxiliary Option Injection (§ [6.2](#S6.SS2)).
The qualitative findings from our experiments are discussed in § [6.3](#S6.SS3).

Figure: Figure 6: BNP Analyses. (a) BNP improves the base performances (dashed lines) regardless of the number of nodes pruned. The number of nodes to prune can be adjusted to achieve better performance. More figures are in Appendix [D](#A4). (b) Each metric improvement ($\%$) from its base Llama-3 performance when using the average bias vector from different sources is shown in heatmaps.
Refer to caption: /html/2409.18857/assets/C_Figures/img/bnp_analysis5.png

#### 6.1 Analyzing Bias Node Pruning

###### BNP is not sensitive to the number of nodes pruned.

Figure [6](#S6.F6)(a) reveals how the performance metrics change as the number of pruned nodes varies.
Regardless of the number of nodes pruned from 8 to 128, our method improves the base performance (dashed lines in the figure) by great margins.
While our method is robust to the amount of nodes pruned, searching for the adequate level of pruning may achieve better debiasing performance on the downstream task.
Full list of the figure is provided in Appendix [D.2](#A4.SS2).

###### The average bias vector can be generalized across datasets.

The average bias vector represents the direction of selection bias in the embedding space.
If the bias vector captures pure information about selection bias, it should generalize across datasets.
To test this hypothesis, we used the bias vector from one dataset on another.
Figure [6](#S6.F6)(b) shows a heatmap of the improvement in each performance metric.
Interestingly, there is no diagonal pattern, indicating bias vectors retrieved from one dataset can reduce selection bias in other datasets.
For instance, the bias vector from the ARC-Challenge dataset improves the CKLD value of the CSQA dataset by 36$\%$, which is even higher than the 22$\%$ improvement using the bias vector retrieved from its own CSQA dataset.

#### 6.2 Analyzing Auxiliary Option Injection

**Table 4: AOI with arbitrary option content on the MMLU-Redux dataset.**
| Method | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ |
| --- | --- | --- | --- | --- |
| Llama-3 | 41.8 | 46.7 | 1.021 | 0.589 |
| Llama-3 + Arbitrary | 45.6 | 46.5 | 0.790 | 0.366 |
| Llama-3 + Ours | 48.3 | 50.5 | 0.531 | 0.288 |
| Bloomz | 28.0 | 32.8 | 1.003 | 0.661 |
| Bloomz + Arbitrary | 28.0 | 26.1 | 0.618 | 0.314 |
| Bloomz + Ours | 32.0 | 33.3 | 0.672 | 0.205 |
| Mistral | 46.4 | 47.6 | 0.366 | 0.186 |
| Mistral + Arbitrary | 9.7 | 3.9 | 0.762 | 1.888 |
| Mistral + Ours | 48.6 | 49.3 | 0.309 | 0.140 |

###### Content of the auxiliary option matters.

Our experiments above used “I don’t know” as the auxiliary option, but other options are also possible.
We conducted an experiment where we substituted it with “I know the answer”.
In Table [4](#S6.T4), ‘Arbitrary’ refers to this auxiliary option.
For Llama-3 and Bloomz, the inclusion of the arbitrary auxiliary option improves performance and reduces selection bias relative to the baseline (Table [4](#S6.T4)), but the “I don’t know” (Ours) performs better in most cases.
With the Mistral model, however, the arbitrary options degrades model performance and increase selection bias.
A full table with other datasets and more ablation experiments are in Appendix [E](#A5)

#### 6.3 Qualitative Evaluation

Figure: Figure 7: Effect of our methods on choice distributions. Our methods reduce the level of selection bias, and the choice distributions become flatter. Dashed lines are the uniform ratios (gold standard).
Refer to caption: /html/2409.18857/assets/C_Figures/img/dist_change3.png

###### Impact on choice distributions.

In Figure [7](#S6.F7), we show how the distribution of the selected answer choices changes when we introduce BNP and AOI.
In all three datasets, the distribution becomes more uniform when BNP and/or AOI are applied, indicating lower levels of selection bias.
More qualitative examples are provided in Appendix [F](#A6).

###### Qualitative examples.

In addition to disclosing the distributional effect, we provide below the qualitative question-response examples of Llama-3 and Bloomz on the ARC-Challenge dataset. As in Figure [3](#S2.F3)(a), Llama-3 often showed a preference for choice ‘D’, regardless of the order of choices.
Our method successfully corrects such errors.
Bloomz, on the other hand, showed a preference for choice ‘A’.
Again, our methods corrected the model’s response.

### 7 Related Works

###### Selection Bias.

The large language models’ tendency to favor choices in a certain order or with a specific symbol has been discussed in many previous works.
Some of the works investigated the skewed pattern of responses for MCQs (Zheng et al., ; Wei et al., ; Pezeshkpour & Hruschka, ), emphasizing that selection bias is a critical problem.
Many works have approached this problem by calibrating the output probabilities (Wang et al., ; Zheng et al., ; Reif & Schwartz, ; Wei et al., ; Pezeshkpour & Hruschka, ; Wang et al., ; Balepur et al., ; Li & Gao, ; Gupta et al., ),
while others change the way queries are input (Li et al., ; Robinson et al., ).
Additional approaches include debiasing the LLM through distillation training (Liusie et al., ) and training the model to enforce its multiple choice symbol binding (MCSB) property (Xue et al., ).
While parameter pruning methods are often used for efficient deep learning (Srinivas & Babu, ; Han et al., ; Zhu & Gupta, ; Molchanov et al., ; ) or to have the LLM unlearn certain factual knowledge (Liu et al., ; Pochinkov & Schoots, ), parameter pruning has rarely been discussed for debiasing.
Thus, our Bias Node Pruning is a novel approach in the context of the selection bias.

###### Auxiliary Options.

Inclusion of the “I don’t know” option can improve the quality of data collected in surveys (Schuman & Presser, ) but does not meaningfully impact the labels assigned by annotators (Beck et al., ). Recent research has drawn attention to the similarities between surveys, labeling tasks, and model responses to MCQs (Tjuatja et al., ; Eckman et al., ; Chen et al., )
Further research into LLMs’ response behavior would benefit from incorporating insights from the survey science domain: see the discussion in (Eckman et al., ).

### 8 Conclusion

When LLMs answer MCQs, selection bias is a critical problem. Previous research has predominantly focused on modifying the LLM’s input and/or output. In contrast, we uncover the internal source of the bias by scrutinizing the embedding-level discrepancies introduced by this bias. Building on these insights, we propose Bias Node Pruning (BNP) and Auxiliary Option Injection (AOI). Additionally, we address the limitations of existing performance-based evaluation metrics by introducing a new distribution-based metric, Choice Kullback-Leibler Divergence (CKLD), which addresses the insensitivity of prior metrics to imbalance of choice labels. Our approach improved MCQ answering performance by reducing the level of selection bias across widely used MCQ datasets using both open-source (white box) and closed-source (black-box) models.
BNP and AOI work alongside other debiasing/decoding methods to improve the base performance of Llama-3 by up to 33.8$\%$ on the ARC-Challenge dataset.
We also conducted in-depth analyses to better understand the effect of each component, along with case studies to provide qualitative insight.
Overall, our method provides a novel intuition in scrutinizing the internal source of selection bias, and also provides a new approach in debiasing LLMs.

## Appendix

### Appendix A Further Experimental Details

#### A.1 Datasets

We experiment on three datasets: ARC-Challenge (Clark et al., ), MMLU-Redux (Gema et al., ), and CommonsenseQA Talmor et al. ().
We also provide the ground-truth choice ratios in the test dataset in Table [5](#A1.T5).

###### ARC-Challenge

is a dataset from the AI2 Reasoning Challenge, which contains grade-school level multiple-choice science questions.
Among the ‘Challenge’ and the ‘Easy’ sets, we use the former set with 1.17K test and 1.12K training questions.
The training questions are used to extract the average bias vectors.

###### MMLU-Redux

is a dataset derived from the original Massive Multitask Language Understanding (MMLU) (Hendrycks et al., ) dataset, which comprises multiple-choice questions from 57 different branches of knowledge.
Gema et al. () discovered that this original version contains numerous errors, and curated the dataset to have 3,000 manually re-annotated questions across 30 subjects in the original MMLU dataset.
In the case of MMLU-Redux, there is no training set available.
So we utilize the validation set from the original MMLU dataset to pre-compute the average bias vectors.

###### CommonsenseQA

is a dataset of multiple-choice questions that require commonsense knowledge to respond.
The dataset questions are extracted using the knowledge graph, ConceptNet (Speer et al., ), which consists of 9.74K training and 1.22K validation questions.
We use the training set to retrieve the average bias vectors and evaluate on the validation set.

**Table 5: Ground-truth Label ratios of each dataset.**
| Datasets | A ratio | B ratio | C ratio | D ratio | E ratio |
| --- | --- | --- | --- | --- | --- |
| ARC-Challenge | 22.4$\%$ | 25.7$\%$ | 25.9$\%$ | 24.1$\%$ | - |
| MMLU-Redux | 22.3$\%$ | 24..6$\%$ | 25.4$\%$ | 27.7$\%$ | - |
| CSQA | 19.6$\%$ | 20.9$\%$ | 19.7$\%$ | 20.6$\%$ | 19.2$\%$ |

#### A.2 Implementation Details

Here, we detail how we retrieve model predictions and list hyperparameters used for each model-dataset experiment.

###### How are predictions retrieved?

As discussed in the main paper, we use the token output probability distribution to select a token ID for prediction.
For instance, if $\mathbf{z}\in\mathbb{R}^{|\mathcal{V}|}$ is the output logit vector of the first output token, we use $\mathbf{z}$[‘A’] + $\mathbf{z}$[‘_A’] to retrieve the logit value for choice ‘A’, and do the same for other choices as well.
Then, we take the softmax over all the choice logits to retrieve the final probability distribution over the choices.

###### System prompt.

We use the same system prompt across all experiments: “You are an AI assistant that answers multiple choice questions. Please respond with capitalized alphabet(s) that correspond to the correct answer”.
For Chain-of-Thought reasoning baseline experiments, we use a slightly different version of “You are an AI assistant that answers multiple choice questions. Please think step by step and respond with capitalized alphabet(s) that correspond to the correct answer” to encourage the model to output a step-by-step reasoning process.

###### Hyperparameters.

The number of nodes pruned is the main hyperparameter of our experiments.
As disclosed in the main paper, we pruned 32 nodes in all experiments with Llama-3 and Mistral, and pruned 128 nodes in experiments with Bloomz.
We did a simple hyperparameter search among {16, 32, 64, 128} nodes.
Results can be found in Figure [6](#S6.F6)(a) and Figure [9](#A4.F9).
Another noteworthy hyperparameter is the choice delimiter, which refers to the type of token used to separate choices.
In our preliminary experiments, we found that different choice delimiters such as space (‘ ’), line break tokens (‘$\backslash$n’), multiple lines (‘$\backslash$n$\backslash$n$\backslash$n’), or special tags (‘$<$c$>$’) have varying impact on performance.
As there were no consistent results, however, we chose to use the basic space delimiter in all our experiments, e.g. ‘What is 1 + 1? (A)   2 (B) 3   (C) 4’.
Although we do not discuss this in depth as it is beyond the scope of our work, we believe that analyzing the effect of different choice delimiters in multiple choice question answering would introduce an interesting viewpoint.

#### A.3 Baselines

In this section, we provide further details on how the debiasing baselines in Table [2](#S5.T2) are designed.

###### Chain-of-Thought (CoT)

first generates the model response that includes explanations by prompting with “Let’s think step by step” as follows.

Using the explanation that is generated with the prompt, we query the LLM once more with

and identically use the first token output probability distribution to retrieve the predictions.
Note that the actual prompt format depends on the model and the template above is a generic form.

###### In-Context Learning (ICL)

takes one question out-of-bag sample and retrieve $N!$ choice-permuted questions, where $N$ is the number of choices.
Then, three of the choice-permuted questions among the $N!$ pool are randomly chosen to be used for the ICL demonstrative examples.
Concretely, we design the prompt as follows.

Again, the prompt template is generic, and the actual input format depends on the model type.

###### Decoding by Contrasting Layers (DoLa)

is a language model decoding method proposed by Chuang et al. (). Following their implementation, we measure the Jensen-Shannon Divergence between the final (or mature) output probability distribution and intermediate (or premature) outputs to select the layer with the highest divergence.
Then, we use the selected layer output to divide the final output.
Since this is similar to calibration, we expected DoLa to have debiasing effects.
However, the results in Table [2](#S5.T2) show that DoLa alone does not reduce the level of selection bias.

#### A.4 Metrics

In this section, we provide a full list of selection bias metrics, including RStd, RSD, our CKLD, and other existing metrics that were not discussed in the main paper.
We taxonomize the metrics into three groups: brute-force evaluation, performance-based evaluation, and distribution-based evaluation.

##### A.4.1 Brute-force Evaluation

Brute-force evaluation metrics utilize all possible choice permutations to retrieve the metric value.
Since we need to infer the output for each of the choice-permuted questions, the computation increases by a factor of $N!$, where $N$ is the number of choices in the question.
Here, we list two brute-force evaluation metrics, Proportion of Plurality Agreement (PPA) and Permutation Sensitivity (PS), and one semi-brute-force metric that additionally computes only the reverse-order permutation, Fluctuation Rate (FR).

Definition 1. (Proportion of Plurality Agreement) is the proportion of the plurality choice among all possible choice orderings of a multiple-choice question:

$$ $\text{PPA}=\frac{1}{|\mathcal{X}|}\sum_{\mathcal{X}}\frac{\underset{n}{\max}\;\bigg{(}\underset{j=1}{\overset{N!}{\sum}}y_{j}=o_{n}\bigg{)}}{N!},$ (10) $$

where $\mathcal{X}$ is the set of test samples, $N$ is the number of choices in each question, $n$ is the index of the choices, $y_{j}$ is the choice content of the $j$-th choice-permuted sample prediction, and $o_{n}$ is the $n$-th choice content. (Robinson et al., )

Definition 2. (Permutation Sensitivity) is the expected divergence in output probability distributions of the choice-permuted questions:

$$ $\text{PS}=\mathbb{E}_{\sigma_{i,j}}\big{[}d(P(\cdot\;|\;q,\mathcal{A}_{\sigma_{i}});P(\cdot\;|\;q,\mathcal{A}_{\sigma_{j}})\big{]},$ (11) $$

where $\sigma_{i}$ is an arbitrary permutation of choices, $\mathcal{A}_{\sigma_{i}}$ is the answer choice with the choice permutation, $q$ is the input question, $d(\cdot\;;\cdot)$ is the divergence function (e.g., KL-divergence), and $P(\cdot\;|\;\cdot)$ is the output probability distribution function. (Liusie et al., )

Definition 3. (Fluctuation Rate) is the rate of inconsistent model responses to the original input question and the question with choices presented in reversed order:

$$ $\text{FR}=\frac{1}{M}\sum_{i=1}^{M}\;\mathbf{1}(\overset{\rightarrow}{y_{i}}\neq\overset{\leftarrow}{y_{i}}),$ (12) $$

where $M$ is the number of test questions, $\mathbf{1}$ is the indicator function, $\overset{\rightarrow}{y}$ is the model prediction to the original question, and $\overset{\leftarrow}{y}$ is the prediction to the question with reversed choice order. (Wei et al., )

##### A.4.2 Performance-based Evaluation

Performance-based evaluation tries to capture the consistency of model performance when measuring selection bias.
The two metrics discussed in the paper, RStd and RSD, fall under this category.

Definition 4. (Standard Deviation of Recalls) is the standard deviation of the class-wise recall:

$$ $\text{RStd}=\sqrt{\frac{1}{k}\sum_{i=1}^{k}(r_{i}-\bar{r})^{2}},$ (13) $$

where $k$ is the number of choices, $r_{i}$ is the recall of the $i$-th class, and $\bar{r}$ is the arithmetic mean of $r_{i}$ values. (Zheng et al., )

Definition 5. (Relative Standard Deviation) is the class-wise accuracy standard deviation normalized by the overall accuracy:

$$ $\text{RSD}=\frac{\sqrt{\frac{1}{k}\sum_{i=1}^{k}(s_{i}-\bar{s})^{2}}}{\bar{s}},$ (14) $$

where $k$ is the number of choices, $s_{i}$ is the accuracy of the $i$-th class, and $\bar{s}$ is the mean accuracy averaged across classes. (Croce et al., ; Reif & Schwartz, )

##### A.4.3 Distribution-based Evaluation

Existing performance-based evaluation metrics are insensitive to imbalance of chocie labels, and manually adjusting the label distribution does not guarantee fair evaluation and may severely influence performance.
Thus, we propose a new distribution-based evaluation metric, Choice Kullback-Leibler Divergence (CKLD), to complement evaluation of the selection bias.

Definition 6. (Choice Kullback-Leibler Divergence) is the KL divergence between the ratio of each predicted choice and the ratio of each ground truth choice label:

$$ $\text{CKLD}=\sum_{i=1}^{k}\,p_{i}\log{\frac{p_{i}}{q_{i}}},$ (15) $$

where $k$ is the number of choices, $p_{i}$ is the ratio of ground truth label choices, and $q_{i}$ is the ratio of each predicted choice label.

### Appendix B Complexity of Bias Node Pruning

Bias Node Pruning is a two-step process that includes the (1) average bias vector computation, and (2) node pruning.
The first phase utilizes $M$ out-of-bag samples with $N$ choices.
This step requires computing the outputs of $N!$ choice-permuted questions, translating to a complexity of $O(N!\cdot M$).
Once we retrieve the average bias vector, we use it to compute the top-$k$ nodes that activate selection bias (equation [4](#S3.E4)).
This is also a one-time process whose node-pruned parameters are applied throughout all test-time inference tasks.
The complexity of inference itself is identical to the original model without Bias Node Pruning, which is proportional to the number of test samples evaluated.

### Appendix C Proof of CKLD’s label ratio sensitivity

We want to prove that CKLD is minimized when the prediction has no bias towards a certain choice, and matches the ratio of ground-truth labels.
From the CKLD definition (equation [15](#A1.E15)) of

$$ $\text{CKLD}=\sum_{i=1}^{k}\,p_{i}\log{\frac{p_{i}}{q_{i}}},$ (16) $$

let $q_{i}=p_{i}r_{i}$, where $r_{i}$ is the selection bias multiplier applied to the ground-truth choice ratio for each $i=1,\ldots,k$.
As we want to find out when CKLD is minimized, we formulate the objective as follows:

$$ $\begin{split}\text{minimize }&\sum_{i=1}^{k}\,p_{i}\log{\frac{p_{i}}{q_{i}}}\\ \text{s.t. }q_{i}=p_{i}r_{i}&\text{ and }\sum_{i=1}^{k}p_{i}r_{i}=1.\end{split}$ (17) $$

By rewriting this as a Lagrangian function $\mathcal{L}$,

$$ $\begin{split}\mathcal{L}(r_{1},\ldots,r_{k},\lambda)&=\sum_{i=1}^{k}\,p_{i}\log{\frac{p_{i}}{p_{i}r_{i}}}+\lambda(\sum_{i=1}^{k}p_{i}r_{i}-1)\\ &=-\sum_{i=1}^{k}\,p_{i}\log{r_{i}}+\lambda(\sum_{i=1}^{k}p_{i}r_{i}-1),\end{split}$ (18) $$

where $\lambda$ is the Lagrangian multiplier, we take the partial derivative of each variable as:

$$ $\frac{\partial\mathcal{L}}{\partial r_{i}}=-\frac{p_{i}}{r_{i}}+\lambda p_{i}=0$ (19) $$

$$ $\frac{\partial\mathcal{L}}{\partial\lambda}=\sum_{i=1}^{k}p_{i}r_{i}-1=0.$ (20) $$

Then, from equation [19](#A3.E19),

$$ $r_{i}=\frac{1}{\lambda},$ (21) $$

and by substituting this to equation [20](#A3.E20), we get

$$ $\begin{split}0&=\sum_{i=1}^{k}\frac{p_{i}}{\lambda}-1\\ &=\frac{1}{\lambda}-1.\end{split}$ (22) $$

Therefore, the objective is minimized when $\lambda=1$, which translates to $r_{i}=1\;(\because$ equation [21](#A3.E21)).
This is equivalent to saying that CKLD is minimized when $q_{i}=p_{i}r_{i}=p_{i}$, i.e., when the prediction ratio matches the actual label ratio and there is no selection bias towards a certain choice.
$\square$

### Appendix D More Experiments and Analyses

Here, we provide further experiments and analysis results that were not included in the main manuscript.
In § [D.1](#A4.SS1), we demonstrate an extended experiment result on another dataset.
In § [D.2](#A4.SS2), an extended list of figures of Figure [6](#S6.F6) (a) is provided.

#### D.1 Further experiments on HellaSwag dataset

Beyond the three datasets tested in our main paper in Table [1](#S5.T1), we disclose results on another widely used benchmark dataset, HellaSwag (Zellers et al., ).
HellaSwag is a commonsense natural language inference (NLI) dataset that contains 4-way MCQ samples that asks the model to select the option that best ends the given sentence.
The experimental results are in Table [6](#A4.T6).
Bloomz is not included in the table because the model failed to reasonably respond to most of the questions.

Figure: Figure 8: More figures on different models other than Llama-3. Left is the bias vector magnitude heatmap from Mistral-7B-Instruct, and right is from Bloomz-7b1.
Refer to caption: /html/2409.18857/assets/C_Figures/img/apdx_bias_heatmap.png

**Table 6: Further experiments are done on the HellaSwag dataset.**
|  | HellaSwag |  |  |  |
| --- | --- | --- | --- | --- |
| Method | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ |
| Llama-3 | 35.9 | 42.3 | 0.988 | 1.416 |
| Llama-3 + BNP | 38.6 | 43.6 | 0.861 | 0.998 |
| Llama-3 + AOI | 47.6 | 51.2 | 0.599 | 0.611 |
| Llama-3 + BNP + AOI | 50.8 | 52.9 | 0.487 | 0.363 |
| Mistral | 46.7 | 48.7 | 0.558 | 0.341 |
| Mistral + BNP | 46.5 | 48.6 | 0.563 | 0.345 |
| Mistral + AOI | 51.7 | 53.0 | 0.414 | 0.206 |
| Mistral + BNP + AOI | 51.6 | 52.9 | 0.415 | 0.207 |

Figure: Figure 9: Full list of plots on the number of nodes pruned.
Refer to caption: /html/2409.18857/assets/C_Figures/img/prune_apdx.png

#### D.2 Extended List of Figures

Here, we provide a comprehensive table of figures on the sensitivity test on the number of nodes pruned (§ [6.1](#S6.SS1), Figure [6](#S6.F6)(a)).
In Figure [9](#A4.F9), the effect of the number of pruned nodes is shown across the three models and datasets, as its value is varied from 16 to 128.
We also provide the heatmap of the average bias vector magnitude in Figure [8](#A4.F8).
Similar to what has been shown in Figure [3](#S2.F3) (b), selection bias seems prominent in the latter part of the decoder layers.

### Appendix E Different AOI Setup

**Table 7: Different AOI setups. The content, location, and number of auxiliary options are varied to see its effect with ARC-Challenge (top table), MMLU-Redux (middle table), and CSQA (bottom table).**
|  | Llama-3-8B-Inst. | Bloomz-7b1 | Mistral-7B-Inst. |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ | Acc. $\uparrow$ | F1 $\uparrow$ | RSD $\downarrow$ | CKLD $\downarrow$ |
| Model | 52.3 | 54.1 | 0.562 | 0.494 | 43.9 | 44.2 | 0.461 | 0.283 | 67.4 | 67.6 | 0.156 | 0.040 |
| Model + Ours | 65.3 | 65.1 | 0.262 | 0.124 | 48.8 | 48.9 | 0.208 | 0.088 | 69.5 | 69.5 | 0.108 | 0.019 |
| Arbitrary AOI | 63.4 | 61.2 | 0.572 | 0.179 | 50.1 | 50.2 | 0.548 | 0.077 | 11.4 | 3.9 | 1.008 | 2.075 |
| 2-Choices AOI | 70.2 | 69.9 | 0.175 | 0.067 | 46.3 | 47.6 | 0.381 | 0.198 | 69.0 | 69.0 | 0.131 | 0.031 |
| 3-Choices AOI | 71.9 | 71.7 | 0.130 | 0.039 | 45.1 | 46.6 | 0.418 | 0.243 | 68.3 | 68.3 | 0.140 | 0.038 |
| 4-Choices AOI | 72.4 | 72.3 | 0.130 | 0.036 | 43.9 | 45.6 | 0.438 | 0.266 | 68.4 | 68.4 | 0.138 | 0.036 |
| First Choice AOI | 67.9 | 67.6 | 0.222 | 0.106 | 44.2 | 45.3 | 0.455 | 0.232 | 68.1 | 68.1 | 0.109 | 0.025 |

In this section, in addition to all three dataset ablation studies on the content of auxiliary options in § [6.2](#S6.SS2), we provide further ablation study results on the number and location of the auxiliary options.

###### More auxiliary options have mixed effects on performance.

We find that controlling the number of auxiliary options has a notable impact on performance.
That is, we tried adding multiple auxiliary options, all with the same “I don’t know” content.
In most cases in Table [7](#A5.T7), adding more auxiliary options did not help improve performance (see $n$-Choices AOI).
Interestingly, however, both the question-answering and debiasing performance of Llama-3 significantly improved when using more options.
This seems to be a peculiar property of Llama-3 that we can enhance its performance by simply adding multiple auxiliary options.

###### Location of the auxiliary option does not decide performance.

The location of the auxiliary option is another factor to consider.
In our main experiments, we have appended the “I don’t know” option to the end of the choice list.
In comparison, we try placing it in the first choice option (i.e., with choice symbol ‘A’), corresponding to ‘First Choice AOI’ in Table [4](#S6.T4).
Overall, there were mixed results, indicating that the location of the auxiliary option is not a decisive factor in determining performance.

### Appendix F Qualitative Examples

Here, we provide more qualitative examples to show how model response changes when our methods are applied.
The examples are retrieved using the Llama-3-8B-Instruct model on the ARC-Challenge dataset.
As observed in Figure [3](#S2.F3)(a), the original Llama-3 response is skewed towards ‘D’.
The provided examples align with the result, and such ungrounded preference is debiased via our BNP+AOI.

We also provide results with Bloomz-7b1 on ARC-Challenge.
Similar to the trend shown in Figure [3](#S2.F3) (a), the original response is biased towards ‘A’, which is corrected through our debiasing approach.

### Appendix G Limitations and Broader Impact

###### Limitations.

The limitation of this work (and also most works on mitigating selection bias) is that we still do not know the root cause of the selection bias.
While there have been various hypotheses on the reason behind this phenomenon, most focused on the superficial effect of it without considering what in the first place triggered such ungrounded preferences.
Future research will need to unravel the core of selection bias by answering questions like, What data points cause selection bias? or What makes the difference in choice preferences between heterogenous model families?
These questions will be critical in understanding LLMs in general, as it is closely related to how the models choose the next tokens to output.

###### Broader Impact.

This work reveals and mitigates a type of bias present in recent large language models (LLMs).
Considering that LLMs have become an integral part of various applications from customer service to science, the presence of any type of bias can negatively impact the reliability of systems and degrade precision in model- or data-driven decision-making.
By addressing the bias, our research not only improves the accuracy and fairness of these models but also has the potential to enhance the trustworthiness of LLMs in general.
Moreover, this work serves as a foundation for ongoing efforts to scrutinize and enhance LLM-automated systems, introducing a new perspective on analyzing performance.

###### Future Application.

One closely related application of selection bias debiasing is data annotation.
Many works discussed ways to leverage LLMs for automated annotation (He et al., ; Eckman et al., ), or devised human-machine collaborative frameworks (Li et al., ).
We expect our work to benefit such annotation systems by reducing the selection bias in answering MCQs.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - What is the impact of subset size on performance? Does a larger or smaller subset size affect the effectiveness of the proposed method?
soundness: 3
strengths: - The paper is well-written, and the methods are straightforward.
- The authors conduct experiments with different model architectures
summary: The authors find that large language models exhibit a selection bias in question answering (QA) tasks, where the choice of options influences their outputs. To address this, they propose two orthogonal methods: one that removes linear layer parameters responsible for this behavior, and another that adds an "I don't know" option for the model to choose from. Experiments using three different model architectures across three QA datasets demonstrate improvement over baselines.
weaknesses: - Pruning model weights can influence model behavior in unforeseen ways, especially since large language models (LLMs) are intended to be general-purpose. 
- The BNP method appears unstable and offers only marginal performance improvements.
- Simply adding the "I don't know" option provides the best overall results; this is expected, given similar behavior observed in older models with the SQuAD v1 and SQuAD v2 datasets.
- The paper uses a limited number of datasets to demonstrate the method's effectiveness.
- The method addresses only one form of bias; the model may still use or learn to rely on other biases for predictions. See, for example, Mikula et al. (2024).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * Have you tested the proposed method on the same model families at larger scales? Will the benefits still be the same?
soundness: 2
strengths: * This paper is well written.
* The selection bias problem of LLMs is well-motivated.
* The proposed techniques could effectively mitigate the selection bias on MCQs on the considered models and tasks.
summary: This work investigates the selection bias of LLMs for multiple-choice questions. They identify two sources of selection bias: (1) LLMs tend to be more biased when they make incorrect predictions; (2) Upon examining the parameter space, the selection bias mainly arises from the last decoder layer. Based on these two observations, they proposed (1) BNP to debias LLMs by zeroing out the rows of final embedding projection that signifies the pre-computed bias vector; and (2) Adding an auxiliary option. The experimental result shows that the proposed techniques are effective across three MCQ datasets.
weaknesses: * To me, Section 2 is lacking some details and needs more discussions and clarification. For example, what is the specific setting used to produce Figure 2? Is it evaluated based on zero-shot or in-context learning (or does this matter in terms of selection bias)? Are there any scaling trends since some works suggest that smaller LMs struggled to answer MCQs with the correct format under zero-shot, which might have different behavior on selection bias? Are the open-weight LMs and black-box LMs evaluated using the same criterion? (based on the probability of choice token or based on the similarity between choice option and response?) If not, does the evaluation criterion matter? I think the current evaluation is too brief to draw a holistic conclusion.

* The scope of the proposed BNP method is limited to open-weight models with choice token-based evaluation. This could possibly be addressed by demonstrating its effectiveness on the black-box settings with open-weight models, i.e., similarity-based evaluation.

* The motivation and rationale for introducing an auxiliary option to mitigate the selection bias when the model is incorrect is unclear and should be explained more than currently provided. It would also be helpful to demonstrate how the auxiliary option affects the distribution of the choices, especially for the incorrect questions.

* In Appendix A.2, line 763 -- 768, it is unclear what z['A'] + z['_A'] (especially '_A') means.

* Regarding the proposed CKLD metric, it is unclear to me why an LM needs to be able to match $p_i$, the ground truth label ratio of each specific downstream dataset, which appears to be totally agnostic to the LM. Should a uniform prior makes more sense in this case?

* Some details about how you evaluate the MCQ benchmarks are unclear. For example, are you using a fixed, specific permutation? Is it possible to draw any form of statistical significance regarding the improvements, such as using different permutations?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: see weaknesses.
soundness: 1
strengths: With their framework, they could improve the performance of LLMs on multiple choice QA
summary: This paper proposes a new method to remove selection bias and introduces a new metric to quantify it. Unfortunately, the paper has fundamental methodological problems and incorrect claims. See weaknesses for more details.
weaknesses: - The primary problem of the paper is the experiment claiming that selection bias stems from the final decoder layer. The authors analyze the embedding differences between correct and incorrect questions within the permutation and observe a significant norm difference only in the last layer. They look at the embedding differences at each position but only report the last 50 token positions. Firstly, there cannot be any difference in the embeddings at earlier positions because LLaMA3 is a decoder-only model, meaning that the embeddings of previous tokens are not affected by future tokens. If the only difference is the order of the options, then no difference should be observed until the options are reached. It is unlikely that four options cover 50 tokens, which raises the need for an explanation from the authors regarding why a difference is still observed in early tokens in Figure 3b. Secondly, observing different embeddings at the last layer is expected because it essentially predicts the next token. How does this show bias? If the order of options is changed, and the next token after option c is analyzed, it is very likely to differ. Since the next token is different, the last layer’s embedding should differ as well. For example, the next token after the sequence “What is the capital city of France? a) Los Angeles, b) Paris, c)” and “What is the capital city of France? a) Paris, b) New York, c)” will likely differ after option c, and this difference should appear in the last layer. The authors must clarify the connection between their experimental setup and bias. The claim that “selection bias stems from the last decoder layer” is unsupported and likely untrue. It is similar to saying, “All predictions of the model stem from the decoder layer.”The last layer obviously has to be in the circuit causing this bias but this is a trivial conclusion. No experiments are needed to establish this.

- The second major issue with the paper is its proposed metrics and criticism of previously suggested metrics, namely Rstd and RSD. The authors appear confused about the definition of selection bias. They use the definition “discrepancy between the model’s response to the original choice ordering and the expected model response over all choice orderings.” However, where is the “dataset” in this definition? This definition applies to a predictive model, and a robust metric should provide similar scores across different datasets, as seen with Rstd and RSD. These two metrics are resilient to different ground truth distributions, as expected. However, the authors likely made an error in their code because while RSD appears consistent under different ground truth distributions, Rstd varies. With the proposed predictive model, recall values can be analytically calculated without needing a simulation. For option \(i\), this is equal to \(0.5 + 0.5 \times (\text{predictor's probability of sampling option } i)\). Additionally, the authors are confused about selection bias and the predictor’s performance in this experiment. For example, if a classifier randomly outputs one of the options with a 0.25 probability, according to their definition, this model would have no selection bias, even though its performance is poor. However, under the authors’ proposed Kullback-Leibler Divergence metric, the model would not receive a good selection bias score because it deviates significantly from the dataset’s ground truth distribution. Finally, the authors state that RSD gives the lowest score when the selection rate for “A” is 0.25, as if this is problematic. In reality, this is expected (Rstd would yield the same result if calculated correctly). If the model outputs uniformly at random for questions it cannot answer, it does not exhibit bias against any choice. This can be confirmed using the authors’ own definition: “discrepancy between the model response to the original choice ordering and the expected model response over all choice orderings.” This discrepancy would be 0 for such a model.

I would be happy to discuss with authors to change my score.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The motivation for AOI could be strengthened by exploring additional theoretical or empirical justifications. For instance, could the authors test alternative auxiliary options, or provide analysis to show that the IDK option specifically reduces bias compared to other options?
2. BNP selectively prunes nodes, but the potential impact of this pruning on model accuracy across other types of tasks is not fully addressed. How stable is the pruning process across different datasets? Including a sensitivity analysis on how much accuracy is affected by varying pruning parameters could help further clarify BNP’s general applicability.
3. Since the model’s selection bias is observed to be prominent in the final layer, could additional experiments clarify why this layer is particularly sensitive to bias? This might provide insights that could further refine the BNP approach.
soundness: 3
strengths: * The paper’s parameter-level debiasing through BNP and simple input modification with AOI offer new perspectives on reducing selection bias beyond typical input/output adjustments.
* The CKLD metric is well-motivated provides good insights by accounting for class imbalance, which is important for accurate and robust evaluation.
* The extensive empirical studies justify most of the paper's claims. Also, the experiments span multiple datasets and models, demonstrating the applicability of the methods across different settings.
summary: This paper addresses selection bias in large language models (LLMs) used in multiple-choice question answering (MCQ) tasks. It introduces two methods: Bias Node Pruning (BNP), which reduces bias by pruning specific parameters in the final layer of the model, and Auxiliary Option Injection (AOI), which includes an “I don’t know” (IDK) option to the choices provided. Additionally, the paper proposes Choice Kullback-Leibler Divergence (CKLD) as a new metric for measuring selection bias, aiming to improve accuracy and adaptability over existing metrics. Experimental results demonstrate the effectiveness of BNP and AOI across various datasets and LLMs.
weaknesses: **1. Motivation 1 in Section 2.2.**

The claim that “Selection bias is amplified when the model is incorrect” does seem to misrepresent selection bias by framing it as a consequence of incorrectness rather than a pre-existing condition that affects the model’s choices. Selection bias in model predictions typically arises from the model’s predisposition toward certain options regardless of their correctness. Thus, it is more reasonable to view selection bias as a factor that can influence or even lead to incorrect responses, rather than something that is “amplified” by incorrect answers.

**2. Weak Justification for AOI.**

In Section 3.2, the hypothesis that an “I don’t know” (IDK) option would reduce selection bias due to its presence in incorrect answers is not straightforwardly justified. The logic does seem somewhat circular: if bias is inherent and present regardless of correctness, introducing an IDK option may not directly address or reduce selection bias without a clearer rationale for how it interacts with the bias mechanism. The approach could use a more robust theoretical basis to explain why AOI would specifically reduce the preference for biased options. An alternative explanation or additional empirical evidence supporting this choice would strengthen the section’s argument.

Meta-review: The paper shows certain value in terms of innovation and experimental verification, but the deficiencies in experimental design and analysis, as well as writing and expression, have affected the quality of the paper, which suggests a reject.
The authors need to further improve the explanation of the experiments, optimize the description of the evaluation metrics, and enhance the clarity of writing to make the research more convincing.

Decision: Reject