Large Language Models Are Not Robust Multiple Choice Selectors
Authors: Chujie Zheng
Hao Zhou
Fandong Meng
Jie Zhou
Minlie Huang
Venue: ICLR 2024 spotlight (2024)
Keywords: large language model
bias
robustness
multiple choice question
evaluation
Primary area: generative models
TL;DR: We investigate LLMs' bias and robustness in multiple choice evaluation, and propose an efficient, interpretable, and transferable debiasing method.

Abstract: Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs). This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent “selection bias”, namely, they prefer to select specific option IDs as answers (like “Option A”). Through extensive empirical analyses with 20 LLMs on three benchmarks, we pinpoint that this behavioral bias primarily stems from LLMs’ token bias, where the model a priori assigns more probabilistic mass to specific option ID tokens (e.g., A/B/C/D) when predicting answers from the option IDs. To mitigate selection bias, we propose a label-free, inference-time debiasing method, called PriDe, which separates the model’s prior bias for option IDs from the overall prediction distribution. PriDe first estimates the prior by permutating option contents on a small number of test samples, and then applies the estimated prior to debias the remaining samples. We demonstrate that it achieves interpretable and transferable debiasing with high computational efficiency. We hope this work can draw broader research attention to the bias and robustness of modern LLMs.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODELS ARE NoT ROBUST MULTIPLE CHOICE SELECTORS
Chujie Zhengi Hao Zhou Fandong Meng! Jie Zhou Minlie Huang Thee CoAl Group, DCST, BNRist, Tsinghua University, Beijing 100084, China Patterr Recognition Center, WeChat AI, Tencent Inc., China chuj wwuijiieencncn@mil.cccmmm
sihuangttsinghut edu. cn
ABSTRACT
Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs). This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent "selection bias", namely, they prefer to select specific option IDs answers (like "Option A"). Through extensive empirical analyses with 20 LLMs on three bench- marks, we pinpoint that this behavioral bias primarily stems from LLMs' token bias, where the model priori assigns more probabilistic mass to specific option ID tokens (e.g., A/B/C/D) when predicting answers from the option IDs. To mit- igate selection bias, we propose label-free, inference-ting called PriDe, which separates the model's sss prior bias for option IDs from the over- all prediction distribution. PriDe first estimates the prior by permutating option contents on small number test samples, and then applies the estimated prior to debias the remaining samples. We demonstrate that it achieves interpretable and transferable debiasing with high computational efficiency. We hope this work can draw broader research attention to the bias and robustness of modern LLMs
debiasing method,
INTRODUCTION
Question: In an latch built from NOR gates, which condition is not allowed Options:
Multiple choice question (MCQ) prevalent input format of large language models (LLMs). An MCQ typically encom- passes a question accompanied by multiple candidate options, from which the model is tasked to select the most suitable an- swer, as exemplified in Figure Current M.M.eertricc scenar- ios have widely utilized the task format of MCQ, for instance, within benchmarks targeted at assessing LLMs (Hendrycks)
A. S-0, R=0
B. S=0, R=1
C. S=1, R=0
D. S=1, R=1
Answer: D
Figure 1: A multiple choice ques- tion (MCQ) example from MMLU.
ct al. 2020 Zhong et al. 2023 Huang et al. 2023) and in LLM- automatic evaluation frameworks (Chiang et al. 2023
Zheng et al. 2023b). In any scenario, we always expect LLMs to robustly select reliable answers MCQs.
Unfortunately, we observe that modern LLMs are vulnerable to option position changes in MCOs
(Robinson & Wingate 2022). We show in Table thatt, in the 0-shot MMLU evaluation (Hendrycks)
et al. 2020), a simple "answer-moving attack" by always moving the golden answers to a specific position causes L.MMs' dramatic performance fluctuations. For instance, moving the golden answers to position D degrades the accuracy of gpt-3, 5-tturbo by 6.3 (from 67. 2 to 60.9). When moving A, 1lama-30b is boosted by 15.2 and surpasses gpt-3. 5-turbo (68.2 vs. 65.3), which starkly contrasts with their original performance (53.1 vs. 67.2).
LLMs' poor robustness option position changes results from their biased behavior: they prefer to select specific option IDs as answers (like "Option A"). which we term as selection bias. As simple verification, we randomly sampled 1,000 MMLU test samples, where we controlled the number of correct answers being A/B/C/D as 250 each. Among these samples, llama-30B selects A/B/C/D 34.6% 27.3% 22.3% 15.8% of the time, while 3pt-3.55-trbo for 22.5% / 25.6%
"Corresponding author.
Project repository: http::///ghhuu..cc////m/eeeeeeee

---

Published as a conference paper at ICLR 2024
Table 1: Simply moving the golden answers of MCQs to
Acc (96)
10
specific option position can cause dramatic performance fluctuations (0-shot MMLU), suggesting LLMs' poor ro- bustness to option position changes in MCQs.
-10
Recall (%)
75
Move Golden to
Orig
A
C
50
1lama-30B
53.1
(+15.2)
68.2
(+1.1)
54,1
(-2.9)
50.1
(-11.9)
41.2
llama-308
vicuna-v1. 1.3333
falcon-408
falcon-Inst-4ol
Ilama-2:77
9pt-3 5- 5-turbo
59.5
58.6
65,8
44.8
3icuma-v1.3-380
$7.0
(+2.5)
(+1.5)
(+8.8)
(-12.3)
46.3
45.2
64.8
47.9
Figure 2: The performance fluctua-
falcon-40B
51.8
(-5.5)
(-6.7)
(+13.0)
(-3.9)
tion (left) correlates with the balance of different option IDs' recalls in the original inputs, as the original recall of option ID X is actually calculated on a subset of samples where all the golden answers are moved to X.
38.3
38,9
55.7
69.1
fal calconiinst-400
51.5
(-13.3)
(-12.7)
(+1.1)
(+17.6)
61.5
68.6
64.1
62.0
2-70B
64.0
(-2.6)
(+4.5)
(+0.1)
(-2.1)
65.3
68.5
74,2
60.9
5pt-3.5-turbo
67.2
(-1.9)
(+1.3)
(+6.9)
(-6.3)
32.3% / 19.6%, respectively (averaged over 10 runs). These proportions are statistically nonuniform 6x2 Test, p-value < 10 10-4) and align well with the performance fluctuations in Table !
Through extensive empirical evaluation ($2.3), with 20 LLMs, on three benchmarks, and with vary- ing option numbers (from two to five). we show that selection bias is prevalent across various LLMs and cannot be well mitigated by simple prompting strategies ($2.6). like Chain-of-Thouuht prompt- ing (Wei et al., 2022] Kojima et al.] (022). It varies with models but manifests a cross-domain similarity within the same model 2.3). With careful ablation analyses (.,.4) we find that, con- trary to the common view in previous work (Wang al., 2023a; Pezeshkpour & Hruschka (023), selection bias arises less from LLMs' position bias, where they are deemed to favor options pre- sented at specific ordering positions (like first or last). In contrast, we pinpoint one more salient intrinsic cause of selection bias the model's token bias when predicting answers from the option IDs given the standard MCQ prompt, where the model a priori assigns more probabilistic mass to specific ID tokens (e.g., A/B/C/D).
To efficiently mitigate selection bias, we propose method called PriDe ($3), referring to Debiasing with Prior estimation. In PriDe, we assume the model's prior bias for option IDs can be separated from the overall prediction distribution. PriDe first estimates the prior by permutating option con- tents on a small number of test samples (e.g.. 5%), and then applies it to debias the remaining samples. The whole debiasing procedure needs no sample labels, takes place during the inference time, and requires only negligible extra computational costs, which is especially suitable for modern LLLMs. We demonstrate that PriDe achieves superior debiasing effectiveness to strong baselines, es- pecially in the low-cost scenario ($4.I). Furthermore, the prior estimated by PriDe provides a good interpretation for selection bias ($4. and can transfer well across different domains 4.2) which highlights its practical potential in broader scenarios.
Summary of Contributions (1) We identify the ubiquitous selection bias in LLMs and provide extensive empirical analyses (with 20 LLMs, on three MCQ benchmarks) and valuable insights on this problem. (2) We pinpoint LLMs' token bias as one primary intrinsic cause of selection bias. (3) We propose label-free, inference-time dehiasing method PriDe, which demonstrates notable effectiveness and efficiency, interpretability, and cross-domain transferability. We hope this work can inspire future research on the bias and robustness of LLMs.
2
INVESTIGATION ON SELECTION BIAS
2.1
EXPERIMENTAL SETUP
Models Our study focuses on the causal, decoder-only LLMs since this architecture has become
the dominant choice for modern LLMs. We experiment with 20 LLMs from popular LLM families across various sizes: 1
(Touvron et al. 2023a). 1lama-2 - (--chat) -7/mat)07
Touvron et al. 2023b), vicuna-vi www......7//1 3 7/3/333B, vicuna-vis 5-7/13B (Chiang al. 2023 ) 2022
falcon (instt)- -7/40B (Almazrouei et al.) (023). and 3. 2ot-3.5-turbbe 5...5-turoo0000 0613 (OpenAl
The models except gpt-3. 3.5 5-turbo are all open source on the HuggingFace website, and can
2

---

Published as a conference paper at ICLR 2024
access their output probabilities. gpt-3. 3pt-3.5-turbo is the commercial API of ChatGPT. It accepts textual prompts and returns generated texts without providing access to output probabilities.
Benchmarks We conduct experiments MMLU (Hendrycks et al. 2020). 2 A...RCCCCC ERC-Challenge (Clark ct al. 2018), and CommonsenseQQ (CSQA) (Tamoor et al. 2019). w which are all MCQ benchmarks widely used for LLM evaluation. Our selection of benchmarks takes into account the diversity of tasks and domains. Specifically, MMLU and ARC consist of 4-option MCQs, while CSQA consists of 5-option ones, and MMLU covers tests from domain categories spanning 57 subjects. The diverse domains facilitate us to derive general observations and enable cross-domain transfer exploration. See Appendix detailed data statistics.
Evaluation Our evaluation protocol follows the mainstream LLM evaluation frameworks, such as HuggingFace LLM Leaderboard, EleutherAI Im-harness, the original MMLU implementation, and OpenAl Evals (see Appendix FF) Specifically, for open-source models, we access the output probabilities of option ID tokens A/B/C/D/E and use the maximal one as the model prediction. For gpt-3. 5-turbo, we compare the golden answer with the first generated token, with the decoding temperature set to See Figure 7 in Appendix A for the input formats.
Our evaluation mainly considers the O-shot setting, which excludes biases introduced by in-context examples, but we also conduct 5-shot experiments. The in-context examples come from the devel- opment sets and are shared across all the test samples within the same task.
2.2 MEASUREMENT OF SELECTION BIAS
In our study, selection bias is defined as the model's behavioral bias to select specific option IDs as answers. To measure selection bias, one naive way based on the counting for model predictions, which, however, is susceptible to label imbalance. We instead propose measure selection bias based on the balance of recalls of different option IDs and use the standard deviation of recalls (RStd) quantitative metric. This measurement is intuitive that greater recall imbalance indicates more pronounced selection bias and is not as susceptible to label imbalance as the counting- based measurement. More importantly, recall balance well reflects the model's robustness to option posi- tion changes, as illustrated in Figure 2 (measured by recall balance) will improve L.MS' robustness option position changes in MCQs.
Hence, we reasonably expect that reducing selection bias
Note that measuring selection bias with recall balance implies the premise that the golden answers should be randomly placed. We show in Figure Appendix (Cthat randomly shuffling the options does not obviously change selection bias, validating the above premise.
2.3 KEY OBSERVATIONS
We first conduct an extensive evaluation of LLMS on various benchmarks to gain a preliminary understanding of selection bias. We show partial results in Figure for brief presentation and put the full results in Appendix
B We draw the following main observations and insights:
Selection bias is prevalent across various LLMs and varies with model families and sizes. In- tuitively, selection bias is likely to originate from LLMs' training data, where some answers (c.g., C) may occur more frequently than others. However, we do not observe consistent patterns of selection bias within the same model family where the models are trained with the same training data (c.g.. 11ama77/3//00/651
We speculate that selection bias arises as a product of complex interactions between training data composition and ordering, model capacity (number of parameters), and other factors like hyperparameters.
Selection bias within the same model displays moderate similarity across different domains. For instance under the 0-shot setting, 1lama-30B consistently prefers A/B on various benchmarks, while gpt-3. 5-ttubb favors C/B more. While the preference ranking may not strictly persist across tasks or domains, there is an overarching tendency for each model to lean towards certain option IDs (e.g., A and B) and away from others (e.g., C and D). It suggests that selection bias is an inherent behavioral bias of LLMs that is less impacted by tasks or domains.
In-context examples can reduce but may meanwhile alter selection bias. As exemplified in Figure 3 11ama-30B disfavors C under the 0-shot setting but becomes biased towards under the 5-shot setting. We find that this alteration still does not display noticeable patterns within the same
3

---

Published as a conference paper at ICLR 2024
0-shot
5-shot
0-shot, Removing IDs
20
Iarr 308
10
gpt -tttrtt
5
10
Social Science
STEM
Humaratie
Others
MMLU
ABC
CSDA
Social Science
STEM
Humanites
Others
MMLU
ABC
CSDA
Social Science
STEM
Humanaties
others
MALL
ARC
CSDA
Figure 3: Selection bias of 11 1lam--30B and gpt-3. t-tub on various benchmarks, see Appendix B for all the 20 LLMs' results. Y-axis: the recall score (%) normalized by subtracting the overall accuracy. MMLU is additionally split into four domains (STEM, Social Science, Humanities, Others) based on its subject categories.
model family. It indicates that in-context examples can introduce new biases that will be intertwined with the inherent selection bias, making the latter complex and less regular.
2.4 WHAT CAUSES SELECTION BIAS?
Given the ubiquity of selection bias in various LLMs, we now seek figure out the intrinsic causes resulting in this behavioral bias. We propose two hypotheses: (1) Token bias. In the standard MCQ prompt (Figure D. when selecting answers from the option IDs, the model may a priori assign more probabilistic mass to specific ID tokens (such as A or C). (2) Position bias. The model may favor options presented at specific ordering positions (such as the first or last one). Note that in a recent work. [Wang et al. (2023a) similarly found that GPT-4 exhibits the bias towards the responses from "Assi "Assistant 1". However, it is still unclear whether it is because the preferred responses are "selected via the ID token 1" or because they are "presented first".
One challenge here is that option IDs are bound with options' ordering positions, e.g., the ID B is naturally tied with the second-presented option.
Table 2: Preliminary debiasing results (O-shot, gpt-3.5-turbel measured by the standard deviation of recalls (RStd) and accuracy (Acc).
To distinguish the impacts of the two hypothesized causes, we conduct two ablation experiments. (I) Shuffling option IDs. We randomly shuffle the default ID ordering A/B/C/D, for instance, into B/D/A/C or C/A/D/B, etc. this way, can de- note the option presented at any ordering position, thus eliminating the impact of position bias and leaving only token bias. Note that this ablation will obviously impair the naturalness and qual- ity of the MCQ prompt, and may consequently de- grade model performance (as shown in Table [
MMLU
ARC
Methods
RStd
Acc
RStd
Acc
Default a/b/c/d 1/2/3/4 (C)
5.5
67.2
3.3
84.3
6.8
67.0
2.1
83.1
3.8
65.8
2.1
82.3
8.1
66.5
4.0
82.4
Debiasing Instruct
6.1
66.3
3.9
84,2
Chain-of-Thought
4.5
66.8
3.4
84.5
Shuffling IDs Removing IDs
5.1
63.9
3.7
80.3
1.0
66.7
0.6
84.9
(2) Removing option IDs and asking the model to directly select option contents. In this way, the change of selection bias would indicate the impact of token bias, while the remaining part corre- sponds position bias. When evaluating LLMs without option IDs, we require gpt-3. 5-turbo to generate the whole selected option, which is then compared with the golden answer. For open-source models, we compute the likelihoods of options. normalized by their lengths, and use the maximum one as the model prediction. See Figure
and g in Appendix A for the input formats.
As shown in Figure 3 and Table 2 the removal of option IDs notably reduces selection bias (RStd decreases), while RStd is little changed by shuffling option IDs. The former observation, in most cases, holds for various LLMs, on different benchmarks, and with varying option numbers (from two to five, where 2/3-option settings are constructed from the original data), see Appendix [ and Table 3 in Appendix C for detailed results. We also try to replace the default ID symbols A/B/C/D with several reasonable alternatives, including a/b/c/d, 1/2/3/4, 4, and (A) / (B) /C) / (D), but observe no remarkable reduction in RStd from the default one, as shown in Table 2 These results confirm that the model's 's taken bias is one primary intrinsic cause of selection bias.
4

---

Published as a conference paper at ICLR 2024
However, with option IDs removed, the remaining selection bias (corresponding the impact of position bias) varies with models and tasks, see Appendix B and Table 3 in Appendix C For in- stance, the remaining selection bias of 11ama-13B, vicuna 3-BB and gpt 3.t-3.5-turbe is only marginal, while that of 1lama-30B. vicuna-v1.3-1m though having been reduced much). The selection bias of 1ama-2-13/70B even slightly increases in MMLU and ARC after option IDs being removed while still decreasing in CSQA, implying the potential counteraction between token bias and position bias. These results suggest that the model's position bias is somewhat present but quite irregular, largely depending on models and tasks.
and falcon-40B is still pronounced (al-
2.5 CAN WE DEBIAS LLMS BY REMOVING OPTION IDs?
Despite the notably reduced selection bias, we find that removing option IDs usually degrades model performance (except in few cases under the 5-shot setting), see Table and in Appendix C This performance degradation results from the way we leverage LLMs to answer MCQs without option IDs, i.e., calculating and comparing the likelihoods of options, which is referred to as the "cloze prompt" format in Robinson & Wingate (0022). Their study demonstrates that asking LLMs to predict option IDs forms better MCQ prompt than the "cloze prompt", which is consistent with our observation. Besides, selecting answers by calculating and comparing the likelihoods of options not as convenient and straightforward to implement as directly predicting option IDs. We thus suggest that removing option IDs is not practical method to mitigate selection bias.
2.6 CAN SIMPLE PROMPTING STRATEGIES MITIGATE SELECTION BIAS?
As a preliminary debiasing attempt, we apply two simple prompting strategies to 3pt-3.5-turbor 5-turbo: (1) Explicit debiasing instruction: We append an explicit debiasing instruction in the system mes- sage of 5-tturb ("Please note that the provided options have been randomly shuffled, so it is essential consider them fairly and without bias."). (2) Chain-of-Thouugh prompting (Weil et al. 2022 Kojima et al. 2022): gpt-3. 5---3..-uubooo is first prompted with "Let's 's think step by step:" to generate its thought process and then produces the final answer. We follow the implementation in OpenAl Evals, see Figure 10 in Appendix for details. As shown in Table 2 the two prompting strategies cannot mitigate selection bias well. It suggests that selection bias is an inherent behavioral bias of LLMs that cannot be addressed by simple prompt engineering.
METHODOL.OGY
3.1 PERMUTATION DEBIASING BASELINE AND FORMULATION
Before proposing our debiasing method, we first introduce a strong permutation-basee debiasing baseline that our method builds upon. It averages the model's prediction distributions under various option permutations (Wang et al.) 2023a] [heng et al.) 2023b), which intuitively cancels out both the model's 's token bias and position bias.
Formally, we use q to denote the MCQ question. Suppose the n default-ordered option IDs (e.g., A/B/C/D) are d, and the default-ordered option contents are o,, E a permutation of 21...... n), I to a set of possible Ts. We use gr( to denote the index of in I, and x' to the concatenation of the default-ordered option IDs and the I-permuted option contents, so that o; is tied with (1) in x². The permutation-based debiasing baseline can be formulated as:
n). We use to denote
P Pashiissed (or q, x.)
Σ
(d x'), i iE (1,2..... IEI
I
where x is the default input of option IDs and option contents, Pobserved (dgr()) (6) 44, x² ) is the observed prediction probability for the option ID dgr(1) (meaning o, being correct) under the option permu-
tation J, and Paetiassa (o: (ollg.a0 x)) denotes the debiased prediction probability for the option content o;- Since computing full permutations is prohibitively expensive (xn! costs), we adopt practical alternative, called Cyclic Permutation, where I = { + 1,..., n, 1,..., i - 1) })} It reduces the computational cost (e.g.. LLM forward times) from xn! x and ensures one pairing between each option ID d; and option content 0j. In Figure [6] in Appendix C we show that selecting other permutations I Cyclic Permutation, where we still ensure one pairing between each d, and oy, leads to similar debiasing results. However, the overhead of Cyclic Permutation is still somewhat high (xn inference costs), which stimulates us to design more efficient debiasing methods.
5

---

Published as a conference paper at ICLR 2024
3.2 PREDICTION PROBABILITY DECOMPOSITION
The core idea of our method PriDe is to obtain debiased prediction distribution by separating the model's prior bias for option IDs from the overall prediction distribution. Equivalently, it assumes that the observed prediction distribution P Pbssevvdd over d; can be decomposed as a prior distribution Prroor over d; and a debiased distribution Pdebiused Over O;:
Pobserved (d. (g, x)) = Zz.z Pprir (d:1g, x² ( Pdebbiased (ieebiazed()))))) (0),
EI (1.
n), (2)
where fi(i) denotes i-th element in J. Note that we can rewrite the form of Pobserved as a joint proba- bility P(d,, 0j x²x²) for d, and oj. which equals to Pobserved (d. [q, xx') if = (8) and 0 otherwise. Therefore, the above prediction probability decomposition can be interpreted as the conditional in- dependent assumption (ignore the normalization item Zz, where the model holds independent beliefs about d; and 0j- Specifically, Pachiased (0; /4, 1q,x²) reflects the model's true belief about the option content Oj, which is not influenced by the option II) dj- In contrast, Pprior (d: l²) indi- cates the model's prior bias for the option ID d, which actually involves not only the model's token bias but also position bias (..44) due to the natural binding between option IDs and options' ordering positions. Hence, under this formulation, we do not need to strictly distinguish the two biases laboriously, but can instead address them together by eliminating Pprior.
For tractable derivation, we reasonably assume that Paeblased is not affected by how the options are ordered, which implies its invariance to option permutation so we can replace x' with the default input I. We also assume P peior to be independent of x', which means that the prior for option IDs depends on only the question q. Equation 2 is then simplified to:
P Pobserved (dbeervedddsgg.... x²) = q,a! prioor (dx (O
VI E EIE (1, n}.
(3)
3.3 DEBIASING WITH PRIOR ESTIMATION
Taking the logarithm of both sides of Equation 3
and summing over all 7 E I, obtain:
Σ TeI
log Pobserved (dslg, x!) = [Z] log Perior (delg)
Σ
log Pacbiased (0) +C
(4)
lEI
[Z
I log Pptior (dilg) +
log P Paebiased ( (0,1g,z) + C
(5)
n
[Z] log prior (d.lg) + C², iE (1,2,....
We derive Equation 5 because Σ1er log Paebiased (0f, |4, x) actually involves [I// iterations over each Pdebiaseed (o: q, .) (given I contains either full eyclie permutations), whose aggregation over j E 2 n is a constant. Therefore, without any sample labels, we can obtain:
P Pprior (d.lg) = softmax
Σ TET
log P Pobserved (d:lg.z)) i 21,2,..., n..
7
Recall our observation in 22. that selection bias within the same model displays moderate cross- domain similarity. This implies that the prior for option IDs is likely to transfer across different samples and domains, which motivates to compute the priors of partial test samples and use them as an approximation for the remaining samples. It can largely improve debiasing efficiency since no more computational overhead is needed for the remaining samples once the prior is estimated.
Drawing the above inspiration, PriDe first takes test samples D. from the test set D, where K can be adjusted based the estimation budget. Each sample in D. undergoes the standard permutation- based debiasing in Equation [] during which we estimate each sample-specific prior Pprior (d, (g)
using Equation For the remaining samples D, DiDe. we compute the "global prior" P Pirio (d,) by averaging the previously computed priors as an approximation to the new sample's Pprior (d, (g)
in Equation| We can thus compute the approximated Pasbiased (ox (oslg,22 x) and obtain the debiased prediction (omit the superscript as we only use the default input here):
Paebissed g, æ) x Posserved (d.lg. (g. x) (prior(d)) i ie (1, n..
(8)
When K < (D). the overhead for prior estimation will become negligible compared to the whole inference cost. The overall procedure of PriDe is summarized as Algorithm

---

Published as a conference paper at ICLR 2024
Algorithm PriDe: Debiasing with Prior Estimation
Require: Language model, test samples D {(q., x;) option number n, estimation budget K
Ensure: Model predictions )/
1: Initialize the model prediction set y = Ø and the prior set P
> Initialization
2: Sample the estimation samples De under K and the remaining samples D, D.D.
3: for (q.z) x)) E De do
4: Debias the model prediction using Equation 1 add the predicted answer
5: Estimate the sample-specific prior prior (da ( using Equation add into P
6: end for
7: Estimate the global prior Ppeior (d,) by averaging P
Prior Estimation
8: for (q.x)) E D, do
9: Debias the model prediction using Equation g
> Efficient Debiasing
Add the predicted answer to y
11: end for return y
MMLU
ARC
CSQA
-9
-9
Recall Ska (96)
-9
-6
-6
-3
Full Perm
3
Full Perm
Cyclic Parm
Cyclic Perm
-3
Cyclic Perm
6-
Cyclic
A Accuracy (%)
PriDe
2-
Cyclic Parm
Cyclic Perm
PriDe
PriDe
x1
X2
x24
x1
X2
x4
x24
x1
X22
X3
X4
Figure 4: Debiasing results (0-shot, averaged over all the 20 LLMs) under varying computational costs (X- axis), and Appendix for detailed breakdowns. We control the costs of Cyclic/Full Perm via the ratio B of the debiased test samples, where we take 3 E (0%, 10,, 20%, 40%, 60%, 80%, 100%). PriDe, we control the costs via the ratio a of test samples for prior estimation (these samples are meanwhile directly debiased via Cyclic Perm), where we take (2%, 5%, 10%, 20%, 40%, 60%, 80%). Note that when 100%. PriDe degenerates Cyclic Perm, do not plot them.
EXPERIMENTS
4.1
MAIN RESULTS
We compare PriDe with two strong permutation-base debiasing baselines: Cyclic Permutation and Full Permutation. The latter experimented on the 5-option CSQA benchmark due the extremely high cost. Since gpt-3. 3.5-3355440 5-turbo does not return the output probability, we sample 100 generated answers
Pobserved- For PriDe, randomly sample K al D test samples as D. and report the average results over runs. Here, we use a E (0, as the ratio of test samples for prior estimation to control the estimation overhead.
approximation
Figure 4 presents the debiasing results (averaged over all the models) versus the computational costs under the 0-shot setting, see detailed breakdowns in Table 3 and 4 in Appendix C PriDe achieves superior debiasing effectiveness and performance improvement to Full/Cyclic Perm, especially in the low-cost scenario. This also holds under the 5-shot setting, see Figure 19 in Appendix C In Figure 17 in Appendix C we show that the estimated prior manifests clear correlation with the empirical selection bias (i.e., the recalls of different option positions before debiasing). suggests that PriDe can provide good interpretation for the model's 's selection bias. Furthermore, observe that the priors are stable when estimated with different sizes of test samples (from 2% to 20%). suggests that we are able to obtain reliable estimate of prior even with a limited computational

---

Published as a conference paper at ICLR 2024
A Recall Std (%)
À Accuracy (%)
2.0
STEM
-9.9
-5.8
-5.5
-5.5
+1.3
+0.7
+0.5
+0.5
-0.9
-4
1.5
Source Domain
Social
+2.2
1.0
Humanities
0,5
Others
5.2
4.9
-3.4
6.5
4.4
+2.1
+1.4
+0.9
+0.8
0.0
ARC
-4.1
4.0
-2.8
4.1
.588
+1.4
+1.2
+0.8
+1.2
+1.4
-0.5
STEM
Social Science
Humanities
Others
ARC
STEM
Social Science
Humanities
Others
ARC
Target Domain
Figure 5: Cross-domain debiasing results of PriDe (0-shot, averaged over all the 20 LLMs), with priors esti- mated using 5% test samples from the source domains.
budget. It also implies that the model's prior bias for option IDs exhibits similar pattern across different samples, confirming our design motivation PriDe in $3.3
Note that one may propose to debias with fewer permutations (such or 3 random permutations, with x2 or x3 costs) as a low-cost alternative to Cyclic (xn) or Full (xn!) Perm. In Figure [ and 22 in Appendix [ we show that PriDe still be combined with these methods and notably boost debiasing effectiveness and efficiency.
4.2 TRANSFERABILIT ANALYSIS
In practical scenarios, we may not guarantee that the test samples always come from the same or similar domains. We hope the prior estimated by PriDe to be transferable: Once the prior is estimated using small number of samples from domain X, it can be used to debias not only other samples from domain X but also samples from domain Y. To this end, we evaluate the cross-domain transfer performance of estimated priors on the four category domains of MMLU and ARC (5 domains in total, are 4-option MCQs). We first use PriDe to estimate the prior with a test samples from source domain X, and then apply it to debiasing the test samples from a target domain Y. As shown in Figure 5 the estimated priors exhibit reasonable transferability across different domains.
However, we find that although the transfcrability in terms of debiasing is promising, there may be slight degradation in model performance when the domain gap is large (e.g., from STEM/Humanities to ARC). While PriDe is not designed to improve model performance but rather mitigate selection bias and thereby improve LLMs' robustness, we suggest updating the estimated prior using new samples when there are predictable domain shifts in test samples, whose overhead is still negligible compared to the whole inference cost.
4.3 HOW DEBIASING AFFECTS MODEL PREDICTIONS?
We notice that although it is not our initial intention, the debiasing methods (PriDe and Cyclic/Full Perm) usually improve the model performance. With PriDe (a 5%) and Cyclic Perm examples, we seek further insights on how debiasing affects model predictions and consequently improves model performance. We are especially interested in how PriDe works with the estimated prior and how it works differently from Cyclic Perm.
We put the breakdown of the model prediction changes Table and 3and rize our main observations follows: (1) For both PriDe and Cyclic Perm, the samples with predic- tions changing from wrong to correct always surpass those from correct to wrong, which illustrates the improvement in model performance. (2) For the samples where the predictions are altered after PriDe debiasing, the model usually holds low confidence in the original predictions. Meanwhile, the debiased predictions typically rank top-2 in the original prediction distribution, which holds pecially for the larger models. It is because the predictions that the model is uncertain about are more susceptible the model's prior bias for option IDs, which is exactly what PriDe aims to al-
Appendix C and summa-
8

---

Published as a conference paper at ICLR 2024
leviate. (3) Cyclic Perm even alters nigh-coofiidene predictions and brings more drastic prediction changes, e.g., the lowest-ranked option becomes the debiased prediction more frequently than in PriDe. It also remarkably flattens the prediction distribution, i.c., Paebiased (o: (g, x)) is smoother than vee (dg [g, x). This probably because the permutation-base debiasing is essentially a kind of "mixture of experts", where each "expert model" prepends the LLM with option permutation op- eration. It leads to more deliberate and moderately confident predictions (corresponding flatter distribution), but on the other hand, requires more computational overhead than PriDe.
5 RELATED WORK
Large Language Models (LLMs) The realm of large language models (LLMs) has been undergo- ing rapid and significant developments since the launch of GPT-3 (Brown al. 2020). Prominent examples, such as ChatGPT (OpenAl 2022). LLaMA (Tovvron et al. 2203a bj. A Alpaca (Taori et al.) 2023), and Vicuna (Chiang et al. ) have emerged successively over the past year. These mod- els usually boast billions of parameters and are trained to comprehend natural language and follow
human instructions (Ouyang et al. 2022 Wang et al. 2023b).
Multiple Choice Questions (MCQs) As concise task format, MCQs are widely adopted in LLM-centric scenarios. For instance, in the automatic evaluation framework of et al.] (2023);
Zheng et al. (2023b), GPT-4 (OpenAl 2023) is presented with a question and two model answers and is tasked to determine which one is better. Numerous standard language model benchmarks also employ the task format of MCQs, such MMLU (Hendrycks et al. 2020), ARC (Hendrycks) et al. 2020), AGlEval (Zhong et al. 2023). C-Eval (Huang et al. 2023). thus piques our research interest in the robustness of LLMs in the context of MCQs.
Bias and Robustness of LLMs In our study, we use "bias" to refer to the systematic error within LLMs rather than the prejudice in culture or gender studied in the field of LLM safety (Zheng al., 2023a 2024). The bias in language models has always been an important research area closely related to model robustness. For instance, [Zhao et al.) (2021) showed that GPT-3 is sensitive to task instructions and in-context examples, which arises from its bias towards certain answers, Chen et al. (2022); ; Si et al. ( Pan al. (2023) explored how I.LMs' few-shot learning is influenced by the construction of in-context examples. Wang et al. (2023a): Zheng et al. (2023)) found that GPT-4
leans towards the first presented answers and may produce unfair evaluation results.
Contemporaneous with our work, Pezeeskkoour & Hruschka (2023) similarly observed that LLMs are sensitive to option position changes in MCQs and verified position bias as one cause of sensi- tivity. However, their study does not ablate the impact of option IDs, making their investigation on "position bias" less convincing. In fact, our study finds that position bias is less regular and largely depends models and tasks, while token bias is more salient intrinsic cause of LLMs' selee- tion bias and, consequently, poor robustness. Their study is also conducted on very limited models (only GPT-4 and InstructGPPT without open-source models, somewhat hindering reproducibility) and tasks (only three MMLU subtasks, one Big-Bench subtask (Srivastava et al. 2023), and CSQA). In contrast, we draw more general observations through extensive cross-model and cross-task empir- ical evaluation, which further inspires our proposal of the computation-efficieet interpretable, and transferable debiasing method PriDe.
CONCLUSION
This work studies the inherent selection bias of large language models (LLMs), which makes them vulnerable to option position changes in multiple choice question (MCQ) evaluation. Through ex- tensive empirical analyses, we pinpoint that this behavioral bias stems primarily from token bias, where the model priori assigns more probabilistic mass to specific option ID tokens when predict- ing answers from the option IDs, and partially from position bias, where the model favors options presented at specific ordering positions. In particular, token bias is a more salient intrinsic cause of selection bias, while position bias is less regular and depends on models and tasks. Our proposed debiasing method PriDe estimates the model's prior bias for option IDs and separates it from the overall prediction distribution. remarkably mitigates selection bias, with no need for sample la- bels and only negligible computational overhead. We especially highlight PriDe's 's high efficiency, interpretability, and cross- transferability. We hope the empirical analyses in this work and debiasing method can inspire future research the bias and robustness of LLMs.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We have released the evaluation data, code, and experimental results at httpp:://iihub.. com/ facilitate reproducible research. Our experiments were run on A 40GB GPUs (for 70B models) and V100 32GB (for other models). The open-source models and data used in this work are listed in Appendix G Appendix respectively.
chujiezheng/ MCMBgg/LLMMMMMMMMMMMM
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604). This work was also supported by the NSFC projects (Key project with No. 61936010).
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large lan- guage model with state-of-the-art performance, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, volume 33, pp. 1877- 1901, 2020.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in-context learning. arXiv preprint arXiv:2209.07661 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
hammsys.org/blog///ee
30- vicuna/
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: (00:::803000000 054557 2018.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, al. C-eval: A multi-level multi-diseipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08. 08322, 2023.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021. doi: 1 URL
httpss://aclantholoyy 22tprroooooggooog2222111.111111 ta51-1.57
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Maafield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint (iXix:2207.05222 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in neural information processing systems, volume 35, pp. 22199-222111 2022.
OpenAl https://chat. openai. aopenai.com com. 2022.
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
10

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744, 2022.
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in- context: Disentangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8298-8319, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10.18653/v 121///3//1223...// hww.://....nnnnssa....... //aclanthology. . org/oog// 2023. iandings-acl.55
URL https:
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of op- tions in multiple-choice questions. arXiv preprint arXiv:2308. 1483, 2023.
Joshua Robinson and David Wingate. Leveraging large language models for multiple choice ques- tion answering. In The Eleventh International Conference on Learning Representations, 2022.
Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. Measuring inductive biases of in-context learning with underspecified demonstrations. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11289-11310, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1:555s/w/2223..ccoooog..... URL
2wwg//223..c0 acl mocl-long.66 632
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga- Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseCQ:: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume (Long and Short Papers), pp. 4149 4158, Minneapolis, Min- nesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/11/N19-1 1421. URL attts://aclantholn
org/N19-1/111
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followee llama model. 2023.
https://github..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09988888, 2023b.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 17926, 2023a.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self- generated instructions. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023b. Association for Computational URL htttts::/aclaattoolll. www.gyy..gg.2..../.
Linguistics. doi: 10. 12t5533/1/1222.aalllooo
scl-long.-77 754
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of th-thught prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 04824-24833 2022.
11

---

Published as a conference paper at ICLR 2024
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp- 12697-12706. PMLR, 2021.
Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation with sequence likelihood contrastive learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, 2023a.
URL httpp:://aclatthhlleee .rgg org//023....2 findings-acl. 65
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. arXiv preprint arXiv:2401.18016 2024.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprini MxXi::300.006683. 2023b.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, arXiv preprint arXiv:2304 2000:266444004, 2023.
12

---

Published as a conference paper at ICLR 2024
A PROMPTS USED IN EXPERIMENTS
The following multiple choice questions electrical engineering. You should directly answer the question by choosing the option.
in-context examples feu-shot)
Question:
In an latch built from NOR gates, which condition is not allowed
Options:
A. S=0, R=0
B. S-0, R=1
S=1, R=0
D. S=1, R=1
Answer: D
Figure Input formats for open-source models (e.g., 1lama(-2) and vicuna). The black text templated imput. The next-token prediction probabilities of the option IDs at the red text is used as the observed prediction distribution. Note that for all the open-source models, we do not prepend the input text with bos_token.
"role": "system", "content": "The following multiple choice questions about electrical engineering. You should directly answer the question by choosing the correct option."
ii-context examples (if few-shot) ]
"role": "user", "content": """Question: In SR latch built from NOR gates, which condition is not allowed
Options:
A. S-0, R-0
B. S-0, R-1
C. S-1, R-0
D. S-1, R=1
Answer: nn*
"role":
"assistant", "content"; "D"
The black text is templated request. blue text in-context or test samples. The red text is the expected API-retumed result. The orange text is the subject in MMLU (57 different ones in total).
Figure 7: Input formats for 3pt-3.5-turbo.
5-turbo.
The following are multiple choice questions about electrical engineering. You should directly answer the question by choosing the correct option.
iin-contxt examples (if few-shot) ]
Question:
In an SR latch built from NOR gates, which condition is not allowed
Options:
S=0, R-0
S=0, R-1
S=1, R=0
S=1, R=1
Answer: S=1, R=1
Figure 8: Input formats for open-source models with option DDs removed. We compute the likelihoods of options, normalized by their lengths, and the maximum the model prediction.
13

---

get the Published as a conference paper at ICLR 2024
"system", "content"; "The following are multiple choice questions about electrical engineering, should directly answer the question by choosing the correct option."!
"role":
iin-context examples few-shot) ]
"role": "user", "content": """Question: an latch built from NOR gates,
which condition is not allowed
Options:
$=0, R=0
s=0, R=1
8-1, R=0
S-1, R-1
Answer:
"role":
"assistant",
"content": "S=1, R=1"
Figure 9: Input formats for gpt-3, 5-turbo with option IDs removed. We require it to generate the whole selected option, which is then compared with the golden answer.
The first API call
"role": "system", "content": "The following are multiple choice questions about
electrical engineering, You should reason in step-by-step manner to get the
right answer.
"role": "user", "content": """Question: In an SR latch built from NOR gates,
which condition is not allowed
Options:
A. S-0, R=0
B. S=0, R=1
C. S=1, R=0
D. 8=1, R=1
Answer: Let's think atep step:""> }
"role": "assistant", "content"; "[ thought ressoning process ]"
The second API call
"The following multiple choice queations about electrical engineering. You should in atep-by-step
"role": "syatem", "content":
right answer."
"role": "user", "content": """Question:
latch built from NDR gates,
which condition is allored
Options:
A. S=0, R=0
B. S=0, R=1
S=1, R=0
S-1, R=1
Answer: Let's think step by step:""a
"role": "assistant", "content": "[ thought reasoning process } above,
"role": "assistant", "content": "Given all
of
question is:"
"role": "assistant", "content";
Figure 10: Chain-of-Thoure prompt for gpt-3. 5-turbo, generally following the implementation of OpenAl Evals, where the decoding temperature is set to 0.
14

---

Published as a conference paper at ICLR 2024
B EVALUATION RESULTS OF SELECTION BIAS
0-shot
5-shot
0-shot, Removing IDs
25
Iaara 13B
Fama 308
-10-
20
Tama- 698
vicuna-v1 3-7B
20
1000
3-333
10
Social Sciance
STEM
Humanatioo
Cehero
MMLU
ABC
CSDA
Social Science
STEM
Aumanalion
Others
Metu
ARC
CR04
Social Science
STEM
Humantlied
Othora
MMLU
ARC
CSDA
Figure 11: Selection bias of 1lama and vicuna-vi 3. models. Y-axis: the recall score (%) normalized by subtracting the overall accuracy.
15

---

Published as a conference paper at ICLR 2024
O-shot
5-shot
0-shot, Removing IDs
Bame 2-78
OFF
llama-2 133
llama-2-788 70.
-10
Itans 3-hak-7B
10
-0
Bamn 2 2--chh--138
-10
Ilama 2 103--haa--00 708
Vicuna-1.5577 5-78
vicuna 5-138
100
STEM
Stcial Sciance
Humanitia
Otfere
MALJ
ABC
CSDA
STEM
Secial Science
Humanitan
Otfhrr
MELL
ARC
cogA
Social Scierice
STEM
Humentte
Othera
HMLU
CoOA
Figure 12: Selection bias llama-2 2, 1lama chatht and vicuna-v1.5 models.
16

---

Published as a conference paper at ICLR 2024
0-shot
S-shot
0-shot, Removing IDs
75
falcon-7 78
20
talcon-888
ralcan-nns--7 78
falcon- -int--8
20
10
000 3.5bbbb
-10
STEM
Social Scierce
Humanitees
ariers
MALU
ARC
eSDA
Social Scienice
STEM
Humenittes
Offiess
Metu
ARC
csgA
Social Sciencee
STEM
Humerttes
Others
MiL
ARC
CSDA
Figure 13: Selection bias of falcon and gpt-3.5-turbo b-tubo models. Note that conjecture falcon-7B and falcon-inst-7B are undertrained. This evidenced by: (1) they exhibits abnormally pronounced selection bias various benchmarks, as reflected their Y-axis scales, their performance on MMLU almost random selection (even 5-shot, see Table and Table in Appendix C and (3) their recalls of position D are close to zero (see Figure 17 in Appendix
17

---

Published as a conference paper at ICLR 2024
0-shot
D-shot, Removing IDs
D-shot
O-shot, Removing Ds
Lame-11
20
-20
2000
4000
10-
-20
10-
1000000
430
20
Hama-008
10
-10
20
vicumen 1-78
10000
400g
sttunee 3-330
20
2000
2000
2000
38
2000
2000
0000
5000
0000
2000
80
198
lume-22200
100000
400000
900
20
-10
-10-
2000
20
-10
Social Rlinnce
Offerr
Seesal Stannce
Chiere
HIEL
Soccal Scarce
STMM
Hoseenteer
Offery
Social Sraence
STTER
Cuters
ABc
Figure 14: Selection bias evaluated under the 3-option setting, where randomly sampled three options (including the golden answer) from the original candidate options. Removing option IDs notably reduces selection hias most L.I.M, except 11ama-30B, vicuna-vi. 3-33B, and 1lama-2-13/70B
18

---

Published as a conference paper at ICLR 2024
0-shot
D-shot, Removing IDs
D-shot
O-shot, Removing Ds
800
380
-20
20
10
600
10
5000
100
300000
-20
20
Farre
%
100
%
190
%
-10
Hame-553
1000 Spger
S000000 500000
600
380
300 2000
100
200
-20-
20
0008000 G00008
68
2008
3
80
vitune 5-330
50000
300
2008
2000
3009
1000
100000
2000
500000
5000
VL
6
80000 yoou
2008
5000 20008
1000
Rac 888
20880
10000
10
20000
1000 2008
2008 8000)
2008
000
100
10
10
1009 200
50000 2000
M00S 5008
80000000
20
100
-10-
20
5-
508
2001
1003
1000
100
Social Plinnc
Offerr
coe
Secsal Stannce
STeN
Humanatie
Chbere
HILL
Seccill Scarrv
STMM
Offeryy
etc
Social Sraence
SITER
Cutere
ABcc
Figure 15: Selection bias evaluated under the 2-option setting, where randomly sampled two options (in-
cluding the golden answer) from the original candidate options. Removing option IDs notably reduces selection bias most L.MMs, except 1lama-2 13B and falcon (-inat) -7B (the latter likely he undertrained).
19

---

Published as a conference paper at ICLR 2024
C SUPPLEMENTARY EXPERIMENTAL RESULTS
MMLU
25
10.12
2.22331 0
1.
11.3. 2)
10.1.2.3
11 0.
12. 3.0,
0.
Sta (%5
-20
-15
10
-5
MMLU
Accuracy (8)
ARC
-55
Recall Std
-10
ARC
20
4 Accuracy (8)
15
10
liama--70 7B
Hama- 13g
llam--308
Btam
Ilama-2-78
llamb-2 2-138
Haam 708
Ilams 2-chat.7
Ilama chat- 138
Hamm chat 708
vicuma 3-78
vicuna- v 138
vicura vi 3-336
vicuna 5.78
vicura 3.5
falcon-7j
falcon
falcon-irat-1 7B
falcon-inst-Ang
average
Figure 16: Debiasing results (0-shot, MMLU and ARC) of Cyclic Permutation with different selected permu- tations I. The selection of permutations ensures one pairing between each option ID and option content o,. We do not observe remarkable differences between these selections.
20

---

Published as a conference paper at ICLR 2024
lama-78
llama-138
lama-308
Estimated Priir
60
0.2
40
0.2
Pecall
0.1
20 0.1
20
0.0
Ilama 65B
vicuna-v1.5-78
vicuna v1.5 138
Estinated Prier
0.3
0.4
60
-60
Rocal (16)
0.2
40
0.1
20
20
0.0
0.0
0.0
vicuna-v1.3-77
vicuna-v1.3-12
Souna-v1.3-338
Estinateed Prter
0.3
60
Recall 196
0.2
0.2
0.1
20
20
0.0
0.0
0.0
1lama-2-708
Essinated Prior
75
60
-60
0.2
0.2
40
Reca
0.1
20 0.1
20
0.0
0.0
0.0
lama-2-chat-78
Ilama-2-chat-138
lama-2-chat-708
Estimated pilor
60
0.4
60
40 0.2
500
0.2
0.2
20
20 0.1
23
0.0
0.0
0.0
falcon-7B
falcon-408
gpt-3.5-turbo
Estimated Frior
0.3
60
0.2
40
0.1
20 0.1
20
0.0
10%
20%
Empirical
falcon-inst-77
falcon-inst-40B
0.4
Estmated Prer
60 0.3
0.2
40 0.2
20 0.1
Empirical
Empirical
Figure 17: Prior preferences estimated by PriDe with varying sizes of test samples (2%, 5%, 10%, 20%) versus the empirical selection bias of the original model predictions (0-shot MMI.U..
We observe that the estimated priors usually correlate strongly with the empirical selection bias, except in a few where the correlation is moderate. like in 11ama-2-7B, 1lama-2-char 22aat-2- and vicuna-v1 3- 33B. Specifically, the estimated prior probabilities may necessarily align perfectly with the frequency-hased recalls. Even in the cases where rankings are the same, the scales may still differ, such as 11ama-7B/11
Our preliminary investigation suggests that may he related to the calibration of L.MSs (Kadavath et al. 2022) We conjecture that an ideal alignment requires the model to be well-calibrated, i.e. the probabilistic predictions match up with frequencies of correctness (Jiagg al. 2021
We leave this research question for future work.
21

---

Published as a conference paper at ICLR 2024
Table 3: Breakdown of the debiasing results (0-shot), measured by standard deviation recalls (RStd) and accuracy (Acc).
Default RSid
Removing
Cyclic Perm RStd
PriDe (5%) RStd
PriDe (40%) RStd
PriDe (80%)
Models
Methods
Acc
RStd
Acc
Acc
Acc
Acc
RStd
Acc
MMLU
Cost:
x1
Cost: x1
Cost: x4
Cost: x1.15
Cost: x2.2
Cost: x3.4
11ama-78
18.5
33.7
5.2
33.4
4.7
43.5
5.5
35.3
5.1
38.4
4.9
41.8
11ama-138
2.6
2.9
40.4
2.6
11ama-308
8.5
53.1
7.0
49.3
3.2
57.2
3.6
54.0
3.4
55.1
3.1
56.4
11ama-658
8,3
56.9
3.6
51.4
3.1
61.8
1.9
58.5
1.2
59.8
2.3
61.1
11ama-2-78
23.0
11.2
37.0
40.2
5.5
11sma-2-138
7.5
50.8
9.6
44.7
4.5
54.7
2.3
51.8
2.2
52.9
3.5
54,0
1lama-2-708
4.1
64.0
4,9
58.9
2.5
66.6
5.7
64.3
3.5
65.2
2.1
66.2
11ama-2-chat-78
5.7
48.7
11ame-2-chat-13B
14.3
52.1
2.9
51.0
3.9
55.4
8.2
53.5
4.7
54.2
2.9
55.0
1lama-2-chat-00B
12.8
59.4
7.0
53.4
2.5
62.1
9.0
60.2
6.1
60.9
3.3
61.7
vicuma-v1.3-77
vicuna-v1.3-13B
10.5
50.2
2.0
49.8
4,4
53.5
3.6
50.8
3.5
51.8
4.1
53.0
vicuna-r1.3-385
8.2
57.0
6.0
57.1
2.2
60.6
7.0
57.6
5.0
58.6
2.9
59.9
vicuns-v1.5-77
48.7
7,9
46.5
5.2
50.5
vicuns-v1.S-138
8.2
54.4
4,3
53.1
4.8
56.6
2.5
55.2
2.2
55.8
3.7
56.2
falcon-7B
24.2
24.7
12.6
30.7
4.3
33.3
4.2
27.11
3.4
29.2
3.8
31.9
falcon-40B
44.2
55.4
2.6
52.3
53.4
falcon-inst-78
28.7
25.0
13.7
29.9
1.7
34.4
7.3
28.1
4.8
30.4
2.2
33.0
falcon-inst-4o
51.5
4.8
45.1
4.7
53.7
1.7
51.8
1.0
$2.5
3.3
53.3
gpt-3.5-tune
67.2
1.0
66.7
1.1
2.8
68.0
2.2
68.5
69.3
A Average
-6.4
-2.1
-8.7
+4.9
-7.6
+1.2
-8.9
+2.6
-9.1
+4.1
ARC
Cost: x1
Cost: x1
Cost: x4
Cost:
x1.15
Cost:
x2.2
Cost: x3.4
11
9.9
38.6
4.9
36.0
2.7
48.0
4.0
40.1
2.3
43.3
2.6
46.4
1lama-138
51.3
2.3
27.8
59.6
7.7
51.6
4.2
55.2
58.3
11ama-308
4.6
67.6
5.3
63.5
2.8
73.0
3.7
67.1
2.2
68.7
2.0
71.6
11ama 658
8.8
72.8
4.4
67.6
1.8
77.9
1.9
73.0
1.7
74.6
1.6
76.9
11ama-2-78
27.4
36.0
8.5
40.7
55.8
7.8
45.8
4.5
49.4
2.1
53.7
11ama-2 22113
6.0
62.9
11.5
$3.0
L.4
66.3
4.8
63.9
3.1
64.9
2.1
66.0
11ama-2-708
80.7
3.0
78.7
2.3
5.2
80.9
3.6
2.3
83.1
11ama-2-chat-7B
12.4
56.5
4.8
S3.7
1.8
59.7
7.3
57.5
4.7
58.4
2.6
59.9
11ama-2-chat c3B
13.7
64.4
9.2
64.0
3.0
69.6
8.5
66.1
5.6
67.2
3.2
68.9
11ama-2-chat-700
78.0
7.3
71.7
3.0
81.1
7.4
78.2
5.6
79.3
3.5
vicuma-r1.3-7B
53.5
3.1
S3.6
55.7
4.2
53.8
54.7
55.1
vicuma-v1.3-138
8.3
62.9
4.3
61.5
2.9
66.8
4.7
64.1
2.0
64.6
1.7
66.2
vicuma-r1.3-399
72.9
3.6
74.3
2.2
4.1
2.1
vicuna-v1.5-78
9.8
58.5
3.8
$6.5
1.9
62.1
5.8
59.0
4.2
60.3
2.4
61.5
vicuma-v1.5-135
6.2
69.7
3.9
67.7
L.3
71.1
3.7
69.8
2.8
70,2
1.8
71.0
falcon-78
29.2
24.9
29.5
4.7
3.7
falcon-40B
6.8
62.9
6.4
55.7
3.3
67.4
5.6
64.1
4.0
65.4
3.2
66.7
falcon-inat-778
23.4
25.2
13.0
29.7
3.5
37.4
10.2
27.8
6.5
31.2
3.2
35.8
falcon-inst-440
56.1
65.3
gpt-3.5-turbo
3.3
84.3
0.6
84.9
L.6
85.0
2.3
84.2
2.1
84.1
1.7
84.9
A Average
-5.1
-2.9
-8.6
+5.7
-5.6
+1.3
-7.3
+2.9
-8.5
+4.9
CSQA
Cost: x1
Cost: x1
Cost:
Cost: x1.2
Cost:
x2.6
Cost: x4.2
36.4
7.9
26.6
53.0
38.5
4.9
45.2
11ama-198
19.9
55.3
1.6
39.4
2.3
64.2
9.5
59.4
6.7
60.6
3.7
63.1
11 ama-300
11.2
66.0
7.6
52.6
2.7
71.1
9.0
65.8
6.8
68.2
3.9
70.0
56.9
11-m
28.4
31.9
9.9
32.2
0.8
56.0
9.9
41.5
6.0
46.3
2.6
52.9
11ama-2-13B
10.2
57.0
5.8
43.8
1.3
66.6
7.4
58.4
5.7
61.4
2.3
64.8
7.3
70.9
4.9
65.5
75.5
71.6
4.7
73.0
74.8
1lama-2-chat-7B
15.2
56.5
7.4
$3.5
1.2
64.8
10.1
59.5
6.3
61.6
2,4
63.4
11ama-2-chat-133
9.8
64.0
7.5
$4.2
2.01
68.8
6.3
65.2
3.6
66.4
1.3
68.0
11sma-2-chat-705
74.6
4.4
56.2
3.1
74.8
75.5
76.4
vicuma-v1.3-7B
8,3
56.9
0.9
$7.2
1.8
62.8
4.0
57.9
2.4
59.4
1.4
61.5
vicura-v1.3-13
12.9
63.4
S.0
63.2
3.8
69.9
6.6
65.3
5.3
66.9
4.0
68.9
vicuma-r1.3-39g
68.9
8.8
70.1
72.8
5.1
2.7
72.0
vicuna-v1.6-78
9.1
60.4
6.2
$6.4
0.8
64.7
4.1
61.1
2.9
62.3
1.5
64.2
vicuna-vi.5-155
2.8
67.1
5.6
62.1
1.3
2.5
67.1
2.1
68.4
1.4
69.6
20,9
8.2
30.8
3.3
28.9
6.2
20.4
3.7
23.5
2,9
27.2
falcon-408
15.9
61.4
6.0
54.1
1.8
70.4
9.2
63.2
5.8
65.8
2.8
69.0
falcon-inat-78
25.7
21.7
13.1
34.8
3.3
39.4
8.7
25.5
4.9
29.9
3.2
36.1
falcon-inatt-40B
6.5
67.2
4.5
54.3
1.4
71.4
3.1
66.9
1.9
68.7
70.6
3ot-3.5-turis.
2.2
76.1
2.1
76.1
1.2
78.0
2.2
76.4
1.9
77.0
1.6
77.8
A Average
-6.2
-7.0
-11.2
+7.9
-6.9
+1.7
-8.9
+4.0
-10.7
46.6
22

---

Published as a conference paper at ICLR 2024
Table 4: Breakdown the debiasing results (5-shot).
Default RStd
Removing IDs
Cyclic Perm RStd
PriDe (5%)
PriDe (40%)
PriDe (80%)
Models
Methods
Acc
RStd
Acc
RStd
Acc
RStd
Acc
RStd
Ace
MMLU
Cost:
x1
Cost: x1
Cost:
Cost:
x1.15
Cost:
x2.2
Cost:
x3.4
11ama-7B
12.5
34.5
2.8
39.3
4.8
44.5
2.6
35.4
2.4
38.6
4.2
42.4
11 11ama-138
14.3
43.5
2.6
45.0
4,7
51.1
4,6
45.7
4.4
47.7
4,4
49.8
2.9
55.9
2.8
60.6
11ama-658
1.8
63.5
3.2
62.0
2.4
65.5
1.8
63.5
1.4
64.2
2.0
65.1
11ama-2-78
44.6
4.1
45.7
4.7
7.7
45.8
6.3
47.4
5.1
49.1
11sma-2-138
4.1
55.5
5.6
S4.7
57.2
3.6
54.5
3.6
55.4
3.9
S6.6
11ama-2-708
1,9
69.0
3.2
70.2
3.5
70.5
2.4
68.9
0.4
69.4
2.2
70.2
11ama-2-chat-77
11.1
46.2
3.6
49.3
4.2
50.7
5.7
47.1
4.5
48.5
4.0
49.9
11sma-2-chat-133
53.8
2.8
56.5
5.7
54.4
$5.2
4.2
S6.4
11ama-2-hhat-7CB
7.7
63.0
3,6
65.0
2.3
64.8
3.5
63.5
2.0
63.9
1.7
64.6
vicuma-v1.3-72
7.5
46.6
5.9
45.9
4.7
47.0
48.2
3.7
49.5
vicuna-r1.3-13B
51.6
0.9
52.4
55.7
4.2
52.3
3.3
53.6
3.0
$5.0
vicuna-v1.3-33B
5.4
59.2
3.2
60.7
2.7
62.4
4.3
59.2
3.3
60.3
2.7
61.7
vicuns-v1.5-7E
50.0
5.3
50.8
4.4
50.6
3.8
$1.5
3.7
$2.6
vicuna-v1.S-13B
55.9
S7.9
58.2
5.4
55.7
4.7
56.7
4.4
57.7
falcon-78
27.3
27.1
3.4
34,0
4.1
36.2
7.8
30.2
6.6
32.4
5.0
34.9
falcon-408
55.6
1.9
$1.0
56.2
2.7
56.7
3.4
57.
falcon-innt-778
27.8
25.0
4.2
31,6
1.0
33.5
4.3
28.8
2.7
30.4
1.2
32.5
fslcon-inst-400
9.9
54.1
3.6
52.4
4.4
56.2
2.3
54.5
2.5
55,1
3.8
55.8
3pt-3.5-turbo
70.9
0.7
71.8
2.1
71.1
71.9
0.3
72.8
Average
+1.2
-5.8
-5.3
-6.1
-6.1
+3.3
ARC
Cost: x1
Cost: x1
Cost: x4
Cost:
x1.15
Cost:
x2.2
Cost:
x3.4
11ama-78
18.8
38.8
6.7
46.2
1.9
$1.6
4.7
39.6
3.3
44.1
1.9
49.5
11ama-138
12.9
55.4
1.5
56.0
2.6
10.2
6.9
58.7
3.7
60.8
11ama-308
3.2
73.0
3.1
74.8
2.2
76.7
3.7
73.4
2.4
74.9
2.1
76.0
1lama-658
2.1
78.5
2.3
80.2
0.9
81.8
4,0
78.4
2.6
79.8
1.3
81.4
11ama-2-78
2.5
11ans -2-18B
4.5
66.8
6.6
68.4
2.3
69.8
4.9
67.2
3.8
68.5
2.9
69.0
11ame-2-708
1.4
84.4
2.0
85
0.6
87.2
3.1
84.1
1.8
85.3
0.7
86.3
11ama-2-chat-78
1
9.2
68.1
8.3
70.0
3.7
71.1
4.9
68.6
3.1
69.4
3.2
70.5
11ame-2-chat-700
5.8
80.0
5.3
81.4
2.8
83.9
4.3
80.9
3.4
81.8
2.8
83.3
vicuna-v1.3-78
57.4
1.7
55.9
$7.3
$7.8
2.9
$8.9
vicuna-v1.3-13B
5.6
65.8
3,9
67.0
2.2
68.4
5.4
66.0
3.1
67.2
1.4
67.6
vicmaar11--339
3.8
74.7
2.4
78.0
3.5
79.2
3.2
74.9
2.4
76.
2.5
78.1
vicuna-v1.5-78
62.4
61.6
vicuna-v1.5-13B
3.7
71.6
4.7
72.8
2.5
72.6
4.8
71.4
2.9
71.9
1.5
72.6
falcon-78
24.0
29.5
6.1
34.7
5.3
38.3
10.5
30.0
8.9
33.0
6.6
36.4
falcon-40B
69.9
2.8
66.4
72.5
2.0
69.2
70.8
2.0
72.0
falcon-inat-78
33.2
24.4
6.4
33.5
3.4
35.3
4.6
29.3
2.0
31.4
2.1
33.7
falcon-inst-400
8.9
66.2
7.1
64.3
1.9
2.9
68.9
1.8
69.8
1.6
69.9
gpt-3.5-turbo
86.6
0.9
86.1
1.6
86.8
87.3
87.7
A Averaee
4.66
+1.6
-6.1
+4.5
-3.7
+0.7
-5.2
+2.1
-6.2
+3.6
CSQA
Cost: x1
Cost: x1
Cost: x5
Cost: x1.2
Cost: x2.6
Cost: x4.2
11ama-7B
22.4
42.3
5.0
49.8
1.4
58.5
5.6
44.6
3.8
49.4
2.2
55.4
11ama-138
62.3
3.4
66.4
62.9
4.1
64.7
2.5
67.1
11ama-308
4.8
73.6
2.0
76.5
2.8
75.1
3.4
74.0
3.1
74.7
2.8
75.0
11 11ama-658
4.5
76.2
1.9
77.9
1.3
78.7
3.6
76.1
2.7
77.1
1.8
78.3
11ama-2-78
21.7
56.4
5.9
64.6
2.4
65,2
8.8
58.0
6.5
60.2
3.8
63.8
11ama-2-155 139
7.3
68.2
2.3
70.7
4.2
71.4
5.3
68.5
4.8
69.8
4.5
71.2
11ame-2-708
4.2
77.9
3.3
79.8
1.9
79.3
79.8
1.7
80.8
1lama-2-chat-7B
59.8
3.9
67.5
1.4
65.5
12.3
60.5
8.3
62.3
3.3
64.2
11ama-2-chat-13B
3.7
68.3
3.5
72.2
3.1
71.7
6.4
69.6
4.9
70.7
3.6
71.5
11ama-2-chat-7P
11ama-2-cce
4.9
76.7
2.2
78.0
2.0
79.9
4.7
76.8
3.1
78.1
2.0
79.5
vicuna-r1.3-7B
14.4
58.5
5.0
62.1
1.7
64.7
6.3
60.2
3.9
61.8
1.4
63.7
vicuma-vi.3-133 3-B
7.5
67.1
4.4
70.3
3.8
69.9
6.8
66.7
5.2
68.0
4.1
69.0
vicuna-r1.3-38
4.2
74.8
7.4
77.1
6.0
74.3
4.4
74.7
1.7
75.9
vicuna-r1.6-7B
13.0
61.7
9.0
67.1
0.8
66.4
7.8
62.9
5.2
64.1
2.4
65.8
vicuna-vi.5-135
5.7
70.6
2.5
74.3
2.0
73.8
5.4
70.4
4.5
71.8
3.0
73.2
falcon-7B
35.7
19.8
3.2
42.2
13.1
24.7
8.4
26.7
3.2
29.1
falcon-40B
8.0
69.1
2.6
72.1
1.0
73.8
2.5
70.6
2.2
71.9
1.4
73.4
falcon-inst-77
31.4
21.1
4.0
39.6
3.5
34.7
7.1
24.5
4.5
27.9
3.0
32.6
falcon-inat-408
4.9
71.1
69.7
2.9
70.5
2.5
71.9
5pt-3.5-turbo.
4.0
79.6
2.7
82.1
1.6
82.1
2.8
80.5
2.3
80.8
1.9
81.8
à Average
-5.5
+4.0
-9.5
+5.5
-5.6
+1.1
-7.3
+2.7
-9.0
+4.6
23

---

Published as a conference paper at ICLR 2024
MMLU
20
ARC
20
CSQA
20
Bama-1
Hamm- 138
Itamar 308
laama453 458
Blama Bama-2-78 2-78
Hama-22 2-138
Hamm 2-708
Ilama-2 2-zhstt-t
ilama -aaa---do 2-ch----1at
Ilama 2-ca----hh---2 708
vicuna 3-78
vicuna-422 3-138
vicura-22 9-330
vicuma 5.78
vicuna- 5.11.9
falcon-1
falcon-408
talcon-inst- 78
falcon 408
opt 3.5 t.tubb
average
Figure 18: Selection bias before (dashed bars) and after (solid bars) shuffling the default options. We observe consistent and remarkable changes in the recall standard deviation (Y-axis).
MMLU
ARC
CSQA
-1000
-6-
Recall Std (96)
7.5
-4-
55.0
-2-
Full Perm Cyclic Perm
-2
Full Perm
Cydk Perm
-25
Cyclc Perm
0
0.0
Cyclic
Accuracy (%)
4-
PriDe
4
4
2
Full Perm
2
Full
Cyclic Perm
Cydic Perm
PriDe
PriDe
x1
X2
x8
x24
x2
x4
x24
x1
x2
Figure 19: Debiasing results (5-shot) under varying computational costs.
Recall Std (%)
Accuracy (%)
STEM
-8.7
-3.4
-5.0
-5.3
-4.1
+1.3
+0.9
+0.3
+0.2
-1.4
1.5
Source Dornin
1.0
Social Science
3.2
5.0
2.8
3.7
2.2
+1.5
+1.3
+0.4
+0.7
+0.4
-4
0.5
Humanities
-4.0
-2.2
-6.6
-2.7
-2.8
+1.4
+0.9
+0.4
40.1
-1.5
-6
Others
3.6
4.1
2.4
5.1
+1.8
+1.5
+0.6
+0.7
+0.1
0.5
-1.0
-1.5
-1.0
2.00
-.7
+0.8
+0.7
40.7
+0.8
-1.5
STEM
Social Science
Humanities
Others
ARC
Social Science
STEM
Humanities
Others
ARC
Target Domain
Figure 20: Cross-domain debiasing results (5-shot). We notice slightly degraded transferability under 5-
shot setting, which can be expected due to both the impact of in-context examples themselves and the alteration of co-textex examples.
24

---

Published as a conference paper at ICLR 2024
MMLU
ARC
CSQA
Recall Std (%6
99
-6
66
Perm
Perm
Perm
-3
Perm
Perm
33
Perm
PriDe Perm
Prize Perm
PriDe 3 Perm
Accuraacy (%)
3 Perm
3
PriDe +22 Perm
Perm
2
2 Perm
Perm
Perm
PriDe Perm
1
PriDe Perm
PriDe Perm
PriDe Perm
x1
x1
x3
x1
x2 x3
Figure 21: Debiasing results (0-shot) of combining PriDe with two low-cost permutation- based debiasing al- ternatives 2/3 Perm. Specifically, we randomly sample another or permutations from standard Cyclic Perm I. which forms subset C I containing or permutations in total. For instance, one possible 2-Perm is {00, 2, 3), (1, 3,3,0) })} and one possible 3-Perm is {0, 1, 3), (2, 3, 0, 1), (3, 0, 1) In 2/3 Perm, we debias the model prediction with Z using Equation We control the costs of 2/3 Perm via the ratio 8 of the debiased test samples, where we take 3 E (0%, 10%, 20%, 40%, 60%, 80%, 100%].
We combine PriDe with 2/3 Perm in three steps: (1) We first estimate the prior Parior with 5% test samples D. (these samples are meanwhile directly debiased via Cyclic Perm). (2) We then debias a 5% samples through 2/3 Perm, with I and using Equation 1 (3) We finally debias the remaining
samples D, with the estimated prior Ppeior, using Equation & take E (5%, 10%, 20%, 40%, 60%, 80%].
MMLU
ARC
CSQA
Recall Std (96)
-6
-8
-4
2 Perm Perm
Perm
Perm
Perm
Perm
-2
PriDe Perm
PriDe 3 Perm
PriDe Perm
& Accuracy 136)
P
2
PriDe Perm Perm
1
2 Perm
1
Perm
1
PriDe + Perm
PriDe 2 Perm
PriDe
PhDDe Perm
x1
x2
x3
x1
x2
x3
x1
x2
x3
Figure 22: Debiasing results (5-shot) of combining PriDe with 2/3 Perm
25

---

Published as a conference paper at ICLR 2024
Table 5: Breakdown of the model prediction changes (0-shot MML.U) after PriDe (single run, prior estimated with a 5% samples and statistics on the remaining samples) or Cyclic Permuta- tion debiasing (statistics on the same samples PriDe).
Upper:
We show the proportions of test samples where the model predictions change after debiasing (Changed). Among these samples, count the proportions of predictions changing from wrong to wrong (F + F, but no such samples for all the models), from correct to wrong (T - F), and from wrong correct (F + T).
Lower: We show the prediction probabilities of the predicted answers before and after debiasing, i.e, max; Pobserved (d: g x) and max; Pdebiased (o: q, æ), averaged over only prediction-changed samples or all the samples.
PriDe Debiasing
Cyclic Permutation Debiasing
Models
Changed
F F F
T
F
F T
Changed F F
F
T T
T
1lama-7B
27.2%
43.7%
25.6%
30.7%
50.6%
38.1%
21.3%
40.7%
1lama-138
24.9%
41.7%
26.3%
32.0%
54.8%
35.1%
20.4%
44.5%
1lama-300
11.8%
43.4%
25.2%
31.4%
24.6%
35.0%
24.2%
40.8%
11ama-68 65B
17.0%
39.9%
26.0%
34.2%
29.7%
33.6%
25.0%
41.4%
1lama-2 2-7B
37.2%
44.5%
22.2%
33.3%
45.6%
35.7%
20.2%
44.1%
11ama-2-11 2-3BB
11.2%
39.4%
27.1%
33.5%
28.6%
35.0%
25.8%
39.2%
1lama-2-708
9.7%
37.9%
30.3%
31.8%
17.3%
27.1%
28.9%
44.0%
1lama-2-chat-77
13.1%
39.6%
28.9%
31.5%
34.4%
37.0%
26.6%
36.4%
1lama-2-chat-11
14.5%
37.9%
26.6%
35.5%
33.6%
37.2%
26.5%
36.3%
1lama-2-chat-70B
5.5%
29.3%
28.4%
42.3%
22.4%
29.6%
41.4%
vicuma-r1.3-7B
15.9%
46.8%
25.7%
27.5%
33.0%
41.0%
25.5%
33.4%
vicuma-v1.3-13B
17.7%
42.9%
27.7%
29.3%
32.5%
37.4%
26.5%
36.1%
vicuma-v1.3-
4.0%
39.7%
26.1%
34.3%
23.0%
35.8%
24.7%
39.6%
vicuma-v1.5-70
13.8%
43.5%
27.5%
29.0%
33.4%
38.3%
27.5%
34.2%
vicuma-v1.5-13B
12.6%
37.9%
27.7%
34.4%
27.7%
39.5%
25.9%
34.6%
falcon-7B
50.0%
48.4%
23.5%
28.1%
65.9%
43.5%
21.5%
34.9%
falcon-40B
10.7%
39.7%
28.7%
31.6%
33.3%
34.9%
27.0%
38.1%
falcon-inst-7B
46.4%
47.4%
23.6%
29.0%
71.7%
44.4%
21.3%
34.2%
fal falcon-inst-40B
16.6%
36.8%
30.5%
32.7%
34.9%
37.2%
28.2%
34.6%
4.0%
24.8%
34.5%
40.8%
15.1%
27.8%
29.1%
43.1%
Original Proh
PriDe-debiased Prob
Cyclic-debiased Prob
max, Pobserved (d. (g, x)
max; Parhissnaa (o:]g,
max; Pashisona (0:|g,
Models
Over
Over
Over
Over Cyclic-
Over
Over
Over
PriDe-
Cyelic-
PriDe-
Changed
Changed
Changed
Changed
1lama-7B
0.289
0.306
0.314
0.283
0.304
0.255
0.257
1lama-13B
0.341
0.434
0.453
0.329
0.421
0.261
0.270
1lama-30B
0.313
0.361
0.497
0.308
0.493
0.271
0.309
1lama-65B
0.332
0.358
0.514
0.328
0.514
0.271
0.315
1lama-2-7B
0.311
0.337
0.362
0.295
0.344
0.260
0.266
1lama-2-138
0.312
0.360
0.459
0.313
0.455
0.271
0.297
1lama-2-70B
0.345
0.411
0.612
0.353
0.617
0.286
0.345
1lama 2-chat-7B
0,503
0.627
0.493
0.743
0.304
0.345
11ama-2-hatt-13B
0.538
0.631
0.780
0.546
0.767
0.313
0.362
11ama-2-chat-708
0.496
0.658
0.810
0.483
0.795
0.317
0.378
vicuna-v1.3-7B
0.339
0.425
0.578
0.337
0.579
0.285
0.326
vicuma-v1.3-139
0.421
0,510
0.683
0.398
0.677
0.295
0.347
vicuma-v1.3-33B
0.388
0,562
0.756
0.382
0.757
0.304
0.372
vicuna-vi.5-7B
0.436
0,537
0.412
0.676
0.299
0.345
vicuma-v1.5-138
0.450
0,529
0.423
0.703
0.307
0.365
falcon-7B
0.299
0.312
0.319
0.286
0.294
0.251
0,251
falcon-40B
0.339
0.417
0.556
0.345
0.553
0.280
0.320
falcon-inst-7B
0.376
0.413
0.419
0.330
0.341
0.255
0.255
falcon-inst-408
0.387
0.455
0.593
0.366
0.577
0.284
0.326
5pt-3,5-turbo
0.463
0.630
0.860
0.486
0.857
0.318
0.408
26

---

Published as a conference paper at ICLR 2024
Table 6: Breakdown of the model prediction changes (0-shot MMLU) after PriDe or Cyclic Per- mutation debiasing (cont.). We count where the debiased/original predicted answers are ranked in the original/debiased prediction distributions, i.e., the ranking of arg max; Paobiased (0: x) in Poseervdd (d: g, x) and the ranking of arg max; Pobeervdd (d: x) ) in Paebiasod (0ilg, Ranking counting presented in descending order, i.e., first second third last %.
PriDe Debiasing
Debiased Prediction
Original Prediction
Models
arg max, dabbinsed (o, qq.z)
arg max, P Pobserved (d, qq.a)
Ranked in Original Distribution
Ranked in Debiased Distribution
Pahservea q q.)r)
Pachiased (o, Q ())
11ama-7B
72.8% / 18.7% / 6. 6.7% / 1.8%%
72.8% / 22. 1 %: 5. % 0.0%
11ama-133
75.1% / 16.7% / 6.6% 1.7%
75.1%% % 12. 8.9% 3.2%
1lama-30B
88.2% / 8.9% / 2.2% / 00.7%
88.2% / 99.9% 1.7% 0..2%
11ama-65B
83.0% / 11.4% / 44.2% / 1.4%
83.0% / 12.3% 3.7% 1.0%
1lama-2-78
62.8% / 17.3% / 14.8% 5.2%%
62.8% / 24.2% 11.3% 1.7%
11ama-2-138
88.8% 10.0% / 1.2% / 00.0%
88.8% / 8.0% / 3300% / 00.2%
12ama-2-708
90.3% / 8.8% / 0..9% / 001%%
90.3% / 7.7%% 11.6% 0..4%
1lama 2-chat7-
86.9% 10.6% / 2.2% / 0.33
86.9% / 10.6% 2.4% 0.0%
11cha-3-c
85.5% 12.6% / 1.7% / 0.11%
85.5% / 10.7% 3.7% 0.11%
1lama-2-chat-70B
94.5% / 5.0% / 00.4% / 0.0%%
94.5% / 4.8% / 0.7% 0.0%
vicuna-v1.3-7B
84.1% 11.0% / 4.66% 0.33%
84.1% / 9.3% / 6.35 0.33%
vicuna-v1.3-13B
82.3% 12.2% / 44.6% 1.0%
82.3% / 15.4% / 2.0% 0.4%
vicuna-v1.3-33B
96.0% / 3.5% / 0.5% / 0.1%
96.0% / 3.7% / 0.3% / 0.0%
vicuma-v1.5-7B
86.2% 11.9% / 1.4% 0.44%
86.2% / 1 / 1.3% 0.4%
vicuna-v1.5-13B
87.4% 11.2% 1.3% 0.1%%
87.4% / 11.9% 0.6% 0.0%
falcon-7B
50.0% / 19.1% 14.3% 6.7%
50.0% / 18.6% / 11.1% / 20.3%
falcon-40B
89.3% 99.1% 1% 1.4% / 0.2%%
89.3% / 8.3% 1.9% 0.5%
falcon-inst-7B
53.6% / 27.8% / 13.7% 4.99%
53.6% / 16.9% / 17.2% 12.3%
falson-inst-408
83.4% 11,49 11,4% / 44,3% / 0.9%
83.4% / 13.3% 3.3% 0.0%
96.0% / 3.8% / 0.2% / 0.0%
96.1% / 3.5% / 0.4% 0.0%
Cyclic Permutation Debiasing Original Prediction
Debiased Prediction
Models
arg max, Pashiased (o, gg.z)
arg max, Pabserved (d, (d,1g.22 (.
Ranked in Original Distribution
Ranked in Dehiased Distribution
Pahwervel (1, q.2)
Pastiimee q;
11ama-7B
49.4% / 23.8% 15.2%
11.6%
49.4% 24.5% 15.1%
11.0%
1lama-133
45.2% / 25.6% 17.6% 11.6%
45.2% / 23.8%
17.5% 13.5%
11ama-30B
75.4% / 16.4% / 5.5% 22.7%
75.4% / 15.6% 6..2% 2.8%
1lama-655
70.3% 14.6% / 6.7% 88.3%
70.3% / 15.00% /9.5% 5.2%
1lama-2-78
54.4% / 21.5% / 14.4% / 99.8%
54.4% / 24.1 1% 13.5% / 8.0%
1lama-2--12)
71.4% / 17.9% / 7.5%% 33.2%
71.4% / 17.6% 77.8% 3.3%
11ama-2-708
82.7% / 13.2% / 1.1 1.0%
82.7% / 12. 12.1% 3.9% 1.3%
11ama 11ama-2-chat-78
65.6% 24.4% 8.0% 2.00%
65.6% 14.1% 12.4% 7.99
1lama-2-chat-1
66.4% / 23.5% 9900% / 1.1%
66.4% / 12.1% 11 11.44 / 10.1%
11ama-2-chat-70B
77.6% 17.6% / /3.9% / (0.9%
77.6% / 11.9% 7.55% 1..5%/33.0 3.0%
vicuna-v1.3-7B
67.0% / 17.6% / 11.0% 4.44
67.0% 15.1% / 7.88% 10.2%
vicuna-v1.3-13B
67.5% 22.6% 7.44% / 2.6%%
67.5% 15.2% 12.7% 4,7%
sicua-v11.3-33p
77.0% 18.2% 3.6% / 1.1%
77.0% / 12.6% 7.7%%
2.7%% 2.7%
vicuna-v1.6-7B
66.6% / 20.7% / 10.9% / 1.7%
66.6% 13.8% 7..6% /1.6% 11.9%
vicuna-v1.5-138
72.3% 21 .6%% 5.0% 1.2%%
72.3% / 13. 6.6%% 99.0% /5.1
falcon-7B
34.1% 25.0% / 21.3% 19.6%
34.1 23 23.9% / 21.2% 20.8%
falcon-40B
66.7% / 19.9% 11..7% 7.7% 1.7%
66.7% / 16.7% 8.9% 7.7%
falcon-inst-7B
28.3% 29.5% / 22.4% 19.8%
28.3% / 28.2% / 24.3% 19.2%
falcon-inst-408
65.1% / 19.3% / 13.4% 2.2%
65.1% 18.0% 1 11.8% 5.1%
gpt-3.5-turbb.
84.9% 12.2% 2.4% 0.5%
84.9% / 10.3% 3.4% 1.4%
27

---

Published as a conference paper at ICLR 2024
D SUPPLEMENTARY PROOF FOR PERMUTATION-BASED DEBIASING BASELINE
Our probability decomposition assumption (Equation can also give a theoretical proof of the soundness of the permulation-basee debiasing baseline (Equation T. Note that gr is the inverse mapping of fi= 91 = fI so we can rewrite Equation
as:
Pobserved (dg (e (1) 1//g,,1100 x²) - 2 prior (dg)()) (1) debiasad x), VI (1, (9)
By taking the logarithm and summing over we can similarly derive:
Σ
log Pobserved (d (d (0) = TET
Σ TET
log
|Z] log Paebiased (2) +C
(10)
7
log P prior (dyla) [Z] log Pdebiased (O.lg.)
(11)
I log Paebiased (0:1g, (0
+ C', i n),
(12)
and obtain:
Paebiasod (o: (x) softmax
1
Σ IEI
log Pobserved (dgrevodddg
(13)
Pobserved (d (d9r())9,
(14)
I
IEI
debbiasoa (0:1g, x), i f1.2.....
(15)
where the approximation holds when the variance of {P (Pobeerved (dgn xf)) E I3 is not too large.
E
EVALUATION DATA AND STATISTICS
MMLU http:://gttabb..cmmnnnadtyyyyy//
ARC http:://alleeaa..crggatttta/
CSQA https://allenaii.org ht/dddda/coooooeeeeeeee
Data statistics of benchmarks used in experiments. Due to the budget constraints and rate limits, for 3.5-tbo in the MMLU evaluation, we randomly sampled 100 test samples from each subject (5,700 in total, accounting for 40% the original size).
Table 7:
"We excluded few MMLU samples (about 3.2% in total), where the options refer to each other like "A and B" "none of the above", in all the experiments. The exclusion of these samples is to ensure the validity of option position changes the "answer-moving attack" and option shuffling or permutation.
Benchmarks
Samples
Evaluated
# Options
Golden Answer Distribution
STEM
3,018
2,925
21.4% / 23.8% / 25 25.9% 22..99
Social Science
3,077
2,992
21.7% 23.4% 23.8% 31.1%%
4
MMLU
Humanities
4.705
4,476
24.2% / /24.5% 27.1% /24.2%
Others
3.242
3,199
23.8% 26.8%%
24.4% 25.0%
Overall
14.042
13,592
4
22.9% 24.7% 25.5% 26.9%
ARC
1,165
1,165
4
22.6% / 26.5% / 26.5% / 24.4%
CSQA
1,216
1,216
5
19.6% / 22.8% 19.8% 20.6% 19.2%
28

---

Published as a conference paper at ICLR 2024
F REFERENCE PROJECTS
HuggingFace LLM Leaderboard
EleutherAl Im-harness
Original MMLU implementation http:://gthubb.comaaatttyy/////
OpenAI Evals
G EVALUATED OPEN-SOURCE MODELS
Models
URLs
11 ama-m 7b
||
https:
11 ama-130
httpa:
11 ama-30b
11 66b
11 ama 2-7b
11 ama 2-133b
11ama-22 -1amm-2-70b
11ama-2-chat-1
11ama-2-chat-13b
1lama 1lcha-2
vicuma-v1.3-70
vicuna-v1.3-132
vicuna-v1.3-33b
httpp///wwuguunngggeccc////////r vicuna-v1.5-13b
vicuna-v1.5-7tb
falcon-7b
falcon-40b
falcon-inst-7b
falcon-inst-40b
29


Reviews:
[{"review_id": "hIXDKsGzbg", "reviewer": "Reviewer_CGVT", "summary": "This paper investigated the LLMs' sensitivity to position changes in multiple-choice questions, discovered that token bias is the main cause/ Furthermore, the authors proposed a way to efficiently suppress this bias and improve accuracy.", "questions": "1. In deriving the method, there are a few key assumptions, e.g. Prior for option IDs depends mostly on q. Is it possible to empirically verify this assumption?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. It flows! The writing is perfect. All sections follow each other naturally, from problem to observation, to diagnosis, to ruling out simplistic solutions, to proposed solutions. In each step, there are corresponding experiments to substantiate it.\n2. There are some clever experiment designs in diagnosing the cause, and the experiments are carried out with caution (e.g. replacing symbols to confirm).\n3. Comprehensive experiments on many models and datasets.", "weaknesses": "1. When the compute budget is unbounded, the proposed method sometimes has a slight accuracy disadvantage compared to full perm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fCk5YNTStz", "reviewer": "Reviewer_rhkS", "summary": "This paper experimentally discovers an issue that LLMs are vulnerable to option position changes, or the Option-Order Sensitivity problem, in MCQs due to their inherent “selection bias.” It proposes a label-free, inference-time debiasing method(PriDe) to mitigate the selection bias. The experimental results demonstrate the claim and the usefulness of the PriDe.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I really appreciate the paper conducted extensive experiments to demonstrate and analyze the Option-Order Sensitivity problem. Some observations are really interesting; for example, even the same models with different parameter sizes but trained using the same data exhibit different position preferences. \n\nThe PriDe is intuitive but also effective.", "weaknesses": "It would be better to cite \"Leveraging large language models for multiple choice question answering\" or other related papers when mentioning the Option-Order Sensitivity problem since they have found the problem earlier than the work of this paper.\n\nIt would be better to analyze more technicals, including self-consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZwkJ7MIRFr", "reviewer": "Reviewer_G9aZ", "summary": "This work presents a comprehensive analysis of the selection bias issue in large language models (LLMs) when dealing with multiple choice questions (MCQs).\nThe experimental results identify the root cause of this bias as the LLMs' token bias, which leads to a preference for specific option IDs when predicting answers.\nBased on these observations, this work proposes a label-free, inference-time debiasing method called PriDe, which effectively mitigates selection bias.", "questions": "1. The generalization analysis indicates that the bias for a certain model is consistent across different tasks.\nCould you further demonstrate this with more statics or results?\nIt would also help to enhace the claimed interpretability.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The empirical analysis is thorough, involving 20 LLMs and three benchmark datasets. This extensive evaluation provides strong evidence for the existence of selection bias in LLMs and its impact on their performance in MCQ tasks. The identification of token bias as the primary source of this issue is a valuable insight that can inform future research on LLMs and their limitations.\n\n2. The proposed PriDe method is effective when the computing cost is limited. Further analysis on generalizability reveals that the prior estimated by PriDe can be generalized across tasks.", "weaknesses": "1. It seems that PriDe achieves comparable performance with simple baselines when the computation cost is not limitated. In application scenarios, we always first estimate the prior without concerning the computation cost, then apply this prior to serve applications.\nIt would be better if PriDe could have a higher upper boudn performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVw5kdLNJO", "reviewer": "Reviewer_VDjF", "summary": "The paper studies the issue of sensitivity to answer option order in large language models (LLMs), which can lead to biased predictions. It introduces a new method called PriDe to mitigate this sensitivity by estimating and correcting for the model's bias during inference. The results show that PriDe can reduce the prediction sensitivity.", "questions": "- The proposed method basically follows estimate-then-mitigate, which is somewhat similar to the calibrate-before-use (Zhao et al. ICML 2021) paper, though this one targets a different setting and is not directly applicable to MCQs. But it would be interesting to compare the differences and know if calibrate-before-use can also help with MCQ sensitivity.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper studies the LLMs' sensitivity to order of answer options, which is an important problem in current LLM evaluation, and provides empirical analysis of the underlying reasons.\n- The proposed method PriDe operates on test time without introducing extra computational cost, which is suitable for current LLMs. \n- The authors conduct extensive experiments including different models, tasks, ablation studies, cross-task evaluation, etc.", "weaknesses": "- The proposed method requires sampling test samples first to estimate the prior, which may introduce another dimension of sensitivity of the selection of the test samples. The accuracy of this estimation might vary based on the quality and representativeness of these samples.\n- I understand the procedure of cyclic permutation and full permutation, but how are they used as the debiasing methods? Do the authors take the best result of the permutations as the prediction?\n- The authors use the balance of recalls and Rstd as the major metrics throughout the paper. Can the authors formally define this? I didn't immediately get it.\n- The writing and presentation need more improvement, e.g., I think the proposed PriDe is quite intuitive but the authors introduce too many unnecessary notations ($d_i, o_i, g_i, x_i, .... $) before getting into the real introduction of the method, which makes the reading difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)