Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models (2024)

Abstract: In this paper, we investigate the phenomena of "selection biases" in Large Language Models (LLMs), focusing on problems where models are tasked with choosing the optimal option from an ordered sequence. We delve into biases related to option order and token usage, which significantly impact LLMs’ decision-making processes. We also quantify the impact of these biases through an extensive empirical analysis across multiple models and tasks. Furthermore, we propose mitigation strategies to enhance model performance. Our key contributions are threefold: 1) Precisely quantifying the influence of option order and token on LLMs, 2) Developing strategies to mitigate the impact of token and order sensitivity to enhance robustness, and 3) Offering a detailed analysis of sensitivity across models and tasks, which informs the creation of more stable and reliable LLM applications for selection problems.

heading: 1 Introduction
text: Large Language Models (LLMs) have demonstrated remarkable abilities across various tasks (OpenAI, 2023; Gemini Team, 2023; Touvron et al., 2023), leading to their widespread adoption in downstream applications. In particular, the utilization of zero-shot or few-shot prompting techniques emerged as a highly convenient approach in harnessing the potential of LLMs, since these techniques empower end-users to solve a wide range of tasks without the need for extensive fine-tuning.
Despite LLMs’ impressive performance and convenience, empirical investigations have found that their output is highly sensitive to the choice of prompts, and even subtle modifications of instructions or demonstrations have considerable influence on their performance. To address this issue, several works have been dedicated to the identification and mitigation the inherent biases in LLMs, aiming to enhance their robustness and reliability (Zhao et al., 2021; Si et al., 2023; Fei et al., 2023).
In this study, our focus is directed towards the domain of “selection problem”, where LLMs are instructed to select the optimal choice from an ordered sequence of choices. This problem encompasses a variety of downstream applications, including but not limited to classification, multiplechoice questions (Zheng et al., 2024; Pezeshkpour and Hruschka, 2023), and model evaluation scenarios (Zheng et al., 2023; Wang et al., 2023; Shen et al., 2023). In our analysis, we identifies specific biases within the context of selection problems, which we call “selection biases” to encapsulate these discernible tendencies. These biases manifest as systematic deviations in LLMs’ preferences, and a thorough understanding of these biases is pivotal for enhancing the robustness of LLMs across the spectrum of applications under the scope of selection problems. Our subsequent exploration delves into the characterization, quantification, and mitigation strategies to address these biases. It is crucial to highlight that our analysis centers on the zero-shot setting. This choice distinguishes our work from previous endeavors, which predominantly concentrate on few-shot settings, making it difficult to disentangle biases stemming from in-context demonstrations.
Our contributions can be summarized as follows: 1) We quantify the influence of option order and token on the decision-making processes of various LLMs when tackling selection problems, providing clear insights into how these factors affect model performance; 2) We introduce strategies to mitigate the effects of token and order sensitivity, leading to performance improvements across a broad spectrum of tasks; 3) We offer a thorough understanding of the sensitivity landscape through an empirical study encompassing different models, tasks, and sensitivity settings. The analysis enables us to identify the most effective strategies for addressing sensitivity issues in diverse task scenarios.
5598

heading: 2 Related Work
text: Biases of LLMs. Several studies have delved into the biases of LLMs. Zhao et al. (2021) identifies three notable biases: majority label bias, where LLMs exhibit a propensity to output the most frequent label in few-shot demonstrations; recency bias, which is the tendency to repeat the label appearing towards the end of the prompt; and common token bias, manifesting as the inclination to output tokens prevalent in the pre-training distribution. Fei et al. (2023) further identifies the domainlabel bias, which could be detected and estimated using random in-domain words from the task corpus. Additionally, Si et al. (2023) focuses on feature bias, which is the tendency to use one feature over another to predict the label, even when both features in the prompt are equally effective for predicting the label. However, these works mainly focus on the few-shot settings, which fails to disentangle the effects of selection biases from biases caused by in-context examples. Selection Problem of LLMs. Previous studies have explored the use of LLMs in tackling selection problems. In Multiple Choice Questions (MCQs), Robinson and Wingate (2023) demonstrated the application of LLMs to MCQs, focusing on how different prompting techniques influence the model’s decision-making process. Pezeshkpour and Hruschka (2023) highlighted how LLMs are affected by position bias when addressing MCQs, while Zheng et al. (2024) pinpointed token bias as the primary reason LLMs are not robust selectors in this context. In evaluation scenarios, Shen et al. (2023) employed LLMs to assess the abstractive summarization outcomes of models, introducing three distinct settings: reason-then-score (RTS), MCQ scores, and head-to-head comparison (H2H). Zheng et al. (2023) applied LLMs as evaluators in chatbot interactions, employing a two-round approach rather than a single-step evaluation. Wang et al. (2023) discovered that LLMs’ evaluation fairness is significantly compromised by option position bias, indicating that LLMs can be heavily influenced by the positioning of options.

heading: 3 Experimental Setup

heading: 3.1 Evaluation Tasks
text: We experiment on six multi-choice tasks with the number of choice options varying from two to five. The six benchmarks are: ARC-Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019),
MMLU (Hendrycks et al., 2021), Winogrande (Sakaguchi et al., 2019), MathQA (Amini et al., 2019), and OpenBookQA (Mihaylov et al., 2018).
We select these datasets due to their coverage of a wide range of domains, including commonsense reasoning, STEM, social sciences, humanities, etc. This diversity ensures a comprehensive evaluation across various fields of knowledge. Data statistics details are shown in Table 5 in Appendix A due to space constrains.

heading: 3.2 Models
text: We adopt six instruction-tuned LLMs in our experiment, encompassing both commercial APIs and open-source models. From the commercial side, our selection included PaLM 2 (Anil et al., 2023), Gemini Pro (gemini-pro) (Gemini Team, 2023), and ChatGPT (gpt-3.5-turbo-1106) (OpenAI, 2022). For open-source models, we employ LLaMA 2 (Touvron et al., 2023) with different model sizes (Llama-2-chat-7b/13b/70b).

heading: 3.3 Notaions
text: For a given question q, the number of options available is denoted by k. Each option within this range, from position 1 to k, is characterized by an option symbol si and the corresponding option content ci, where si ∈ Sq and ci ∈ Cq. Here, Sq denotes the option symbol set, and Cq represents the option content set. For instance, consider a question q that offers four possible answers with the symbol set Sq = {s1, s2, s3, s4} and Cq = {c1, c2, c3, c4}; in this scenario, the representation of q can be expressed as q = {(s1, c1), (s2, c2), (s3, c3), (s4, c4)}.

heading: 3.4 Other Details
text: Following HuggingFace Open LLM Leaderboard (Beeching et al., 2023), we utilize the EleutherAI lm-harness (Gao et al., 2023) tool to manage datasets for our experiments. For commercial APIs, we set the temperature to 0 to guarantee reproducibility. For open-source models, we employ Azure AI Studio to deploy various sizes of Llama-2-Chat for parallel processing, optimizing our experimental setup for efficiency and scalability. Additionally, all experiments in this study are conducted in the zero-shot setting, with the prompts being consistent with those used in prior research (Zheng et al., 2023; Wang et al., 2023). Details of the prompts can be found in Appendix B.

heading: 4 Investigation on LLM Sensitivity
text: While prior research has touched upon biases in LLMs concerning MCQs, with notable findings on position bias and token bias, our work stands out by delving deeper into unexplored territories of the combined impact of option order and token usage within MCQs. We uncover novel insights into the decision-making processes of LLMs that have yet to be extensively explored in the existing literature.

heading: 4.1 Setups
text: We adhere to the notations established in Section 3.3, allowing for a more coherent and precise description of the experimental setups. Token Sensitivity. To assess the impact of token sensitivity, we employ the default option symbol set Sq = {A,B, . . . , Sqk} for each question q, where k represents the number of option contents for question q and and Sqk represents the k-th letter of the alphabet from A to Z. For each question, we conduct experiments with two distinct requests to the LLM. The first request is defined as follows:
rforward = {(si, ci) | i = 1, 2, . . . , k} (1)
Here, si refers to the i-th option symbol in Sq, and ci represents the i-th option content of Cq, indicating the i-th answer candidate for the question.
Conversely, the second request, rbackward, introduces a reversed arrangement of the option sym-
bols, as detailed below:
rbackward = {(sk−i+1, ci) | i = 1, 2, . . . , k} (2)
Subsequently, the results of rforward and rbackward are analyzed. Order Sensitivity. To determine the influence of order sensitivity, we adopt a strategy of coupling each option symbol with its corresponding option content, thereby aiming to nullify the effects of token sensitivity. Consistent with the settings described previously, the option symbol set Sq is {A,B, . . . , Sqk}. rforward and rbackward are:
rforward = {(si, ci) | i = 1, 2, . . . , k} (3)
rbackward = {(si, ci) | i = k, k − 1, . . . , 1} (4) Both Sensitivity. In practical scenarios, a common remediation strategy involves rearranging the order of option content. This maneuver inherently addresses both order and token sensitivities. It is anticipated that if the biases induced by these sensitivities align, their cumulative effect on sensitivity will be magnified. Conversely, if they are in opposition, their effects will likely be mitigated. Following the previously described setting, where the symbol set Sq is {A,B, . . . , Sqk}, we define rforward and rbackward as follows:
rforward = {(si, ci) | i = 1, 2, . . . , k} (5)
rbackward = {(si, ck−i+1) | i = 1, 2, . . . , k} (6)

heading: 4.2 Measurement of Sensitivity
text: To assess the model’s sensitivity, we introduce the Fluctuation Rate (FR), a metric designed to quantify the variability in responses between rforward and rbackward. The equation for FR is given by:
FR =
∑N i=1(rforward(i) ̸= rbackward(i))
N (7)
where ∑N
i=1(rforward(i) ̸= rbackward(i)) denotes the number of instances where the outcomes of rforward and rbackward are not identical, and N represents the sample size of that task. Thus, FR reflects the fraction of all questions where the two requests yield divergent results.

heading: 4.3 Overall Observation
text: Table 1 comprehensively summarizes our sensitivity experiments across various LLMs. We also provide detailed breakdowns of the MMLU’s performance across its 57 subtasks in Appendix C. In powerful LLMs, PaLM 2, Gemini Pro, and GPT3.5, we observe a notable trend: they are more sensitive to option order than to symbols/tokens in 17 out of 18 cases. An exception to this trend is observed with the Winogrande dataset, where PaLM 2 shows increased sensitivity to token variations. In both sensitivity setting, which examines the joint effects of token and order sensitivities, we find that in 11 out of 18 cases, the combined influence is the most pronounced. This indicates that in more than half of the cases, the directional impacts of token and order sensitivities tend to align.
Conversely, the open-source LLM, LLaMA 2 (Llama-2-chat), across its varying sizes, does not exhibit a consistent sensitivity trend towards token and order. For instance, while the 7B model appears more sensitive to order, the 13B and 70B models do not follow this pattern. Although Table 1 indicates that the 13B and 70B models are more sensitive to token differences in 9 out of 12 instances, the discrepancy in the fluctuation rate between token and order sensitivity is marginal.

heading: 4.4 Relation between Difficulty and Sensitivity
text: Table 1 reveals an interesting pattern: tasks with higher accuracy levels, such as ARC Challenge, HellaSwag, and OpenBookQA, tend to exhibit lower fluctuation rates. This observation prompts us to question whether there is a relationship between the difficulty of a task and the sensitivity of a model to it. To further investigate this hypothesis,
we analyze the sensitivity across 57 MMLU subtasks. For detailed results per model, we refer to Appendix C, as mentioned in Section 4.3, due to space constraints.
Figure 1 illustrates the correlation between task accuracy and fluctuation rates across six models, encompassing three advanced commercial LLMs and three open-source models of varying sizes. This comparison offers a unique opportunity to assess the impact of scaling model parameters. For comprehensive insight, we integrate the previously discussed settings—token sensitivity, order sensitivity, and both sensitivity—into a single diagram per model, facilitating a clear understanding of how task difficulty correlates with model sensitivity.
Results from PaLM 2, Gemini Pro, GPT-3.5, and LLaMA 2 70B appear to support our hypothesis: more challenging tasks, characterized by lower accuracy, tend to exhibit greater sensitivity, as indicated by higher fluctuation rates. This aligns with our intuition that models are more confident and thus less sensitive to fluctuations in easier questions. A notable observation pertains to the smaller LLaMA 2 models, specifically the 7B and 13B versions. Tasks that are more straightforward for other powerful LLMs pose significant challenges to these models, leading to lower accuracy and a muted trend in sensitivity as tasks vary in difficulty. However, a closer analysis of the 7B, 13B, and 70B models reveals a gradual manifestation of the expected trend. The shift from the 7B to the 13B model, for instance, corresponds with our expectation in the both sensitivity setting. With further increases in model size, the 70B model exhibits the predicted correlation between task difficulty and model sensitivity across all examined settings.

heading: 4.5 LLMs’ Option Tendency
text: To understand LLMs’ behavior, we calculated the option proportion statistics to analyze their tendencies. Specifically, we calculated the answer distri-
bution of rforward and detailed the information for each option alongside the ground truth label proportion. Table 2 shows the results for the ARC dataset, highlighting the similarities and differences in selection biases among various LLMs. According to the results, most models, except for LLaMA2-7B, exhibit a notable bias towards option C compared to the ground truth proportion. Due to space limitations, statistics for the other five datasets, including HellaSwag, MMLU, Winogrande, MathQA, and OpenBookQA, are included in Appendix D. Generally, most models, except for LLaMA2-7B, show a bias towards options B or C.

heading: 5 Methodology
text: To mitigate the impact of sensitivity to tokens and/or order and improve model stability, we propose three methods tailored to different contexts of LLMs. We categorize these contexts into two scenarios: Gray-Box and Black-Box. In a Black-Box scenario, the LLM provides only the generated text upon request, without additional information. Conversely, a Gray-Box scenario allows access to more detailed output, such as token probability information. In our experiments, GPT-3.5 falls into the Gray-Box category, as the OpenAI API enables retrieval of the top 5 token log probabilities, whereas other models are Black-Box ones. Furthermore, all experiments adhere to the sensitivity settings mentioned in Section 4.

heading: 5.1 Gray-Box Probability Weighting
text: For each question q, the requests rforward and rbackward are:
rforward = {(sfi , c f i ) | i = 1, 2, . . . , k} (8)
rbackward = {(sbj , cbj) | j = 1, 2, . . . , k} (9)
Let function p(·) represents the probability of a token generated by the LLM. For instance, p(sf3) represents the probability that the model selects the third option symbol of rforward. We calculate the weighted probability for each option content cfi in the first query set rforward. The weighted probability of a specific option content cfi is derived by integrating the probabilities of its corresponding symbol sfi in rforward with the symbol s b j in the second query set rbackward. The formulation of this computation is as follows:
Pweighted cfi = p(sfi )×p(sbj) where c f i = c b j (10)
The final choice is cfi∗ , the option content with the highest weighted probability, determined by:
i∗ = argmax i Pweighted cfi , (11)
where f∗ maximizes Pweighted cfi .

heading: 5.2 Gray-Box Probability Calibration
text: Due to the sensitivities of LLMs to both the order and tokens in MCQs, their outputs frequently show biases, leading to preferences for specific options. To address this issue and promote a fairer and more accurate answer selection process, we calibrate the output probabilities. This calibration aims to enhance the precision of which the model selects answers.
Let the output distributions for each option symbol in rforward and rbackward are denoted by Dforward and Dbackward, respectively, and are formulated as follows:
Dforward = {pd(sfi ) | i = 1, 2, . . . , k} (12)
Dbackward = {pd(sbj) | j = 1, 2, . . . , k} (13) where pd(si) represents the probability distribution of option symbol si , defined by:
pd(si) = count(si)
N (14)
Here, N denotes the total sample count, and count(si) indicates the number of samples for which the model selects si as the answer. Thus, pd(si) reflects the percentage of selections for si. For real-world applicability, we calculate these distributions using the validation set of each task.
To calculate the calibrated probabilities, we use the following formulations:
P calibratedforward =
{ p(sfi )
pd(s f i )
| i = 1, 2, . . . , k } (15)
P calibratedbackward =
{ p(sbj)
pd(s b j)
| j = 1, 2, . . . , k } (16)
Here, P calibratedforward and P calibrated backward represent the sets of calibrated probabilities for each option symbol in rforward and rbackward, respectively. The calibration process of P calibratedforward involves dividing
the original probability of selecting each symbol p(sfi ) by its corresponding output distribution probability pd(s f i ), for i = 1, 2, . . . , k. This approach ensures that each option’s probability is adjusted in light of its observed selection frequency, aiming to align the model’s output more closely with an unbiased selection criterion.
Considering the three distinct sensitivity settings, we identify three specific distribution sets: (Dtokenforward, D token backward), (D order forward, D order backward), and (Dbothforward, D both backward). These distributions underpin our calibration strategy, allowing us to adjust the model’s outputs to reduce bias and enhance answer accuracy across different sensitivity contexts.

heading: 5.3 Black-Box Two-Hop Strategy
text: In practical applications, we often encounter blackbox scenarios while using commercial LLM APIs. To mitigate the impact of model sensitivity in these situations, we propose a black-box two-hop strategy that leverages the model’s output distributions Dforward. Given the constraints of black-box scenarios, where recalculating the token probability p(si) is impossible, we adopt an alternative strategy. Our approach intentionally avoids selecting the most biased option symbols in the first request rforward, opting for responses from rbackward instead. Firstly, we identify the most probable option symbol sfi∗ based on the distribution Dforward, using the equation:
i∗ = argmax i pd(s f i ), (17)
where pd(s f i ) denotes the distribution probability of selecting symbol sfi from rforward. Subsequently, the two-hop strategy is implemented as follows:
Final Selection =    cfj∗f
if i∗ ̸= j∗f , cbj∗b if i∗ = j∗f , (18)
where we select cfj∗f from rforward if it does not exhibit bias most. Otherwise, we opt for cbj∗b as our final answer. Here, j∗f and j ∗ b are determined by:
j∗f = argmax j p(sfj ), (19)
j∗b = argmax j p(sbj), (20)
where j∗f , j ∗ b indicate the indices of the symbols with the highest probabilities in rforward, rbackward, respectively.
This method aims to utilize responses that potentially minimize bias by considering the model’s preference patterns indicated by Dforward, thereby ensuring the accuracy of selections.

heading: 6 Experiment Results

heading: 6.1 Gray-Box Results
text: Tables 3 and 4 show the results of our methods. In the gray-box scenario, only GPT-3.5 is included since other models do not provide the information of token probability information. Conversely, in the black-box scenario, all models are considered in our experiment. In the subsequent analysis, we compare the performance improvements achieved by our methods against the baseline established in Section 4, aiming to underscore the enhancements or limitations observed across tasks and models.
As Table 3 illustrates, gray-box methods, including both probability weighting and calibration approaches, significantly improve performance across
six distinct tasks under three sensitivity settings. Notably, the probability weighting method demonstrates considerable enhancements in all scenarios, surpassing the baseline. It benefits not only more challenging tasks such as MathQA, Winogrande, and MMLU but also shows improvements in easier tasks. Interestingly, the probability calibration method outperforms the weighting method in two specific tasks out of the six: Winogrande and MathQA. These tasks are unique in their format; Winogrande presents only two options, whereas MathQA offers five options per question. We speculate that the number of options may influence the LLM’s preference distribution, thereby affecting the performance of different methods.

heading: 6.2 Black-Box Results
text: Table 4 displays the results of the black-box method applied to various models, sensitivity settings, and tasks. Notably, the stronger models, PaLM 2 and Gemini Pro, show significant benefits from the twohop strategy. They improved in five out of six tasks, with Winogrande being the only exception. Similarly , GPT-3.5 also shows improvements in most tasks, succeeding in four out of six. The exceptions are Winogrande and MathQA, with MathQA noted as the most challenging one across all tasks. The LLaMA 2 models, spanning the 7B, 13B, and 70B variants, improve in half of the evaluated tasks. Their performance is consistent across model sizes. They excel in ARC Challenge, HellaSwag, and MMLU but face challenges in the other three tasks.
A noteworthy observation is that all models, re-
gardless of their capability, exhibit reduced performance on the Winogrande task after applying our two-hop strategy. This includes both the high-end models like PaLM 2 and Gemini Pro, as well as the smaller scale LLaMA 2-7B model. The reasons for this decline are not immediately apparent, as factors such as model strength, task difficulty, and specific sensitivity to Winogrande do not directly explain the reduced effectiveness. Winogrande stands out due to its unique characteristics: it offers only two options per question and employs a cloze-test format rather than standard questionanswering. We hypothesize that the limited number of options or the specific task type might alter the LLM’s preference distribution, impacting the efficacy of our black-box strategy.

heading: 6.3 Breakdown MMLU Subtasks
text: To gain deeper insights, we conducted a detailed breakdown of the MMLU’s 57 subtasks, examining closely how each method we proposed affects these subtasks. Figure 2 offers a comprehensive view of how the MMLU’s 57 subtasks respond to both the probability weighting and calibration methods within a gray-box scenario. Consistent with the findings reported in Table 3, most of subtasks improve after applying our proposed methods. Specifically, within the probability weighting analysis, declines are observed in only 1, 6, and 3 subtasks across the token, order, and both settings, respectively. This translates to an average of merely about 6% of tasks not deriving benefits from the weighting method. Upon closer examination, virology emerges as the subtask experiencing a decline across all three sensitivity settings. Among sub-
tasks with notable decreases, machine learning in the both setting shows a 3.57% drop, while moral scenarios and business ethics in the order setting decline by 3.46% and 3%, respectively.
Regarding the probability calibration method, on average, more than 78% of the subtasks improved with our approach, with over 30% of them having at least a 1% increase in accuracy. Recall that, in Table 3, the calibration method significantly outperforms the weighting method in the MathQA task. This trend extends across MMLU subtasks, with STEM-related tasks showing the most substantial gains. For instance, in the both setting, the top beneficiaries include elementary mathematics, high school mathematics, college physics, and college chemistry, with improvements of 14.29%, 12.22%, 11.27%, and 7.00%, respectively, outshining other subtasks. This phenomenon is shown in the bottom three diagrams of Figure 2. Due to space constraints, detailed breakdowns of the graybox method are presented in Appendix E, within Table 17 and 18. Figure 6, 7, and 8 in Appendix F displays the results of the black-box two-hop strategy, with LLMs, where only generated text is accessible. Despite this limitation, more than half of the MMLU subtasks show improvement after our method’s application. This enhancement is observed across various models, from the strongest to smaller ones like the 7B and 13B models.

heading: 6.4 Cost Analysis
text: Our method prioritizes cost-effectiveness by minimizing the need for numerous permutations or voting on costly chain-of-thought (CoT) candidates. For the probability weighting method, each ques-
tion q needs two requests to calculate the weighted probability. In contrast, the probability calibration and gray-box methods require a validation set of approximately 200 samples to compute the distributions Dforward and Dbackward. Calibration of either rforward or rbackward alone requires just one request per question; The black-box method also demands two requests per question q. Furthermore, the total expense for all experiments conducted in this study was under $400 USD, covering six models and six benchmarks. Notably, the use of PaLM 2 and Gemini Pro was temporarily free. For specific costs associated with the OpenAI API and Azure AI Studio, please refer to the official documentations.

heading: 7 Conclusion
text: This study investigate into the effects of token and order sensitivity on LLMs when addressing selection problems, incorporating an empirical analysis of both powerful commercial LLMs such as Gemini Pro and GPT-3.5 and open-sources models like LLaMA 2. By concentrating on zero-shot settings, we aim to isolate and better understand biases that previous works identified in in-context demonstrations, thereby offering a clearer perspective on how these sensitivities influence LLM decision-making processes. Our findings underscore the significance of task difficulty as a crucial determinant of sensitivity impact on LLM performance.
Moreover, we introduce cost-effective mitigation strategies, including gray-box and black-box approaches, tailored for practical application scenarios. The results demonstrate that our gray-box methods, namely probability weighting and probability calibration, outperform baselines with minimal additional expenditure, contrasting with more complex methods like majority voting. Additionally, our two-hop strategy for black-box scenarios proves to be effective in a significant portion of tasks. We anticipate that our contributions will aid future research in enhancing the robustness of LLMs across various types of selection problem applications.

heading: 8 Limitation
text: While this study contributes valuable insights into mitigating selection biases in LLMs, we acknowledge several limitations that warrant consideration. Firstly, the gray-box strategies proposed for alleviating selection biases may encounter constraints
when applied to certain black-box LLM APIs. The efficacy of these strategies heavily relies on the availability of probability information, which may be restricted in externally hosted APIs.
Secondly, the exploration of mitigation strategies primarily focuses on the gray-box and black-box settings, leaving the examination of further mitigation strategies in white-box open-source models unexplored, and we recognize it as a potential avenue for future research. Investigating mitigation strategies within white-box open-source models could provide a more comprehensive understanding of how selection biases manifest and can be addressed in models where internal workings are transparent.

heading: Acknowledgements
text: This work was supported by National Science and Technology Council, Taiwan, under grants MOST 110-2221-E-002-128-MY3, NSTC 112-2634-F002-005 -, and Ministry of Education (MOE) in Taiwan, under grants NTU-113L900901.

heading: A Data statistics details

heading: B Prompt templates
text: We list all the prompt templates used in our experiments, including three different sensitivity settings: token, order, and both. These templates are presented in Figures 3, 4, and 5, corresponding to each setting respectively.

heading: C Detailed Sensitivity Experiment Results
text: Tables 6 through 11 provide detailed experimental results for the MMLU, covering its 57 subtasks for each of the following models: PaLM 2, Gemini Pro, GPT-3.5, LLaMA 2 7B, LLaMA 2 13B, and LLaMA 2 70B, respectively.

heading: D LLMs’ Option proportion statistics
text: Table 12 through 16 provide detailed information on the tendency of each option and the ground
truth label proportion across five datasets: HellaSwag, MMLU, Winogrande, MathQA, and OpenBookQA.

heading: E Gray-Box Results of MMLU Subtasks
text: Tables 17 and 18 detail the results for the MMLU’s 57 subtasks following the application of our graybox strategies, including probability weighting and calibration.

heading: F Black-Box Results of MMLU Subtasks
text: Figure 6, 7 and 8 show the distribution of accuracy differences resulting from the black-box approach, specifically within the token, order and both sensitivity settings.

heading: Future Work
text: Secondly, the exploration of mitigation strategies primarily focuses on the gray-box and black-box settings, leaving the examination of further mitigation strategies in white-box open-source models unexplored, and we recognize it as a potential avenue for future research. Investigating mitigation strategies within white-box open-source models could provide a more comprehensive understanding of how selection biases manifest and can be addressed in models where internal workings are transparent