Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors (2024)

Abstract: Multiple-Choice Questions (MCQs) constitute a critical area of research in the study of Large Language Models (LLMs). Previous works have investigated the selection bias problem in MCQs within few-shot scenarios, in which the LLM’s performance may be influenced by the presentation of answer choices, leaving the selection bias during Supervised Fine-Tuning (SFT) unexplored. In this paper, we reveal that selection bias persists in the SFT phase , primarily due to the LLM’s inadequate Multiple Choice Symbol Binding (MCSB) ability. This limitation implies that the model struggles to associate the answer options with their corresponding symbols (e.g., A/B/C/D) effectively. To enhance the model’s MCSB capability, we first incorporate option contents into the loss function and subsequently adjust the weights of the option symbols and contents, guiding the model to understand the option content of the current symbol. Based on this, we introduce an efficient SFT algorithm for MCQs, termed Point-wise Intelligent Feedback (PIF). PIF constructs negative instances by randomly combining the incorrect option contents with all candidate symbols, and proposes a point-wise loss to provide feedback on these negative samples into LLMs. Our experimental results demonstrate that PIF significantly reduces the model’s selection bias by improving its MCSB capability. Remarkably, PIF exhibits a substantial enhancement in the accuracy for MCQs‡.

heading: 1 Introduction
text: Multiple-Choice Questions (MCQs) are ubiquitously employed in the realm of Large Language models (LLMs). They typical comprise a query accompanied by an array of potential options, wherein the model’s assignment is to discern and
*These authors contributed equally to this work. †Corresponding author. ‡The code of this work is available at https://github.
com/berryxue/PIF
select the most appropriate solution. Given the significance, numerous studies (Pezeshkpour and Hruschka, 2023; Zheng et al., 2023a; Robinson et al., 2023) have explored the challenges faced in the few-shot phase of MCQs. One of the main challenges is the selection bias (Pezeshkpour and Hruschka, 2023; Zheng et al., 2023a), which refers that LLMs are sensitive to variations in the arrangement of options within MCQs. Figure 1 demonstrates an example of selection bias.
Foundation LLMs such as LLaMA2-7B can seldom acquire data-sensitive domain knowledge. For such scenarios, it is crucial to perform Supervised Fine-Tuning (SFT) on LLMs. Thus we hope LLMs to ensure accuracy and also robustly select reliable options in MCQs during SFT. Unfortunately, we find that the selection bias still exists during the SFT phase. Following the methodology of Zheng et al. (2023a), we conduct “answer-moving attack” experiment by always moving the golden option content to a specific symbol, and the results are displayed in Table 1. In detail, we train two models using the training dataset, and select the bestperforming model on the validation set and predict results for the test set. Then, we implement the
4331
answer-moving attack by moving all the correct options to the same symbol for the test dataset and predict results again, which cause significant changes in the models’ performance. For example, when moving all correct options to symbol A, the accuracy of the LLaMA2-7B model increases by 11.1, while LLaMA2-13B model decreases 4.4.
Why do LLMs show selection bias during the SFT stage? We propose a hypothesis that "Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors". We utilize Multiple Choice Symbol Binding (MCSB) capability (Robinson et al., 2023) to represent the LLM’s ability to bind option symbols and their corresponding option contents, and employ Proportion of Plurality Agreement (PPA) metric to compare the relative MCSB ability of two LLMs. Through comprehensive experimental validation, we discover that improving the LLMs’ performance on the PPA metric alleviates the LLMs’ performance on selection bias.
Based on the relationship between the LLMs’ selection bias and its MCSB capabilities, we expect to mitigate selection bias by enhancing the model’s MCSB capability. We first incorporate option contents into the loss function, guiding the model to understand the content of the current symbol. However, the results are less than satisfactory. Considering that label words are anchors (Wang et al., 2023a), we adjust the weights of the option symbols and contents in the optimization objective, termed Reweighting Symbol-Content Binding (RSCB). Subsequently, inspired by Reinforcement Learning from Human Feedback (Stiennon et al., 2020), we propose Point-wise Intelligent Feedback (PIF), in which we construct negative samples by randomly combining the contents of incorrect options with all option symbols, and design a pointwise loss to feedback these negative samples into SFT. Finally, PIF not only ensures the stability of
model performance but also enhances it. In summary, our contributions are as follows: (1) We conduct experiments to demonstrate that there is still the selection bias when performing SFT on LLMs for MCQs. We hypothesize that Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors. In Section 2.3, we validate this hypothesis by investigating the correlation between the MCSB capability and the selection bias. (2) We mitigate selection bias by enhancing the model’s MCSB capability. We propose the Point-wise Intelligent Feedback (PIF) method, which constructs negative samples by randomly combining the content of incorrect options with all candidate symbols and designing a point-wise loss to provide feedback on these negative samples into LLMs. (3) We conduct extensive experiments to validate that our PIF method can significantly alleviate the selection bias of LLMs during the SFT phase for MCQs. Meanwhile, the experimental results prove that PIF could also enhance the accuracy performance of the model.

heading: 2 Exploration of Selection Bias During Supervised Fine-tuning

heading: 2.1 Experimental Background
text: Datasets To investigate the selection bias in the SFT stage of LLMs for MCQs, we conduct experiments on several commonly used MCQs benchmarks: Massive Multitask Language Understanding(MMLU) benchmark (Hendrycks et al., 2021) and CommonsenseQA(CSQA) (Talmor et al., 2019). Our choice of benchmarks considers the variety of tasks and fields involved. Explicitly, MMLU comprises 4-option multiple-choice questions, whereas CSQA contains 5-option variants. As for domains, MMLU encompasses a wide range of tasks from fields including STEM, humanities, social science and others, coverage 57 subjects; while CSQA conduct questions draw upon ConceptNet (Speer et al., 2017) concepts, enabling variety and integration of worldly knowledge within the queries. It’s worth noting that the CSQA dataset does not have standard answers for the test data. Thus in our experimental setup, we use the dev set as the test set and extract a portion of samples from the train set to serve as the validation set. MMLU is additionally split into four domains (STEM, Social Science, Humanities, Others) based on its subject categories. Details can be found in Appendix A. Models We conduct experiments on LLMs origi-
nating from well-known LLM families, spanning various sizes. LLaMA (Touvron et al., 2023) constitutes a compilation of widely-used foundation models designed to promote research on LLMs for training, inference, and extensive applications. In this paper, taking into account the model size, we assess the 7B and 13B variant of LLaMA2 (LLaMA2-7B, LLaMA2-13B). Projects used in this work is illustrated in Appendix B.
Taking into account the limited computational resources, we fine-tune LLMs with Low-Rank Adaptation(LoRA) (Hu et al., 2022) and set its rank to 16, the alpha parameter to 64, and the dropout rate to 0.1. To investigate the selection bias, we design two distinct output configurations for LLMs during the training process. One generates only the option symbols, defined as πSymbol, while the other produces both the option symbols and contents concurrently, defined as πSCB. During the testing process, the model is configured to output only the symbol, which is sufficient for determining whether the answer is correct or not.

heading: 2.2 Selection Bias During SFT
text: We conduct experiments using two LLMs on six datasets. Similar to Zheng et al. (2023a), we conduct "answer-moving attack" experiment to measure the selection bias. During testing, we move all the correct answers to A|B|C|D|E respectively (Answers are A|B|C|D in MMLU. For the sake of convenience, we will use the unified notation of A|B|C|D|E) for the test set, and then display the model’s accuracy after the relocation. Subsequently, to provide a concrete numerical representation of the impact of the answer-moving attack, we calculate µbias, the average absolute value of the
difference between the accuracy after the answermoving attack and the original accuracy tested on the standard test set. Which can be formulated as:
µbias =
∑K i=1(|Acci − Acc0|)
K , (1)
Acci refers to the accuracy after answer-moving attack, Acc0 is the accuracy on the standard testing set, K is the number of options, and is set as 5 for CSQA, 4 for MMLU. Through the experimental results in Figure 2, we make following observations: Selection bias varies among different sizes of the model parameters. As seen from Figure 2, LLaMA2-7B tends to choose C, while LLaMA213B prefers to choose D. Although these two models belong to the same model family, their performances are not identical, which aligns with the findings in Paper Zheng et al. (2023a). We also observe that, when fine-tuning the model only with option symbols, LLaMA2-13B has a smaller µbias on MMLU compared to LLaMA2-7B, while reversed on the CSQA dataset. Therefore, during SFT, there is no absolute relationship between selection bias and the size of the model parameters. SFT training examples influence both the magnitude and distribution of selection bias. Regardless of the LLMs, the overall bias value µbias for CSQA is always smaller than that of MMLU. As seen from Appendix A, the proportions of the five options in the CSQA benchmark training set are roughly equal, while this is not the case for MMLU. Additionally, when we fine-tune LLaMA2-13B with option symbols, the model shows a stronger inclination to predict option B on MMLU, while on CSQA, it predicts B with a significantly lower probability than others. Therefore, we speculate
that the selection bias during SFT is closely related to the fine-tuning dataset used in the SFT stage.

heading: 2.3 Why Do LLMs Suffer Selection Bias in MCQs’ SFT
text: After analyzing the selection bias during SFT in various LLMs and datasets, we now focus on understanding why LLMs exhibit selection bias during the SFT stage. We identify two reasons why LLMs may choose the wrong answer. Firstly, the LLM does not know the correct option, which can be attributed to its capability. Secondly, the LLM is aware of the correct option content. However, selection bias makes it choose a preferred option symbol instead of one corresponding to the correct option. This indicates the model’s weak ability to associate option content with the appropriate symbol. According to it, we propose a hypothesis: Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors.
In this paper, we utilize Multiple Choice Symbol Binding (MCSB) capability (Robinson et al., 2023) to represent the LLM’s ability to bind option symbols and their corresponding contents. Similar to Robinson et al. (2023), we also employ the Proportion of Plurality Agreement (µppa) metric to compare the relative MCSB ability of two LLMs.
µppa = 1 |D| ∑
D
max k∼K ( K!∑ j=1 yj = ok)
K! . (2)
As shown in Table 15, given a question with K options, there are K! distinct arrangements of these options in a fixed set of symbols. For ease of expression, we use ok to represent the k-th option content and yj to describe the content of the predicted arrangement. During testing, we present each question to the model using each unique ordering, and then PPA for this question is the frequency corresponding to the most frequently predicted option content. For a dataset D, µppa is calculated as the average of the PPAs for all individual questions.
To demonstrate the relationship between the MCSB capability and the selection bias, we cal-
culate the differences of µppa and µbias between πSymbol and πSCB, which are defined as follows:
∆µbias = µ πSCB bias − µ πSymbol bias , (3)
∆µppa is defined in a similar approach. As delineated in the prior analysis, if there exists a correlation between selection bias and the LLM’s MCSB capacity, then a positive ∆µppa corresponds to a negative ∆µbias, indicating that increasing the MCSB capability resulting a reduction in the LLM’s bias. We conduct experiments with LLaMA2-7B and LLaMA2-13B. The result is illustrated in Table 2. Except for the performance of LLaMA2-13B on MMLU-Others, all the remaining results indicate that when ∆µbias is positive, ∆µppa is negative and vice versa. Additionally, the performance of LLaMA2-13B on MMLU-Others can be considered as having a relatively constant µppa, rather than being contrary to our conclusion. Appendix H also demonstrates a theoretical proof.

heading: 3 Methodology
text: According to the previous analysis, we can mitigate the selection bias of LLMs during SFT by enhancing their MCSB abilities.

heading: 3.1 Symbol-Content Binding
text: Initially, we incorporate the option symbols as the LLM’s target tokens during training, resulting in a model πSymbol. Given input x and output ySymbol, we define optimization objective as:
LSymbol = − ∑ t logPπSymbol(y Symbol t |x, ySymbol<t )
|ySymbol| . (4)
As shown in Table 1, πSymbol suffers from severe selection bias. Since enhancing the MCSB capabilities can effectively alleviate the selection bias, given output ySCB, we propose the Symbol-Content Binding (SCB) debiasing method πSCB, which incorporates both the option symbols and contents as the LLM’s target tokens during training:
LSCB = − ∑ t logPπSCB(y SCB t |x, ySCB<t )
|ySCB| . (5)
However, the results are not as expected. As shown in Table 2, there is no clear pattern indicating that πSCB has a lower bias compared to πSymbol.

heading: 3.2 Reweighting Symbol-Content Binding
text: Actually, label words are anchors (Wang et al., 2023a), which LLMs pay more attention to. On the other hand, the answer content merely plays an auxiliary role, assisting the model in comprehending the actual content of the corresponding symbol. Thus we adjust the weights of the option symbols and contents in the optimization objective, termed Reweighting Symbol-Content Binding (RSCB). The objective function is defined as:
LRSCB = LSCB + (1− ps)α · β · (− log ps). (6)
Where ps is the predicted probability of the symbol token, β is the re-assigned weight for the symbol token. Considering that LLM itself has already focused on the symbol tokens for simple samples, there is no need to emphasize the symbol tokens specifically in such cases. Therefore, we ultimately employ the Focal loss (Lin et al., 2017).

heading: 3.3 Point-wise Intelligent Feedback
text: Human feedback allows LLMs to identify issues with accuracy, fairness, and bias (Liu et al., 2024). Previous studies have explored how to incorporate
human feedback with various stages during LLM’s training process, such as pre-training (Korbak et al., 2023), SFT (Yuan et al., 2023), Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al., 2020; Xue et al., 2023), and others.
In the context of MCQs, we possess knowledge of both the positive options’ symbols and contents, as well as the negative options. We can also easily acquire negative symbol-content binding examples. We call this process Intelligent Feedback without human preference annotations. Moreover, our feedback is intrinsically point-wise, i.e., with absolute scores, the reward score for positive samples should be 1 and be λ for negative samples. We refer to this approach as Point-wise Intelligent Feedback (PIF). In this method, we aim to maximize the probability of positive examples, approaching 1, which can be optimized by cross-entropy loss, and minimize the likelihood of negative examples falling below λ. Inspired by RRHF (Yuan et al., 2023), we optimize the object of negative samples as follows:
LPIFn = max(0,
log λ− ∑ t logPπPIF(y PIFn t |x, yPIFn<t )
|yPIFn | ). (7)
Consequently, the overall optimization objective can be summarized as follows:
LPIF = max(0,−γk·
(
∑ t logPπPIF(y PIF t |x, yPIF<t )
|yPIF| − log λk)), (8)
where k is utilized to differentiate between the positive and negative samples. In our experiments, γpos = 1, γneg = −1, λpos = 1, λneg is a hyperparameter defined in Section 4.1.
We also find that the performance of πPIF is related to its initial parameters. We ultimately initialize πPIF using the parameters from πRSCB. Appendix D demonstrates the ablation study.

heading: 4 Experiment

heading: 4.1 Implementation Detail
text: Samples For PIF When employing PIF, the inclusion of negative samples in the optimization objective is essential. Given that the ultimate goal is to enhance the LLMs’ MCSB performance, negative samples are constructed by randomly combining the incorrect option contents with all candidate option symbols. For instance, we can construct a negative sample "B: 33KV." for the example presented in Figure 3. Due to the limited computational resources, we randomly select one negative sample for each instance. What’s more, in our experiment, the parameters of πPIF are warmed up from πRSCB. Metrics We finally employ four evaluation metrics. µbias is defined by Equation 1 to measure the LLM’s selection bias. µppa is defined by Equation 2 to represent the LLM’s MCSB capability. "Acc" refers to the LLM’s accuracy performance on the standard test set of the current benchmark. Accmin is the minimum accuracy after performing the answer-moving attack, in which case LLMs with higher Accmin exhibit greater robustness.
Fine-tuning Hyper-parameters and prompts used in our experiment can be found in Appendix C.

heading: 4.2 Main Results
text: Table 3 contains the main experimental results of comparison between four methods defined in Section 3, which are called "Symbol", "SymbolContent Binding (SCB)", "Reweighting SymbolContent Binding (RSCB)" and "Point-wise Intelligent Feedback (PIF)". Figure 4 demonstrates a more apparent trend in the results’ changes. Appendix G also reports the accuracy after the answermoving attack for each method on each benchmark.
The SCB method combines the option symbols and option contents as the prediction target tokens. However, this method can not reduce selection bias or enhance the LLM’s MCSB capabilities across all datasets and LLMs. In fact, SCB can even exacerbate selection bias for LLaMA2-13B.
Conversely, RSCB balances the weight of option symbols and contents in the loss function based on SCB. This method enhances the abilities of MCSB on almost all datasets, which helps to reduce selection bias. This finding implies that symbols play a more significant role in MCQs learning process.
Finally, we conduct experiments on our PIF method. This method further reduces selection
bias for nearly all datasets. It is noteworthy that for LLaMA2-7B’s performance on CSQA and LLaMA2-13B’s performance on the Social Science benchmark, the inherent bias of the models is already relatively small, at 1.3 and 1.9, respectively. Under these circumstances, the bias of our PIF method is essentially on par with the bias of the Symbol method. All these results validate the effectiveness of our constructed negative samples and the proposed point-wise objective function.
Moreover, based on the results presented in Table 3, it is evident that an enhancement in MCSB capability leads to a reduction in selection bias. This finding reaffirms our initial hypothesis:Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors.

heading: 4.3 Impact of Our Methods on Accuracy
text: Essentially, while ensuring the stability of LLMs in MCQs, we also aspire to achieve higher accuracy. In Table 4, we present the performance of four methods in terms of accuracy and also the minimum accuracy after the model has been subjected to an answer-moving attack. This aims to demonstrate whether the current method can guarantee accuracy when faced with adversarial attacks.
As we can see, the simple SCB method can not lead to an improvement in accuracy, and even causes a decrease in accuracy on LLaMA2-13B. On the other hand, by emphasizing the importance of symbols, RSCB achieves a similar level of accuracy as the method trained solely on symbols. Moreover, due to the significant reduction of selection bias in the RSCB method, it outperforms Symbol in terms of the Accmin metric.
We are pleased to report that the PIF model out-
performs the other three methods in terms of both Acc and Accmin metrics, yielding the most favorable results. This outstanding performance is attributed to the process of randomly selecting contents from incorrect options and combining them with all symbols while creating negative instances in the PIF model. This approach not only highlights incorrect binding relationships but also exposes the wrong option contents to the model, resulting in an enhancement of the model’s accuracy.

heading: 4.4 Discussion
text: What would happen if there were no bias in SFT training data? Although PIF has mitigated selection bias, there is a more aggressive debiasing approach. Considering that enhancing the model’s MCSB capability can alleviate selection bias, similar to the calculation of µppa, we randomly combine symbols and contents to obtain all K! possible arrangements as training data and train the model. We refer to this method as "Perm". Due to the limited computational resources, we randomly select 1 K! of the original training set for Perm. The results are shown in Appendix H. Perm indeed significantly alleviates selection bias, demonstrating that LLM’s selection bias during SFT primarily stems from the training data utilized in the SFT. This is also consistent with the conclusion in Section 2.2.
It is practically impossible to implement the Perm approach as it requires using all possible random combinations, which is unrealistic due to the high computational resources and training time required. In contrast, PIF can effectively alleviate the selection bias by introducing just one negative example during the SFT process. This further confirms the effectiveness of our approach.
PIF vs. Data Argumentation. Perm method can significantly reduce selection bias, but this approach becomes unfeasible due to its high computational resources. How about we only use a portion of the data in PIF for Data Augmentation? The resources consumed by this method are comparable to those of PIF. To verify the feasibility of this method, which is called Argum., we add an additional experiment based on LLaMA2-7B. Considering that PIF generates a random negative sample for each sample during training, we also augmented the original training data by randomly shuffling the combination of symbols and contents for each sample. The results are as Tabel 7. As can be seen, under the same time consumption, simple data augmentation does not perform better than PIF. Since our PIF method is derived from RLHF, the rationale for its effectiveness is the same as that of RLHF, we introduce negative feedback information to the model, informing it that such negative examples cannot be generated, thereby improving the model’s performance. Point-wise vs. Pair-wise. We propose Point-wise Intelligent Feedback to resume the constructed negative samples. How about incorporating these feedback data using a Pair-wise method? We implement DPO (Rafailov et al., 2023), and the results in Appendix F show that PIF outperforms both alleviating selection bias and achieving higher accuracy compared to DPO. The results prove that it is rational to make the prediction scores of the negative examples approach a minimal value λ. However, when the samples can be optimized with absolute scores, the pair-wise method will lose the information of the gap between the pairs (Cai et al., 2023).

heading: 5 Related Work
text: Large Language Models In recent years, the scale of the model parameters has progressively increased with the rapid development of deep learning, from 1.5 billion in 2018 (Radford et al., 2018) to 540 billion in 2022 (Chowdhery et al., 2023). A significant number of large language models have swiftly surfaced, especially after the emergence of chatGPT(OpenAI, 2022), such as LLaMA(Touvron et al., 2023), Vicuna(Chiang et al., 2023), and BLOOM(Workshop et al., 2022). These large language models have a vast amount of parameters. After being trained with massive data using generative methods, they possess the impressive ability of natural language understanding and can follow hu-
man instructions effectively (Ouyang et al., 2022). Selection Bias in Multiple-Choice Questions The robustness and vulnerabilities of large language models have always been an important research realm. Many researchers have explored how large language models are influenced by modifications or adversarial attacks that impact individual instances in few-shot learning. For example, Zhao et al. (2021) found that large language models are easily affected by changes in task instructions and context when performing tasks. Wang et al. (2023b) and (Zheng et al., 2023b) reveal that GPT-4 tends to choose the option in the first position.
MCQs are widely used to assess the capabilities of large language models. Various MCQs datasets are introduced as standard language model benchmarks, such as MMLU(Hendrycks et al., 2021), C-Eval(Huang et al., 2023), CSQA(Talmor et al., 2019). LLMs have achieved human-like performances on various MCQ benchmarks. However, many researchers have noticed that LLMs suffer from selection bias in MCQS. Pezeshkpour and Hruschka (2023) shows that large language models are sensitive to the order of choices. Zheng et al. (2023a) found that LLMs are vulnerable to option symbol changes in MCQs due to their inherent "token bias". Robinson et al. (2023) shows that a model with high multiple choice symbol binding(MCSB) ability performs better in MCQs. However, previous studies have only explored the selection bias of LLMs in few-shot scenarios. Our work demonstrates that the selection bias still exists in SFT and mainly stems from the LLM’s inadequate multiple choice symbol binding capability. Human Feedback in LLMs OpenAI (2022) and Schulman et al. (2017) have demonstrated the potential of applying reinforcement learning in LLMs and its impressive performance. One part of the main techniques is learning a reward function from human feedback for reinforcement learning, which is often dubbed as RLHF(Christiano et al., 2017) (MacGlashan et al., 2017) (Lee et al., 2021). However, RLHF is a complex, expensive, and often unstable procedure. Lee et al. (2023) offers a promising alternative that uses a powerful LLM to generate preference instead of human annotators. To mitigate the problem of PPO’s sizeable computational cost, Rafailov et al. (2023) introduces a stable, computationally lightweight method to solve the standard RLHF problem. Liu et al. (2024) convert all types of feedback into sequences of sentences, which are then used to fine-tune the model
to learn human preference. The methods above mainly focus on pair-wise preference data. To use point-wise preference data, Cai et al. (2023) develop a point-wise preference learning method. In this paper, we propose PIF which designs a pointwise loss to incorporate negative samples into SFT.

heading: 6 Conclusion
text: This paper highlights that selection bias persists in the SFT of LLMs in MCQs. To explore why LLMs suffer from selection bias, we suppose that "Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors". We utilize MCSB capability to represent the LLMs’ ability to bind option symbols and the corresponding content and design experiments to establish the relationship between MCSB capability and the selection bias. Finally, we eliminate selection bias by enhancing the model’s MCSB capability. We propose PIF, constructing negative samples by randomly combining the content of incorrect contents with all candidate symbols and designing a point-wise loss to resume these negative samples. Comprehensive experimental results demonstrate that PIF significantly reduces selection bias and substantially improves the accuracy of LLMs for MCQs.

heading: 7 Limitations
text: Due to limited computational resources, there are some deficiencies in our work and we list here for future reference. Firstly, we conduct experiments on LLMs with 7B and 13B parameter sizes. In the future, we are eager to understand the patterns of selection bias during the SFT phase in LLMs with even larger parameter sizes such as 70B. Secondly, during our experiments, we select one negative sample randomly for each instance. We would like to investigate the correlation between the severity of selection bias and the percentage of negative samples introduced. Finally, our method has a broader range of applications. For instance, when using LLMs as human preference annotators, they prefer the responses in a specific position. In this case, we can assign some symbols to all responses and then use PIF method to eliminate the bias. We leave these limitations for future work.

heading: A Statistics of Benchmarks
text: Statistics of all benchmarks used in our paper is illustrated in Table 5.

heading: B LLMs Project
text: LLaMA2-7B https://huggingface.co/metallama/Llama-2-7b-hf LLaMA2-13B https://huggingface.co/metallama/Llama-2-13b-hf Implementation Framework https://github.com/hiyouga/LLaMA-Factory
C Implementation Details
Fine-tuning Hyper-parameters We assign α and β of LRSCB to 2 and 0.1, respectively. For λ of LPIF, we conduct experiments with a set of values {0.0001, 0.001, 0.01, 0.1}, and select the one that achieves the best performance on the validation set. We configure the sequence length, epoch, the maximum number of new tokens generated as 1024, 3, 4, respectively. For learning rate, we experiment with the value set {1e− 5, 5e− 5, 1e− 4, 2e− 4}, and select the one that yielded the best performance on validation. In most cases, it is set to 1e− 4. We conduct experiments on 8 GPUs. For the MMLU benchmark, for LLaMA2-7B and LLaMA2-13B models, the batch size per device is set to 4 and 2, respectively, while the gradient accumulation step is set to 2 and 4, ensuring a final total batch size of 64. For CSQA, we adjust the gradient accumulation step to achieve a final total batch size of 32. Additional details can be found in our code. Prompts We don’t use specific prompt engineering in our experiments. Instead, all the experiments are conducted using a simple prompt: "The following are multiple choice questions. You should directly answer the question by choosing the correct option. Question: {{text}} Options: {{text}} Answer: {{text}}"

heading: D Ablation Study on PIF Parameter Initialization
text: In this part, we investigate the effectiveness of the parameter initialization for πPIF, the results are contained in Table 6. Compared to PIFraw, our proposed PIFRSCB performs better in terms of selection bias, MSCB capability and accuracy.

heading: E Results of Permutation
text: Figure 5 illustrates the selection bias of Perm. Perm indeed significantly alleviates selection bias, which demonstrates that LLM’s selection bias during SFT primarily stems from the training data for SFT.

heading: F Results of The Point-wise and Pair-wise Method
text: Table 8 demonstrates the different results between point-wise method PIF and pair-wise method DPO. PIF outperforms both alleviating selection bias and achieving higher accuracy compared to DPO.

heading: G Evaluation Results of Selection Bias
text: All accuracy results after answer-moving attack can be found in Table 9, 10, 11, 12, 13, 14.

heading: H Proof of The Relationship Between MCSB Capability and Selection Bias
text: For a single instance, we denote its PPA as vppa, its selection bias as vbias, its accuracy as vAcc. From Equation 1, vbias = ∑K i=1(|vAcci−vAcc0 |) K . Considering that vAcc0 can only be 0 or 1 for a single instance, then vbias is:
vbias =   
∑K i=1 vAcci K vAcc0=0
1− ∑K
i=1 vAcci K vAcc0=1
(9)
On the other hand, vppa = max k∼K
( K!∑ j=1 yj=ok)
K! . Given a question with K options, there are K! distinct arrangements of these options, we now use a subset, always moving the golden option content to a specific symbol, to represent the overall distribution,
then vppa = max k∼K
( K∑
j=1 yj=ok)
K . If the most frequently predicted option is the correct answer, vAcc0 will have a probability of vppa to take the value of 1, and vppa = ∑K i=1 vAcci K ; else, vppa = ∑K i=1(1−vAcci )
K . Thus we define:
vppa =   
∑K i=1 vAcci K vAcc0=1
1− ∑K
i=1 vAcci K vAcc0=0
(10)
From Equation 9, 10, we can observe that vbias and vppa are inversely related with a probability of vppa. Generally, after SFT, LLMs tend to have higher vppa, Therefore providing theoretical support for our work.
Benchmarks #Samples #Options Golden Answer Distribution
MMLU
train - 99842 4 22.2%/25.8%/26.9%/25.1%
test
STEM 3018
4
21.4%/23.8%/25.9%/28.9% Social Science 3077 21.7%/23.4%/23.8%/31.1%
Humanities 4705 24.2%/24.5%/27.1%/24.2% Others 3242 23.8%/26.8%/24.4%/25.1% Overall 14042 22.3%/24.7%/25.5%/26.9%
CSQA train 8971
5 19.4%/20.3%/20.1%/20.4%/19.9%
test 1221 19.6%/20.9%/19.7%/20.6%19.3%
Figure 5: Selection bias of Perm.
Model Method STEM Social Science Humanities Others MMLU µbias↓ Acc↑ µbias↓ Acc↑ µbias↓ Acc↑ µbias↓ Acc↑ µbias↓ Acc↑ LLaMA2-7B Argum. 2.0 44.1 2.8 62.7 5.9 51.1 1.5 60.4 2.7 54.3 PIF 3.7 44.0 2.3 62.7 3.9 53.1 1.3 60.9 2.7 55.0
Table 7: PIF vs. Data Argumentation.

heading: Future Work
text: Due to limited computational resources, there are some deficiencies in our work and we list here for future reference. Firstly, we conduct experiments on LLMs with 7B and 13B parameter sizes. In the future, we are eager to understand the patterns of selection bias during the SFT phase in LLMs with even larger parameter sizes such as 70B. Secondly, during our experiments, we select one negative sample randomly for each instance. We would like to investigate the correlation between the severity of selection bias and the percentage of negative samples introduced. Finally, our method has a broader range of applications. For instance, when using LLMs as human preference annotators, they prefer the responses in a specific position. In this case, we can assign some symbols to all responses and then use PIF method to eliminate the bias. We leave these limitations for future work