Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think
Authors: Xinpeng Wang
Chengzhi Hu
Bolei Ma
Paul Rottger
Barbara Plank
Venue: COLM (2024)
Keywords: Robustness,  LLM Evaluation, Multiple-Choice Question
TL;DR: We show text answers from instruction-tuned language models have less selection bias than first token-based answers and are more robust to various prompt perturbations.

Abstract: Multiple choice questions (MCQs) are commonly used to evaluate the capabilities of large language models (LLMs). One common way to evaluate the model response is to rank the candidate answers based on the log probability of the first token prediction. An alternative way is to examine the text output. Prior work has shown that first token probabilities lack robustness to changes in MCQ phrasing, and that first token probabilities do not match text answers for instruction-tuned models. Therefore, in this paper, we investigate the robustness of text answers. We show that the text answers are more robust to question perturbations than the first token probabilities, when the first token answers mismatch the text answers. The difference in robustness increases as the mismatch rate becomes greater. As the mismatch reaches over 50%, the text answer is more robust to option order changes than the debiased first token probabilities using state-of-the-art debiasing methods such as PriDe. Our findings provide further evidence for the benefits of text answer evaluation over first token probability evaluation.

the log Published as a conference paper at COLM 2024
Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think
Xinpeng Wang Chengzhi Hu', Bolei Mat Paul Röttger", Barbara Plank
LLU Munich, Bocconi University
Munich Center for Machine Learning (MCML)
[xinpeng. wang, bolei.m ma, b. plankk) @lmuudd de, chengzhi. hu@campus. 1mu de, rou..
Abstract
Multiple choice questions (MCQs) are commonly used to evaluate the capabilities of large language models (LLMs). One common way to evaluate the model response is to rank the candidate answers based probability of the first token prediction. An alternative way is to examine the text output. Prior work has shown that first token probabilities lack robustness to changes in MCQ phrasing, and that first token probabilities do not match text answers for instruction-tuned models. Therefore, in this paper, we investigate the robustness of text answers. We show that the text answers are more robust to question perturbations than the first token probabilities, when the first token answers mismatch the text answers. The difference in robustness increases as the mismatch rate becomes greater. As the mismatch reaches over 50%, the text answer is more robust to option order changes than the debiased first token probabilities using state-of-the-arr debiasing methods such PriDe. Our findings provide further evidence for the benefits of text answer evaluation over first token probability evaluation.
1 Introduction
The open-ended nature of autoregressive language generation complicates the evaluation of Large Language Models (LLMs). One popular solution to this problem is prompt LLMs with Multiple Choice Questions (MCQs), which limit the answer space to few candidate options, thus enabling evaluation by comparing model choices against gold labels. For MCQs, there are two main ways of extracting model choices from their generated text answers: 1) In the text-based approach, the choice is automatically extracted from the text answer either by pattern matching (Wang et al., 2022) or by prompting strong LLM (Chiang et al., 2023; Li et al., 2023) such as GPT4 (OpenAl et al., 2024). However, pattern matching can often be inaccurate because each model has different response styles, requiring manual feature engineering. This makes pattern matching infeasible when evaluating many tasks and models at the same time. Using a strong proprietary model like GPTT4 as an evaluator, on the other hand, is more flexible but lacks transparency and reproducibility. and also creates high financial costs when we running large-scale evaluations. 2) In comparison, the probability-based approach simplifies the evaluation by ranking the log probabilities assigned to the option IDs (e.g. A/B/C/D) from the first token prediction of the model. This approach is widely adopted in many different benchmarks and LLM evaluation studies (Hendrycks et al., 2020; bench authors, 2023; Liang et al., 2022; Santurkar al., 2023). How- ever, recent works point to mismatch between the text- and probability-base approaches (Lyu et al., 2024; Wang et al., 2024), showing that first token probabilities do not match text answers given by models, especially for models fine-tuned on conversational safety data.
Moreover, recent studies have shown that the probability-based approach lacks robustness that it is sensitive to linguistic properties of the MCQ prompt (Leidinger et al., 2023) or perturbations such as typos, adding options, word swapping (Tjuatja et al., 2023) and option position (also called selection bias, Dominguez-Olmeda et al., 2023; Zheng et al., 2023).

---

Published as a conference paper at COLM 2024
In this paper, investigate the robustness of the text-based approach. We compare it with the first token answer as well as the debiased first token answer using the first token debiasing method PriDe (Zheng et al., 2023), by measuring the answer robustness under various prompt perturbations. To avoid the labor of feature engineering and the high cost of using proprietary model as evaluator, fine-tune Mistral-7b-Insse show that it can reliably and accurately detect the choice from the text answers across models and datasets.
model and
By comparing the choice made in the text answer and the first token probability, we show that: (1) The text answer shows small selection bias ($3.3) and high robustness to various sentence perturbations ($3.5) across all the models we examined; (2) The robustness dis- crepancy between the first token probability and the text answer increases as the mismatch rate between them increases (63.3, $3.5); (3) When the mismatch rate is high (over 50%), the text answer shows smaller selection bias than the state-of-art first token debiasing method PriDe ($3.3). As a whole, our work provides further evidence for the benefits of text-based over probability-basee MCQ evaluations of LLMs.
2 Mismatch between the probability and the text-based MCQ evaluation
Before the rise of instruction-tunel large language models, it was appropriate to check the probabilities assigned to the options in MCQs by the models for several reasons: 1) Discriminative models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are forced only give probabílities with task-specific classification head; 2) liqueenee-o-sequeeee
the available opinions by finetuning on the task
models such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) are good giving the option ID in the next-word prediction since they were specifically trained to perform NLP tasks of various formats but they are not good at "following a user's intent" real-life scenarios (Ouyang et al., 2022); 3) Foundation models such as GPT3 (Brown et al., 2020), OPT (Zhang et al., 2022) have poor Intructtionfollowiing ability which makes the text completion uninterpretable, thus, the prediction of model is measured by ranking the log probability of the available options.
Techniques like Instruction-Tuning (Wei et al., 2021) and Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) improve the model's ability to follow the instructions and give helpful responses. Therefore, the model can directly answer the MCQs and give its choice in the text answer. This makes it less appropriate to probability- based evaluation since the first token probability doesn't 't represent the text answer, even specifically asking to start with the option ID in the prompt (Wang al., 2024). Figure shows an example of the mismatch between the first token probabilities and the text answer model. This mismatch leads the different performance of
given by the Llama2-7b- model on benchmark depending on the evaluation method, as shown in Table 1.
Prompt:
General Instruction: Please read nutiile--hoice question balow caratully and select ONE of the listed options and only give single
First Token Logits:
A
Model (0-shot)
First Token
Text Answer
B
Gemma-70-Inst
30.2
50.8
effectively invented
Question: The Web was
C
Llama2-76-Chin
43.1
BemeersLLee year?
D
Llama2-136-CCB
40.2
47.6
Options:
Mistral-7--Inst-0.2
53.2
53.6
A. 1991
Sure
C. 1989
Table 1: Performance of the models on MMLU with different evaluation meth- ods: First token probability and Text answer. Text answer achieves better performance in general.
D. 1983
b. Toxt Answer:
Sure! The answer is (A) 1991.
Answer:
Figure 1: An example mismatch case between the first token probabilities and the text answer given by the Llama2-76-Che model.
We share the dataset and trained classifiers https: //gi thub. hwt.c///iinlll/M/CCRRRouutteeet
2

---

Published as a conference paper at COLM 2024
3 Robustness evaluation of the probability and text-based approaches
Several works (Dominguez-Olmeet et al., 2023; Zheng et al., 2023) have shown that large language models have selection bias such as preferring " "option A", therefore they are not robust under position change of the options. To mitigate this bias, PriDe (Zheng et al., 2023) was proposed as a debiasing method to improve the robustness of the model by disentangling the token bias from the first token probability. Tjuatja et al. (2023) shows that LLMs are also sensitive to perturbation in the question such as typos, word swapping and adding more options. However, this line of work only focuses on the first token evaluation. It intuitively makes sense that the probabilistic mass on a single token is sensitive to its context and is biased towards certain token ('A') which leads to selection bias (Zheng et al., 2023). However, it is unknown whether the text answers given by the instruction-tuned models are also sensitive to such changes. Therefore, we answer this question by comparing the robustness of the text answers with the answer estimated by original/ /ebiased first token probability (referred to first token in the following).
3.1 Experimental setup
Models Our study focuses on open-source instruction-tunee language models since we are interested in the text response to the user's 's instruction, as well as committing to open science and reproducibility. We cover three model families including Llama2 (Touvron et al., 2023), Mistral (fiang et al., 2023) and Gemma (Team et al., 2024). We do greedy decoding during the inference time. We also trained free and robust MCQ evaluators using Mistral-7b models which we plan to make public. For simplicity, we drop the postfix "Inst/Chat" later.
Benchmark We conduct the experiments on MMLU (Hendrycks et al., 2020) which consists of 4-option MCQs with tasks covering 57 subjects ranging from elementary mathematics to US history. Note that the original implementation of the debiasing method PriDe was done on the aggregated dataset level. In our robustness evaluation experiments, we believe it is instructive to go beyond the aggregate level: we employ and compare with PriDe on each task individually in order to gain deeper insight into the behavior of the model. We additionally use OpinionQA (Santurkar et al., 2023) inspired by Wang et al. (2024) to test the robustness of our trained classifier which will be introduced later.
Prompting To give a fair comparison between the text answer and the first token evalua- tion, we specifically guide the model to respond directly with the option ID. We adopt the system prompt used in Wang et al. (2024) where the model is specifically asked to answer with a "single letter": Please read the multiple-choice question below carefully and select ONE of the listed options and only give a single letter. Our preliminary experiments show that it is imporlant to add the constraint to respond with single letter instead of just asking it to "choose the option" used in prior works. Without such a constraint, the model will prefer to repeat the option content instead of the option ID.
Probability- evalution We extract the log probabilities assigned to the token corre- sponding to the option IDs ("A", "B" "D") from the model and take the one with the highest probability as the answer. In our experiment, we see issues of only looking at the first token since different models have diverse response patterns. For example, the Llama2 models will always start the response with space token " and the Gemma model starts its answer by repeating the word "Answer:" (e.g. " "Answer: C") most of the time. Therefore, we also use the second token of the Llama2 model and the third token of the Gemma model, which indeed leads to slightly higher robustness results, but still lower than the text answer.
Text-based evaluation To extract the choice from the text answer automatically, we follow Wang et al. (2024) to construct a classifier by training a language model. We test five different language models: T5-small, T5-base, T5-large, Mistral-7b-Base- 1 and Mistral-7b-Instrude v0.2. We finetune the Mistral models using QLoRA (Dettmers et al., 2024) implemented by Huggingface (Mangrulkar et al., 2022). See Appendix A.1 for training hyperparameters.
3

---

Published as a conference paper at COLM 2024
Test Data
Original Options
Extra Options
Training
Model
Aoc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Mistral
Llamaz
Gemma"
Mistral
Llama2
Gemma"
AVG
Mistral-7b: Base
0.993
0,870
1.000
1,000
0.997
0.995
1.000
1.000
0.993
0.711
1,000
1,000
0.997
0.929
Mistral 7bb Inst
0.993
0.870
0.997
0.874
0.993
0.992
1.000
1,000
0.997
0.997
0.997
0.997
0.996
0.955
MMLU (ID)
TS-small
0.986
0.987
0.919
0.913
0.963
0.965
0.986
0.570
0.878
0.665
0.916
0.380
0.941
0.747
T5 base
0.979
0.960
0.997
0.997
0.963
0.964
0.986
0.667
0.878
0.665
0.990
0.664
0.966
0.823
TB T5-large
0.990
0.870
0.997
0.997
01980
0.860
0.986
0.570
0.875
0.482
(1983
0.663
0.968
0.740
Mistral-7--Base
0.993
0.592
0.997
0.997
0,950
0.870
0,979
0.689
0.875
0.537
1,000
1,000
0,972
0.842
Mistral--b--Inst
0.997
0.873
0,986
0.983
0.977
0.856
0.986
0.665
0.878
0.551
1,000
1,000
0.971
0.821
OAinion
T5-small
0.973
0.992
0.989
0.953
(1.86
0.6997
0.986
0.745
0.871
0.353
0.920
0.220
0.916
0.673
(OOD)
15-base
0.986
(1.866
0.986
0.863
0.906
0.728
0.977
0.689
0.875
0.345
01963
0.355
0,953
0.664
T5-lange
0.979
0.860
0.993
0.671
0.980
0.768
0.983
0.569
0.864
0.435
0.993
0.795
0.965
0.716
Table 2: Performance of trained text answer classifiers. Both Mistral-7b-Bine and Mistral-7b- Inst achieve good performance on test data. *We trained our classifiers only on responses from Mistral-7b and Llama2-7b models while also testing on responses from Gemma-7b.
Annotation scheme and classifier training
Table 3:
Example of manually annotated data used for training our classifier.
The annotated data used for training the clas- sifier is constructed as this:
Input
Sure! The least common multiple (LCM) of 4 and 10 is 40, so the answer (C) 40.
Input: The model text response, alongside the multiple choice options.
References:
Label: The ID of the correct option as deter- mined by the model.
A. 14
B. 20
Table 3 is an example of the annotated model out- put. It important to add references to the input since there are cases where the model gives an answer outside of the references ("No correct answer", "Refusal", "I don't know"). We opt to annotate such cases as: {X: No correct answer, Y: Refusal, Z: I do not know). Therefore, our classifier is required to be able to detect such cases when the model gives an answer outside of the options. In our answer floating experiments where we test if the model will change its answer with additional options, we add these three cases into the options. Therefore, we need to train two classifiers for the two cases, each of which has 7 classes.
40
D. 60
Label C. 40
We annotated total of 600 samples for Gemma and 1600 samples each for the two 7b models of Llama2 and Mistral. For each model, we have the same amount of samples from two subsets: one with (Extra Options) and the other without (Original Options) additional options. See Appendix A.2 for data details. From these annotated samples, we used total of 2000 samples for training our classifiers on Llama2 and Mistral outputs. Subsequently, tested the classifiers on outputs from Llama2, Mistral, and Gemma models, presented in Table 2. To test the robustness of our classifier, we also evaluate the model trained on out-of-distribution data (responses to OpinionQA) provided by Wang et al. (2024).
The result in Table shows that the Mistral models provide better results than the T5 models as classifiers. The models trained on in-distribution (ID) data have near-perfect results. Even when trained on ciito-ddiitrriuuion (OOD) data, the performance gap in accuracy remains minimal, ranging from 1% to 4% in most cases. The successful performance of our classifier shows that it can be used in other domains as an evaluator. Furthermore, we trained our classifiers only on responses from Mistral and Ilama2 models. Remarkably, when tested on responses from Gemma models, which were not included in the training data, the classifier performed comparably well, underscoring its cross-model robustness. should be noted that the Gemma model's 'ss regular response pattern can explain the good performance on the Gemma's 's output. Our classifier is not optimized for extracting MCQ answers from any models, especially the ones with significantly different response styles. It is important to include diverse response styles from more model families into the training data to improve the robustness further.
4

---

Published as a conference paper at COLM 2024
3.2 Metrics
Standard deviation of recalls We follow the metric proposed by Zheng et al. (2023) to measure the selection bias by calculating the standard deviation of recalls (RStD) of each option. The imbalance of the recall of the option IDs indicates that the model has selection bias towards a certain option ID, lower RStD score indicates greater model robustness to changes in MCQ option positions.
--B-P2- (rA - +) (rB- f)2 + (rc - F)2 (rD- P)2
(1)
4
where the mean recall and YA, rc and rp are the recall for options A, B, Cand
Entropy The RStD score based on the assumption that the correct answer is placed randomly. We introduce a second metric to assess the model's robustness by measuring the entropy of answers from multiple runs with random perturbations. For example, to test the model's robustness to option position change, we shuffle the positions of option contents N times while keeping the positions of option IDs and record the answer A, each time. The entropy is calculated as follows:
m H(A) = Σ P(A.)) P(A;)
(2)
where m is the number unique answers of the runs, and P(A,) is the probability of answer A; occurring (which is the number of times A; occurs divided by N). A lower entropy means higher answer consistency under perturbation.
0.6
First Token
First Tokan
Text Answer
0.30
0.4
0.25
0.20
0.2
0.1
0.0
0.00
Gemma:78 7b
Liama2-7b
Llama2-13b
Mistral-7b
Gemma- 7b
Lluma2-76
Llama22-33
Mistral-7b
Figure 2: Accuracy and selection bias results. A lower RStD score means smaller selection bias. As the mismatch rate decreases from Gemma (56.8%) Mistral (10.2%), the perfor- mance gap between the first token (red) and text answer (blue) decreases. Text answers from Gemma and Llama2 have lower selection bias than the debiased first token answers.
3.3 Selection bias result
We compare the selection bias and accuracy of the text answer, first token answer and the debiased ver- sion of first token answer using PriDe in Figure 2. Except Mistral-7h, both PriDe and text answers show lower selection bias than the first token answer on all models, while text answers perform on par with or better than the debiased first token evaluation. terms of accuracy, text answer archives higher scores than the original and debiased first token answers among all the models except for Mistral-7b where the first token answer is slightly higher than the text answer. To understand why Mistral-7b behaves differently, we calculate the mismatch rate between the first token and text answer shown in Table 4. Mistral-7b has the lowest mismatch rate among all the models since it has good instruction-follloing ability to start its first token with the option ID. Therefore, its first token answer shows
Model
Mismatch Rate
Mistral-7b-Insse
10.2%
Llama2-13b-Che
35.3%
Llama2-7b-Chn
51.4%
Gemma-7b-Inst
56.8%
Table 4: Mismatch rate of first token probabilities and text outputs.

---

Published as a conference paper at COLM 2024
significantly lower selection bias compared to other models with high mismatch rate and performs similarly with its text answer. Using the first token debiasing method can indeed further lower the selection bias and improve the accuracy of the model. On the contrary, Gemmaa-h shows the highest selection bias and mismatch rate of 56.8%. Surprisingly, there is huge performance gap between the text answer and the first token answer, ter debiasing. The text answer outperforms the debiased first token answer (PriDe) on Gemma-7b and Llama2-7b where the models have a mismatch rate over 50%. This shows that instruction-tuned language models have less selection bias than we think. In many cases, the first token debiasing method may not be necessary since the text answer is already robust, such as Gemma-7b and Llama2-7b.
af-
3.4 Perturbations
To fully test the robustness of the two evaluation approaches, we further incorporate a series of different perturbation types for the prompt, based on the response bias modifications and non-bias perturbations from Tjuatja et al. (2023). Table 6 summarizes the 5 perturbation types we used. For each question, we perturbated the question four times and calculated the answer entropy except Additional options, where calculated the rate of answer change. Given the evidence that the first token answer has a higher selection bias than the text answer, need to disentangle the influence of the question perturbation and the option order. Therefore, we shuffle the option order five times for each perturbation and take the majority-voted option content for calculating the entropy, resulting in 20 runs for each perturbation type except Option Swap, where we study the impact of the option order itself.
3.5 Perturbation robustness results
Answer consistency Table 6 shows answer consistency results for both evaluation ap- proaches under the perturbation Letter Typos, Letter Swap, Word Swaap Option Swap measured by entropy. Similar to the selection bias result, text answer is consistently more robust to all the perturbations on all models except Mistral-7b, where first token shows a minor difference compared to the text answer. The slightly higher entropy of the text answer of Mistral-7b can explained by the special cases where the model claims no correct answers available or refuses to answer the question due to safety reasons (see section 3.6 for detailed discussion), which leads to larger option space than the first token answer. We observe that larger model size leads to more robust first token and text answers: the model size increases from 7b to 13b for Llama2 entropy decreases. Moreover, we note that the position of the option has the largest impact the answer since the entropy of both approaches is the highest under Option Swap for all models. The robustness discrepancy between the first token and the text answer also the largest under the Option Swap than the other perturbation types.
Model
Mode
Letter Typos
Letter Swap
Word Swap
Option Swap
First Token
0.32
0.39
0.25
0.52
Mistral 7b
Text Answer
0.34
0.41
0.27
0.52
First Token
0.60
0.62
0.53
0.87
Gemma 7b
Text Answer
0.40
0.44
0.32
0.56
First Token
0.63
0.66
0.55
1.04
Llama2 7b
Text Answer
0.40
0.44
0.32
0.68
First Token
0.53
0.54
0.48
0.96
Llama2 13b
Text Answer
0.33
0.38
0.26
0.62
Table 5: Answer entropy under different perturbation types. Lower entropy indicates higher answer consistency under the perturbation. Text answer is more robust on all perturbation types across al the models except Mistral 7b where text answer performs similarly as first token asnwer.

---

Published as a conference paper at COLM 2024
Original question
A multiple choice question
Question: Which social psychological principle best explains
prejudice?
Options:
A. self-serving bias
B. in-group bias
C. individualism
D. collectivism
Letter Typos
With low probability of around 0.2, randomly change one letter in each word of the question:
Question: Which social polhagcoiyscl pinrplcie prejudice ?
bset explains
Letter Swap
We randomly swap the characters of each word with length bigger than 3 in the question, excluding the first and the last letter
Question: Which social psychologicas pdinciple best explains
Irejudice ?
Word Swap
We randomly swap the order of four words the question, excluding the first and the last word
Question: Which social psychological explains prejudice ?
best principle
Option Swap
We randomly swap the order of choice options
Question: Which social psychological principle explains
prejudice?
Options:
A. in-group bias
B. collectivism
C. individualism
D. self-serving bias
Additional Options
We add three additional options, which represent three out-of-choice options. They are: "No correct answer", "Refuse", and "I don't know".
Question: Which social psychological principle best explains
prejudice?
Options:
A. self-serving bias
B: correct answer
C: Refuse
D. in-group bias
E. individualism
F. collectivism
G: I do not know
Table 6: Pertubation types inspired by Tjuatja et al. (2023).
Answer floating We also test the model's ro- bustness to adding opinions by checking the percentage of the questions where the model changes its answer (answer floating). Figure 3 shows the answer floating result under the perturbation Additional Options. The first token and text answers here are majority-voted after shuffling the option orders and we use the de- fault option order to assign the option IDs to the answers. For Llama2 and Gemma models, there is significant percentage of questions where the model changes its first token answer after adding three more options (Refused, I do not know and No correct answer is given), reach- ing to nearly 70%, which is much higher than the text answer. For Mistral-7b, both text and first token answers achieve similar answer floating rates at around 25.5%.
Text Answer
RSD
0.2
0.0
Liamad-130
Figure 3: Answer Floating Rate. Text an- swers are more robust to adding options, except Mistral.
7

---

Published as a conference paper at COLM 2024
To further know about the answer floating behaviour, zoom into the answer distribution in Figure 4. In all models, especially Llama2-7b-Chan and Llama2-13b-Chat, the first token answer shifts sigmificantly after adding the three additional options. The number of answers shifted to the three additional options is not negligible, taking up to 14.2% of the total responses for Llama2-133--aB
After adding the options, the distribution of answers going to the three additional options is flatter compared to the text answer result. This could mean that the model selects them by chance. Again, this shows the brittleness of the first token evaluation, whose result can easily be shifted by adding new options.
Looking at the text answer result, we see minor distribution shift after adding the options in all models, Note that the text answer can choose to "refuse" or claim "no correct answer given" when those are not in the original options. Before adding the options, we observe different answer patterns from different model families. The Gemma model rarely gives answer outside of the options, while the Llama2 models have high refusal rate, especially for the 7b model. The Mistral model, interestingly, tends to claim there are correct answers more instead of refusing or showing uncertainty.
After adding the options, Llama2-7b-Cal tends to keep the refusal answers unchanged which likely due to the strong safety guardrail which can easily triggered by the keywords in the question. Mistral-7b-Instt unlike other models, indeed chooses more from the added options than without them, and it is the only model that actually expresses uncertainty in the text answer such as "I do not know", which is also observed during the annotation phase.
First Token
Original Options
4000
Original Options
Extra Options
Extra Options
Gemma-7b
2000
2003
1000
4000
Original Options
Original Options
Options
3000
Extra Options
Llama-78
3000
2003
1000
0
4000
Original Options
4009
Original Options
Extra Options
Extra Options
Llama2-13b
3000
3000
2000
2000
Original Options
Original Options
Mistral-7b
Options
3003
Extra Optiors
2000
2000
A
do not know
No correct answer
Rahuaed
do not know
No correct answer
Rotused
Figure 4: Answer distribution before and after adding additional options. Text answers show less distribution shift after adding additional options. Note that the text answers are not limited to the given options in the original options setting.

---

Published as a conference paper at COLM 2024
3.6 Observations
Psychology
Refusal As discussed in the previous section, refusal behaviour takes a large part the total responses from Llama2-7b-Chat, This has a huge impact on the evaluation result as we inspect the model behaviour in each subcategory of MMLU. Figure shows the selection bias of text, original, and debiased first token answers on 10 subcate- gories based on the top and bottom 5 subcategory performance of PriDe. A detailed overview of the accuracy scores and selection bias of responses from all models in all subcategories shown the Appendix A.3. Surprisingly, the text answer exhibits zero RStD on subcategory Moral Scenarios. As look into the data, it turns out the model refuses to answer all the 895 questions this sub- category due to safety reasons. This reveals the unreliability of the first token accuracy evaluation especially on sensitive topics.
Elementary Mathematics
Miscellaneous
Prehistory
School Mathematics
School Computer
Clinical Knowledge
College Mathematics
Mediccal Genetics
Moral Scenarias
RSID
Rist. Taken
First Tohen (PriDe)
Text Answer
Figure 5: RStD of Llama2-7b-Chat in se- lected subcategories.
Mismatch We see relation between the mismatch rate and the robustness gap between the first token and text answer. As shown in our results in Table 1, 5 and Figure 2 and 3, compared to other models, Mistral-7b exhibits good robustness and low selection bias, with the lowest mismatch rate of 10%. In contrast, the accuracy and robustness level gap is the largest for Gemma-7b-Inst which has the highest mismatch rate of 56.6%. Figure we plot the mismatch rate of the models and their first token text answer difference in terms of accuracy and selection bias evaluated on MMLU. It shows that as the model's mismatch rate increases, the first token answer is less accurate and robust campred to the text answer. Thus, the closer the first token answers are to the text answers, the better the robustness level, whereas the robustness level of the text answer stays high. Therefore, it is recommended to always look at the text answer if the mismatch rate is unknown.
0.30
Gemma-7b-Inst 0.30
Accuracy Difference
0.25
RStD Difference
0.25
Accuracy Difference
0.20
Llama2-7b-Chat
0.20
iffeeeeee
Llama2-13b-Char
0.15
0.15
R5tD
0.10
0.10
Mistral-7b-Inst-vo.2
0.05
0.05
0.00
0.00
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Mismatch Rate
Figure 6: The absolute difference between the first token and text answer in terms of prediction accuracy and selection bias. The text answer achieves higher accuracy and lower RStd score (less selection bias) than the fist token answer across all the models. The gap increases the model's mismatch rate increases.

---

Published as a conference paper at COLM 2024
4
Related work
Large language models The landscape of large language models (LLMs) has undergone dynamic evolution in recent years. With notable contributions from renowned models such as GPT4 (Achiam et al., 2023) and Gemini (Team et al., 2023), which have significantly advanced the field. Other recent advancements, including models like Mistral (Jiang et al., 2023), Gemma (Team et al., 2024), and Llama2 (Touvron et al., 2023), offer distinct settings and characteristics. These models have been equipped with innovative techniques such as Instruction- SeaeetiionTTinnnngggg and RLHF, enabling them to effectively interpret and adhere to given instructions (Ouyang et al., 2022), and to answer with diverse and natural responses.
Multiple choice questions evaluation Multiple choice questions (MCQs) are widely used to assess the capabilities of LLMs across various domains. They play a crucial role in evaluation benchmarks like MMLU (Hendrycks et al., 2020), AGIEval (Zhong al., 2023), HELM(Liang et al., 2022) and Evaluation Harness (Gao et al., 2021), as well as in evaluating moral beliefs, opinions on public issues, and surveys (Santurkar al., 2023; Scherrer al., 2024; Wang et al., 2024). Traditionally, MCQ accuracy has been measured based on the model's first token prediction (Dominguez-Olmedd However, contemporary LLMs often offer nuanced responses, challenging the reliability of first token evaluation. This discrepancy has been revealed by recent studies (Wang et al., 2024; Lyu et al., 2024).
et al., 2023; Santurkar al., 2023).
Robustness Studies by Dominguez-Olmedo et al. (2023) and Zheng et al. (2023) reveal selection bias (like 'Option A') in LLMs, but they focus solely on the first token of the model's response. Meanwhile, Tsvilodub et al. (2024) and Lyu et al. (2024) explore different evaluation methods' impact on LLM robustness, noting performance variations without clear preference. Research prompt brittleness, such Röttger et al. (2024) and Leidinger et al. (2023), highlights the sensitivity of LLMs to prompt variations, impacting performance and reliability. Further investigation is required to grasp prompt brittleness's full extent and its implications for model robustness.
Conclusion
This work studies the robustness of instruction-tuned language models in the multiple- choice question answering settings. Our research builds upon previous studies which have highlighted the brittleness of first token probability evaluation and the mismatch between first token and text answer. Through extensive perturbation experiments, we demonstrate that text answers generated by instruction-tunel language models are more robust. In cases where the first token answer matches the text answer, both approaches exhibit similar levels of robustness-ing in our experiments this was the case only for one out of the tested models, Mistral. Our findings suggest that the inrrucctiontuunel language models are more robust for text answers than what previously shown in works using the first token probabilities as evaluation. Therefore, we suggest to evaluate LMs in a more detailed and realistic way by directly inspecting the text answer, to have better understanding of the full spectrum of model behaviour. We strongly caution against relying only on first token probability.
Acknowledgments
XW, CH and BP are supported by ERC Consolidator Grant DIALECT 101043235 and in parts by Independent Research Fund Denmark (DFF) Sapere Aude grant 9063-00077B. CH is also supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. BM is supported by BERD@NFDI (German Research Foundation grant 460037581). PR is a member of the Data and Marketing Insights research unit of the Bocconi Institute for Data Science and Analysis, and is supported by a MUR FARE 2020 initiative under grant agreement Prot. R20YSMBZ8S (INDOMITA).
10

---

Published as a conference paper at COLM 2024
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiv preprint ar arXiv:2303.08774 2023.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research, 2023, ISSN 2835-8856.
URL
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Adounces in Neural Information Processing Systems, 36, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653 1/////11/// 11/1//1111111113.11 URL httpp:://allanthology org/N19-1422 1423.
Ricardo Dominguez-Olmedo Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. arXiv preprint arXiv:2306.07955) 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https: //doi.org/1e 0 00115528122//// 5371628.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. aiv preprint ar drXiv:2310.06825, 2023.
Alina Leidinger, Robert van Rooij, and Ekaterina Shutova. The language of prompting: What linguistic properties make a prompt successful? In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9210-9232, 2023.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeling of the Association for Compulational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 /1 2020.acl-main.. URL httpss///allanhhhlloooo org/ 2000 acl-main 703.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followingg models. https://gi http:///ggttub.cccc httuub.coo/tattu-laallllaee aalaaaaaaccaaaa eval, 2023.
11

---

Published as a conference paper at COLM 2024
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Xiv preprint ar Xri::2211.091100 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ariiv preprint ar arXiv:1907.1166:: 2019.
Chenyang Lyu, Minghao Wu, and Alham Fikri Aji. Beyond probabilities: Unveiling the misalignment in evaluating large language models. arXiv preprint arXiv:2402.13887, 2024.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-ar parameter-efficient fine-tuning methods. https: l/github.com comm/huggingface///oooooooooo pef 2022.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapin Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reüchiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Iseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei,
12

---

Published as a conference paper at COLM 2024
CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277444 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: httt:://jmlr.org/pppppppp.e wwwgppppps//v1/////// /atps//vv//////// 20021/22007444 004.0074.000 html.
Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. Xrviv preprint ar axXit:2402..6766, 2024.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? ArXiv, abs/ 0bss/2333.1544 2023.
URL https://pppi htt.ppp.ssmaaiiscoollla. wwwpccholaaoorg//oorrooee
2078340488
Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Bapp Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. Xrv preprint ar arX:::2312.11805 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXio preprint #XXt:2403.08295, 2024.
Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. Do Iims exhibit human-like response biases? case study in survey design. arxiv preprint ar 1xix:2311.04076 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Polina Tsvilodub, Hening Wang, Sharon Grosch, and Michael Franke. Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods. Xrviv preprint ar Xrx::24403.0099 00998, 2024.
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. " my answer is c": First-token probabilities do not match text answers in instruction-turee language models. arXiv preprint X14:2400.11499, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ar Xie preprint ar Xiv::2203.11171, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. Xrv preprint ar XrXt::2205001068, 2022.
13

---

Published as a conference paper at COLM 2024
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXio preprint arXio:2304.06364 2023.
A
Appendix
A.1
Hyperparameter for the text answer classifier training
We finetuned five different language models: 15-small, 15-base, 15-large, Mistral-7b-base- v0.1 and Mistral-7b-Instruct-vbb toretrrutttt.tffo for the training text answer classifier. Table 7 shows the hyperparameters and their corresponding values for the T5 models. For the two mistral models, we used QLoRA (Dettmers al., 2024) and the default parameter settings from the Huggingface PEFT repository (Mangrulkar et al., 2022). Table 8 shows the hyperparameters and their corresponding values for the Mistral models.
Hyperparameter
Value
learning -ate
2e-5
train bainhbatch_size
weight decay bfl6 num_train con__trai_eepchhs
0.01
True
Table 7: Hyperparameters for training T5 models
Hyperparameter
Value
lora
64
lora_alpha
16
lora_dropout
0.1
task_type
"CAUSAL.LM"
use-4bit
True
mnb.4bit.compute... ddtpe
"float16"
bnb 4bit quant type
"nf4"
nested quant num_train_epoche train.batch_size
False
8
4
gradient accumulation steps
1
meadient.checkpoone
True
max_grad_nom
0.3
learning rate
2e-4
reiggt_ddeay optim
0.001
adamm "32bit"
It.scheduler.type
"constant"
warmup ratio group by length
0.03
True
Table 8: Hyperparameters for training the Mistral models
A.2 Annotation scheme
The annotation process was carried out by two in-house annotators, who were presented with the model outputs and the relevant multiple-choice questions. By reading the model
14

---

Published as a conference paper at COLM 2024
response, the annotator has to decide which option the model is referring to. In cases where the model contradicts itself, such as choosing 'A' followed by describing the option content of 'B', we annotate this case failure mode 'NaN', which is not considered in this work. After the independent annotation process, two annotators discuss the samples where disagreement occurs and resolve the conflicts. Table 9 shows the data statistics of our annotated data. It is noteworthy that in the original options setting, we never observe cases where the model expresses "I do not know", leading to number of Z. However, when adding it into the options, we do often observe that model choosing "I do not know", especially in Mistral--b-Inst--0.2 The option position in the Extra Options setting is shuffled.
Model
Original Options
Extra Options
Llama2-7b-Chat
A:168, B:163, C:183, D:171, Y:99, NaN:16
A:126, B:152, D:151, E:102, F:73, G:58, NaN:16 D:78, E887, G:106, NaN:25
Mistral-7b-Inst-x022 A-223, B:131, C:248, D:41, X:20, NaN:37
156,
Gemma-7b-Inst
A:112, B77, C:55, D:57, NaN:1
A:88, 8:52, C:23, D:30, E: 32, F: 38, G: 36, NaN:1
Table 9: Data statistics of our annotated data.
A.3 Accuracy and selection bias in all subcategories
The detailed results the accuracy scores and selection bias of responses from all models in all subcategories are shown in the following Figures.
15

---

Published as a conference paper at COLM 2024
World Religions
First Token
Virolggy
(PrIDe)
Foreign Policy
Answer
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional
Professional Law
Professional Accounting
Prehistory
Philosephy
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketiing
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Sexuality
Human Aging
High School World History
Subcategories
High School Us H History
High School Sta tistics
Psychelogy
High Physics
High School Microeronmiiss
High School Mathematics
High School Macroeconomics
High School
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Formal Logic
Elementary Mathematics
Bectrical Engineering
Econometrics
Conceptual Physics
Computer Security
College Physics
College Medicine
College Mathematics
College Computer
College Chemistry
College Biology
Cinical Knowledge
nesi Bhics
Astronomy
Anatomy
Abstract Alpebra
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
RStD
Figure 7: Selection bias on responses from Llama2-76-Chan in all subcategories.
16

---

Published as a conference paper at COLM 2024
World Religions
Virology
Foreign Policy
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional
Professional Law
Professional Accounting
Prehistory
Philosephy
Nutrition
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketinng
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Human Sexuality
Human Aging
High School World History
Subcatogories
High School Us H History
High School Sta Satistics
School Psychelogy
High School Physics
High School Microeconamies
High School Mathematics
High School Macroeconomics
High School Govemment
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Glabal
Formal Logic
Elementary Mathematics
Bectricall Engineering
Econometrics
Conceptual Physics
Computer Security
College Physics
College Medicine
College Mathematics
College Computer
College Chemistry
College Biology
Cinical Knowledge
ness Bthis
Astronomy
First Token
Anatomy
First Token (PriDe)
Abstract Alpebra
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Acc
Figure 8: Accuracy on responses from in all subcategories.
17

---

Published as a conference paper at COLM 2024
World Religions
First Taken
Virology
Taken (PrIDe)
Foreign Policy
Answer
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional Medicine
Professional Law
Professional Accounting
Prehistory
Philosephy
Nutrition
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketinng
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Human Sexuality
Human Aging
High School World History
Subcatogoriis
High School Us H History
High School Sta tisticcs
School Psychelogy
High School Physics
High School onamics
High School Mathematics
High School Macroeconomics
High School Govemment
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Global
Formal Logic
Elementary Mathematics
Bectriccll Engineeringg
Econometrics
Conceptual Physics
Computer Security
College Physics
College Medicine
College Mathematics
College Computer
College Chemistry
College Biology
Ciniccl Knowledge
ness Bhiis
Astronomy
Anatomy
Abstract Alpebra
0.0
0.1
0,2
0.3
0.4
RStD
Figure 9: Selection bias on responses from in all subcategories.
18

---

Published as a conference paper at COLM 2024
World Religions
Virology
Foreign Policy
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional
Professional Law
Professional Accounting
Prehistory
Philosephy
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketiing
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Human Sexuality
Human Aging
High School World History
Subcategories
High School Us H History
High School Sta Satisiics
School Psychelogy
High Physics
High School oicmiis
High School Mathematics
High School Macroeconomics
High School Govemment
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Formal Logic
Elementary Mathematics
Bectrical Engineering
Econometrics
Conceptual Physics
Computer Security
Collegee Physics
College Medicine
College Mathematics
College Computer
College Chemistry
College Biology
Cinical Knowledge
nesi Bhics
Astronomy
First Token
Anatomy
First Token (PriDe)
Abstract Alpebra
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Acc
Figure 10: Accuracy on responses from (allmm--133--Ci in all subcategories.
19

---

Published as a conference paper at COLM 2024
World Religions
First Teken
Virology
Taken (PrIDe)
Foreign Policy
Answer
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional
Professional Law
Professional Accounting
Prehistory
Philosephy
Nutrition
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketiing
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Human Sexuality
Human Aging
High School World History
Subcategories
High School Us H History
High School Sta tistiics
School Psychelogy
High School Physics
High School Micrecoonmics
High School Mathematics
High School Macroeconomics
High School Govemment
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Glabal
Formal Logic
Elementary Mathematics
Bectricall Engineering
Econometrics
Conceptual Physics
Computer Security
Collegee Physics
College Medicine
College Mathematics
College Computer
College Chemistry
College Biology
Ciniccl Knowledge
ness Bthiss
Astronomy
Anatomy
Abstract Alpebra
0.00
0.05
0.10
0.15
0.20
0.25
0.30
RStD
Figure 11: Selection bias on responses from Mistral-7b-Inst-0.22 in all subcategories.
20

---

Published as a conference paper at COLM 2024
World Religions
Virclogy
Foreign Policy
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional Medicine
Professional Law
Professional Accounting
Prehistory
Philosephy
Nutrition
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketinng
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Human Sexuality
Human Aging
High School World History
Subcatogoriie
High School Us H History
High School Sta Satistics
High School Psychelogy
High School Physics
High School onamics
High School Mathematics
High School Macroeconomics
High School Govemment Politics
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Global
Formal Logic
Elementary Mathematics
Bectriccll Engineeringg
Econometrics
Conceptual Physics
Computer Security
College Physics
College Medicine
College Mathematics
College Computer Science
College Chemistry
College Biology
Ciniccll Knowledge
ness Bhhis
Astronomy
First Token
Anatomy
First Taken (PriDe)
Abstract Alpebra
0.0
0.2
0.4
0.6
0.8
Acc
Figure 12: Accuracy on responses from Mistral--7-Instt-022 in all subcategories.
21

---

Published as a conference paper at COLM 2024
World Religions
Virology
Foreign Policy
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional Medicine
Professional Law
Professional Accounting
Prehistory
Philosephy
Nutrition
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketinng
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Human Sexuality
Human Aging
High School World History
Subcatogoriis
High School Us H History
High School Sta tistiics
School Psychelogy
High School Physics
High School onamics
High School Mathematics
High School Macroeconomics
High School Govemment
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Global
Formal Logic
Elementary Mathematics
Bectricall Engineertigg
Econometrics
Conceptual Physics
Computer Security
College Physics
College Medicine
College Mathematics
College Computer Science
College Chemistry
College Biology
Ciniccl Knowledge
ness Bthis
Astronomy
First Token
Anatomy
First Taken (PriDe)
Abstract Alpebra
0.0
0.1
0.2
0.3
0.4
RSED
Figure 13: Selection bias on responses from Gemma-7b--nst in all subcategories.
22

---

Published as a conference paper at COLM 2024
World Religions
Virology
Foreign Policy
Socielogy
Security Studies
Public Relations
Professional Psychology
Professional Medicine
Professional Law
Professional Accounting
Prehistory
Philosephy
Nutrition
Moral Scenarios
Moral Disputes
Miscelianeous
Medical Genetics
Marketinng
Management
Machine Learning
Logical Fallacies
Jurisprudence
International Law
Human Sexuality
Human Aging
High School World History
Subcategories
High School Us H History
High School Sta tistics
School Psychelogy
High School Physics
High School Microeconaiiss
High School Mathematics
High School Macroeconomics
High School Govemment
High Geography
High School Eurapean H History
High School Computer Science
High School Chemistry
High School Biology
Global
Formal Logic
Elementary Mathematics
Bectricall Engineering
Econometrics
Conceptual Physics
Computer Security
Collegee Physics
College Medicine
College Mathematics
College Computer
College Chemistry
College Biology
Cinical Knowledge
ness Bhics
Astronomy
First Token
Anatomy
First Token (PriDe)
Abstract Alpebra
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Acc
Figure 14: Accuracy on responses from Gemma-76--Inst in all subcategories.
23


Reviews:
[{"review_id": "mhTtBFKTOI", "reviewer": "Reviewer_4zY5", "summary": "The paper tackles robustness of LLMs for MCQ answering by evaluating the actual text generated. Most work in MCQ predominantly uses first token probability, but prior work has shown that they have selection bias (Zheng et al, 2024). This paper perturbs the answer choices (editing the prompt, swapping order, and adding answers) and uses a fine-tuned model (Mistral-7b) to help evaluate text answers and measure discrepancy between first token and text answers.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed review and the recognition of our work!\n### **Lack number with PriDe**\nPriDe targets only a single type of bias: Preferring a certain option position, which they also called “selection bias”. Therefore, it can only improve the robustness under the perturbation “Option Swap”, as shown in Fig 2.  PriDe works by gathering the first token probabilities through repeated questioning with shuffled option orders and calculating the prior probability of bias. Since we want to disentangle the influence of the option position when evaluating under other perturbations, we shuffle option orders multiple times for each question. **Therefore, PriDe will behave the same as the original token prob.** We will add more explanation in the final version.\n### **Error Case**\nBy examining the few error cases, we see that they are also hard cases for humans. We show one example: \n```\nQuestion: \nWellman cites a number of \\\"unpalatable implications that follow from denying a country's right to freedom of association,\\\" including\"  A.x B.x C.x D.x \nLLama2-7b:\nI apologize, but I cannot provide a letter answer to this question as it is based on a false premise. The question asks about the implications of denying a country's right to freedom of association, but it is not possible to deny a country's right to freedom of association as it is a fundamental human right recognized by international law. Therefore, none of the options listed are valid.\n```\nWe originally labeled it as Y (Refusal) as it denies the assumption of the question in the first place. Our classifier labeled it as X (No correct answer) which also makes sense since the response says “none of the options are valid” in the end. We will provide more error cases in the final version.\n### **Related Work**\nBoth Wang 2024 and Lyu 2024 reported this mismatch issue at the same time. However, they only opened up a question but didn’t provide a solution. Our work is a direct follow-up with more focus on the robustness and a solution to the problem. \n### **Term Definition**\n“Robustness discrepancy” is the robustness score (RstD and Entropy) difference between the token prob and text answer. The “Mismatch rate” is the frequency of cases where the first token prob doesn’t match the text answer.\nWe will add them in the final version. \n### **Figures and Citation**\nThanks for the suggestion! We will indicate the direction for clearer understanding.\n\nThe double citation was a mistake. Thanks for pointing that out!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experiments are clear and precise and imply that there is strong selection bias in first token probabilities, even when debiased with prior work on perturbations (3.2/3.5) and additional choices (3.5)", "reasons_to_reject": "- Some of the experiments seem lack numbers with PriDe, which make the paper weaker; this could be a mis-reading and could be clarified in the next section.\n- Some of the terms could be defined better ( especially something like \"robustness discrepancy\" and\"mismatch rate\" which are in the main claims of the paper). \n- Experiments largely hinge on their fine-tuned Mistral model on human annotated texts that pick out the intended MC answer. Their results  seem strong enough to be convincing (acc generally 98%+ excepting Gemma), but some more analysis of their fine-tuned model would be helpful, especially for the few cases the model was wrong, would help quell some worries. \n- The paper could be seen as less exciting with other related work like Lyu, 2024 which they cite. However, the paper goes further into depth about the discrepancy and offer a potential solution with a text-based model instead.", "questions_to_authors": "- Are the results in Table 6, Figure 3, and Figure 4 solely without PriDe? If so, what are the numbers on those?\n\n- nitpicks:\n  - Overall, the figures are very helpful to understanding the paper! But for some figures like Figure 2, it might be clearer to indicate a \"better\" direction, especially since metrics like accuracy and entropy are \"better\" in different directions. \n  - It seems that PriDe is cited twice as a and b; is that intentional?", "extra_scores": {}}, {"review_id": "ibnZsXPyze", "reviewer": "Reviewer_k5FQ", "summary": "When asking a LLM to answer a multiple-choice question with, say, four answer alternatives \"a\", \"b\", \"c\", and \"d\", it is a non-trivial problem to determine whether or not the model has given the correct response. This is because the model might generate variations of the answer, like \"a\", \"Alternative a\", \"Answer: a\", \"The correct answer is a\", etc. This might happen even if the prompt to the model explicitly asks for a single-letter answer.\n\nThis paper examines various methods for determining the given answer, including first token probability (assuming that the model is well-behaved and answers with a single letter), and text-based evaluation, which amounts to training a classifier based on the text outputs from models responding to multiple-choice questions. The classes are the answer alternatives, e.g. \"a\", \"b\", \"c\", and \"d\". The investigations in the paper concern several models, datasets, and evaluation metrics. The authors have also studied the effect of perturbations, like changing the word order in the question, changing the order of the answer alternatives, adding extra answer alternatives, etc.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing the quality of our writing and our systematic experiments. We appreciate the reviewer's positive feedback and constructive criticism.\n\n ### **Contribution of our work**\n\nWhile we understand the concern regarding the perceived size of our contribution, we would like to emphasize the significance of our findings. **Many current works in the field rely heavily on the first-token probability for evaluating LLMs without being aware of the potential misalignment and robustness issue we have highlighted.**\n\nOur work addresses a critical oversight in the evaluation of LLMs, providing evidence that relying solely on first-token probabilities can lead to significant inaccuracies. This issue impacts a wide range of applications that utilize multiple-choice questions for model evaluation.\n\nOur study advocates for a more realistic and comprehensive evaluation approach by considering the entire text output with easy to fine-tune text classifier. This shift could (greatly) enhance the accuracy and relevance of future LLM assessments, leading to better model development and deployment. By inspecting the text answer, we show that instruction-tuned language models are more robust and perform better than previously shown prob-based evaluation which shows the benefit of instruction-tuning and RLHF. This raises important questions: **Does evaluating token probability on the base model really reflect its capability in real life?**  **Should we move to a more realistic evaluation framework by evaluating the text answers of the instruction-tuned models instead?** Since a large scale of users directly interact with the instruction-tuned models in the text space, it is important to move to a more human-centric evaluation paradigm.\n\nGiven that one more page is allowed in the camera-ready version, we will add more discussion and implications for future works.\n\nWe hope the reviewer finds our responses satisfactory. Once again, we extend our gratitude to the reviewer for their valuable suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written, providing a good introduction to this research question. The experiments are systematically conducted, and the results are interesting, although not surprising", "reasons_to_reject": "The contribution of the paper is not very big.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uwc8sAKBzg", "reviewer": "Reviewer_vYwj", "summary": "This paper studies the robustness of instruction-tuned LLMs in multiple-choice question (MCQ) evaluation. The authors found that compared to the traditional way, where LLMs are straightforwardly prompted to predict the option IDs as answers (using the next-token prediction probability), prompting the instruction-tuned LLMs with natural instructions to select the answers leads to better robustness. More empirical analyses consolidate the claim and findings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your recognition of the importance of our work and our contribution to the community. With our findings, we encourage people to choose better methods when evaluating LLMs, especially instruction-tuned models.\n\nThanks for pointing the writing out and we will incorporate your suggestion into our camera-ready version.\n\n**Clarifying \"Mismatch\"**: The mismatch rate refers to the frequency of cases where the fist token answer doesn’t match the text answer. We will add this definition to the camera-ready version to enhance clarity. \n\n**Scatter Figures for Robustness Correlation**: We appreciate the suggestion to present the correlation with robustness more effectively using scatter figures. Although we initially considered this, we were constrained by page limitations. Given that we have one additional page available for the camera-ready version, we will include the scatter figures. \n\nWe thank the reviewer once again for the recognition of the importance of our work and for the valuable suggestions. We hope these modifications will enhance the clarity and accessibility of our paper and address the reviewer's concerns satisfactorily.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper focuses on an important problem, i.e., the selection bias of instruction-tuned LLMs in MCQ evaluation. This is somewhat paid less attention to in previous research.\n* The evaluation and analyses are thorough and sound. They involve comparison with baselines (PriDe), multiple benchmarks (MMLU and its multiple categories; OpinionQA), and multiple models. These efforts consolidate the findings in this paper and justify the effectiveness of properly prompting instruction-tuned LLMs in MCQ.", "reasons_to_reject": "I do not see clear reasons to reject it, but I suggest the authors to polish the writing to make it more accessible. For instance, the term \"mismatch\" is quite unclear and confusing across abstract, intro, and Section 3.3. I think more formal definitions are needed. Also, its correlation with the robustness (Figure 2) can be better presented in scatter figures (e.g. the y-axis is a metric of robustness while the x-axis is the mismatch value).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2cX90ZNi4H", "reviewer": "Reviewer_6RFZ", "summary": "This paper analyses the robustness of multiple-choice QA evaluation of instruction-tuned LLMs. It compares two approaches:\n\n1. Token probability: Ranking answer choices based on the model's predicted probability of an early token (varies by model).\n2. Text-based: Extracting the answer choice from the generated text with a classifier.\n\nThe paper compares three model families (Llama2, Mistral and Gemma) on the popular MMLU task.\n\nThe authors argue that finetuning techniques have reduced the robustness when relying on token probabilities, and provide empirical evidence for this. \n\nThey provide informative experiments, and show that token probability systematically underestimates model performance on MMLU tasks. They also provide evidence for improved robustness with their classifier.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback!\n### **Reliability of the Classifier**\nWe acknowledge the concern, however, the focus of this paper is not to design the best MCQ answer extractor applicable across all models. There are other papers focusing on creating a reliable LLM-based answer extractor for any model, which is a task beyond the scope of this paper. **Our focus is to analyse the behavior of the chosen models with the help of the classifiers we trained, which have good enough performance to be relied on, as shown in Table 3.**\n\nHowever, we are happy to give further analysis: the classifier needs to encounter diverse response styles to enhance its robustness. Llama2 exhibits a wider range of response patterns, which makes classifying Llama2 responses more challenging. To verify this, we compare the classifier accuracy when trained on different model responses:\n|Test data | Llama2-7b | Mistal-7b | Gemma-7b\n| -| -| -| -|\n|Trained on  Mistral-7b | 84.7 | 1.0  | 99.5|\n|Trained on Llama2-7b |99.5 | 99.5 | 1.0|\n\nBoth models can generalize well to Gemma-7b because of its regular output pattern. The model trained on Llama2 can generalize to Mistral, but not the other way around. \nTherefore, it is better to include more diverse training samples.\n### **Closed-sourced model result**\nWe are happy to provide results on a close-sourced model.  Note that closed-sourced models don’t provide logits over the vocabulary, which makes token probability evaluation infeasible. We compare the text answer by asking ChatGPT-3.5 and Gemma 7b 10 questions, under the “swap option” perturbation. Both models demonstrated similar scores. **While it is a small experiment, it suggests that close-sourced models might not be more robust.**\n|| Entropy | \n|-|-| \n|ChatGPT-3.5 | 0.468 | \n|Gemma-7b    | 0.459 | \n\n### **Difference to Wang et al.[1]**\n[1] only showed the mismatch problem but did not provide a solution. We further investigated the robustness of the two approaches and showed that text-based evaluation is more beneficial. Therefore, **our paper is a direct follow-up and a solution to the problem raised by [1].**\n### **Finetuning Suggestion**\nA straightforward solution is to add more perturbation types to the training dataset. Another solution could be using debiasing methods. Similar to PriDe, we can also design bias estimation methods to de-bias the text answer by conducting multiple runs. As PriDe uses only first token probabilities, a text-based debiasing method is required.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is very well written and well structured.\n* The included experiments provide strong empirical evidence for the author’s case to prefer text-based methods over token probabilities.\n* The findings are very relevant and important for LLM evaluation practices.", "reasons_to_reject": "* Using a trained model for answer extraction raises concerns about the generalisability of the findings to models with significantly different instruction tuning / rlhf. Their discussion of Gemma doesn’t alleviate this, because they neither show how a model trained on Gemma data would behave, nor provide an error analysis that systematically compares the model answers. Something like this could have been achieved with a data ablation study without collecting more examples.\n \n* The paper is very similar to ​​https://arxiv.org/pdf/2402.14499 (that they cite throughout) in both findings and methods. The main additions are 1) including the Gemma model and 2) using the MMLU benchmark instead of just OpinionQA.", "questions_to_authors": "* The fine-tuned Mistral model used for answer extraction could introduce its own biases, affecting the evaluation results. Have you considered measuring and mitigating this?\n* Have you considered evaluating the robustness of closed, but popular models in comparison to the open models studied in the paper? (Even small samples could be interesting.)\n* Do you have recommendations for finetuning methods based on your findings that could alleviate the robustness issues?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the robustness of instruction-tuned LLMs in multiple-choice question (MCQ) evaluation. The authors found that compared to the traditional way, where LLMs are straightforwardly prompted to predict the option IDs as answers (using the next-token prediction probability), prompting the instruction-tuned LLMs with natural instructions to select the answers leads to better robustness. More empirical analyses consolidate the claim and findings.

The evaluation and analyses are thorough and sound. They involve comparison with baselines (PriDe), multiple benchmarks (MMLU and its multiple categories; OpinionQA), and multiple models.

The writing of the paper could be improved, specially to distinguish better from prior work (Wang et al.)
Decision: Accept