Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think

Authors: xinpeng  wang
chengzhi  hu
bolei  ma
paul r öttger
barbara  plank
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 11:22:29

Abstract: Multiple choice questions (MCQs) are commonly used to evaluate the capabilities of large language models (LLMs). One common way to evaluate the model response is to rank the candidate answers based on the log probability of the first token prediction. An alternative way is to examine the text output. Prior work has shown that first token probabilities lack robustness to changes in MCQ phrasing, and that first token probabilities do not match text answers for instruction-tuned models. Therefore, in this paper, we investigate the robustness of text answers. We show that the text answers are more robust to question perturbations than the first token probabilities, when the first token answers mismatch the text answers. The difference in robustness increases as the mismatch rate becomes greater. As the mismatch reaches over 50%, the text answer is more robust to option order changes than the debiased first token probabilities using state-of-the-art debiasing methods such as PriDe. Our findings provide further evidence for the benefits of text answer evaluation over first token probability evaluation.

Main paper: 
 introduction: 
 The open-ended nature of autoregressive language generation complicates the evaluation of Large Language Models (LLMs). One popular solution to this problem is to prompt LLMs with Multiple Choice Questions (MCQs), which limit the answer space to a few candidate options, thus enabling evaluation by comparing model choices against gold labels. For MCQs, there are two main ways of extracting model choices from their generated text answers: 1) In the text-based approach, the choice is automatically extracted from the text answer either by pattern matching (Wang et al., 2022) or by prompting a strong LLM (Chiang et al., 2023; Li et al., 2023) such as GPT4 (OpenAI et al., 2024) . However, pattern matching can often be inaccurate because each model has different response styles, requiring manual feature engineering. This makes pattern matching infeasible when evaluating many tasks and models at the same time. Using a strong proprietary model like GPT-4 as an evaluator, on the other hand, is more flexible but lacks transparency and reproducibility, and also creates high financial costs when we running large-scale evaluations. 2) In comparison, the probability-based approach simplifies the evaluation by ranking the log probabilities assigned to the option IDs (e.g. A/B/C/D) from the first token prediction of the model. This approach is widely adopted in many different benchmarks and LLM evaluation studies (Hendrycks et al., 2020; bench authors, 2023; Liang et al., 2022; Santurkar et al., 2023) . However, recent works point to a mismatch between the text-and probability-based approaches (Lyu et al., 2024; Wang et al., 2024) , showing that first token probabilities do not match text answers given by models, especially for models fine-tuned on conversational or safety data.Moreover, recent studies have shown that the probability-based approach lacks robustness -that it is sensitive to linguistic properties of the MCQ prompt (Leidinger et al., 2023) or perturbations such as typos, adding options, word swapping (Tjuatja et al., 2023) and option position (also called selection bias, Dominguez-Olmedo et al., 2023; Zheng et al., 2023) .In this paper, we investigate the robustness of the text-based approach. We compare it with the first token answer as well as the debiased first token answer using the first token debiasing method PriDe (Zheng et al., 2023) , by measuring the answer robustness under various prompt perturbations. To avoid the labor of feature engineering and the high cost of using a proprietary model as an evaluator, we fine-tune a Mistral-7b-Instruct model and show that it can reliably and accurately detect the choice from the text answers across models and datasets. 1By comparing the choice made in the text answer and the first token probability, we show that: (1) The text answer shows small selection bias ( §3.3) and high robustness to various sentence perturbations ( §3.5) across all the models we examined; (2) The robustness discrepancy between the first token probability and the text answer increases as the mismatch rate between them increases ( §3.3, §3.5); (3) When the mismatch rate is high (over 50%), the text answer shows a smaller selection bias than the state-of-art first token debiasing method PriDe ( §3.3). As a whole, our work provides further evidence for the benefits of text-based over probability-based MCQ evaluations of LLMs. 
mismatch between the probability and the text-based mcq evaluation: 
 Before the rise of instruction-tuned large language models, it was appropriate to check the probabilities assigned to the options in MCQs by the models for several reasons: 1) Discriminative models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are forced to only give probabilities to the available opinions by finetuning on the task with a task-specific classification head; 2) Sequence-to-sequence models such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) are good at giving the option ID in the next-word prediction since they were specifically trained to perform NLP tasks of various formats but they are not good at "following a user's intent" in real-life scenarios (Ouyang et al., 2022) ; 3) Foundation models such as GPT3 (Brown et al., 2020) , OPT (Zhang et al., 2022) have poor instruction-following ability which makes the text completion uninterpretable, thus, the prediction of a model is measured by ranking the log probability of the available options.Techniques like Instruction-Tuning (Wei et al., 2021) and Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) improve the model's ability to follow the instructions and give helpful responses. Therefore, the model can directly answer the MCQs and give its choice in the text answer. This makes it less appropriate to use probabilitybased evaluation since the first token probability doesn't represent the text answer, even specifically asking it to start with the option ID in the prompt (Wang et al., 2024) . Figure 1 shows an example of the mismatch between the first token probabilities and the text answer given by the Llama2-7b-Chat model. This mismatch leads to the different performance of a model on a benchmark depending on the evaluation method, as shown in Table 1 .General Instruction: Please read the multiple-choice question below carefully and select ONE of the listed options and only give a single letter.The Web was effectively invented by Berners-Lee in which year?A . 1991 B. 1980 C. 1989 D. 1993 Answer:Sure! The answer is (A) 1991.  
robustness evaluation of the probability and text-based approaches: 
 Several works (Dominguez-Olmedo et al., 2023; Zheng et al., 2023) have shown that large language models have selection bias such as preferring "option A", therefore they are not robust under position change of the options. To mitigate this bias, PriDe (Zheng et al., 2023) was proposed as a debiasing method to improve the robustness of the model by disentangling the token bias from the first token probability. Tjuatja et al. (2023) shows that LLMs are also sensitive to perturbation in the question such as typos, word swapping and adding more options. However, this line of work only focuses on the first token evaluation.It intuitively makes sense that the probabilistic mass on a single token is sensitive to its context and is biased towards a certain token ('A') which leads to selection bias (Zheng et al., 2023) . However, it is unknown whether the text answers given by the instruction-tuned models are also sensitive to such changes. Therefore, we answer this question by comparing the robustness of the text answers with the answer estimated by original/debiased first token probability (referred to first token in the following).Models Our study focuses on open-source instruction-tuned language models since we are interested in the text response to the user's instruction, as well as committing to open science and reproducibility. We cover three model families including Llama2 (Touvron et al., 2023 ), Mistral (Jiang et al., 2023) and Gemma (Team et al., 2024) . We do greedy decoding during the inference time. We also trained free and robust MCQ evaluators using Mistral-7b models which we plan to make public. For simplicity, we drop the postfix "Inst/Chat" later.Benchmark We conduct the experiments on MMLU (Hendrycks et al., 2020) which consists of 4-option MCQs with tasks covering 57 subjects ranging from elementary mathematics to US history. Note that the original implementation of the debiasing method PriDe was done on the aggregated dataset level. In our robustness evaluation experiments, we believe it is instructive to go beyond the aggregate level: we employ and compare with PriDe on each task individually in order to gain deeper insight into the behavior of the model. We additionally use OpinionQA (Santurkar et al., 2023) inspired by Wang et al. (2024) to test the robustness of our trained classifier which will be introduced later.Prompting To give a fair comparison between the text answer and the first token evaluation, we specifically guide the model to respond directly with the option ID. We adopt the system prompt used in Wang et al. (2024) where the model is specifically asked to answer with a "single letter": Please read the multiple-choice question below carefully and select ONE of the listed options and only give a single letter. Our preliminary experiments show that it is important to add the constraint to respond with a single letter instead of just asking it to "choose the option" used in prior works. Without such a constraint, the model will prefer to repeat the option content instead of the option ID.We extract the log probabilities assigned to the token corresponding to the option IDs ("A", "B", "C", "D") from the model and take the one with the highest probability as the answer. In our experiment, we see issues of only looking at the first token since different models have diverse response patterns. For example, the Llama2 models will always start the response with a space token " ", and the Gemma model starts its answer by repeating the word "Answer:" (e.g. "Answer: C") most of the time. Therefore, we also use the second token of the Llama2 model and the third token of the Gemma model, which indeed leads to slightly higher robustness results, but still lower than the text answer.To extract the choice from the text answer automatically, we follow Wang et al. (2024) to construct a classifier by training a language model. We test five different language models: T5-small, T5-base, T5-large, Mistral-7b-Base-v0.1 and Mistral-7b-Instruct-v0.2. We finetune the Mistral models using QLoRA (Dettmers et al., 2024) implemented by Huggingface (Mangrulkar et al., 2022) . See Appendix A.1 for training hyperparameters. The annotated data used for training the classifier is constructed as this:The model text response, alongside the multiple choice options. Label: The ID of the correct option as determined by the model. Table 3 is an example of the annotated model output. It is important to add references to the input since there are cases where the model gives an answer outside of the references ("No correct answer", "Refusal", "I don't know"). We opt to annotate such cases as: {X: No correct answer, Y: Refusal, Z: I do not know}. Therefore, our classifier is required to be able to detect such cases when the model gives an answer outside of the options. In our answer floating experiments where we test if the model will change its answer with additional options, we add these three cases into the options. Therefore, we need to train two classifiers for the two cases, each of which has 7 classes.We annotated a total of 600 samples for Gemma and 1600 samples each for the two 7b models of Llama2 and Mistral. For each model, we have the same amount of samples from two subsets: one with (Extra Options) and the other without (Original Options) additional options. See Appendix A.2 for data details. From these annotated samples, we used a total of 2000 samples for training our classifiers on Llama2 and Mistral outputs. Subsequently, we tested the classifiers on outputs from Llama2, Mistral, and Gemma models, as presented in Table 2 . To test the robustness of our classifier, we also evaluate the model trained on out-of-distribution data (responses to OpinionQA) provided by Wang et al. (2024) .The result in Table 2 shows that the Mistral models provide better results than the T5 models as classifiers. The models trained on in-distribution (ID) data have near-perfect results. Even when trained on out-of-distribution (OOD) data, the performance gap in accuracy remains minimal, ranging from 1% to 4% in most cases. The successful performance of our classifier shows that it can be used in other domains as an evaluator. Furthermore, we trained our classifiers only on responses from Mistral and Llama2 models. Remarkably, when tested on responses from Gemma models, which were not included in the training data, the classifier performed comparably well, underscoring its cross-model robustness. It should be noted that the Gemma model's regular response pattern can explain the good performance on the Gemma's output. Our classifier is not optimized for extracting MCQ answers from any models, especially the ones with significantly different response styles. It is important to include diverse response styles from more model families into the training data to improve the robustness further. EQUATION)where r is the mean recall and r A , r B , r C and r D are the recall for options A, B, C and D.The RStD score is based on the assumption that the correct answer is placed randomly. We introduce a second metric to assess the model's robustness by measuring the entropy of answers from multiple runs with random perturbations. For example, to test the model's robustness to option position change, we shuffle the positions of option contents N times while keeping the positions of option IDs and record the answer A i each time. The entropy is calculated as follows:EQUATION)where m is the number of unique answers of the N runs, and P(A i ) is the probability of answer A i occurring (which is the number of times A i occurs divided by N). A lower entropy means higher answer consistency under perturbation. As the mismatch rate decreases from Gemma (56.8%) to Mistral (10.2%), the performance gap between the first token (red) and text answer (blue) decreases. Text answers from Gemma and Llama2 have lower selection bias than the debiased first token answers. We compare the selection bias and accuracy of the text answer, first token answer and the debiased version of first token answer using PriDe in Figure 2 . Except Mistral-7b, both PriDe and text answers show lower selection bias than the first token answer on all models, while text answers perform on par with or better than the debiased first token evaluation. In terms of accuracy, text answer archives higher scores than the original and debiased first token answers among all the models except for Mistral-7b where the first token answer is slightly higher than the text answer. To understand why Mistral-7b behaves differently, we calculate the mismatch rate between the first token and text answer as shown in Table 4 . Mistral-7b has the lowest mismatch rate among all the models since it has a good instruction-following ability to start its first token with the option ID. Therefore, its first token answer shows significantly lower selection bias compared to other models with a high mismatch rate and performs similarly with its text answer. Using the first token debiasing method can indeed further lower the selection bias and improve the accuracy of the model. On the contrary, Gemma-7b shows the highest selection bias and mismatch rate of 56.8%. Surprisingly, there is a huge performance gap between the text answer and the first token answer, even after debiasing. The text answer outperforms the debiased first token answer (PriDe) on Gemma-7b and Llama2-7b where the models have a mismatch rate over 50%. This shows that instruction-tuned language models have less selection bias than we think. In many cases, the first token debiasing method may not be necessary since the text answer is already robust, such as Gemma-7b and Llama2-7b.To fully test the robustness of the two evaluation approaches, we further incorporate a series of different perturbation types for the prompt, based on the response bias modifications and non-bias perturbations from Tjuatja et al. (2023) . Table 6 summarizes the 5 perturbation types we used. For each question, we perturbated the question four times and calculated the answer entropy except Additional options, where we calculated the rate of answer change.Given the evidence that the first token answer has a higher selection bias than the text answer, we need to disentangle the influence of the question perturbation and the option order. Therefore, we shuffle the option order five times for each perturbation and take the majority-voted option content for calculating the entropy, resulting in 20 runs for each perturbation type except Option Swap, where we study the impact of the option order itself.Table 6 shows answer consistency results for both evaluation approaches under the perturbation Letter Typos, Letter Swap, Word Swap, Option Swap measured by entropy. Similar to the selection bias result, text answer is consistently more robust to all the perturbations on all models except Mistral-7b, where first token shows a minor difference compared to the text answer. The slightly higher entropy of the text answer of Mistral-7b can be explained by the special cases where the model claims no correct answers available or refuses to answer the question due to safety reasons (see section 3.6 for a detailed discussion), which leads to a larger option space than the first token answer. We observe that a larger model size leads to more robust first token and text answers: as the model size increases from 7b to 13b for Llama2 entropy decreases. Moreover, we note that the position of the option has the largest impact on the answer since the entropy of both approaches is the highest under Option Swap for all models. The robustness discrepancy between the first token and the text answer is also the largest under the Option Swap than the other perturbation types. We also test the model's robustness to adding opinions by checking the percentage of the questions where the model changes its answer (answer floating). Figure 3 shows the answer floating result under the perturbation Additional Options. The first token and text answers here are majority-voted after shuffling the option orders and we use the default option order to assign the option IDs to the answers. For Llama2 and Gemma models, there is a significant percentage of questions where the model changes its first token answer after adding three more options (Refused, I do not know and No correct answer is given), reaching to nearly 70%, which is much higher than the text answer. For Mistral-7b, both text and first token answers achieve similar answer floating rates at around 25.5%.To further know about the answer floating behaviour, we zoom into the answer distribution in Figure 4 . In all models, especially Llama2-7b-Chat and Llama2-13b-Chat, the first token answer shifts significantly after adding the three additional options. The number of answers shifted to the three additional options is not negligible, taking up to 14.2% of the total responses for Llama2-13b-Chat. After adding the options, the distribution of answers going to the three additional options is flatter compared to the text answer result. This could mean that the model selects them by chance. Again, this shows the brittleness of the first token evaluation, whose result can easily be shifted by adding new options.Looking at the text answer result, we see a minor distribution shift after adding the options in all models. Note that the text answer can choose to "refuse" or claim "no correct answer is given" when those are not in the original options. Before adding the options, we observe different answer patterns from different model families. The Gemma model rarely gives answer outside of the options, while the Llama2 models have a high refusal rate, especially for the 7b model. The Mistral model, interestingly, tends to claim there are no correct answers more instead of refusing or showing uncertainty.After adding the options, Llama2-7b-Chat tends to keep the refusal answers unchanged which is likely due to the strong safety guardrail which can be easily triggered by the keywords in the question. Mistral-7b-Inst, unlike other models, indeed chooses more from the added options than without them, and it is the only model that actually expresses uncertainty in the text answer such as "I do not know", which is also observed during the annotation phase. Refusal As we discussed in the previous section, refusal behaviour takes a large part of the total responses from Llama2-7b-Chat. This has a huge impact on the evaluation result as we inspect the model behaviour in each subcategory of MMLU.Figure 5 shows the selection bias of text, original, and debiased first token answers on 10 subcategories based on the top and bottom 5 subcategory performance of PriDe. A detailed overview of the accuracy scores and selection bias of responses from all models in all subcategories is shown in the Appendix A.3. Surprisingly, the text answer exhibits zero RStD on subcategory Moral Scenarios.As we look into the data, it turns out the model refuses to answer all the 895 questions in this subcategory due to safety reasons. This reveals the unreliability of the first token accuracy evaluation especially on sensitive topics.We see a relation between the mismatch rate and the robustness gap between the first token and text answer. As shown in our results in Table 1 , 5 and Figure 2 and 3, compared to other models, Mistral-7b exhibits good robustness and low selection bias, with the lowest mismatch rate of 10%. In contrast, the accuracy and robustness level gap is the largest for Gemma-7b-Inst which has the highest mismatch rate of 56.6%. In Figure 6 , we plot the mismatch rate of the models and their first token/text answer difference in terms of accuracy and selection bias evaluated on MMLU. It shows that as the model's mismatch rate increases, the first token answer is less accurate and robust campred to the text answer. Thus, the closer the first token answers are to the text answers, the better the robustness level, whereas the robustness level of the text answer stays high. Therefore, it is recommended to always look at the text answer if the mismatch rate is unknown. Figure 6 : The absolute difference between the first token and text answer in terms of prediction accuracy and selection bias. The text answer achieves higher accuracy and lower RStd score (less selection bias) than the fist token answer across all the models. The gap increases as the model's mismatch rate increases. 
related work: 
 Large language models The landscape of large language models (LLMs) has undergone dynamic evolution in recent years. With notable contributions from renowned models such as GPT4 (Achiam et al., 2023) and Gemini (Team et al., 2023) , which have significantly advanced the field. Other recent advancements, including models like Mistral (Jiang et al., 2023), Gemma (Team et al., 2024) , and Llama2 (Touvron et al., 2023) , offer distinct settings and characteristics. These models have been equipped with innovative techniques such as Instruction-Tuning and RLHF, enabling them to effectively interpret and adhere to given instructions (Ouyang et al., 2022) , and to answer with diverse and natural responses.Multiple choice questions evaluation Multiple choice questions (MCQs) are widely used to assess the capabilities of LLMs across various domains. They play a crucial role in evaluation benchmarks like MMLU (Hendrycks et al., 2020) , AGIEval (Zhong et al., 2023) , HELM (Liang et al., 2022) and Evaluation Harness (Gao et al., 2021) , as well as in evaluating moral beliefs, opinions on public issues, and surveys (Santurkar et al., 2023; Scherrer et al., 2024; Wang et al., 2024) . Traditionally, MCQ accuracy has been measured based on the model's first token prediction (Dominguez-Olmedo et al., 2023; Santurkar et al., 2023) . However, contemporary LLMs often offer nuanced responses, challenging the reliability of first token evaluation. This discrepancy has been revealed by recent studies (Wang et al., 2024; Lyu et al., 2024) . et al. (2023) , highlights the sensitivity of LLMs to prompt variations, impacting performance and reliability. Further investigation is required to grasp prompt brittleness's full extent and its implications for model robustness. 
conclusion: 
 This work studies the robustness of instruction-tuned language models in the multiplechoice question answering settings. Our research builds upon previous studies which have highlighted the brittleness of first token probability evaluation and the mismatch between first token and text answer. Through extensive perturbation experiments, we demonstrate that text answers generated by instruction-tuned language models are more robust. In cases where the first token answer matches the text answer, both approaches exhibit similar levels of robustness-in our experiments this was the case only for one out of the tested models, Mistral. Our findings suggest that the instruction-tuned language models are more robust for text answers than what previously shown in works using the first token probabilities as evaluation. Therefore, we suggest to evaluate LMs in a more detailed and realistic way by directly inspecting the text answer, to have a better understanding of the full spectrum of model behaviour. We strongly caution against relying only on first token probability. 
 
 Appendix: We share the dataset and trained classifiers at https://github.com/mainlp/MCQ-Robustness. 


Decision: This paper studies the robustness of instruction-tuned LLMs in multiple-choice question (MCQ) evaluation. The authors found that compared to the traditional way, where LLMs are straightforwardly prompted to predict the option IDs as answers (using the next-token prediction probability), prompting the instruction-tuned LLMs with natural instructions to select the answers leads to better robustness. More empirical analyses consolidate the claim and findings.

The evaluation and analyses are thorough and sound. They involve comparison with baselines (PriDe), multiple benchmarks (MMLU and its multiple categories; OpinionQA), and multiple models.

The writing of the paper could be improved, specially to distinguish better from prior work (Wang et al.)

Reviews: review: paper_summary: paper_summary: The paper tackles robustness of LLMs for MCQ answering by evaluating the actual text generated. Most work in MCQ predominantly uses first token probability, but prior work has shown that they have selection bias (Zheng et al, 2024). This paper perturbs the answer choices (editing the prompt, swapping order, and adding answers) and uses a fine-tuned model (Mistral-7b) to help evaluate text answers and measure discrepancy between first token and text answers.
strength_weakness: strength_weakness: - Some of the experiments seem lack numbers with PriDe, which make the paper weaker; this could be a mis-reading and could be clarified in the next section.
- Some of the terms could be defined better ( especially something like "robustness discrepancy" and"mismatch rate" which are in the main claims of the paper). 
- Experiments largely hinge on their fine-tuned Mistral model on human annotated texts that pick out the intended MC answer. Their results  seem strong enough to be convincing (acc generally 98%+ excepting Gemma), but some more analysis of their fine-tuned model would be helpful, especially for the few cases the model was wrong, would help quell some worries. 
- The paper could be seen as less exciting with other related work like Lyu, 2024 which they cite. However, the paper goes further into depth about the discrepancy and offer a potential solution with a text-based model instead.
questions: questions: - Are the results in Table 6, Figure 3, and Figure 4 solely without PriDe? If so, what are the numbers on those?

- nitpicks:
  - Overall, the figures are very helpful to understanding the paper! But for some figures like Figure 2, it might be clearer to indicate a "better" direction, especially since metrics like accuracy and entropy are "better" in different directions. 
  - It seems that PriDe is cited twice as a and b; is that intentional?
paper_summary: paper_summary: When asking a LLM to answer a multiple-choice question with, say, four answer alternatives "a", "b", "c", and "d", it is a non-trivial problem to determine whether or not the model has given the correct response. This is because the model might generate variations of the answer, like "a", "Alternative a", "Answer: a", "The correct answer is a", etc. This might happen even if the prompt to the model explicitly asks for a single-letter answer.

This paper examines various methods for determining the given answer, including first token probability (assuming that the model is well-behaved and answers with a single letter), and text-based evaluation, which amounts to training a classifier based on the text outputs from models responding to multiple-choice questions. The classes are the answer alternatives, e.g. "a", "b", "c", and "d". The investigations in the paper concern several models, datasets, and evaluation metrics. The authors have also studied the effect of perturbations, like changing the word order in the question, changing the order of the answer alternatives, adding extra answer alternatives, etc.
strength_weakness: strength_weakness: The contribution of the paper is not very big.
paper_summary: paper_summary: This paper studies the robustness of instruction-tuned LLMs in multiple-choice question (MCQ) evaluation. The authors found that compared to the traditional way, where LLMs are straightforwardly prompted to predict the option IDs as answers (using the next-token prediction probability), prompting the instruction-tuned LLMs with natural instructions to select the answers leads to better robustness. More empirical analyses consolidate the claim and findings.
strength_weakness: strength_weakness: I do not see clear reasons to reject it, but I suggest the authors to polish the writing to make it more accessible. For instance, the term "mismatch" is quite unclear and confusing across abstract, intro, and Section 3.3. I think more formal definitions are needed. Also, its correlation with the robustness (Figure 2) can be better presented in scatter figures (e.g. the y-axis is a metric of robustness while the x-axis is the mismatch value).
paper_summary: paper_summary: This paper analyses the robustness of multiple-choice QA evaluation of instruction-tuned LLMs. It compares two approaches:

1. Token probability: Ranking answer choices based on the model's predicted probability of an early token (varies by model).
2. Text-based: Extracting the answer choice from the generated text with a classifier.

The paper compares three model families (Llama2, Mistral and Gemma) on the popular MMLU task.

The authors argue that finetuning techniques have reduced the robustness when relying on token probabilities, and provide empirical evidence for this. 

They provide informative experiments, and show that token probability systematically underestimates model performance on MMLU tasks. They also provide evidence for improved robustness with their classifier.
strength_weakness: strength_weakness: * Using a trained model for answer extraction raises concerns about the generalisability of the findings to models with significantly different instruction tuning / rlhf. Their discussion of Gemma doesn’t alleviate this, because they neither show how a model trained on Gemma data would behave, nor provide an error analysis that systematically compares the model answers. Something like this could have been achieved with a data ablation study without collecting more examples.
 
* The paper is very similar to ​​https://arxiv.org/pdf/2402.14499 (that they cite throughout) in both findings and methods. The main additions are 1) including the Gemma model and 2) using the MMLU benchmark instead of just OpinionQA.
questions: questions: * The fine-tuned Mistral model used for answer extraction could introduce its own biases, affecting the evaluation results. Have you considered measuring and mitigating this?
* Have you considered evaluating the robustness of closed, but popular models in comparison to the open models studied in the paper? (Even small samples could be interesting.)
* Do you have recommendations for finetuning methods based on your findings that could alleviate the robustness issues?

Comments: title: Thank you!
Reply to Reviewer 6RFZ
Author response
comment: Thank you for your valuable feedback which helps improve the quality of our paper!
Thank you for your response, the rebuttal has clarified some areas I was concerned about and I'll increase my score (and expect that a final version of the paper would also be clear about these).
Thank you for the clarification. I will retain the current rating.
Thank you for your comments and further questions!

**Reliability**
Thank you for your suggestions. We agree it is important to address that our classifier may not perform perfectly on models that are significantly different.  If one wants to apply our classifier to their own models, we suggest inspecting the model's response first, and further finetuning the classifier if the response style is highly different from the training data, to further improve its robustness. Therefore, we will share our classifiers so that other people can continue improving them based on their results. We will include this in the final version of the paper.


**Solution to Wang et al.**
Sorry for causing any confusion. By "solution", we don't mean *"how to extract the text answer"*. The solution here means *"which one to choose when facing the choice between the first token and text answer evaluation".*
Wang et al. showed the mismatch issue but didn't give a clear suggestion on which one is clearly better. **How much benefit do we get by doing text evaluation instead of first token probability evaluation?**
Given the mismatch problem shown by Wang et al, we want to know which one should we choose by comparing them systematically in terms of robustness and accuracy. In both cases, the text answer evaluation shows superior properties. Therefore, we provide an answer to the mismatch problem raised by Wang et al. 

We hope this can address your concern adequately. We are happy to give further explanations if there is still any uncertainty. If you find the revisions and clarifications satisfactory, we would appreciate your consideration in re-evaluating the manuscript.
Thank you for your response.

**Reliability of the Classifier** Thanks a lot for adding the comparison between Mistral and Llama. The stark difference in classifier accuracy between Mistral and Llama highlights the difficulty in training reliable LLM-based answer extractor that works across models. I appreciate that solving this problem is out of scope for your paper, but I think it's important to emphasize this limitation in the paper. (e.g. in your reply, in the discussion of differences to Wang et al. you describe your approach as a solution, which seems too strong given the strong bias in this method.)

**Difference to Wang et al.** I'm not sure what solution your paper provides that the paper from Wang et al. doesn't. If I understand correctly, that paper also trains a classifier. Can you be more specific to what the solution is your paper provides but that paper doesn't?
Thanks for the authors' response. I will maintain the rating.
Hi reviewers, 
Please take a look at author's responses and other reviews of the paper. If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum.

Overall, most of the reviewers are positive about this paper. 

Reviewer 6RFZ: 
- One of your concerns is that the generalizability of the results to other closed models or those trained with other instruction/rlhf training. Does the author response address your concerns? 

Reviewer 4zY5 also raised similar concerns. Other reviewers should also feel free to weigh in on the above points. 

Thanks!