When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024)

Abstract: Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value — we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Centerfor-AI-Saudi-Arabia/lm-evaluation-harness.

heading: 1 Introduction
text: The advent of transformer-based Large Language Models (LLMs) (OpenAI, 2023; Deepmind, 2023; Anthropic, 2023; Anil et al., 2023; Touvron et al., 2023) has led to a generational leap in generative models, enabling interaction with computing devices through natural language. This advancement encompasses improvements that have rendered many earlier benchmarks and leaderboards obsolete (Laskar et al., 2023; Shen et al., 2023), leading to the compilation of more challenging and comprehensive tests. However, the current generation of leaderboards still does not satisfy many of
*Core contributor †Corresponding author: haidark@sdaia.gov.sa
the requirements of researchers and practitioners looking to build on LLMs (Ethayarajh and Jurafsky, 2021; Dehghani et al., 2021). Since LLMs are extremely expensive to both train and inference, selecting the LLM (or LLM training recipe) is often the most costly decision for the entire project. Stable leaderboards are critical to making the right decision.
Leaderboards based on multiple choice questions (MCQ) for evaluation (Wang et al., 2018, 2019; Nie et al., 2019; Zhong et al., 2023; Hendrycks et al., 2020) present both convenience and significant limitations (Pezeshkpour and Hruschka, 2023; Zheng et al., 2023). While MCQs offer an automated and quantifiable means to assess certain aspects of model ability (e.g., knowledge), they fall short as a stable means to measure performance. Figure 1 demonstrates the instability of the leaderboard ranking of one popular benchmark, Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2020), under small perturbations.
Moreover, the reliance on MCQs raises concerns about the models being overfit to these benchmarks, potentially excelling in structured tests while lacking real-world applicability. This discrepancy highlights the need for more holistic and diverse evaluation methods that transcend the simplicity of MCQs (Liang et al., 2023). It also prompts critical reflection on how these models might inadvertently be trained to achieve high scores through spurious correlations, pattern recognition, and optimization for specific question formats rather than genuine language comprehension or knowledge. As LLMs continue to evolve, it is imperative to develop evaluation frameworks that can more accurately assess their abilities in a way that mirrors the complexity of real-world use.
Despite being widely used, benchmarking with MCQs has turned out to be anything but simple. It requires the full synchronization of eval-
13787
uation frameworks and results often vary wildly due to nuanced differences. For example, minor changes in prompting and scoring can produce invalid results for particular LLMs1. Recent studies have investigated the issue of sensitivity in evaluating LLMs, with some demonstrating that LLMs are susceptible to the ordering of answer choices and bias towards specific tokens/symbols (Zheng et al., 2023; Pezeshkpour and Hruschka, 2023; Lu et al., 2022), and others exploring different prompt modifications and their effects in benchmarking LLMs (Mizrahi et al., 2024; Sclar et al., 2023; Weber et al., 2023; Zhao et al., 2021).
In this work, we conduct a broad range of minor perturbation experiments to MCQ benchmarks and observe the disruption it causes to model rankings on leaderboards. We also take additional steps to precisely identify the limitations of LLMs on this measurement approach.
The contributions of this paper can be summarized as follows:
1https://huggingface.co/blog/ evaluating-mmlu-leaderboard
1. Existing model rankings on popular benchmarks break down under slight perturbations, particularly in the medium to small model sizes.
2. This behavior can be explained by the susceptibility of all tested LLMs to various forms of bias in MCQ.
3. Some families of LLMs have an over-reliance on format, pointing to potential benchmark leakage.
4. We find that LLMs also exhibit bias to the scoring method for answer choices in MCQ.
5. We demonstrate that some categories of modifications do not affect the benchmark rankings.

heading: 2 LLM Evaluation with MCQs
text: Evaluating LLMs with MCQs has rapidly become a standard for measuring the knowledge and reasoning capabilities of the model (OpenAI, 2023; Anil et al., 2023; Deepmind, 2023; Jiang et al., 2023).
Many such MCQ benchmarks have been used to measure LLMs, including Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2020), Ai2 Reasoning Challenge (ARC) (Clark et al., 2018a), and Common-sense Question Answers (CSQA) (Saha et al., 2018).
Mechanically, testing LLMs with MCQs is accomplished by presenting the question along with the answer choices to the model and selecting the choice deemed most probable by the model. Although this setup appears straightforward, LLMs react in unpredictable ways to formatting and other minor changes to the questions or the answers. LLM performance on an MCQ test can change with the introduction of an extra space (e.g., between the question and answer) or adding an additional instructional phrase (e.g., "Choices:"). In addition to this brittleness, Pezeshkpour and Hruschka (2023) found changes to the order in which answer choices are presented to GPT4 and instructGPT can change the model’s prediction.
These findings lead us to take a deeper look at how MCQ-based benchmark results are affected by small perturbations to question formats, LLM prompts, presentation of few-shot examples, and other dimensions. In particular, we introduce variations in three categories:
• Answer choice format and ordering: testing the limits of LLM sensitivity to ordering and formatting (Section 3.1).
• Prompt and scoring modifications: changing text included in the prompt and analyzing different scoring schemes (Section 3.2).
• In-context knowledge manipulation: inserting relevant/irrelevant information in the prompt and/or few-shot examples (Section 3.3).
Our main aim is to quantify how these small perturbations/variations change the rankings of a set of models on a particular benchmark. As MCQ benchmarks-based leaderboards are often used to compare models and guide model selection, we investigate the robustness of benchmarks for this purpose. Figure 1 demonstrates how existing benchmarks exhibit significant undesirable shifts in rankings under small perturbations.

heading: 3 Methods
text: In this section, we describe and justify the perturbations we apply in each category. We note that
some MCQ test changes, like modifying the order of answer choices, can change performance even for humans but the effect is typically not pronounced (Lions et al., 2021). In general, our modifications are designed to be small perturbations to the MCQ and prompts that should not affect performance. The exception to this is some of the in-context knowledge manipulations described in Section 3.3, which are designed to improve or degrade performance drastically.

heading: 3.1 Answer choice format and ordering
text: In light of earlier findings related to selection bias (Zheng et al., 2023; Pezeshkpour and Hruschka, 2023; Lu et al., 2022), we investigate the effects of changes to the presented order of answer choices and changes to the symbols associated with answer choices.
Random choice order Our first study aims to uncover how dependent MCQ benchmark performance and rankings are on the original ordering of the answer choices. We apply two simple schemes to randomly change the order of answer choices presented to the model: (i) swapping choices using a fixed set of swaps for all questions and (ii) randomly assigning new positions to each choice while ensuring each choice is moved to a different position.
Biased choice order In this setting, the correct answer choice is set to a fixed position across the entire test to measure bias toward predicting answers at particular positions. For zero-shot, we simply set the correct answer choice to each of the positions in turn.
In the few-shot case, we examine the influence of biasing the correct answers in the examples to the model’s inherent bias to particular positions. For each question, we fix the correct answer of the examples to each position in turn. We then modify the test question in two ways: (i) unchanged answer choices and (ii) correct choice fixed to the same position as examples. This setup is shown in Figure 2.
Answer choice symbols The symbols used for the answer choices (e.g. A, B, C, D) also play a role in model bias (Zheng et al., 2023). Thus, we experiment with replacing the symbols with alternative and less common tokens. This aims to decouple the bias to particular positions from the bias to symbols or the relative ordering in natural
symbols. We replace [’A’, ’B’,’ C’, ’D’] with the following two sets of symbols:
• Set 1: ["$", "&", "#", "@"] comprising of common tokens that are languageindependent.
• Set 2: ["œ", "§", "Ze (Cyrillic)", "ü"] consisting of rare tokens in the vocabulary without any implicit relative order.
In the few-shot setting, we test both assigning fixed ordering for the replaced symbols in the examples as well as changing the ordering across examples.

heading: 3.2 Prompt and scoring modifications
text: LLMs exhibit high sensitivity to variations in prompt formatting (Sanh et al., 2021; Mishra et al., 2022), forcing benchmark developers to unify prompt templates within the same evaluation scheme. However, it remains unknown if certain models have an affinity towards any specific prompt templating style. It is unclear how benchmarking prompt choices advantage/disadvantage different models. In addition to that, the scoring style may change depending on how we are prompting the context of a query. We distinguish three major categories of scoring methods for MCQs.
• Symbol scoring: Prompt template is structured as question followed by answer choices. The model chooses the answer based on the likelihood scores for the answer choice symbol. Used in Hendrycks et al. (2020).
• Hybrid scoring: Prompt template is structured as a question followed by answer choices. The model chooses the answer based on the likelihood scores for the answer choice content normalized by length. Used in Raffel
et al. (2020); Sanh et al. (2021); Chowdhery et al. (2022)
• Cloze scoring: Prompt templates are structured as a question followed by a single answer choice. Maximum normalized likelihood scores over all answer choices define the prediction. Used in Clark et al. (2018a).
Figure 3 gives an overview of each scoring method. In addition, we also investigate further modification of instruction and sentinel tokens in the prompt template.
Prompt instructions To assess the impact of subtle token alterations in prompt instructions, we conduct experiments on (i) removing question subject information and (ii) adding "Correct" alongside the answer. These targeted changes aim to identify the robustness in response to certain tokens, particularly when they carry crucial information, as well as to evaluate the influence of contextual bias introduced by minor modifications of the instruction text.

heading: 3.3 In-context knowledge manipulation
text: In this category, we attempt to measure model and benchmark robustness in the few-shot setting by testing the entire spectrum of knowledge injected in the few-shot examples. In particular, we experiment under the following settings:
Correct answer provided We provide the target question and the correct answer in the prompt as an example to the model. This corresponds to the simplest setting for the model, where it only needs to look up the answer in the context.
Incorrect answer provided This setting is the opposite of the former. The target question is provided with an incorrect answer as an example. It is challenging as the model must ignore the context and determine the correct answer independently.
Trivial examples We replace few-shot examples with simple questions the model is known to be able to answer (typically related to the language/text of the question itself). The only information the examples convey is related to formatting (Soltan et al., 2022). We create three versions of these questions and answers using GPT-4 and ensure the model can answer them correctly (as shown in Figure 8).
Out of domain examples Instead of providing examples from the same subject as the target question, we add out-of-domain questions (from another subject) as the few-shot examples. This setting corresponds to a difficulty level between the original format and providing trivial examples.

heading: 4 Experiments
text: In the bulk of our experiments, we focus on the MMLU benchmark due to the extensive nature of our experiments (11 models, 22+ settings), and extend some experiments to ARC-challenge to show generalizability.
MMLU (Hendrycks et al., 2020) is a commonly used benchmark for comparing LLMs, consisting of 57 subjects spanning four domains: humanities, STEM, social sciences, and others. Each subject includes at least 100 multiple-choice questions with 4 answer choices. The entire benchmark contains 14,042 questions (Tables A.29 and A.30 have a breakdown of the MMLU subjects and their distributions).
Ai2 Reasoning Challenge (Clark et al., 2018b) is a benchmark consisting of 7787 grade school science questions. The benchmark is split into two sets: Easy and Challenge. We conduct experiments on the Challenge set (ARC-C) which is proven to contain harder questions for existing models. The questions in ARC-C have 3-5 answer choices.
Unless otherwise stated, the reported score for each experiment/model combination on MMLU is the mean accuracy across all 14,042 questions. All tested model tokenizers encode the multiple-choice answers as single tokens. Hence, the accuracy is equivalent to the normalized accuracy. All baseline and modified MMLU benchmarks were performed
using the LM Evaluation Harness (Gao et al., 2023) library. Their implementation of MMLU measures the log-likelihood of each of the answer tokens [’A’,
’B’, ’C’, ’D’] after the input prompt and chooses the letter with the highest probability as the model’s answer.
Some of our experiments require permuting the answer choice order (Table 1, A.8, A.9, and A.10), however, this can be confusing for questions where the answer choices are dependent on their position, such as “D. All of the above.”, or reference other choices, such as “C. Both A and B.”. To circumvent this dependency, we manually inspected and modified the questions from three subjects to ensure their answers are permutation-independent for a subset of our experiments. The modified subjects are college chemistry, college mathematics, and global facts.
For each variation introduced to the MCQ benchmarks, we calculate the change in accuracy (∆Acc) and recall standard deviation (RStd) for each model. RStd measures the bias of a model to a particular answer choice by computing the standard deviation of recalls for each answer choice (Zheng et al., 2023). This metric quantifies how much the model favors particular positions for the correct answer choice. We typically observe whether RStd changes (∆RStd) are significant across experimental settings.
To measure the change in ranking induced by an applied perturbation to a benchmark, we measure the normalized Kendall’s τ distance between two rankings of n models (Kendall, 1938). Kendall’s τ computes the number of swapped pairs between two rankings normalized by the total number of pairs n(n−1)2 . We report kτ = τ+1 2 , where kτ = 1.0 indicates total agreement between rankings, and kτ = 0 indicates complete disagreement by reversing the original rankings.

heading: 5 Results & Analysis
text: In this section, we highlight the major findings of our work and combine the results of multiple lines of experimentation (detailed in Section 3) into concise observations. Additional observations and complete experimental results can be found in the appendix (Section A.1).

heading: 5.1 MCQ benchmarks are not robust to perturbations
text: As shown in Figure 1, there exist perturbations that cause dramatic shifts in the order of models with respect to commonly accepted leaderboard rankings. We find a significant number of small perturbations demonstrate this effect, while other perturbations are more benign.
Sensitive perturbations Shuffling/changing the presented order of the choices, swapping choice symbols, and alternative scoring methods all cause major shifts to the rankings (determined by thresholding kτ ≤ 0.75). For example, in a controlled experiment using a subset of MMLU, we randomly shuffle the answer choices presented to the models (Table 1). 5 out of 11 models change in ranking after the perturbation and kτ drops to 0.564. A similar pattern is seen for perturbations like fixing the correct answer to a particular position (Table 2), replacing the default choice symbols with other sets (Figure 4), and alternative scoring methods (Figure 7).
Some models elicit this behavior much more strongly. For example, we observe that Yi-6b drops from 3rd place to 7th or 8th place under some benchmark perturbations in the group of 11 models we tested (namely, the rare symbol and cloze perturbations). Other models in the same size range are more stable (e.g., Mistral-7b, Llama2-7b), not shifting more than one or two ranks under all perturbations. The reasons for this are unclear but could indicate overfitting to aspects of the benchmark style. Since training data for these models is not public, it is difficult for us to verify this hypothesis.
Unsensitive perturbations Changes that have little effect on the model rankings are discussed in Section 5.4.

heading: 5.2 Revisiting selection bias: token bias vs. position bias
text: Prior and concurrent work finds that LLMs answering MCQs are highly sensitive to the order that choices are presented (Pezeshkpour and Hruschka, 2023; Robinson et al., 2023) (position bias) as well as the symbols used as choice IDs (Zheng et al., 2023) (token bias). We find selection bias is apparent in all LLMs we test both in 0 and 5-shot setups, as shown in Tables 2 and A.6. This confirms earlier findings and highlights a major weakness of the current methods of evaluating LLMs on MCQs.
To disentangle these two sources of bias, we first measure the change in bias (measured by RStd) as we randomly shuffle the entire choice and symbols together, as performed in (Zheng et al., 2023). We
find that simply shuffling entire choices is inconclusive in ruling out the effect of position bias (vs. token bias) as there is a wide variance in the bias change across LLMs (Figure 5, Table A.7 ). In light of this, we opt to isolate token bias from position bias by replacing the default symbols (A/B/C/D) with new/rare symbols from the LLM’s vocabulary (without an implicit relative ordering) and shuffling them. This experiment, displayed in Figure 6 and Table A.8, shows that (i) LLMs always bias toward the symbols representing the choice IDs and (ii) even after shuffling the symbols, bias changes in unpredictable ways.

heading: 5.3 Another source of bias: scoring bias
text: Beyond the ordering of choices and the symbols associated with them, LLMs exhibit varying amounts of bias under the choice of scoring method for MCQs. We studied the three scoring methods described in Section 3.2: symbol scoring, cloze scoring, and hybrid scoring. Symbol scoring has become the dominant method for evaluating LLMs on MCQs, largely due to the high accuracy achieved by LLMs (Robinson et al., 2023). This, however, comes at the cost of high selection bias. Cloze scoring can essentially eliminate bias since the choices are never presented to the model, but LLMs tend to score poorly when using this method. This also does not reflect a true MCQ setting. Figure 7 and
Table A.13 detail the results of these experiments. Hybrid scoring, where cloze scoring is combined
with a prompt that reveals all answer choices to the model, represents an acceptable balance between the two, reducing bias over symbol scoring on MMLU and ARC-C, as shown in Figure 7. In light of this, we recommend practitioners to replace symbol scoring with hybrid scoring to mitigate the effects of bias on model rankings.

heading: 5.4 Minor few-shot and prompt changes have little effect on benchmark rankings
text: We ran several experiments to assess the effect of the initial prompt on model performance and rankings. We find that changing the informativeness of in-context examples, e.g. providing irrelevant/trivial examples (Figure 8, Tables A.17-A.19) or examples from subjects other than the target subject (Figure 9, Tables A.24- A.25), slightly changes performance across models and reduces bias compared to zero-shot settings but does not change rankings drastically. This finding leads us to conclude that adding few shot examples to benchmark evaluations can help reduce, but not eliminate, leaderboard sensitivity.
We also experiment with removing subject information from instructions and ending the prompt with "Correct Answer:" instead of "Answer:" (Figure 10, Tables A.20-A.23). We see little changes
(kτ > 0.9) in these prompt modification experiments.

heading: 5.5 LLMs readily reference knowledge provided in-context (even if it is misleading)
text: In our study of in-context knowledge injection, we find that LLMs can, expectedly, read off answers
to questions when the answer is provided in the context (Table A.27). However, when the question is answered incorrectly in the LLM’s context (Table A.26), all models (regardless of size) are unable to reason correctly.
To investigate whether this behavior is due to in-context knowledge acquisition or "incorrect an-
swer" pattern following, previous work has shown that smaller models tend to rely more on priors learned during the pretraining stage while larger models tend to be more influenced by knowledge given in-context (Wei et al., 2023). This result suggests that we would find that the magnitude of changes in accuracies for small models is smaller than that of larger models. However, we do not observe a significant effect to support that finding conclusively (when comparing A.1 with A.26).
This behavior is further studied in Wang et al. (2023); Xie et al. (2023); Min et al. (2022); Yoo et al. (2022) and indicates answer leakage in this way could affect benchmark results.
To test whether LLMs can infer subtler patterns in the few-shots examples, we fix all answers in the few-shot examples to each of the positions A/B/C/D. The results (Table 3) suggest that LLMs also bias their answers to these kinds of (potentially inadvertent) patterns in the context.
While we have not observed these vulnerabilities in current benchmarks, we highlight them here as (potential) sources of benchmark instability.

heading: 6 Related Work
text: Benchmarks for the evaluation of LLMs (Chang et al., 2023) such as MMLU (Hendrycks et al., 2020), HELM (Liang et al., 2023), and BigBench (Suzgun et al., 2022) have seen widespread adoption recently. Depending on the ability that is being assessed (e.g., language generation, knowledge understanding, complex reasoning) some benchmarks are designed in the form of close-ended problems like MCQs. To facilitate comparisons among LLMs, a number of leaderboards aggregating these benchmarks have been established, such as the OpenLLM Leaderboard (Beeching et al., 2023) and OpenCompass (Contributors, 2023).
However, issues with the leaderboards and the underlying benchmarks have emerged. In a case study, Deng et al. (2023) discovered contamination/leakage of the MMLU benchmark in the training sets of multiple models. A significant portion of models memorized benchmark questions and was able to perfectly reconstruct the removed part of some benchmark questions or answers. For instance, GPT-4 correctly completed the questions in 29% of the prompts with URL hinting.
Even under the assumption of uncontaminated data, the performance of models on the underlying benchmarks are not robust to minor perturbations. Pezeshkpour and Hruschka (2023) showed that specific orderings of MMLU answer choices resulted in up to ±30% deviations in GPT-4 performance on various subjects. Similarly, Zheng et al. (2023) demonstrate that models are biased to certain answer letters. On llama-30B, they showed a 27% difference in MMLU accuracy by forcing all correct answers to either position A or D. As well, (Robinson et al., 2023) find that the accuracy of LLMs improve (without regard to bias) when evaluating using a pure multiple choice question style vs a cloze question answering style.
While prior work has highlighted weaknesses in LLMs themselves (Zheng et al., 2023; Pezeshkpour and Hruschka, 2023), evaluation method (Robinson et al., 2023), or the contents of benchmarks (Dehghani et al., 2021) in our work we thoroughly study the effects these factors have on existing leaderboards and demonstrate where leaderboards lack robustness.

heading: 7 Conclusion
text: Building robust leaderboards is a major challenge for the community, as leaderboards help practitioners select the best methods and models for continued research. Given this importance, it is critical to address the breakdown of existing leaderboards to the slight perturbations we demonstrated in our work. In addition to building our understanding of the causes of this sensitivity (e.g., bias in LLMs and bias in scoring methods), future work should aim to adopt and design benchmark practices that avoid these pitfalls.

heading: 8 Limitations
text: The limitations of our work fall into two main categories: (i) understanding the causes of LLM bias and (ii) our limited success at overcoming leader-
board sensitivity. To explain LLM bias, we attempted to design
experiments that isolate each source of bias under MCQ but were unable to quantify the relative effects of bias or conclude why they occur. This was further complicated by our inability to access the pretraining datasets of the LLMs to rule out benchmark contamination. Future work in this direction will most likely require tools from interpretability research (e.g. mechanistic interpretability).
One of our main contributions was to highlight where MCQ-based leaderboards fail to deliver stable rankings. Although we succeeded in showing this, we were unable to demonstrate a robust solution to this problem. Our recommendation to, for example, use hybrid scoring methods is still not completely robust to perturbations.

heading: 9 Potential Risks
text: In this work, we do not present a new leaderboard. There is a risk that Figure 1 is interpreted as a leaderboard or used for model selection. Our intention was to demonstrate how minor changes can affect model rankings.

heading: Future Work
text: Future work in this direction will most likely require tools from interpretability research (e. g. mechanistic interpretability). One of our main contributions was to highlight where MCQ-based leaderboards fail to deliver stable rankings. Although we succeeded in showing this, we were unable to demonstrate a robust solution to this problem. Our recommendation to, for example, use hybrid scoring methods is still not completely robust to perturbations