Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks (2024)

Abstract: Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs). The research community often relies on a model’s average performance across the test prompts of a benchmark to evaluate the model’s performance. This is consistent with the assumption that the test prompts within a benchmark represent a random sample from a real-world distribution of interest. We note that this is generally not the case; instead, we hold that the distribution of interest varies according to the specific use case. We find that (1) the correlation in model performance across test prompts is non-random, (2) accounting for correlations across test prompts can change model rankings on major benchmarks, (3) explanatory factors for these correlations include semantic similarity and common LLM failure points.

heading: 1 Introduction
text: Since the introduction of the Transformer architecture (Vaswani et al., 2017), Large Language Models (LLMs) have progressed into sophisticated systems with an outstanding ability to comprehend and generate text that mimic human language. Notable models in this domain include ChatGPT1, utilizing the GPT-3.5-TURBO or GPT-4 architectures2, LLaMA (Touvron et al., 2023), ChatGLM (Zeng et al., 2023), Alpaca (Taori et al., 2023), and Falcon (Penedo et al., 2023).
Due to their effectiveness, LLMs are becoming very popular in both academia and industry, making their evaluation crucial. However, this effectiveness comes at the cost of increased complexity, which makes their evaluation very challenging. Although prior research has introduced benchmarks for different tasks along with evaluation measures,
†These authors contributed equally to this work. 1New chat: https://chat.openai.com/ 2Models - OpenAI API: https://platform.openai.
com/docs/models/
these assessments often overlook potential biases. When a benchmark includes multiple prompts with similar characteristics, it can increase or decrease the average performance of a model, so model comparisons can become brittle with respect to benchmark composition (see Figure 1 for an illustrative example). In this work, we show that the inherent connections between the prompts in current benchmarks impact the models’ performance and their relative rankings.
The standard approach for evaluation on a benchmark is to (i) obtain model responses for each prompt in the benchmark, (ii) compute the performance metrics for each response, (iii) aggregate (usually average) the performance metrics to obtain a single performance metric over the benchmark, and (iv) compare models by comparing their aggregate performance.
When aggregating performance metrics in step iii above, each prompt is generally weighted equally (Yang and Menczer, 2023; Peña et al., 2023). However, using equal weights reflects the assumption that prompts in the benchmark are “equal”, in the sense that prompts are representative samples of a target distribution of interest. In the case of LLMs, the notion of a target distribution (i.e., the distribution of all possible prompts for a given use case) is usually not well-defined. For example, different Natural Language Inference (NLI) applications may have very different target distributions, and we should not expect a single benchmark to capture every one. Therefore, one must ask: What distribution do the prompts in the benchmark represent? Would considering different distributions fundamentally change model comparisons? In this work, we present a novel approach to assess the robustness and adequacy of benchmarks used in evaluating LLMs, by analyzing the performance of multiple LLMs on a set of four major benchmarks.
Our key contributions are outlined below:
10406
1. For each considered benchmark, we observe that the correlation of model performance across prompts is significant (p-value < 0.05). This demonstrates the existence of relationships between prompts within the investigated benchmarks.
2. We explore the robustness of model comparisons to different distributional assumptions based on correlation structure, and we observe shifts in performance as large as 10% and rank changes as large as 5 (out of 14 models).
3. We provide a characterization of performance over the distribution of all possible prompt weights. This constitutes a robustness check that can be incorporated in comparative studies.
4. We show that model performance similarity across prompts can be explained by semantic similarity, but it is most likely derived by common failure points of the LLM.

heading: 2 Related work
text: Evaluating the performance of LLMs has become a critical area of research, drawing significant attention in recent years. Comprehensive surveys of LLM evaluation and benchmark quality can be found in Chang et al. (2023); Guo et al. (2023); Perlitz et al. (2023), and Liang et al. (2022).
When assessing the quality of LLMs, the robustness aspect is becoming of increasing importance (Wang et al., 2022; Goel et al., 2021). Robustness investigates the stability of a model when confronted with unforeseen prompts. Robustness research can be divided into four main lines of work (Li et al., 2023): (i) robustness under distribution shift (Wang et al., 2021; Yang et al., 2023), (ii) robustness to adversarial input (Zhu et al., 2023; Wang et al., 2023a), (iii) robustness to prompt formats, including instruction templates (Mizrahi et al., 2023; Voronov et al., 2023; Weber et al., 2023; Sclar et al., 2023), and (iv) robustness to dataset bias (Gururangan et al., 2018; Le Bras et al., 2020; Niven and Kao, 2019). Our work falls into the latter category.
Reducing bias on benchmarks is a long-standing area of research spanning many diverse fields. Applications range from weighing survey responses to match a target population (DeBell, 2018), to accounting for language biases in visual questionanswering (Goyal et al., 2017). In the context of NLI, researchers have looked into improving the quality of prompts in order to mitigate certain types of biases. Work in this area has focused on determining the quality of prompts by generating optimal prompts (Pryzant et al., 2023; Deng et al., 2022) or by clustering prompts based on semantic similarity (Kuhn et al., 2023). Additionally, researchers have investigated data leakage between benchmarks and LLM training data (Zhou et al., 2023; Oren et al., 2023).
Limited research has been conducted to study inherent biases in LLM benchmarks. Among existing works, Gururangan et al. (2018) and Niven and Kao (2019) have shown that models leverage spurious statistical relationships in the benchmark datasets and, thus, their performance on the benchmarks is overestimated. In the same spirit, Le Bras et al. (2020) propose to investigate AFLITE (Sakaguchi et al., 2023), an iterative approach to filter datasets by removing biased data points to mitigate overestimation of language models’ performance. More recently, Alzahrani et al. (2024) show that
performance of LLMs is highly sensitive to minor changes in benchmarks with multiple-choice questions. Other studies demonstrate that benchmarks often include redundancy and effective LLM evaluation can be achieved with a significantly smaller sample size (Polo et al., 2024; Vivek et al., 2024).
Our work is orthogonal yet complementary to previous work. In particular, we propose a new method to identify biases in a benchmark by looking at the performance of multiple recent LLMs on that benchmark. We show that similarity in performance correlates with similarity in prompts. To the best of our knowledge, our work is the first approaching benchmark biases by analyzing and leveraging the performance of a collection of models on a set of major benchmarks; as well as investigating the impact of inherent distributional biases in benchmarks used on LLM comparative studies.

heading: 3 Proposed method
text: In this section, we outline the problem setup and introduce the notation and expressions that will be employed throughout the paper. Second, we present the approach to evaluate whether relationships between prompts (based on models’ performance) are statistically non-random. Furthermore, we describe our method for analyzing how sensitive model comparisons are with respect to different distributional assumptions of the benchmark. Finally, we present our proposed methodology for exploring the origins of relationships between prompt performance vectors.

heading: 3.1 Problem setup
text: Consider a benchmark containing n prompts {p1, . . . , pn}, and a set of k LLMs {m1, . . . ,mk} being evaluated. We define the performance matrix Q as an n× k matrix, where every cell Q[i, j] represents the performance of model mj on prompt pi. We refer to the i-th row of that matrix, qi, as a performance vector for prompt pi. To measure how similar two prompts are with respect to model performance, we compute the similarity between their performance vectors sperf (pi, pj) := s(qi,qj), where s(·, ·) is a similarity function. Here, we consider cosine, Jaccard, and Hamming similarity. Given a performance matrix Q and a similarity function s, we compute a n × n similarity matrix Ts(Q), where every cell T [i, j] is the performance similarity for prompts pi, pj : T [i, j] = sperf (pi, pj).
Semantic meaning from text is commonly understood through the use of embeddings. An embedding of a prompt is a numerical vector that contains the learned representations of semantic meaning. Measuring semantic similarity between two prompts is achieved by measuring the distance between their embeddings. In this paper, we use ada-2 embeddings from OpenAI3. The ada-2 embeddings are widely used and have been proven effective in various NLP tasks. These embeddings have shown strong performance in assessing semantic similarity between texts (Aperdannier et al., 2024; Kamalloo et al., 2023; Freestone and Santu, 2024). For a set of prompts {p1, . . . , pn}, we compute a matrix of embeddings E = {e1, . . . , en}. E is a n × s matrix, where s is the size of the embedding vectors. To measure semantic similarity between pairs of prompts, we compute similarity metrics between the corresponding rows: ssem(pi, pj) = s(ei, ej).

heading: 3.2 Determining if performance vectors are correlated
text: Given a benchmark, we assess whether the observed similarity among performance vectors is significant. If the observed similarity is significantly high, this implies the existence of specific connections between prompts. These connections lead to similar model behavior when responding to these prompts.
To test this hypothesis, we perform permutation tests. We generate permutations of the performance matrix Q by randomly shuffling the cells of each column. In this way, we permute the values of the model responses across prompts, while holding constant the overall performance of each model (i.e., the column averages of Q). We then compute a similarity matrix Ts(Q) for the observed performance matrix Q, as well as for each permutation Q′ of the performance matrix: [Ts(Q′1), Ts(Q ′ 2), . . .]. We compare the distribution of values from Ts(Q) with the distribution of values from the permuted tables [Ts(Q′1), Ts(Q ′ 2), . . .]. We conduct a permutation test to compare the average, 75th, and 95th percentiles of these distributions. The p-value of the permutation test is calculated as the proportion of permuted tables for which the statistic is greater than the one obtained with the observed table. Additionally, we use the Kolmogorov-Smirnov (KS) test to compare the entire distribution of values between observed and permuted similarity matrices.
3 https://openai.com/blog/new-and-improved-embedding-model
To further support our findings, we cluster the observed and permuted performance vectors. If there are non-random correlations between performance vectors, we would expect the clustering of the observed vectors to have higher clustering quality metrics, such as silhouette score.

heading: 3.3 Effect of non-uniform weights in aggregate performance metrics
text: So far, we have focused on aggregate performance measures that treat prompts as if they are independent and identically distributed (i.i.d.) samples from some real-world distribution of interest—i.e., each prompt is given equal weight in calculating aggregate performance metrics. In this section, we examine the implications of relaxing this assumption for ranking models based on their performance. Generally, there is no universally correct distribution of interest—it depends on each user’s application. Here, we look into three different ways of capturing distributional assumptions (i.e., of defining weights) for a given benchmark.
Cluster-based: We leverage the clustering of performance vectors described above. We consider the following variants for evaluating performance:
1. Only include prompts that are cluster representatives (i.e., the medoids of the clusters). This effectively decreases the size of the benchmark.
2. Include all prompts, but weigh them based on their distance from their cluster representative. We employ two types of weights: (i) Distance-based: The further away a prompt is from the cluster representative, the larger its weight. This setting gives more emphasis on diversity of the benchmark. More formally, let pi be a prompt in cluster Cj , prj be the representative prompt of cluster Cj , and d(·, ·) the distance function between two prompts. The weight w for pi is:
w(pi) = d(pi, p
r j)
∑ pk∈Cj ( d(pk, p r j) ) |Cj |∑ i |Ci|
The first factor is the within-cluster weight of the prompt (normalized within cluster). The second factor weighs all prompts of a given cluster proportionally to the cluster’s size. (ii) Inverse-distance weights: The closer a prompt is to the cluster representative, the larger its weight. This setting effectively smooths out the hard clustering we produced: all data points contribute to the performance, not just the cluster representatives.
The weight w for pi is computed as:
w(pi) = d−1(pi, prj)
∑ pk∈Cj ( d−1(pk, prj)
) |Cj |∑ i |Ci|
Increasing benchmark size We start with a random prompt and iteratively add new prompts into the benchmark. To select the next prompt to add, we use two methods: (i) most informative: select the prompt with the largest cosine distance (lowest cosine similarity) from the previously selected ones in order to obtain an informative test set with a reduced semantic similarity between prompts, (ii) random: select a random prompt.
Random distributions of weights We weigh each prompt and compute weighted performance, with weights drawn uniformly at random. To achieve that, we sample uniformly at random from the unit simplex using the sampling technique described in Smith and Tromble (2004). This approach aims to provide a characterization over all possible weight configurations.

heading: 3.4 Comparing performance vectors with semantic embeddings of prompts
text: Having established that model performance is similar across prompts, we next investigate where this similarity stems from. Our hypothesis is that for a pair of prompts, similar model performance can occur if the prompts are semantically similar.
We use linear regression to determine if there exists a significant relationship between semantic similarity and model performance similarity:
sperf (pi, pj) = ssem(pi, pj)β + ϵ
where β is the coefficient of how much semantic similarity contributes to the model and ϵ is error.
Using all prompt pairs raises concerns about the data being i.i.d., given that each observation is a pairwise comparison and each member of a pair appears in many observations. To avoid that, we estimate one model for each prompt, including all the pairwise observations of which that prompt is a part. We collect p-values for the coefficients across all models and perform multiple hypotheses adjustment to generate False Discovery Rate (FDR) values. We repeat the same approach for 1000 permutations as described in Section 3.2 for both pairwise performance and semantic similarity vectors. Finally, we compare the distribution of coefficients and FDRs between original data and permutations using the KS test.

heading: 4 Experimental setup
text: In this section, we describe the setting of our experiments. Specifically, we provide details on the benchmarks and evaluation metrics we use, the LLMs we consider, and how we evaluate performance of the LLMs on the benchmarks.

heading: 4.1 Benchmarks
text: We investigate four major benchmarks that are designed for different tasks.
ANLI The Adversarial Natural Language Inference (ANLI) dataset4 is a large-scale dataset for natural language inference (NLI) (Nie et al., 2020). It is collected via an iterative, adversarial humanand-model-in-the-loop procedure, making it more difficult than its predecessors. The dataset used here comprises approximately 100K samples for the training set, 1,200 for the development set, and 1,200 for the test set. Each sample contains a context, a hypothesis, and a label. The goal is to determine the logical relationship between the context and the hypothesis. The label is the assigned category indicating that relationship. In the context of NLI, the labels typically include “entailment”, “contradiction”, or “neutral”. Finally, ANLI makes available a reason (provided by the human-in-theloop), explaining why a sample was misclassified.
HellaSwag This is a commonsense natural language inference dataset (Zellers et al., 2019), tasking machines with identifying the most probable followup for an event description. Comprising 70,000 instances, each scenario presents four potential outcomes, with only one being accurate. Engineered to be challenging for cutting-edge models, the dataset employs Adversarial Filtering to incorporate machine-generated incorrect responses, frequently misclassified by pretrained models. Covering diverse domains, HellaSwag demands a fusion of world knowledge and logical reasoning for successful interpretation.
CommonsenseQA This is a multiple-choice question-answering dataset that requires different types of commonsense knowledge to predict the correct answers (Talmor et al., 2019). It contains 12,102 questions with one correct answer and four distractor answers. The questions are crowdsourced and cover a wide range of topics such as
4 https://huggingface.co/datasets/anli
open-domain-qa, real-life situations, elementary science, social skills.
CNN/Daily Mail The CNN/Daily Mail dataset is a widely used benchmark for text summarization (Nallapati et al., 2016). The dataset comprises news stories from CNN and Daily Mail websites. In total, the corpus contains 286,817 training, 13,368 validation, and 11,487 test pairs.

heading: 4.2 Evaluation measures
text: For ANLI, HellaSwag, and CommonsenseQA, the performance matrix contains binary values (correct / incorrect answer). Hence, we use average accuracy to evaluate the performance of each model, as commonly done with these benchmarks (Nie et al., 2020; Wei et al., 2022; Zellers et al., 2019; Talmor et al., 2019). For CNN/Daily Mail, following previous work (See et al., 2017), we measure model performance using the ROUGE score.

heading: 4.3 Considered LLMs
text: In order to have a diverse collection of LLMs, we include models from several developers, such as OpenAI and Meta. These include GPT LLMs (Brown et al., 2020; OpenAI, 2023), Llama LLMs (Touvron et al., 2023), and other popular LLMs, such as Falcon-180b (Almazrouei et al., 2023), Koala 13B (Geng et al., 2023), Alpaca 7B (Wang et al., 2023b). Table 1 shows the various models used for each benchmark5.

heading: 4.4 Performance evaluation
text: For ANLI, we evaluate each model on the test dataset, which contains 1200 prompts. For each sample, we use 7 few-shot samples extracted from the ANLI dev set. For the remaining benchmarks, we randomly sample 10% of each benchmark for test and use the rest for few-shot selection. This results in 1005, 1221, and 1150 test samples for HellaSwag, CommonsenseQA, and CNN/Daily Mail respectively. For HellaSwag, we use 10 fewshot examples, while for CommonsenseQA and
5Due to constraints in LLMs’ availability, we use different LLMs for each benchmark. This does not impact our work, as each benchmark analysis is standalone and independent of the remaining benchmarks.

heading: 5 Results
text: In this section, we present the results of the experiments described in Section 3 on the benchmarks.

heading: 5.1 Performance vectors are correlated
text: To determine if prompt performance vectors are correlated, we perform the permutation tests described in Section 3.2, using different correlation measures. The obtained p-values for ANLI, HellaSwag, and CommonsenseQA are depicted in Table 2. On ANLI and CommonsenseQA, the permutation tests show strong evidence that the correlations between the prompt performance vectors are significant. For HellaSwag, our findings reveal consistently low p-values across all correlation measures when using the 75th percentile, as well as a low p-value when averaging Jaccard similarities. For the three benchmarks above, the KS test is significant across all correlation measures.
For CNN/Daily Mail the performance matrix
contains ROUGE scores, which are continuous values. Thus, we use cosine similarity to compare the average correlations obtained from the original and permuted performance matrices. The results show that the correlations among original performance vectors are significantly greater.
To further support this finding, we cluster the model responses using spherical k-means (Dhillon and Modha, 2001). We choose the optimal number of clusters to maximize the average silhouette score, computed using cosine distance. Table 3 contains the average silhouette scores of clustering the performance vectors and a random permutation of them. For all benchmarks, the performance vectors produce higher silhouette scores compared to the permuted performance vectors. This provides additional evidence to support the outcome of the hypothesis tests presented above: the performance vectors are similar.

heading: 5.2 Impact of prompt weights on performance and relative ranking of models
text: In this section, we present the results of different weighting schemes for the prompts of a benchmark, as described in Section 3.3.

heading: 5.2.1 Cluster-based evaluation
text: First, we cluster the performance vectors of each benchmark as described earlier. Then, we compute
the average accuracy of models for each benchmark, using only the cluster representatives of that benchmark. We also compute weighted performance using distance-based and inverse-distancebased weights. Figure 2 illustrates how these weighting schemes affect the relative ranking of models for each benchmark. The rows correspond to different weighting schemes, while the columns correspond to the different models and are ordered by increasing original performance (i.e., decreasing rank). Every cell contains the ranking change (compared to the original benchmark) of the model of that column for the method of that row. If there were no ranking changes, all values would be 0. However, we observe that there are multiple ranking changes as great as 5 (model is ranked 5 positions above the original benchmark).

heading: 5.2.2 Increasing size of benchmark
text: Next, we study how performance is affected by the size and diversity of the benchmark. We start with a random prompt and iteratively add new prompts to the benchmark, either by adding the most informative prompt (i.e., the one with the maximum average distance from the current benchmark), or a random one. Figure 3 shows the average performance for each model as the benchmark size increases (maximum benchmark size corresponds to the original benchmark). Looking at the most informative method for ANLI (Figure 3a), the first 400 prompts result in random performance (0.5) for all models. This suggests that the initial prompts chosen with this method are the most “difficult”, in that the models are exhibiting performance close to random (accuracy 50%). Similar results are observed for HellaSwag and CommonsenseQA (see Appendix C, Figure 10), but not for CNN/Daily Mail (Figure 3b), where the performance on the reduced benchmark follows a similar pattern as the performance on the original benchmark. The random method tracks the original performance for all benchmarks (see Appendix C, Figure 11).

heading: 5.2.3 Random distributions of weights
text: We explore the distribution of all weighting schemes and the effect they have on the weighted accuracy and relative ranking of the models. As described in Section 3.3, we sample 100,000 random weight configurations. For each model, we compute the weighted performance based on these weights.
For ANLI, HellaSwag, and CommonsenseQA
the performance of a model can change up to 10%. For CNN/Daily Mail, the range is smaller, up to 3%. Detailed results are included in Appendix D. We note that the range is similar for all models within a benchmark, indicating that it is a property related to the benchmark and not the specific models.
To further demonstrate changes in relative ranking of models, we take a closer look at the pairwise ranking differences. Figure 4 depicts a pairwise comparison of weighted performance for each benchmark. Every cell shows how often the model in the row outperforms the model of the column. For ANLI, approximately for half of the weight configurations the ranking of the top two models is reversed! However, for the CNN/Daily Mail data, there are effectively no reversals (less than 0.01%).

heading: 5.3 Relationship between model performance and semantic similarity of prompts
text: Having established that model performance is correlated across prompts, we investigate what can explain these correlations. Our hypothesis is that it is driven by semantic similarity. We use the method described in Section 3.4 to assess if there is a significant relationship between semantic similarity and model performance similarity.
Our findings show that only CNN/Daily Mail presents a significant relationship between prompt semantic similarity and prompt performance similarity (see Figure 5d). This benchmark is a text summarization task, where the success of the ROUGE metric highly depends on the ability to extract relevant entities from text. For example, we find that prompts referring to the economy or global warming have high correlation in model performance (see Appendix B, Table 5).
ANLI also makes available a reason component: what human agents state as the explanation for why the LLM gave a wrong answer. We find a significant relationship between semantic similarity using the reason component and prompt performance similarity (as seen in Figure 5a). The input prompt—consisting of the context, hypothesis and label components—shows no relationship, which is most likely because the creators of ANLI put great effort into ensuring diversity in the benchmark (Nie et al., 2020). This is also evident in Figure 3. The significance of the reason component indicates that the model performance vectors correlate because of how the model generates a response. We observe prompts where the reasons for similar model performance indicate that the model cannot do math,
e.g., “The system may have missed this as it did not add up the losses from both sets” and “the model might not know math” (see Appendix B, Table 4).
Hellaswag and CommonsenseQA use a multiplechoice format. The lack of strong evidence supporting the correlation in these benchmarks (see Figures 5b and 5c) is likely due to the embeddings picking up similarities between the different choices, rather than the logic the LLMs employ to arrive at their conclusion. This is consistent with our findings for ANLI, where a significant relation-
ship does not stem from inputs to the model, but from the LLMs’ failure points.
Our findings indicate there is a larger question about why the model performance vectors are correlated, and investigating this is central to understanding model performance. Semantic similarity can be a factor, but it depends on the task the benchmark is designed for. Based on our results for ANLI, it appears that the reasoning required for the task (i.e., reasoning types that cause models to fail), can be even more important than semantic similarity.

heading: 6 Conclusion and future work
text: LLMs are commonly evaluated on benchmarks that may include multiple prompts testing similar skills. In this work, we demonstrate this bias on major benchmarks, by showing that model performance across different prompts is significantly correlated. Furthermore, we demonstrate that LLM comparative studies can be significantly altered when using non-uniform weights for prompts during evaluation. The suggested approach can serve as a consistency check in comparative studies of LLMs, ensuring that the results take into consideration benchmark biases. Finally, we show that similar model performance across prompts can be explained by semantic similarity, but is most likely derived from common failure points of the LLM.
Our findings could influence a larger diagnostics tool for evaluating the robustness of model quality comparisons with respect to distributional assumptions of benchmarks. Future work also in-
cludes identifying additional factors that may explain these biases. This information can give rise to solutions for improving benchmarks robustness. These findings could help researchers generating novel benchmarks to identify and eliminate biases.

heading: 7 Limitations
text: Our study requires access to multiple LLMs to generate model performance vectors for each prompt in a benchmark. This can be computationally expensive and require GPUs. Some models, such as OpenAI’s GPT-4, have limited API calls, making data collection time consuming.
While we provide a novel approach for researchers to investigate bias in their own studies, providing a comprehensive de-biasing methodology is not within the scope of this work.
Finally, we have only touched the surface on why prompts have similar performance across multiple LLMs. There are many other components to investigate, such as the length of the prompt and prompt complexity. This information could be leveraged to propose solutions on improving benchmarks, without running prompts through multiple LLMs.

heading: A Prompt structure
text: The prompts used for inference are depicted in Figures 6, 7, 8 and 9 for ANLI, HellaSwag, CommonsenseQA and CNN/Daily respectively.

heading: B Results: Semantically similar prompts
text: For the statistical tests in Section 3.4, we describe a set of linear regression models being generated where each model contains the prompt pairs of a
specific single prompt. Here, we display semantically similar prompts from these models where the semantic similarity coefficient is high and significant in explaining the model performance dependent variable.
In Table 4, the ANLI reason component demonstrates that the prompts are adversarial because the model is unable to perform simple math operations. In other words, the prompts elicit the same mathematical operation task. For CNN/Daily Mail data, the prompts either refer to the economy or global warming as seen in Table 5. This entails that the models’ performance had similar capabilities in extracting text about these subjects.

heading: C Results: Increasing size of benchmark
text: Figure 10 shows results for all benchmarks for our experiments on increasing size of benchmark using the most informative method, as described in Section 5.2.2. Figure 11 shows results for all benchmarks when adding prompts in random order.

heading: D Results: Distributions of weighted performance
text: Figure 12 shows distribution of weighted performance and pairwise ranking changes for all benchmarks.
Pers imm
on 8 B
Llam a-2
13B
Koa la 1
3B Mist ral 7 B Vicu na 1 3B Clau de-2
Cha tGP
T-Tu rbo-
Bas e-05
16 Falc on-1 80b Llam a-2 70B
Cha tGP
T-Tu rbo-
030 1
GPT -4-T
urbo -110
6 (C hat)
GPT -4-0
314 (Ch
at)
0.4
0.5
0.6
0.7
Ac cu
ra cy
Test set size 100 200 300 400 500 600 700 800 900 1000 1100 1200
Alpa ca 7
B H F
Pers imm
on-8 b Koa la 1 3B H F Vicu na 1 3B H F Llam a-213B Llam a-30 B Clau de-2 .0 Llam a-270B
Cha tGP
T-Tu rbo-
Bas e-05
16
Cha tGP
T-Tu rbo-
030 1
Text -Alp
ha-0 02-C
urre nt
GPT -4-T
urbo -110
6 (C hat)
GPT -4-0
314 (Ch
at) 0.4
0.5
0.6
0.7
0.8
0.9
Ac cu
ra cy
Test set size 100 200 300 400 500 600 700 800 900 1000
Per sim
mo n 8
B Lla ma -2 13 B Ko ala 13 B Mis tra l 7 B Vic un a 1 3B Cla ud e-2
Ch atG
PTTur
bo -Ba
se05
16
Fal con
-18 0b
Lla ma
-2 70
B
Ch atG
PTTur
bo -03
01
GP T-4
-Tu rbo
-11 06
(C ha
t)
GP T-4
-03 14
(C ha
t) 0.3
0.4
0.5
0.6
0.7
0.8
anli: Average accuracy of models for increasing size of test set (adding points randomly).
Model
Ac cu
ra cy
Test set size 100 200 300 400 500 600 700 800 900 1000 1100 1200
Alp aca
7B HF
Per sim
mo n-8
b Ko ala 13 B H F Vic un a 1 3B HF Lla ma -2- 13 B Lla ma -30 B Cla ud e-2 .0 Lla ma -2- 70 B
Ch atG
PTTur
bo -Ba
se05
16
Ch atG
PTTur
bo -03
01
Tex t-A
lph a-0
02 -Cu
rre nt
GP T-4
-Tu rbo
-11 06
(C ha
t)
GP T-4
-03 14
(C ha
t)
0.4
0.5
0.6
0.7
0.8
0.9
hellaswag: Average accuracy of models for increasing size of test set (adding points randomly).
Model
Ac cu
ra cy
Test set size 100 200 300 400 500 600 700 800 900 1000
Per sim
mo n 8
B Lla ma -2 13 B Ko ala 13 B Mis tra l 7 B Vic un a 1 3B Cla ud e-2
Ch atG
PTTur
bo -Ba
se05
16
Fal con
-18 0b
Lla ma
-2 70
B
Ch atG
PTTur
bo -03
01
GP T-4
-Tu rbo
-11 06
(C ha
t)
GP T-4
-03 14
(C ha
t) 0.3
0.4
0.5
0.6
0.7
0.8
De ns
ity
(a) ANLI
Pers imm
on 8 B Llam a-2 13B Koa la 1 3B Mist ral 7 B Vicu na 1 3B Clau de-2
Cha tGP
T-Tu rbo-
Bas e-05
16
Falc on-1
80b Llam a-2 70B Cha tGP T-Tu rbo030 1
GPT -4-T
urbo -110
6 (C hat)
GPT -4-0
314 (Ch
at) Persimmon 8B
Llama-2 13B Koala 13B Mistral 7B
Vicuna 13B Claude-2
ChatGPT-Turbo-Base-0516 Falcon-180b Llama-2 70B
ChatGPT-Turbo-0301 GPT-4-Turbo-1106 (Chat)
GPT-4-0314 (Chat)
0.0%
21.9%
0.2%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
78.1%
0.0%
0.5%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
99.8%
99.5%
0.0%
21.2%
2.3%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
78.8%
0.0%
8.7%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
97.7%
91.3%
0.0%
10.7%
0.8%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
89.3%
0.0%
10.4%
0.5%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
99.2%
89.6%
0.0%
12.9%
0.4%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
99.5%
87.1%
0.0%
1.5%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
99.6%
98.5%
0.0%
3.4%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
96.6%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
0.0%
44.9%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
55.1%
0.0%
Proportion where row > column 0 25 50 75 100
(b) ANLI
Ba bb
ag e-0
72 1 (
GP T-3
)
Ch atG
PT20
23 01
Ch atG
PTTur
bo -03
01
Tex t-D
aV inc
i-0 02
Tex t-D
aV inc
i-0 03
GP T-4
-03 14
(C ha
t)
DV 3-F
P8
Tex t-A
lph a-0
02 -Cu
rre nt
0.25
0.30
0.35
0.40
De ns
ity
(c) CNN/Daily Mail
Bab bag
e-07 21 (
GPT -3)
Cha tGP
T-20 230
1
Cha tGP
T-Tu rbo-
030 1 Text -Da Vinc i-00 2 Text -Da Vinc i-00 3 GPT -4-0 314 (Ch at) DV3 -FP8
Text -Alp
ha-0 02-C
urre nt
Babbage-0721 (GPT-3)
ChatGPT-202301
ChatGPT-Turbo-0301
Text-DaVinci-002
Text-DaVinci-003
GPT-4-0314 (Chat)
DV3-FP8
Text-Alpha-002-Current
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
0.0%
0.3%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
99.7%
0.0%
Proportion where row > column 0 25 50 75 100
(d) CNN/Daily Mail
Proportion where row > column 0 25 50 75 100

heading: Future Work
text: Future work also in-
cludes identifying additional factors that may explain these biases. This information can give rise to solutions for improving benchmarks robustness. These findings could help researchers generating novel benchmarks to identify and eliminate biases