Stronger Random Baselines for In-Context Learning
Authors: Gregory Yauney
David Mimno
Venue: COLM (2024)
Keywords: random baselines, order statistics, in-context learning, evaluation
TL;DR: Random baselines for few-shot classification tasks should account for validation set reuse and small datasets.

Abstract: Evaluating the in-context learning classification performance of language models poses challenges due to small dataset sizes, extensive prompt-selection using the validation set, and intentionally difficult tasks that lead to near-random performance. The standard random baseline--the expected accuracy of guessing labels uniformly at random--is stable when the evaluation set is used only once or when the dataset is large. We account for the common practice of validation set reuse and existing small datasets with a stronger random baseline: the expected maximum accuracy across multiple random classifiers. When choosing the best prompt demonstrations across six quantized language models applied to 16 BIG-bench Lite tasks, more than 20% of the few-shot results that exceed the standard baseline do not exceed this stronger random baseline. When held-out test sets are available, this stronger baseline is also a better predictor of held-out performance than the standard baseline, avoiding unnecessary test set evaluations. This maximum random baseline provides an easily calculated drop-in replacement for the standard baseline.

Published as a conference paper at COLM 2024
Stronger Random Baselines for In-Context Learning
Gregory Yauney David Mimno
Cornell University (gyauneyēcs, coinee)
Abstract
Evaluating the in-context learning classification performance of language models poses challenges due to small dataset sizes, extensive prompt- selection using the validation set, and intentionally difficult tasks that lead to near-random performance. The standard random baseline-the expected accuracy of guessing labels uniformly at random-is stable when the evalu- ation set is used only once or when the dataset is large. We account for the common practice of validation set reuse and existing small datasets with a stronger random baseline: the expected maximum accuracy across multiple random classifiers. When choosing the best prompt demonstrations across six quantized language models applied to 16 BIG-bench Lite tasks, more than 20% of the few-shot results that exceed the standard baseline do not exceed this stronger random baseline. When held-out test sets are available, this stronger baseline is also a better predictor of held-out performance than the standard baseline, avoiding unnecessary test set evaluations. This max- imum random baseline provides an easily calculated drop-in replacement for the standard baseline.
Introduction
One of the most exciting applications of contemporary large language models (LMs) is their ability to perform complex tasks given only small number of examples (Brown et al. 2020). In-context learning (ICL), also called few-shot, performance has therefore become a critical tool in LM evaluation (Liu et al. 2022),, but the nature of few-shot tasks makes them hard to contextualize. Because they are intended to evaluate specific abilities, datasets can be small and idiosyncratic. ICL performance is extremely sensitive to small changes in formatting and demonstrations (Zhao et al., 2021 Sclar et al., 2024 Finally, the fact that few-shot datasets are intended to evaluate the outer bounds of LM performance means that they are intentionally designed to be difficult (Suzgun et al. 2023). We study the implications of these characteristics and argue for using a probabilistic baseline that better distinguishes from random performance for small datasets and accounts for searches over prompts.
Downstream users want to find and deploy the best prompt (Mizrahi al. 2024). But ICL performacce of LMs varies greatly across semantically equivalent pr features like the choice of demonstrations and their order (Zhao al. Lu et al., 2022), instruction phrasing (Mizrahi et al., 2024), and template formatting (Sclar et al. 2024 Voronov et al. 2024). Because performance
both variable and unpredictable, standard ICL practice involves searching over large numbers of potential prompts based on a validation set. Researchers often report model performance as the maximum score on validation set or the corresponding performance of that prompt on an additional truly held-out test set to avoid overfitting to the validation set (Brown et al. 2020 Perez et al. 2021). In this work we show that we can better identify prompts that may be overlitting and avoid unneccessary evaluations on test data.
In searching for the best prompts, the simplest and most common comparison is to random baseline. The standard random baseline for classification tasks is the expected accuracy of guessing labels uniformly at random (Mitchell 1997] inter alia). While universally treated as a point estimate, the accuracy of any specific random classifier follows a binomial distribution. For larger datasets, the variance of this distribution is tightly concentrated

---

Published as a conference paper at COLM 2024
around the expectation, but variance can be considerably higher for the small datasets typically used in the ICL setting. We introduce a stronger random baseline that accounts for both variance and validation set reuse by asking fairer question: if we are choosing the best of I different prompts, why not compare that prompt's accuracy to the best of t different random classifiers? shows how these two random baselines can lead to different conclusions on a fixed evaluation set. We also find that the maximum baseline can provide a more accurate estimate of a prompt's generalization to a held-out test set than the standard baseline does. This can reduce premature usage of the test set.
Treating random performance as a distri- bution has two key advantages. First, the stronger random baseline can calculated in closed form as the expectation of the max- imum order statistic of the binomial distri- bution. When choosing the best prompt from even as few as 10 options, this base- line increases the threshold for beating "ran- dom" performance by more than 7 points of accuracy for binary classification task with 100 examples. Second, by using a fam- ily of parametric distributions represent random performance rather than point estimate, we can also calculate a compa- rable performance metric across multiple datasets that takes into account factors like the number of classification options, the number of evaluated prompts, and the num- ber of evaluated examples. We can then use a tail probability to quantify what fraction of random classifiers outperform prompt. This contextualization also applies in the setting where evaluation data is used only
Proportion of classifiers
Accuracy different prompts
Random classiffer
0.2
many random clamifiers
6.0
Standard random baseline
Expected maximum
Best accuracy acroes prompts
Accuracy
Figure 1: 200 different prompts for the emoji_movie task yield a spread of accuracies for OLMo-7B (4-shot, quantized). The best prompt has much higher accuracy than the expected performance of single random clas- sifier. But its performance is worse than the expected maximum accuracy among 200 dif- ferent random classifiers.
once.
Our contributions are the following. First, we propose maximum random baseline that explicitly depends on validation set size and the number of evaluated prompts. We give a simple method for computing this value by taking the expectation of the maximum order statistic of the binomial distribution and show how this quantity varies with task parameters and evaluation setup. Second, we examine quantized LM few-shot performance across the BIG-bench Lite benchmark suite when choosing the best prompt demonstrations, finding in our setting that more than 20% of results that exceed the standard baseline do not exceed the maximum random baseline. Third, we show that when truly held-out test set is available, comparing maximum validation performance to the maximum random baseline, rather than the standard random baseline, is better able to predict whether corresponding test performance will also be above random. This can help prevent prematurely evaluating with held-out set, reducing the chance of overfitting to that held-out test set. The stronger baseline can also allow researchers to use smaller validation sets more confidently. Finally, the maximum random baseline can be easily calculated as drop-in replacement for the standard random baseline | High-quality evaluation datasets remain small, and prompt variability leads to dataset reuse. Our baselines should account for this.
Related work
Robust model comparisons. Dodge et al. (0019) introduce expected maximum validation accuracy to compare model accuracy for given budget of hyperparameter evaluations. This value can be estimated with low mean-squared error (Dodge et al., 2021). Our work casts the number of validation set reuses as the resource of interest and asks more basic question:
1 Code is available at: httpps///gt http:/gghhhhcmmyyyyyya////aa//ea
2

---

Published as a conference paper at COLM 2024
is given model even outperforming random guessing for set budget of validation set reuses? Other work also seeks to compare distributions of model performance (Dror et al. 2019), but it does not do so against a random baseline. [ard et al. (2020) find that many common NLP evaluation datasets are too small to reliably compare models unless there is large improvement in performance. BBragg et al. (2021) advocate for larger few-shot evaluations to reduce confidence intervals around reported performance. Our work finds one more reason to prefer larger evaluation sets.
Overfitting to the evaluation set. Data reuse has been approached from many angles. Some restrict access to the test set, either through submission systems (Wang et al. 2019a|b b Alex et al. 2021 Bragg et a 2021) or differential privacy (Dwork et al. 2015 Perez
et al. 2021 advocate for a true few-shtot setting: model selection using cross-validation and
minimum description length on the training set, with only one held-out set evaluation.
While they compare selected prompts to randomly selected prompts, we instead compare to classifiers that guess at random. Some works create held-out sets for ICL (Wei et al., 2022 Nori et al. 2023), though this is not always standard practice. Rather than stipulate how to access the validation set, our approach contextualizes the current practice of reusing the validation set. Sometimes the prompt template with the best validation performance is chosen for test set evaluation (Wei et al. 2022 Mizrahi et al. 2024). The stronger random
baseline does not apply to the test accuracy, but it does apply to the best validation accuracy.
Permutation tests for classification. Permutation tests have a long history for evaluating the strength of association between features and labels by retraining classifiers on datasets with shuffled labels (Golland et al. 2000 Ojala & Garriga 2010), especially in low-data medical tasks (Golland & Fischl| 2003). In contrast, our approach seeks to determine improvement over random classiher and-since the random performance distribution is known-we do not need to retrain any classifiers. Hypothesis tests have also been increasingly used in NLP to test whether one classifier reliably has better performance than Peyrard et al. 2021).
another Dror et al. 2018
Zmigrod et al. 2022
3 Random baselines
1/m Consider dataset with n validation examples and m possible labels Then p is the probability of guessing one example's 'ss label correctly uniformly at random We primarily consider how to contextualize accuracy in a setting with just one set of π examples (generalization is discussed in Section 5.2). An experiment evaluates f different classiferss (or prompts, or hyperparameter settings) and reports maximum accuracy.
Standard random baseline. Let h be a classifier that guesses labels uniformly at random for each example. Let B(n, p) be the binomial distribution with independent trials and probability p of success on each trial. Let X be the number of correct guesses that h makes evalupte when evaluated on all examples. X B(n, p) models the number of correct guesses, and:
acc(h) -
n
The expected accuracy of a random classifier is straightforward:
1 n (np)) = p
E [acc(h))
E
X
(1)
X~B(np))
Expected maximum random baseline. In this setup, we want baseline comparable to the classifier that achieves the maxinum accuracy on the validation set out of different classifiers. The idea is to take the expected maximum performance among f random classifiers. Let ,,,...,h, h, be classifiers that guess answers independently and uniformly at
"Appendix A extendds the baseline to datasets where mi varies per example. AAppendix A also shows this does not depend on the dataset having balanced labels.
3

---

Published as a conference paper at COLM 2024
random, with corresponding independent numbers of correct guesses X1, ,,,,...,,. X,. Consider the one with the highest performance:
1max
argmax {acc(h;)}
ic[F)
The number of correct guesses maa by Mmax is the 8th order statistic (or sample maximum) of the X,, denoted (()=
Xmax max ( (X:} = (()) iE[t]
(2)
Recall the probability mass function f(k) = P(X; = k) and distribution function F(k) P(X: S k) for all binomial random variables X₁:
=
f(k) = p1
F(k) 1 11 (n
(3)
where is the regularized incomplete beta function. Following the general method for order statistics of discrete random variables (Casella & Berger 2002), the probability mass function
for the tth order statistic, i.e. the sample maximum, is:
(() = k) = p(X ( S - P (()) <k)
(4)
= P(X1 S A X S kk) P(x1 <k.... X <)
(5)
P(X S - -(11 <k)
(6)
= =F(K)* - (F(k)-f(88)
(7)
We now have a closed form for the expected maximum accuracy out of random classifiers:
E Jacc (Amax)]
=
Xmax
E
[((]] = Σ kP
( =
(8)
X 12
n X
-,, -111p((n1ppp---1 (1--(nnk,,++ k
11 (H)* -)))))))) ) pt
k Σ k
n
Tail probabilities against random baselines. For either kind of random baseline, it can be useful compare to the distribution of random performances rather than just the expected accuracy. To get the probability that random classifiers outperform a given classifier ho, the tail probabilities for the accuracy of hg against both baselines are (details in Appendix A):
P Pstandard = P(acc (h) 2 acc(ho))
Pmax = P(acc (Amax) 2 acc(ho))
(10)
1 F(n acc(h)) -
F(n acc(hh) - 1)²
(11)
Properties of the expected maximum random baseline
This section builds intuitions for the values of maximum random baselines and how they are affected by experimental design parameters. The standard random baseline depends only on the number of possible labels. In contrast, the expected maximum random baseline depends on the number of validation examples n, the probability of guessing label correctly p, and the number of validation set evaluations f. Figure 2 shows the expected maximum random accuracy on a binary classification task as function of validation set evaluations t and dataset size n.
Expected maximum random accuracy is higher for smaller и and larger t. For given dataset size n, expected maximum random accuracy increases increases. For given number of validation set evaluations f, smaller datasets have greater expected maximum random accuracies. When there are fewer than several hundred examples the validation set, even a few evaluations of the validation set yields maximum random baseline at least 10% higher than the standard random baseline. For example, a dataset of π = 100 examples has expected max accuracy of 0.575 after only f 10 evaluations. But it requires more than = 10,000 evaluations to reach that expected max accuracy for a dataset with 1,000.

---

Published as a conference paper at COLM 2024
Parameter extremes capture standard set- tings. When 1 (only evaluating one random classifier), the expected maximum is simply the expectation of the binomial distribution. The maximum random base- line therefore subsumes the standard base- line as special case. When is large, the expected maximum random accuracy is nearly the same as the standard random baseline because the binomial distribution very concentrated around its expectation with large иt The implication of this result is that if an experiment has a large t, the standard random baseline is only good in-
Dataset with 2 choices per example
0.50
n= size of dataset
1,000
0.55
0.50
100
0.63
0220
10
100
1,000
10,000
t = number of validation set evaluations
dicator of random guessing if n is also large.
But while increasing is always a good idea theoretically, in practice it may not be feasi- ble for the kind of diverse, fast-moving LM tasks that are the key use case for ICL.
Figure 2: The expected maximum accuracy achieved among random classifiers on bi- nary classification dataset depends on I and the size of the cltaaset.
5 Experiments
We evaluate the extent to which the maximum random baseline recontextualizes in-context learning performance. First, we study a deliberately simple and challenging setting of prompt demonstration selection using validation data: do heavily quantized language models outperform random baselines? We then move to a setting with a held-out dataset and find that comparing maximum validation accuracy to the maximum random baseline rather than the standard random baseline is more indicative of whether held-out accuracy exceeds random chance. Finally, we use the maximum random baseline to re-evaluate published results on prompt template selection and instruction selection
5.1 The standard random baseline overstates performance
Consider the task of choosing which demonstrations include in few-shot prompt. Prior work has shown that choosing different examples can lead to drastic differences in accuracy (Zhao al. 2021]. In this setting, we are going to report the accuracy of the prompt with the highest validation accuracy. Our goal is to get the best validation accuracy possible, and we have a budget of t 200 different prompts to evaluate. Following Dodge al. (2019), we report the expected maximum validation accuracy achieved by any prompt as function of i, the number of evaluated prompts. We compare the best prompt's accuracy to both the
standard random and maximum random baselines. See Appendix B for full details.
Models and datasets. evaluate six LMs at the 7B-parameter scale: Llama-2-7b Touvron et al. 2023), OLMo-7B (Groeneveld et al. 2024), Falcon-7b (Almazrouei et al., 2023), and their instruction-tuned counterparts: Alpaca-7b (Taori et al. 2023), OLMo-7B-Instruuee and Falcon-7b-instruct. We quantize the models to 4-bit for a particularly challenging setting (Dettmers et al. 2023). We use 16 BIG-bench Lite multiple choice tasks with their standard instruction templates (Srivastava et al. 2023). Our goal is to compare differences between standard and maximum baselines due to the nature of different few-shot tasks. Because we know analytically that the size of a validation set influences random baselines, we reduce size as a confounding factor by subsampling tasks to have maximum size of 200 examples All baselines are calculated with respect to the size of the subsampled datasets
*Reproduction code is available at: https: //gi thub. com/ htmm//yyann//storgrrrrnddooo wwwpranddombaaeeenneee ines Using smaller validation sets also results substantial savings in computation when evaluating hundreds of example combinations, an important consideration for practitioners.
For the seven datasets with < 200, there are only possible demonstrations in the 1-shot setting. In these cases compare to the proper maximum random baseline with instead of : 200.
5

---

Published as a conference paper at COLM 2024
OLMo-78 emoji movie
OLMo-7B known unknowns
OLMo-78 bbq. lite json
OLMo-78
hindu kn knowledge
04
0.4
6.2
228
200
120
Coathuung (cundai))
1/0
1.0
3.0
0.5
0.5
0.5
328
320
150
320
nember valeuttinn nut enstuations
namber of vabiittion set ovokuttien
1 number vatieaaioo set evakartions
numberat valteation set evatustiees
Standard random
random baseline
L-nctt across prompts
Seat 2-hot acroas prempts
Beat 4-nnot promots
Figure 3: OLMo-7B 1-, 2-, and 4-shot beats the standard random baseline (dashed line) on four tasks in expected maximum validation accuracy. But accounting for validation set reuse with the maximum random baseline (solid black line), the best accuracies across prompts on the left two datasets are in fact no better than random.
Llama-2-7b Alpaca-7b
OLMo-78
falcon-7b emoji movie
emoji emoji_movin movie
emoji movie
0.5
0.3
03
bace
candom baseline Best T-chot prompts
0.2
022
32
100
150
200
52
502
350
230
52
102
150
220
Bes; 2 hot across
instructio- tunid oc 9os prompps Best 2 Instruciion prompos
1.0
1.0
155
55
502
230
number of validation set ovelustions
1- number validation sot enslustien
1- numberr viditatton ovalustions
Figure 4: Expected maximum validation accuracy compared to the standard random and maximum random baseline for base and instruction-tuned models on a single hard dataset.
Results for individual datasets. Figure shows OLMo-7B performance for selected tasks. The x-axis shows the number of prompts evaluated on the validation set. As increases, the maximum random baseline also increases, sharply at first and then more gradually. For emoji movie and known unknowns, the expected maximum 2-, and 4-shot accuracies across multiple prompts are above the standard random baseline but below the maximum random baseline. To further illustrate how the baselines disagree, we can use their distributions to generate tail probabilities for the observed accuracies. In the middle row of panels, tail probabilities with respect to the standard random distribution are near-zero in almost all cases. The bottom row shows tail probabilities with respect to the maximum baseline: here 1-, and 4-shot all have high tail probabilities. Contextualizing performance against the maximum random baseline shows that these datasets are more challenging than they appear. Performance is high enough on tasks like boq_lite_json and hindu_ kno.leoweeee to be above random no matter which baseline used. In these cases, tail probabilities for both baselines are near-zero. The maximum random baseline gives us additional confidence that performance in fact good.
Figure 4 compares base and instruction-tuned models the emoji dataset. These results show that the standard random baseline substantially underestimates the difficulty of this dataset. Comparing instead to the maximum random baseline reveals that only Llama-2-7b outperforms random guessing. While the instruction-tuned falcon-7b models slightly outperform the standard random baseline, the standard tail probabilities show that a significant percentage of random classifiers still have higher accuracy.
Aggregate results. Table shows that out of 288 total experiments, maximum validation accuracy exceeded the standard random baseline in 255, but maximum validation accuracy

---

Published as a conference paper at COLM 2024
Base model
Instruction-tuned
1-shot
2-shot
4-shot
1-shot
2-shot
4-shot
BIG-bench Lite dataset
L O
L
LOT
Total
novel.concepts
32
known_unknowws
46
9
code_line.escrriptie
60
0
enoji_novie
100
11
conceptua..cmminnationn
103
strange.stor
hindu.knovledge
bbq_lite__jon
200
0
200
12
language.identi ficttin
200
12
Legucindeeouction
200
3
Mlayttialog.same.r..u uodimeoordiiiiii fferent
strategyqa
synbol interpretation
200
7
vitaminc,Fase slamiincaact.verifieee. veccaiiiii
200
1
winowhy
200
0
Baseline disagreements model
5
1
4
2 3.3
3
Total baseline disagreements ! Total percentage flipped (
12
8
5
13
9
9
56
26%
19%
15%
28%
20%
23%
22%
Table 1: Agreement between the standard random baseline and maximum random baseline when evaluating the maximum performance
200 choices of prompt demonstrations on BIG-bench Lite. O: best prompt performed worse than both baselines. D: best prompt performed better than the standard random baseline but worse than the maximum random baseline. best prompt performed better than both baselines. number of examples. L: Llama-2-7b, O: OLMo-7B, F: Falcon-7B, Alpaca-7b.
exceeded the maximum random baseline in 199. The baselines disagreed in 56 experiments. This means that 22.0% of results that are above the standard baseline are not above the maximum baseline. Among base models, there are fewer baseline disagreements number of shots increases. Surprisingly there are not fewer disagreements when moving from base models to their instruction-tunet counterparts. Specific datasets are responsible for many disagreements, the highest number which come from formal.fallacies and language identification For many datasets, both baselines agree for all evaluations. Full per-dataset results are in Figures 10 and in Appendix Appendix Caaiiix/giiii gives results for non-quantized models.
the
5.2 Maximum random baseline predicts held-out accuracy
Now that we have shown that the standard and maximum random baselines differ in many typical ICL settings, we turn to contextualizing held-out test set accuracy. Validation accuracy is often used a first step select model, followed by reporting the selected model's performance on an additional held-out test set. But, of course, the more we look at the test set, the less useful becomes. Here we show that the maximum random baseline is better predictor of whether test performance will exceed random guessing than the standard random baseline and thus can avoid wasted test set evaluations.
Setup. Just as before, we choose the prompt with best performance on the validation data and report: the maximum validation accuracy, the standard random baseline, and the maximum random baseline. Given just the validation accuracy, we have to make decision: is this prompt's test accuracy above below random? Using the same models and 16 BIG-bench Lite tasks as above, we randomly partition each task into a validation set (75%) and test set (25%). For given model and task, we select the prompt from among t = 200 prompts with the highest validation accuracy. This prompt's validation accuracy is compared to the standard random and the maximum random baselines.
7

---

Published as a conference paper at COLM 2024
The prompt is said to outperform random on the corresponding held-out test set when its test accuracy is above the standard random baseline (because there has been no test set reuse). For each of 100 different splits of each dataset, predict whether test set accuracy is above the standard baseline. We are comparing two "classifiers": standard predicts true when validation accuracy is above standard random, and max predicts true when validation accuracy is above the maximum random baseline. In both cases it is possible that a low quality prompt could have high validation accuracy due to random chance, so we do not expect perfect performance. Another reason we cannot expect perfect performance on this task is that just knowing that validation performance is above the standard random baseline means that performance is better than that of at least half of the random classifiers, not that outperforms all of them. But we can evaluate which baseline gives us more insight into held-out accuracy.
validation accuracy to
Results. Evaluating all predictions across all splits of datasets, the held-out accuracy of the best prompt exceeds random 73% of
True positive rate
the time. Figure 5) shows that when pre- dicting whether test performance is above random from validation accuracy, standard achieves an accuracy of 0.82, AUROC 0.67, and AUPR 0.81. max has lower ac- curacy of 0.79, but does better on AUROC 0.80 and AUPR 0.88. standard achieves higher accuracy on this task at the expense of many false positives- often incorrectly predicts that test performance will be above random chance. In fact, the standard ran- dom baseline does only slightly better in this setting than simply predicting that all test accuracies will be above random. The maximum random baseline has higher precision-wher validation accuracy sur- passes the maximum random baseline we can be more confident that the corresponding held-out performance is above random. Re- sults split by model and dataset are in Appendix [ as are different values of t for validation set reuse. As the number of prompts f that are being searched over increases, the maximum random baseline maintains lower false positive rate even as the standard baseline's false positive rate increases (Figure S Appendix D).
0.59
9.50
0.2
0:23
0.30
0.25
0.30
positive
Recall
Standard rd random baseline
Maximum random baseline
Figure 5: ROC and precision-recall curves when using maximum validation accuracy to predict whether held-out test accuracy will be above random chance. standard and maximum curves use binary predictions of above or below the given random baseline. The gray curve uses the distribution func- tions for confidence scores.
We can additionally use each baseline's distribution function as measure of how likely validation performance is to be above random. Instead of directly predicting above or below random as compared to each baseline's expected accuracy, we instead associate each point with the percentage of random classifiers that the validation accuracy above. Both baselines produce the same rankings of likelihood above the baseline. This is because the distribution function the maximum order statistic of the binomial distribution is monotonic function of the distribution function of the binomial distribution (Equations 3 6). Both baselines therefore yield the same ROC and precision-recall curves (AUROC = 0.90, AUPR 0.96). The threshold for predicting above below random performance is what changes between baselines, leading to the difference shown in Figure 5 In both cases, using the additional information provided by treating the random baselines as distributions contextualizes performance better than point estimates alone.
5.3
Choosing instructions and template formatting
The maximum random baseline can recontextualize existing results. Mizrahi et al. (2024 Sclar et al. (2024) demonstrate that ICL performance is highly variable across instruction paraphrases and template formatting, respectively. As they release maximum validation accuracies and the number of prompts they evaluated, we are able to analyze the same experiments from the perspective of comparison to random baselines.
8

---

Published as a conference paper at COLM 2024
Mizrahi et al. 2024) report maximum validation accuracy across different instruction para- phrases for tasks from BIG-bench Lite and BIG-bench Hard. For BIG-bench Hard, they evaluate 11 models on 15 datasets and each time report the maximum validation accuracy across t = 10 prompts. Out of 165 experiments, 152 outperform the standard random baseline, and 147 outperform the maximum random baseline. 3.3% of results that exceed the standard random baseline do not exceed the maximum random baseline. For BIG-bench Lite, models are evaluated on 13 datasets with 10 prompts each. Out of 117 experiments, 116 outperform the standard random baseline, and 109 outperform the maximum random baseline. 6.0% of results that exceed the standard random baseline do not exceed the maxi- mum random baseline. From the perspective of random baselines, these experiments are relatively robust because the models perform very well and is small.
Sclar et al. 2024) use Thompson sampling to evaluate t 320 different prompt templates for Llama- 2-70b on 53 different tasks from Super-Naturallnsttuuiioi
(Wang et al. 2022 that were sampled to have и = 1,000 examples. Using the most generous parameter settings that make the maximum random baseline comparable to their setup, we find that 51 results of maximum accuracy across prompt templates are above the standard random baseline, and 46 are above the maximum random baseline. 9.8% of results that exceed the standard random baseline do not exceed the maximum random baseline.
Overall, the maximum random baseline can make us more confident when maximum valida- tion accuracies are above random performance while also identifying those pairs with especially weak performance.
Discussion and conclusion
If an experiment reports the maximum performance across multiple evaluations dataset, the standard random baseline may significantly underestimate the probability of achieving that performance by random guessing. In-context learning is particularly susceptible to this danger because of its combination of small evaluation datasets, many prompt evaluations, and difficult tasks. In such settings, the expected maximum accuracy across multiple random classifiers is stronger and more appropriate baseline. The baseline is easily calculated, and we release code for a drop-in replacement baseline.
Using the maximum random baseline for calibrating performance on validation sets can avoid inappropriate use of test sets. When a held-ouf test set is available, the maximum baseline can better predict the generalization performance of the best-performing prompt on the validation set when it is evaluated on a truly held-out test set. This can prevent test-set overuse: if the best prompt doesn't 't outperform the maximum random baseline on the validation set, then do not evaluate on the test set yet. The maximum baseline can gracefully account for dataset re-use in other ways, too. Even if an individual researcher strictly uses a benchmark's test set exactly once, benchmarks are re-used across studies (Koch et al. 2021). The maximum baseline can contextualize results on re-used test sets.
The maximum random baseline highlights the need to report more standard information to contextualize maximum accuracy across prompts. It is common to report validation size n1 and reporting the number of prompt evaluations f needed for this baseline is also good practice. Of random sample of 20 papers from EMNLP 2023 that study ICL, six report using multiple prompts, three state that they report the maximum evaluation accuracy over multiple prompts, and only one of these reports how many times the validation set was used. We argue that reporting such parameters allows for proper contextualization of results and should be best practice.
We reiterate the many calls for larger evaluation sets (Card et al. 2020 Bragg et al. 2021) this time to limit the variance of random guessing. But the reality of contemporary LM evaluation is that researchers have limited time and budget for computation and dataset development. Producing thousands of high-quality examples and evaluating many possible prompts them may not always be feasible (Liang et al. 2023 Polo et al. 2024). The maximum random baseline proposed in this paper enables researchers to reduce validation size and therefore computation and cost, while still avoiding falsely positive prompt settings.

---

Published as a conference paper at COLM 2024
We leave it to future work to study the maximum random baseline across more models, tasks, prompt variations, and scoring strategies, as well as extensions to metrics beyond accuracy, such F1. Future work can also study the rigorous specification of hypothesis tests for deciding whether generalization performance is likely to exceed random guessing, where dependence between classifiers poses challenge for analysis. While we limit our analysis to in-context learning with language models, the stronger random baseline applies to any classification setting with evaluation set reuse.
Comparing a model's performance on a new task to random baselines is the first test of the model's capabilities. Random baselines remain relevant even as model performance increases across the board because increasingly difficult tasks are regularly constructed to probe the limits of performance. Ultimately, validation datasets will continue to be reused, especially given the variability of TCL, performance. Baselines should account for this.
References
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, et al. RAFT: A real-world few-shot text classification benchmark. In NeurlPS Datasets and Benchmarks Track, 2021.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The Falcon series of open language models. arviv preprint a:Xit:2311.16867, 2023.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. FLEX: Unifying evaluation for few-shot NLP. In NeurIPS, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In EMNLP, 2020.
George Casella and Roger L. Berger. Statistical Inference. Thomson Learning, 2nd edition, 2002.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In NeurlPS, 2023.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. Show your work: Improved reporting of experimental results. In EMNLP, 2019.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. Expected validation performance and estimation of random variable's maximum. In Findings of EMNLP, 2021.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart.
The hitchhiker's guide to testing statistical significance in natural language processing. In ACL, 2018.
Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance-how to properly compare deep neural models. In ACL, 2019.
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toni Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. In NeurlPS, 2015.
Polina Golland and Bruce Fischl. Permutation tests for classification: towards statistical significance in image-based studies. In Biennial International Conference on Information Processing in Medical Imaging, pp. 330-341. Springer, 2003.
Polina Golland, Feng Liang, Sayan Mukherjee, and Dmitry Panchenko. Permutation tests for classification. JMLRR, 2000.
10

---

Published as a conference paper at COLM 2024
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, lan Magmusson, Yizhong Wang, et al. OLMo: Acceler- ating the science of language models. arXiv preprint ar arXi::2402.00888, 2024.
Charles R. Harris, K. Jarrod Millman, Stéfan J. der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Rio, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumlPy. Nature, September 2020.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn't always right. In EMNLP, 2021.
Yili Hong. On computing the distribution function for the Poisson binomial distribution. Compulational Statistics & Data Analysis, 59:41-51, 2013.
Bernard Koch, Emily Denton, Alex Hanna, and Jacob G Foster. Reduced, reused and recycled: The life of a dataset in machine learning research. NeurlPS Dafasets and Benchmarks Track, 2021.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. TMLR, 2023.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACI., 2022.
Tom Mitchell. Machine Learning. McGraw Hill, 1997.
Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dařna Shahaf, and Gabriel Stanovsky. State of what art? call for multi-prompt LLM evaluation. TACL, 2024.
Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist founda- tion models outcompete special-purpose tuning? Case study in medicine. ar preprint arXiv::311. 16452, 2023.
Markus Ojala and Gemma C Garriga. Permutation tests for studying classifier performance. JMLR, 2010.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In NeurlPS, 2021.
Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. Better than average: Paired evaluation of nlp systems. In ACL, 2021.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. arXiv preprint 17X492240..14992, 2024.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In ICLR, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TMLR, 2023.
11

---

Published as a conference paper at COLM 2024
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challeng- ing big-bencl tasks and whether chain-of-thought can solve them. In Findings of ACL, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpaca: A strong, replicable instruction- following model. 2023.
modell.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint (xXi::2377.09288 2023.
Anton Voronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements. Findings of ACL, 2024.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A stickier benchmark for general- purpose language understanding systems. In NeurIPS, 2019a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019b.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, al. Super-Naturallnnstrcciin on 1600+ NLP tasks. In EMNLP, 2022.
Generalization via declarative instructions
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In EMNLP: System Demonstrations, 2020.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use:
Improving few-shot performance of language models. In ICMI., 2021.
Ran Zmigrod, Tim Vieira, and Ryan Cotterell. Exact pairedd-nr testing for struc- tured test statistics. NAACL, 2022.
12

---

Published as a conference paper at COLM 2024
A
Additional details of the expected maximum random baseline
The probability of randomly guessing a correct answer does not depend on label balance. Consider random classifier h that guesses labels uniformly at random, independently across examples. Given a labeled example (x, y), the probability that this random guesser gets the correct label is 1/m, where is the number of possible labels. It does not depend on the dataset having balanced proportions of labels:
m
P(h(x) = y) = U (y=en((x)
(nh(x) =
Σ 1
P(y (n)())= =t)
Σ
P(y e) P(h(x) 9)
Σ P(y =) m
1M-1 m Σ P(y= E-
Extending to tasks with different numbers of labels per example. Some tasks allow each example have different number of possible labels. For example, the BIG-bench task
has examples with 2 to 5 possible labels each. The number of correct guesses modeled by a Poisson binomial distribution, where each independent trial can have different probability of success, rather than binomial distribution (Hong) 2013). The maximum order statistic of the Poisson binomial distribution can be found by plugging in its distribution function and probability mass function into equation 4 This extension is implemented in our code, though we leave further study to future work.
Tail probabilities against random baselines. The tail probability for the accuracy of classifier hn against the standard random baseline is:
Pstandard P(acc (h) 2 acc(h)))
= 1 P(acc (h) < acc(Ho))
= P( (h) < n acc(hh)))
P(X < n accc)h)))
(F(n (FCa)cc)))) (n acc(hh)))
acc(ho) 1)
Datacet with choices with choices
Dataset with choicos
Dataget chokes
p
-33
p
p 3/5
10.00
2000
12000
1.0 12.99 9C09
2000
3000
11.90 R1.09
number validation avalsation
1 number validation avaluutions
rumber validation avaluations
number validation and evaluations
Figure 6: The expected maximum accuracy for various parameter settings. The leftmost plot the same Figure but with larger parameter ranges.
13

---

Published as a conference paper at COLM 2024
Data set with 2 choices per axample
Datanet with cholces acample
Dacaset with 4 choicss scample
Dutanet with choices example
Accuneay
t=1
10.40 33
10:06
10
22.000 '1
28.03
Acumscy
t=10
2:10
19,748
10/00
10
11.990
1,900
Acorssy
t=100
0.50
6.0
0.50
193
1,99
10/008
188
1.83
10
90
1,206
11.900
1,300
15705
Accumacy
t=200
L300
38.00
Figure Tail probabilities with respect to the expected maximum random baseline for vari- ous accuracies and parameter settings. The top row (t 1) corresponds to tail probabilities for the standard random baseline. Contours are shown at every 0.1 interval, with the 0.5, 0.1, and 0.01 contours labeled. Plots have been smoothed for ease of visualization.
The final line follows because the binomial is discrete distribution over integers, so (k) F(k) = F(k 1) f if we define F(-1) = 0. By similar reasoning, we get the tail probability with respect the maximum random classifier:
Pmax Placc (Imma)) 2 acc(ho))
P P(X1) nccc(h)))
(F(nacc(ho)- - 1))
Additional plots. F shows the expected maximum accuracy for various parameter settings of и, the number of examples in the evaluation dataset, the probability of guessing correct answer each example (one divided by the number of choices), and the number of random classifiers. Figure gives tail probabílities for the expected maximum accuracy for various parameter settings. The top row (t 1) corresponds to tail probabilities for the standard random baseline.
naive implementation of the maximum random baseline depends mainly on the number of examples in the dataset. The most expensive settings that we consider still run in under 1 second for 10,000 examples.
Runtime.
14

---

Published as a conference paper at COLM 2024
B Evaluation details
We evaluate models in the in-context learning setting, where the model is supplied with natural language prompt followed by demonstrations and a possible answer (Brown al. 2020). This is commonly referred to as the few-shot setting, though few-shot can reter to other settings as well (Bragg et al. 2021).
Datasets. We et al.] 2023):
16 datasets from BIG-bench Lite (Srivastaval bbq_lite.jsom code_line_dee diieeddeccripton, conceptual.conb conbalaatninnnnss,.
movie, formal.fallacies.. wtw.lesssyllogiissnneeeeeee logical.deduction, novel_concepts, play.dialog.same..o _iitssaaeeooooeiiaeeeeeeeeeeeeee fferent, strange.stories, strategyqa, symbol..nterpretation,, witaminc.aatt.vrrificca use these because within a dataset, they have the same or very similar numbers of labels across examples. We begin with each full dataset, and for the tasks larger than = 200 examples, we randomly sample 200 examples for evaluation and keep these fixed for all experiments. Lists of which examples were sampled can be found in the supplementary code.
kandu.knowledge, known_unknowe
winowhy. We
Prompts and demonstrations. A prompt consists of natural language instruction with demonstrations in given format. Each demonstration is an example along with its ground- truth label. In our deliberately simple setting, we keep the demonstrations constant across evaluation examples-2 a prompt refers fixed string with fixed demonstrations prepended to validation examples. We use each multiple choice task's 's standard instruction template provided with BIG-bench. We evaluate 200 different prompts for each combination of model, dataset, and number of demonstration shots. Demonstrations are chosen at random and appended each other with \n. All prompts with their chosen demonstrations can be found in the code repository. For given example with multiple choices, select model's answer as the label with the highest average log-likes likeliiikiioo per token (Holtzman et al. 2021).
For tasks with more than 200 examples, we sample 200 examples all parameter combinations. We choose demonstrations from the remaining unsampled examples in the task. For the seven datasets with prompt demonstrations in the 1-shot setting. In these cases we compare to the proper maximum random baseline with 1 instead of / 200. For the same datasets, there is subtlety regarding dataset size. Because the demonstrations come from the pool of available examples, the 1-shot, 2-shot, and 4-shot versions of the dataset differ by few examples. Normally this would not make much of difference, but the maximum random baseline depends on dataset size. For ease of visualization in Figures 4 10 and 11 we plot the maximum random baseline for the dataset's full number of examples camples, which is slightly weaker than each setting's ss exact max random baseline (because the baseline increases with smaller 1t). All baseline judgments in Table 1 are with respect to each setting's exact maximum random baseline.
fixed set across
200, there only possible
Implementation details. All evaluations are implemented using NumPy (Harris al. 2020 and Hugging Face transformers (Wolf et al. 22200. We use bi tsandbyes to quantize models to NF4 4-bit with nested quantization and compute dtype bfloat16 (Dettmers al. 2023). In cases where we compared quantized models to non-quantized models, performance was not consistently better or worse. We use an NVIDIA RTX A6000 with 48GB of RAM.
15

---

Published as a conference paper at COLM 2024
C Results for non-quantized models
The main paper evaluates the performance of quantized models. 2able gives full equivalent results for non-quantized models. In this setting, 15.8% of results that exceed the standard baseline do not exceed the stronger random baseline.
D Held-out set evaluations
For the results in Section 5.2 Table 3 shows per-model results. The maximum random baseline outperforms the standard baseline in AUROC and AUPR across all models on these datasets. Table4 per-dataset results.
Base model 2-shot
Instruction-tuned 2-shot
1-shot
4-shot
1-shot
4-shot
BIG-bench Lite dataset
L O F
I. O
LO
O F
A O F
A O F
Total C
_concepts
known unkenwwss
code_line inedderrroottoonn
60
enoji Lovie
100
7
com
103
0
strange_stories
174
kindu_kno
175
bbq_lite_jjon
forml fallaries sy segissss negation
200
11
language innguageidentiFicatiio
200
7
logical deduction
200
play.dielog.same.c .iiieeeentt
200
strategyqa
200
symool_interpretttion
200
sttannn..act.vvrrffcaattt
200
winowhy
200
Baseline
disagroements per model
4
2
3.3
1
4
2 1 5
Total baseline disagreements
10
40
Total percentage flipped (
21%
10%
14%
15%
17%
15%
16%
Table 2: Non-quantized model results. Agreement between the standard random baseline and maximum random baseline when evaluating the maximum performance over 200 choices of prompt demonstrations on BIG-bench Lite. O: best prompt performed worse than both baselines. C: best prompt performed better than the standard random baseline but worse than the maximum random baseline. best prompt performed better than both baselines. number of examples. L: Llama-2-7b,
O: OLMo-7B, F: Falcon-7B, A: Alpaca-7b.
Accuracy
Precision
Recall
AUROC
AUPR
Model
Standard
Max
Standard
Max
Standard
Max
Standard
Max
Standard
Max
Llama-2-76, Base model (77%)
0.92
0.87
0.91
0.95
0.99
0.88
0.85
0.91
0.93
OLMo-78, Base model (72%)
0.79
0.73
0.79
0.89
0.97
0.72
0.64
0.74
0.79
0.84
Faleon-7h, Base model (67%)
0.77
0.77
0.98
0.74
0.69
0.79
0.77
0.84
Alpaca-7b, Instruelion-tuned (84%)
0.87
0.84
0.96
0.99
0.84
0.84
0.94
OLMo-7B, Instruction-tus -uned (72%)
0.78
0.72
0.78
0.86
0.97
0.71
0.63
0.73
0.78
0.83
Faleon 7h, Instruction tuned (68%)
0.79
0.80
0.77
0.98
0.77
0.67
0.81
0.77
0.86
Total (73%)
0.82
0.79
0.81
0.92
0.98
0.79
0.67
0.80
0.81
0.88
Table 3: Per-model results when using whether maximum validation accuracy is above each baseline to predict whether held-out test accuracy will be above random chance. Percentages in parentheses indicate the proportion trials where the best prompt's test accuracy was above random chance.
16

---

Published as a conference paper at COLM 2024
AUROC
AUPR
BIG-bench Lite
Standard
Standard
Max
codo li deseriiption (100%)
0.50
0.50
1.00
1,00
lite 1son (94%)
knowledge (100%)
0.50
0.50
1.00
1.00
cooee_oo (93%)
0.50
0.46
0.93
0.92
smo11 novio (62%)
0.50
0.53
0.62
0.64
vitaminc fact verification (83%)
0.96
0.94
0,98
0.98
conceptual con.inniioossssss (81%)
fomal_aallaciess.yy htg.ee/yyyylllsssnegglllll (37%)
0.58
0.51
0.41
0.37
unownnunknowwss (59%)
logical deduction (81%)
0.50
0.48
0.81
0.80
play dialog different (59%)
strange.stories (69%)
0.65
0.80
0.77
0.86
symbol, innteraretattionn (17%)
0.78
0.50
0.33
0.17
kinowhy (100%)
n/a
n/a
1.00
1.00
stratesyga (95%)
0.50
0.50
0,93
0.93
Total (73%)
0.67
0.80
0.81
0.88
Table 4: Per-dataset results when using validation accuracy to predict whether held-out test accuracy will be above random chance. Percentages in parentheses indicate the proportion of trials where the best prompt's test accuracy was above random chance.
1
True positive rate
0.75
0.75
0.75
True positive
0.50
0.50
0.59
025
True
0.5
0.25
False positive
False pasitive
False positive rate
10
100
200
True positive rate
0.75
0.75
0.75
True positive
0.50
0.50
0.59
0.25
True
0.5
0.25
positive
False positive
False positive rate
Standard random
Maximum random baseline
Figure 8: ROC curves for multiple values of t when comparing maximum validation accuracy to baselines to predict whether held-out performance is above random guessing.
The final panel is what is shown on the left side of Figure 5
17

---

Published as a conference paper at COLM 2024
E
Full results
This section provides full BIG-bench Lite results for Llama-2-7b and Alpaca-7b in Figure 9
OLMo-7B and OLMo-7B-Intruuct in Figure 10 and Falcon-7b and Falssre7 in
Figure 11
We report expected maximum validation accuracy, as in Section
json
hindu, knowledge
conceptual, comceptual.comiinttion
04
0.2
3.5
055
156
150
folmal_.allaciesspe syllogisms_nneeeeeeeee nnegatiionnnnnnnnnnnnoooooooooon
known_unknen unknowns
logical_ded deduction
baooyddallggsammg differeent
0.2
0.5
04
58
520
350
230
1.0
106
150
200
1.0
56
180
153
220
53
100
180
200
100
202
3.0
100
156
00
50
102
153
232
26
100
150
200
nomberr volidation eveluations
number sotdarios enskattoss
number otvaliaation exaluntiors
sumber vat.aatioo
strange, storles
symbol Interpretation
winowhy
strategyqa
0.5
0.18
320
0.4
0.2
0.50
350
130
150
150
2.
05
35
50
L- momberrad villdation set eveluations
L- number of saldarton set easkartiss
1- number olsaliaation set exaluation
t- sumber valitation set evalustiess
novel, concepts
emoji movie
fact verification
language, Idantification
0.2
0.35
00
(cundardh
0.5
12
530
350
233
1.0
106
130
200
3.0
50
187
152
235
53
350
350
200
randors
random beadine
Beet Tochot prompee
Bee: 2-enet proppo
Dest Siehot prompea
Bast L-shet Intruccson tuned ecoss prompps
Seal: Zuanot 109c-ctoon suned promis
Bust 4abot luned scom prompts
bassinn
Figure 9: Full BIG-bench Lite results for Llama-2-7b and Alpaca-7b.
18

---

Published as a conference paper at COLM 2024
hindu knowledge
conceptual combinations
0.45
0.4
0.25
0.5
0.25
202
1.0
150
2.
120
200
0.5
05
203
0.0
306
350
200
00
50
107
235
150
200
1emmmeer validation evaluations
number valdation evakariion
number otvalidatio ovaluations
sumber valicattoo ovalustions
Tormal fallades syllogisms negation
known, unknowns
Togical deduction
play dialog same or differ ent
0.25
190
203
1.0
106
116
1.0
50
181
212
150
150
(nsadnan))
15
330
350
50
300
330
200
50
187
153
231
53
300
150
200
ta mimmmbrr validation not evaleatioon
number of sstcatio ast evakustion
t- number otvallsation eraluntions
<m sumberotvvlltationn evalustion
Srange_stories
symbol interppettrtnnnnonnn
winow by
strategyqe
04
0.58
535
0.50
15
199
100
0.0
166
110
00
10
191
150
300
validation
numberb ofvaliaationnn evalsation
ovalustions
concepts
emoji
fact_verification
language, identificatiion
035
0.35
0.2
0.05
330
283
155
56
200
0.5
0.5
055
35
150
150
Best Lisho promptx
Besi 2 prompox
Best 4-ghor peompes
Beet 1-47et tured ocs promipss
Ree 2-cnet
Best 5-hoo
bassliee
boodine
uuned croos
tuned ac00s
Figure 10: Full BIG-bench Lite results for OLMo-7B and OLMo-7B-Instrruct
19

---

Published as a conference paper at COLM 2024
hindu knowledge
conceptual combinations
655
0.5
02
202
1.0
150
50
120
200
0.5
05
203
3.0
306
350
00
50
107
235
150
200
n.mbmer validation evaluations
number valdarionn evakariion
number otvalidatio ovaluations
sumberat valicaaioo
Tormal fallades syllogisms, negation
known, unknowns
Togical deduction
play dialog same or different
0.20
190
203
1.0
166
116
1.0
50
181
212
150
150
(nsainan)
18
330
350
50
300
350
200
50
180
153
53
300
150
200
ta mimmmbrr validation not evaleaaiion
number of satdatos ast evakustion
t- number otvallsation eraluntions
<m sumberot valiaaaaaiiooooo evalustion
Srange_storiess
symbol_interprette interrpeetrtinnoonn
winow by
strategyqe
320
15
199
100
0.0
116
110
00
16
191
150
300
validation
number ofvaliattion evalsation
ovalustions
concepts
emoii
fact_verification
language, identificationn
125
155
200
0.5
0.5
05
35
156
164
150
Best Lisho promptx
Busi 2 prompox
Bost 4.shor prompes
Beet 1447et
Bee 2-cnet
Best 5-toot tuned ac0os
bassliee
boodine
tured acioss
uuned cross
promipss
Figure 11: Full BIG-bench Lite results for Falcon-7b and Falcon-7b-instruct.
20


Reviews:
[{"review_id": "tI3n0Uz6B9", "reviewer": "Reviewer_fFwf", "summary": "This work proposes an alternative approach to LLM in-context learning evaluation against a random baseline by considering the performance of the best random baseline across multiple runs. The proposed method is shown to reframe the performance of several state-of-the-art large models on a subset of BigBench tasks, serve as a better predictor of test set accuracy, and provide more precise information about multi-prompt evaluation results.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review!\n\n***Concern:** Models are more often compared to each other rather than to random baselines.*\n\nWe agree that researchers usually compare to other models. We're saying that they should also make sure they beat a more honest random baseline. Our results show that when datasets are small, even models that perform well above the standard baseline can be beaten by our new baseline. In such cases, it doesn’t matter as much which model is better if none outperform random guessing. We will also emphasize that our goal is not just to support academic papers, but to support practitioners.\n\n***Concern:** The discussion of random baselines assumes that labels are uniformly distributed between all classes. When labels are imbalanced, the probability of a correct guess is not 1/m.*\n\nThank you for encouraging us to look at this! We will add the following to Section 3.\n\nSurprisingly (even to us!), the probability that a uniform random guesser gets the correct label is $1/m$ and does not depend on class balance. The probability that a uniform guesser is correct on an example in a binary task (0/1 labels) where a possibly imbalanced $q$ fraction of examples are labeled class 0 does not depend on $q$. Let $y$ be the correct label and $g$ be the guess:\n$$P[y = g] = P[(y = 0 \\cap g = 0) \\cup (y = 1 \\cap g = 1)] =  P[y = 0]P[g = 0] + P[y = 1]P[g = 1] = q * 0.5 + (1-q) * 0.5 = (q + 1 - q) * 0.5 = 0.5$$\n\nWhen a dataset is imbalanced, a classifier that guesses the majority label will have a higher accuracy than a uniform random guesser. But classifiers that always guess the majority are not randomized, so taking the max of multiple will not increase accuracy. Random guessing proportional to the class imbalance will do better than uniform guessing: the probability of a correct guess becomes the sum of squared label probabilities (which is a constant > 1/m), so we can reuse the max baseline framework with a success probability p > 1/m. We’ll expand this analysis—sorry this is just a sketch due to word limit.\n\n\n***Concern:** p-values comparing to random baselines are not applicable in a setting with multiple models because of multiple comparisons.*\n\nWe are using p-values as a way to summarize how many uniform random guessers a given model outperforms. We don’t use them for any hypothesis testing, so we think the multiple comparisons problem should not apply here. We will rename “p-value” to “tail probability” to prevent confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem of robust evaluation studied in the paper is highly important, especially in light of other advances in the field\n2. The proposed method is simple to understand and implement, yet provides a theoretically grounded way of estimating maximum performance across classifiers\n3. The set of experiments is sufficiently extensive to demonstrate the benefits of the method in several settings", "reasons_to_reject": "1. I am slightly concerned about the potential impact of the method, because in a significant amount of research, language models are compared to each other and not to a random baseline. The maximum random baseline might indeed outperform the accuracy of state-of-the-art models on some tasks, but often the point of comparison is to measure progress in capabilities on a certain task and not to check if the model is meaningfully better than random choice. For instance, p-values against random baselines might see limited applicability because evaluations frequently involve multiple models and therefore should be viewed as multiple comparisons from a statistical perspective.\n2. The approach appears to assume that the ground truth distribution of class labels is uniform: unless I misunderstand something, the probability of the random classifier being correct is not $1/m$ if a label is chosen at random. I wonder if the proposed approach can be extended to accommodate for such cases; at the very least, it would be worth mentioning the potential class imbalance when describing the method", "questions_to_authors": "1. Have you empirically evaluated the impact of class imbalance on the performance of a baseline that samples classes uniformly?", "extra_scores": {}}, {"review_id": "TIkdRE7tj8", "reviewer": "Reviewer_WAy5", "summary": "This paper proposes a well-justified stronger random baseline for in-context learning (ICL), presenting the simplest case-study for the new random baseline: ICL in the context of single-label classification tasks. The idea behind the baseline is taking the expected best performance across multiple random classifiers, where the actual strength of the baseline naturally depends on the choice of the population/validation set, as well as on the number of the evaluated random classifiers. These dependencies are also analysed in the paper with some practical tips provided to the researchers and practitioners.\n\nThe importance of having such a more elaborate baseline especially for ICL-style experiments is nicely motivated and the post-hoc analysis on BIG-Bench Lite reveals that the choice of a different random baseline can even have large impact on how one perceives performance of the actual method/system.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review!\n\n***Suggestion:** Add a plot showing how expected max accuracy varies with n and t.*\n\nWe will clarify that Figure 2 at the top of page 5 is this plot! Figure 6 on page 13 shows the same for 3-/4-/5-way classification. \n\n***Concern:** Does the max random baseline hold when evaluating a selected model on an unseen test set?*\n\nThank you for asking this. You are exactly right about the standard baseline on the test set if you only evaluate one classifier. Section 5.2 shows that the max random baseline can—perhaps surprisingly—predict performance on an unseen test set. Comparing validation accuracy to the max random baseline is a better indicator than the standard baseline of whether the selected classifier will outperform random guessing on an unseen test set. We will expand Section 5.2 and emphasize these findings in the intro.\n\n***Concern:** Many directions for future work (more models/tasks/prompts/scoring strategies/metrics, number of labels, task complexity, calibration) should have been explored already.*\n\nWe agree that it’s always more convincing to have more experimental settings. Do you have specific concerns that additional experiments could answer beyond showing general robustness?\n\nWe think discussing the variety within the experiments we have already run will address some of your concerns:\n- Task complexity, number of labels: We study 6 models on 16 tasks in 3 few-shot settings, with numbers of labels ranging from 2 to 11 labels. The max baseline works well for all tested label numbers. The baseline matters most for hard tasks where models are right at the cusp of beating random guessers.\n- More models, tasks, prompts: Section 5.3 uses published results to analyze the max random baseline for 1) prompt template choice on Llama-2-70b accuracy across 53 tasks from Super-NaturalInstructions, and 2) instruction paraphrasing across 11 models on 15 BIG-bench Hard tasks and 9 models on 13 BIG-bench Lite tasks.\n\nAs a comparison, we take the Batch Calibration paper you linked as a model of good practice. It studies 1 new method (and 4 baselines) across 3 models on 13 tasks and 1 model on 6 more tasks.\n\nOur goal with this paper is to 1) introduce the max random baseline and 2) demonstrate that the baseline makes a difference in important settings. Experiments in future work will add nuance to the max baseline’s impact, but we don’t think any outcome of those experiments would change the correctness or usefulness of the new baseline.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The main idea is nicely motivated and elaborated on from multiple perspectives: (a) practical need; (b) theoretical justification, and (c) empirical findings.\n- The analyses delve deep into the main problems of the standard random baseline for ICL and how different choices of random baselines can provide different experimental conclusions.\n- Some practical tips have been provided, and the paper provides a nice balance between more theoretical insights and its downstream/practical value.\n- The paper is very well written and the main concepts and ideas have been well described and I enjoyed reading the paper.", "reasons_to_reject": "- The choice of tasks could have been made more comprehensive so that the reader also gets a perspective related to task complexity, number of labels, and some calibration issues and properties of ICL (e.g., see https://arxiv.org/pdf/2309.17249 for some work on calibration for ICL)\n- The paper leaves too many directions for future work which it should have ideally explored already. For instance, at least some directions should have been investigated among the ones listed in Section 6:\n1. more models\n2. more tasks\n3. prompt variations\n4. scoring strategies.\n5. other evaluation metrics\n6. classification tasks with different properties (single-label vs multi-label or a small number of labels vs a larger number of labels).\nPut simply, more experiments would have strengthened the main findings of the paper and possibly the wider adoption of the proposed stronger random baseline. \n\n- I am also unsure if the proposal for the stronger random baseline holds in a standard setup where we do selection/optimisation on validation set multiple times and then just port a single best model to the test only once. The random baseline on the test set is then again the standard random baseline? I might be missing something here, so I would definitely like to see this discussed.", "questions_to_authors": "This is not a question, but rather a suggestion. It would be useful to maybe provide a table or a simple plot that shows how expected maximum accuracy changes with respect to its crucial parameters (n and t) so that the reader gets a better overview of these dependencies.", "extra_scores": {}}, {"review_id": "a9CGCqBsJd", "reviewer": "Reviewer_nvye", "summary": "This paper introduces a novel random baseline for evaluating in-context learning performance.\n\nIn contrast to traditional random baselines that rely on prompts achieving the highest validation set performance,\n\nthe proposed method leverages multiple classifiers to generate more general and robust evaluation results.\n\nSpecifically, it first identifies the best-performing random classifier and then reports the expected performance derived from a variety of prompts.\n\nThe experiments demonstrate that this expected random baseline provides stronger and more reliable baselines compared to simple random baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review!\n\n***Concern:** The usefulness of the proposed method is unclear. It replaces a random baseline, with no significant improvement.*\n\nWe show that the standard random baseline is wrong, and that it systematically underestimates random guessing. In our experiments, >20% of \"above random\" results were actually below random. We present a more accurate replacement that may \"outperform\" seemingly strong results. Our goal is to better answer the question: “Does my model outperform random guessing on this task?” We help those who are developing new models, or choosing a prompt to deploy, not fool themselves into overestimating performance.\n\n***Concern:** It is not clear how the empirical results support the claims.*\n\nOur main claim is that if you are choosing the best of multiple real classifiers, the correct baseline is the best of multiple random guessers, not the *average* random guesser. Our results show:\n1. There is a meaningful difference between these baselines in real settings (Figures 1, 2, 3, 4; Table 1)\n2. The stronger baseline better predicts held-out performance, reducing wasted test set evals (Figure 5)\n3. The stronger baseline can re-evaluate published results (Section 5.3)\n\n***Concern:** The computational cost is higher than the standard random baseline.*\n\nWe will add to Section 4 that the max baseline runtime mainly depends on the number of examples, and at its most expensive still runs in under 1 second for 10,000 examples.\n\n***Concern:** t-tests and p-values may not be valid for small sample sizes.*\n\nWe do not conduct any statistical hypothesis tests, such as t-tests. $t$ is a variable for the number of random guessers. We use “p-values” simply to summarize how many random guessers a given model outperforms. We will rename “p-value” to “tail probability” to avoid confusion.\n\n***Question 1:** What do the classifiers mean?*\n\nWe will clarify what “classifiers” means depending on context. The max baseline reports the expected maximum accuracy across uniform random guessers. In Section 5.1 and 5.2, classifiers correspond to prompts with different few-shot demonstrations. In Section 5.3, classifiers are different prompt templates and instruction paraphrases.\n\n***Question 2:** How do results change with input prompt quality?*\n\nSection 5.3 analyzes previously published results that vary the quality of prompt templates and prompt phrasing, finding that the standard and max baselines can disagree there as well. We will expand this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper addresses a critical issue in prompt engineering and in-context learning.\n- Its motivation is simple and straightforward.\n- The authors present extensive empirical results.\n  - However, I could not understand how the empirical results support the claims.", "reasons_to_reject": "- The usefulness of the proposed method is unclear.\n  - It appears to simply replace a random baseline, offering no significant improvement.\n- Additionally, with a small sample size (number of classifiers), t-tests and p-value comparisons may not be appropriate statistical tools.\n- Finally, the computational cost seems significantly higher compared to a simple random baseline. The authors should address these concerns.", "questions_to_authors": "Question 1: What do the classifiers mean exactly?\n- As you mentioned, the model selection in prompt engineering could be different such as backbone models, input prompts, and hyperparameters.\n\nQuestion 2: I would like to see the change of results according to the input prompt quality.\n\nSuggestion\n- I believe this concept could be developed further to create a general evaluation method or metric for prompt engineering.", "extra_scores": {}}, {"review_id": "N7BwynB7wn", "reviewer": "Reviewer_pemZ", "summary": "The primary contribution of this paper is to propose a stronger random baseline when accounting for validation set reuse. The authors argue that the standard random baseline (expected accuracy of guessing labels randomly) is not sufficient when we allow for multiple evaluations on validation set, and when accounting for that (multiple evaluations on validation set and picking the max), the authors derive the closed form expression of the stronger random baseline, and show that many published results outperform the weaker standard random baseline but not the newly proposed baseline. The authors also argue that this baseline predicts held-out performance better.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks for your thoughtful review! We really appreciate that your concerns get to the heart of our work. We think we can address most of them by reframing the intro/conclusion and emphasizing Section 5.2.\n\n***Concern:** Standard practice is to select the best model using the val set and then evaluate that model on an unseen test set. The max random baseline makes no difference when train/val/test splits are properly used.* \n\nWe agree that train/val/test splits are necessary and will state so in the intro. The max baseline should play exactly the same role as the standard baseline does now, where no one would bother evaluating on test if val is below a random baseline. Our work points out the fact that people are using the wrong random baseline here.\n\nSection 5.2 gives evidence that the max baseline on validation can—perhaps surprisingly—provide information on test accuracy. Comparing validation accuracy to the max random baseline is a better indicator than the standard baseline of whether the best classifier will outperform random guessing on an unseen test set. The test set should be used only once, and not wasted on classifiers that don't actually beat random guessing. We will reframe the intro/conclusion to a) emphasize Section 5.2’s findings and b) state that a test set should always be used for a good candidate model and that our goal is to avoid cases where you shouldn't even bother looking at a test set. \n\n***Concern:** The max baseline is trying to correct a flawed practice that would be avoided by train/val/test splits.* \n\nWe will state in the intro that the baseline is just one possible intervention on the widespread flawed practice of not using test sets properly. We will frame the intro and the results in Section 5.1 with the caveats you mention. We also hope that our work can trigger a community discussion about test set reuse across studies. Even when each researcher properly uses a test set only once, benchmark test sets are reused across studies to measure progress. When multiple models across different studies are compared on the same test set rather than redrawing fresh examples from the task distribution, the max baseline applies even to the test set.\n\nPlease let us know if any of this addresses the core of your concerns. We are happy to discuss more!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The new maximum random baseline is theoretically motivated and is intuitive to understand: if we allow multiple draws from the binomial distribution which the random classifier is doing, the expected value of the maximum accuracy will be higher than the accuracy if we only allow one draw. The baseline is built on this principle and I agree with the authors that this is also straightforward to implement and can be used as a drop-in replacement for the simple random baseline.\n- The revelation that many published results do not, in fact, outperform this simple baseline is a potentially impactful finding.", "reasons_to_reject": "One of my biggest concerns is as follows: the standard machine learning practice is *not* to rely on validation accuracy as the result to report. To give an example, if we train an image classifier, the standard practice is to evaluate the validation set at the end of each epoch, and then choosing the argmax for testing *once*. The test accuracy is typically what we care about, and we are only allowed to evaluate on it exactly once -- putting into the context of this paper, this means we are only allow to draw one realization of the binomial distribution (Eq (1)) once rather than $t$ times at *test-time*, which will reduce the proposed baseline back to the standard random baseline. Thus, even though on validation set, which is designed to be evaluated multiple times, the proposed baseline will be much stronger, it will make no difference at test time if standard protocol is followed. The authors seem to suggest that many reported papers do not properly use this elementary train-validation-test split and either report the validation accuracy only or evaluate on test sets multiple times -- while it is certainly not the authors' fault that basic data splitting protocol is not adhered to, I think that providing a stronger validation baseline seems to be the wrong question to answer when the real solution, which is not very difficult to implement and is virtually in any machine learning 101 course, is to use standard data splitting and to report test performance and allow for a single evaluation on test set. \n\nAs such, I felt that the contribution of this paper is somewhat conditional on a flawed practice of not properly using test set being widespread, and to my opinion using the test set properly as told in the textbook is a much better solution (even the authors have acknowledged so in Sec 6 *\"While it is best practice to report test performance on a truly held-out set of examples\"*). While I think it's unfair to harshly penalize the authors for violations of such basic rules by the others, I think it's necessary for this paper to strongly emphasize this caveat.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes an alternative random guessing baseline that sets a stronger bar for meaningful in-context learning (ICL). This new baseline prompts rethinking of the success of current ICL methods, as they do not outperform this stronger baseline on more than 20% of the datasets. Moreover, when considering accuracy on a truly held-out set, this proposed random baseline still outperforms the standard random baseline.

Strengths
* The problem of robust evaluation is crucial, and the paper offers fresh insights into this topic (Reviewer fFwf, WAy5, nvye, pemZ).
* The proposed baseline is very simple and is theoretically grounded (Reviewer fFwf, nvye, pemZ).
* The paper is well-written, with the theoretical justification and practice values (Reviewer WAy5).

Weaknesses
* The main concern is that the standard ML practice does not rely on validation accuracy, so the paper is arguable based on a flawed practice (Reviewer pemZ). Section 5.2 provides evidence that the method generalizes to the held-out test set, but the comparison is made against a basic random guessing baseline rather than ICL methods. It remains unclear how the proposed random baseline would perform compared to current ICL methods if the setting in Section 5.2 were the default.

Diverging opinions
* Experiments are extensive in some dimensions (Reviewer fFwf, nvye) but are not in others (WAy5). The paper includes many datasets and many models which is a plus, but is missing prompt variations and scoring strategies. In addition, the conclusion of the paper mainly holds with 1-shot and 2-shot but less with 4-shot, and less likely with more than 4 (which is a typical ICL setting).
Decision: Accept