Evaluating the Evaluators: Are Validation Methods for Few-Shot Learning Fit for Purpose?
Authors: Luísa Shimabucoro
Ruchika Chavhan
Timothy Hospedales
Henry Gouk
Venue: Accepted by TMLR ()

Abstract: Numerous benchmarks for Few-Shot Learning have been proposed in the last decade. However all of these benchmarks focus on performance averaged over many tasks, and the question of how to reliably evaluate and tune models trained for individual few-shot tasks has not been addressed. This paper presents the first investigation into task-level validation---a fundamental step when deploying a model. We measure the accuracy of performance estimators in the few-shot setting, consider strategies for model selection, and examine the reasons for the failure of evaluators usually thought of as being robust. We conclude that cross-validation with a low number of folds is the best choice for directly estimating the performance of a model, whereas using bootstrapping or cross validation with a large number of folds is better for model selection purposes. Overall, we find that with current methods, benchmarks, and validation strategies, one can not get a reliable picture of how effectively methods perform on individual tasks. However, we find that existing methods already provide enough information to enable selection of few-shot learners on a task-level basis.

Published in Transactions on Machine Learning Research (11/2024)
Evaluating the Evaluators: Are Validation Methods for Few- Shot Learning Fit for Purpose?
Luísa Shimabucoro University of São Pavlo
luisabsh Gusp. br
Ruchika Chavhan
r.chahan Dsmms ed. acd.ac....
School of Informatics, University of Edinburgh
Timothy Hospedales
whooppdales@ed..coom uk
School of Informatics, University of Edinburgh Samsung Al Center, Cambridge
Henry Gouk
henry.gouk@@ed. uk.uk
School of Informatics, University of Edinburgh
Reviewed on OpenReview: https: operreview. net/ forum? id- 4KKY2mDEND
Abstract
Numerous benchmarks for Few-Shot Learning have been proposed in the last decade. How- ever all of these benchmarks focus on performance averaged over many tasks, and the ques- tion of how to reliably evaluate and tune models trained for individual few-shot tasks has not been addressed. This paper presents the first investigation into task-level validation fundamental step when deploying model. We measure the accuracy of performance es- timators in the few-shot sctting, consider strategies for model selection, and examine the reasons for the failure of evaluators usually thought of being robust. We conclude that vross-validation with a low number of folds is the best choice for directly estimating the per- formance of a model, whereas using bootstrapping or cross validation with a large number of folds is better for model selection purposes. Overall, we find that with current mothods, benchmarks, and validation strategies, one can not get reliable picture how effectively methods perform on individual tasks. However, we find that existing methods already pro- vide enough information to enable selection of few-shot learners lask-level basis.
1 Introduction
Deep learning excels many tasks where large-scale datasets available for training (LeCun et al. 2015 However, many standard deep learning techniques struggle to construct high accuracy models when only a very small number of training examples are available. This is a serious impediment to the broader uptake of machine learning in domains where web-scale data are not available. In many domains, such as medicine and security, is common to suffer from data scarcity issues due to multitude of resource constraints and the rarity of the events being modelled. The Few Shot-Learning (FSL) paradigm, which focuses on enabling models to generalise well with little data through the of transferred prior knowledge, has gained relevance in attempt to overcome these challenges. A significant amount of attention has been given to FSL and related meta-learning rescarch in the last decade (Wang et al. 2020 Hospedales et al. 2021 1 with large number of methods and benchmarks proposed in application domains ranging from visual recognition systems for robots to identifying therapeutic properties of molecules (Xie et al. 2018 Stanley et al. 2021
Even though many learning algorithms have been developed this area, and great efforts have been dirccted towards improving model performance in FSL scenarios (Finn et al. 2017 Snell al. 2017 Hospedales
et al. 2021 the best. practices for how to evaluate models and design benchmarks for this paradigm remain

---

Published in Transactions on Machine Learning Research (11/2024)
Hold Out
CV S-Fold
CV 1,00
Bootstrapping
100
100
140
80-
80-
10-
10-
Oradl accuracy
60-
50
60-
-0-
40-
40-
20-
20-
20-
20-
0-
75
Estimatedd accurny
Estimated acourncy
Estimated accumnc
Estimatedd aeturacy
Figure 1: Scatter plots of model accuracy on query set versus accuracy estimated using only the support set. The ideal estimator would have the points almost co-linear and lying approximately on the diagonal line (green). All estimators have very high bias (points are off diagonal) and variance (no co-linearity), indicating they provide poor estimates of performance.
relatively unexplored. In typical academic benchmark setups, performance estimation often relies on the existence of test ("query") sets that are several times larger than training ("support") sets desideratum that is clearly not satisfied in realistic FSL scenarios. Moreover, performance is averaged over a large number of learning problems, leading to accuracy estimates that cannot be used as validation score for a single problem ("episode") of interest. While these experimental design decisions make sense when assessing the general efficacy of a learning algorithm, they are not helpful when considering how one might validate or select model for a real FSL problem with a single few-shot support set.. Even the case where a good estimate of model performance is not required, little attention has been given robust methods for selecting the best model from some prospective pool of models when little data is available. The shortage of model selection procedures for FSL settings means that one cannot reliably perform hyperparameter optimisation or select which FSL algorithm to use.
We hypothesise that exisling model evaluation procedures are not effective in the few-shot learning regime- much like how conventional learning algorithms are not well-suited for direct application to few-shot learning, We experimentally investigate this hypothesis by analysing the behaviour of commonly used model evaluation processes. In addition to studying the accuracy of the performance estimates, we also consider how well these potentially noisy estimates can be utilised in model selection pipelines where one cares only about the relative performance of different models. We analyse failure modes of these methods and show that, should evaluation methods be improved, one could see both a risk reduction for FSL deployments and better models, due to the ability to do model selection reliably.
We answer several concrete questions about the state of methods currently available for evaluating models trained in FSL setting"]
Q1 How accurately can we estimate the performance of task-level models trained in the FSL regime! There are no combinations of learning algorithms and evaluators that are able to produce reliable performance est imates, but we find that 5-fold cross- validation is the best of the bad options.
Q2 Are rankings produced by eristing evaluators well-correlated with the true performance rankings of models, thus allowing for effective model selection? Current model evaluation procedures do not provide reliable rankings at the per-episode level, but methods based on resampling with a large number of iterations are most reliable.
Q3 By how much could performance in FSL be improved by incorporating accurate model selection proce- dures? Our results show that there is still a lot of room for improvement in the case of model selection, as evidenced by the large gap between performance obtained via current model selection methods and perfor- mance from using an oracle evaluator to perform model selection.
The code available at btterr///htthhh//t/tttatttttttta

---

Published in Transactions on Machine Learning Research (11/2024)
2
Related Work
Current practices for evaluating FSL arose from the work on developing new learning algorithms (Lake al. 2015 Ravi & Larochelle 2017 Ren et al. 2018 Triantafillou et al. 2020 Ulla el. al. 2022), and the aim has consistently been to determine the general efficacy of new learning algorithms. The experimental setup used by these works employ a large number of downstream FSL tasks (drawn from a so-called meta-test set), each of which has a test set several times larger than the associated training set. The endpoint measured and compared when making conclusions is the average accuracy across the episodes in the meta-test set. While such an experimental setup is sensible when assessing the "average-case" performance of an FSL method, in this work we address the more classic model validation and selection problem encountered when deploying machine learning models, which is currently neglected in existing FSL benchmarks. Lee, Given a single specific few-shot learning task defined by small teeining/suupors set, how can we estimate the performance on the wnseen lesl/query sel? This is crucial in order to perform model selection, and to validale whether predictions of the few-shot learner are safe to act upon or not. Recent work from application domains that require few-shot learning indicate that the lack of reliable evaluation procedures is a major blocker for deployment (Varoquaux & Cheplygina 2022
Some prior work on studying the failure modes of FSL has identified that the variance of accuracy across episodes is typically very high (Dhillon et al., 2020 Agarwal et al., 2021 [asu et al.] 2023). Work in this area has focused on identifying "hard" episodes by constructing support sets that lead to pathological performance on a given query set. The focus of this existing work has been on discovering trends in the types of downstream tasks where FSLL methods do not work, whereas our motivation is to explore: i) what processes can be followed to delermine whether a particular model will perform as required; and ii) how can we reliably select the best model from some set of potential models. These specific questions are currently unaddressod by existing literature.
3 Few-Shot Model Validation and Selection
Modern few-shot learning research considers a two-level data generating process. The top level corresponds to a distribution over FSL tasks, and the bottom level represents the data generation process for an indi- vidual task. More precisely, consider a top level environment distribution, E, from which one can sample distributions, D,, associated with task indexed by i. One can then sample data, from D, generate examples for task i. We first outline the existing assumptions about data availability and evaluation proce- dures used in the literalure, which we refer to as Aggregated Evaluation (AE). We then discuss how this can be augmented to better match real-world FSL scenarios, and then outline the Tesk=Level Evaluation (TLE) and Task-Level Model Selection (TLMS) protocols that we experimentally investigate.
A Aggregated Evaluation (AE) In this setting, one samples collection of distributions, E)i= and then for each of these distributions one constructs a meta-test episode consisting of a support set, S, = { ~ D= and a query set, Q, {(z), D= It is typical for the size of the query set, m, to be several times larger than the size of the support set, We observe that in a true FSL setting this is an unrealistic assumption. Standard practice when evaluating FSL methods is to sample t episodes. For each episode, a model is trained using the support set and then a performance estimate is computed using the query set, Å'D = 1 En, EQ. 11y = hs, (I,)), where hs, denotes a model trained on S, and 1(-) is the indicator function. Finally, the performance of the learning algorithm is summarised by aggregating over all the meta-test episodes, βog Ei-ip. AD.
(D; ~
E,
Using de to evaluate the performance of a FSL algorithm makes sense if the downstream application involves large number of different FSL problems, and the success of the overall system is dependent being accurate on average. Examples of such applications include recommender systems and personalised content tagging, where each episode corresponds lo a single user session. The success of such personalisation systems depends on being accurate for most user sessions, but a poor experience for a small number of sessions (i.e., episodes) is acceptable. In contrast, applications of FSL in modicine or sccurity are not well-suited to the aggregated evaluation provided by ugg In these settings, each episode might correspond to recognising the presence of a specific pathology or security threat, and poor performance on such episodes would translate to systematic
3

---

Published in Transactions on Machine Learning Research (11/2024)
misdiagnosis and critical security vulnerabilities. Such outcomes would be considered catastrophic system failure.
Pask-Level Evaluation (TLE) We investigate Task-Level Evaluation, which serves a purpose distinct from AE While AE is typically undertaken to assess the general efficacy of FSL algorithm, the purpose of TLE determine the performance of a model trained for a particular episode. In many real-world applications of FSL, one must have accurate estimates of the performance of models trained for each episode, Moreover, in realistic situations there is no labelled query set for evaluating model, so both the model fitting and model evaluation must be done with the support set. Just as FSL requires specialised learning algorithms, we argue that TLE of FSL requires specialised performance estimators.
We investigate three common approaches to evaluating machine learning models, which we summarise below.
Hold-out (Witten et al. 2016 The Hold-out evaluation method, which commonly used in the deep learning literature, requires a portion of the data be set aside so it can be used test the model. In particular, one sets aside n samples from the support set, trains model on the m n remaining samples, and uses the held-out data points to estimate performance.
cross-validation (CV) [Stone, 1974) The cross-validation method splits the support set in & folds and repeats model training k time. For each repetition, a different fold is used for lesting the performance of the model, and the other k --1 folds are used to train the model. The mean performance across all iterations is used as a performance estimate for a model trained on all the data. When set to the number of data points this method referred to as vrossvvaliattion leave-one-out (CV-LOO).
Bootstrapping Efron 1979 Bootstrap sampling is technique which consists of randomly drawing sample data with replacement from given set, resulting in and an out-of-bag set composed of the data points that where not chosen during the sampling process. This method is used in our bootstrapping evaluator to create a new support sample from the support set and a new query set composed by the out-of-bag data points. The process is repeated and the result of each repetition averaged give a final performance estimate.
sample with the same size as the original set,
Estimators typically evaluated in terms of bias and variance, defined respectively as
Bî E'a] - Elu.. Var(ū) = E [( -
where á is a performance estimate and is the true performance. While the variance is mathematically convenient when analysing properties of estimators, we use the Mean Aboslute Frror (MAF) instead, as it is more directly interpretable,
MAB(A)= = E [1p- - Ejull
In practice, the Bias and MAE are estimated using episodes from the meta-test set.
Task-Level Model Selection One of the common use-cases for performance estimators is in the model selection process, where they are used to rank the relative performance of set of candidate models, either part of the hyperparameter optimisation process. We observe that accurate estimates of the performance of models may not be needed for model selection, as long as similar types of errors as made for all the prospective models. For example, a performance estimator is systematically biased, but does not exhibit much variance, then the resulting rankings should be reliable, even though the estimate of the performance is not.
for deployment
4 Experiments
We conduct experiments to determine which evaluation techniques are most reliable for: i) estimating the performance a model; and ii) selecting the best performing model from some pool of prospective models. We also investigate the underlying reasons why evaluators traditionally thought of as being quite robust can fail in the FSL setting, and the extent to which better evaluators could enable improved performance via more reliable model selection.
4

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: 5-way/5-shot few-shot learning accuracy considering a 95% confidence interval: Actual (Oracle)
as prodicted by various estimators. Most methods under-cstimate accuracy compared to Oracle.
Model
Oracle
Hold-Out
5-fold CV
LOO-CV
Bootstrapping
CIFAR-FS
Baseline
71.17 = 0.727
64.50 2.235
69.28 + 1.040
62.47 1.122
54.09 + 0.895
+aseline++
71.38 + 0.755
66.28 2.262
70.41 + 1.017
63.80
1.145
56.91 + 0.969
ProtoNet
71.65 = 0.755
70.30 1.717
70.09 = 1.006
65.43 1.089
57.31 + 0.941
MAML
69.51 + 0.762
69.93 1.819
71.41 1.119
16.79 0.667
42.39 + 0.666
R2D2
72.06 = 0.751
71.37 1.609
70.22 1.021
62.24 1.172
55.12 0.905
MetaAlbum minilmageNet
Baseline
= 5 0.646
57.17 1.765
57.15 1.040
44.80 1.094
38.34 + 0.723
Baselinc+
64.07 0.632
63.30 1.683
63.31 = 0.978
21.28 0.957
23.97 + 0.452
ProtoNet
6
63.73 1.699
63.98 + 1.000
51.69 1.177
37.67 + 0.740
MAML
59.76 + 0.717
48.11 2.286
59.81 + 1.135
17.39 + 0.684
16.28 + 0.350
R2D2
63.58 = 0.658
62.67 = 1.682
62.33 = 1.001
48.58 + 1.126
40.54 + 0.734
Baseline
59.36 + 1.688
54.39 +: 2.658
57.43 + 1.792
51.58 1.919
45.68 1.695
+aseline++
57.99 = 1.724
1.837 56.78 23.11
48.84 2.005
43.85 : 1.701
ProtoNet
47.74 + 1.689
4
44.07 + 1.849
39.44 + 1.665
MAML
47.10 # 1.736
1 2.603
15.24 0.819
28.68 + 1.088
R2D2
52.91 + 1.670
51.30 + 2.242
51.39 + 1.779
44.70 + 1.890
39.39 + 1.599
Table 2: Per episode accuracy MAE between accuracy of oracle and accuracy of each estimalor (5-way 5-shot setting)
Model
Hold-Out
5-fold Cv
LOO-CV
Bootstrapping
CIFAR-FS
Baseline
24.06
12.38
14.41
18.65
Basclinc++
23.75
12.49
14.59
17.27
ProtoNet
16.75
8.67
10.16
15.00
MAML
19.74
14.00
52.72
27.25
R2D2
7.27
4.70
9.94
16.94
Baseline
19.14
13.01
17.90
21.65
MetaAlbum minilmageNee
Baseline++
18.18
11.51
42.82
40.10
ProtoNet
16.55
8.95
16.14
26.46
MAML
26.14
13.61
42.38
43.48
R222
16.83
8.96
16.36
23.06
Bascline
21.50
9.39
11.32
14.50
Baseline++
20.29
9.32
12.22
14.91
ProtoNet
15.78
9.06
9.77
10.58
MAML
27.38
9.75
35.50
17.67
R2D2
16.73
9.50
11.66
14.32
We conduct experiments datasets and learners popular in both the meta-learning (Hospedales et al., 2021) and vision & language transfer learning (Radford et al. 2021 family of approaches to few-shot learning. Specifically, we use:
minilmageNet This is a scaled down version of ImageNet consisting of 100 classes, each with 600 total examples and image scaled down to 84 x 84 pixels. We make use of the standard meta-train/ wtt---aaiinalliiiiooo/ooooo vali/aaalloottess split proposed by (Ravi & Larochelle 2017).
CIFAR-FS Uses the CIFAR-100 dataset KKrizhevsky 2009). with 64, 16 and 20 classes for peta-training, meta-validation, and meta-testing, respectively.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Meta-Album This is recently proposed (Ullah et al. 2022 dataset that has focus providing data from a broad range of application domains, accomplished by pooling data from 30 different existing datasets. We use the Mini version of the dataset, which provides 10 different domains (three source datasets per domain) and 40 examples per class. Three splits are present, each containing one source dataset from each domain. Each split used once each as meta-train, meta-validation, and meta-lest, and the final evaluation results are obtained by computing the mean of these three runs.
CLIP Few-Shot For experiments with CLIP-like few-shot learners, we also use three datasets (EuroSat, Flowers, and Food101) selected from the CLIP few-shot suite (Radford et al. 2021). In this case we follow the CLIP few-shot learning protocol, drawing from 1-16 shots from all categories in each datasct, and evaluating on the full test sets.
Implementation Details We primarily study the classic FSL methods from the meta-learning community ProtoNet (Snell et al. 2017). MAML. (Finn et al.] 2017). R2D2 (Bertinctto et al. 2019). and Basclino(++) Chen et al. 2019), using the implementations provided by the LibFewShot package 2 We use the mela- training hyperparameters suggested in the documentation of this implementation, which were tuned using the meta-validation set of minilmageNNet. We use the standard Couv4 architecture for minilmageNet and CIFAR-FS, ResNet18 for meta-album. From the vision & language community, we also study CLIP (Radford using the VTT-B32 vision encoder architecture.
et al.
2021
4.1 Performance Estimation
Experimental Setup For each meta-test episode, all evaluators are given only the support set and required to produce an estimate of the generalisation error for models trained using each of the FSL algorithms we consider. We additionally construct a query set that is several times larger than the support set, and is used as an oracle estimate for the performance of each model. For each evaluator we construct a sample of performance estimates, where each data point corresponds to a meta-test episode. We report the mean of the accuracy across episodes, and also the mean absolute difference in accuracy between each estimator and the oracle accuracy.
Results For the 5-way 5-shot setting, Table ! compares FSL accuracy as observed by the oracle with the four standard estimators considered. We can see that the estimators lend to under-estimate the actual accuracy of the oracle, to varying degrees. This is due to the performance estimators having to sacrifice some of the training data, order to estimate performance, mcaning the model fit poorer than model that is trained on all available data. This issue also exists in the many-shot setting, but the effect is usually negligible in that case because of the diminishing returns of training on more data.
Table D presents the mean absolute difference between cach estimator and the oracle. We can see that 5-fold CV tends to have the lowest difference, i.c., it provides the best estimate of the oracle accuracy. We note, however, that it is still likely to be too high to be practically useful in many applications. To igate this on a more fine-gres ann-ngr level, we visualise the distribution of absolute differences in Figure 2 From this figure we can see that the distribution of errors is quite dispersed for all combinations of learning algorithm and evaluator: for all polential choices there is at least a 50% chance the validalion accuracy would be wrong by more than 10,, in absolute terms. This means that there is no pipeline that one can use to reliably train and validate models to ensure they are safe to deploy in situations where one must take into account the risks involved in deploying their models.
We next analyse how the error between oracle and estimator varies with number of shots. From the results in Figure 3 we can see that the error of the estimator decreases with shot number as expected, but tends to be substantial in the < 20 shot regime typically considered by FSL. Together, Table 2 Figure 2 and Figure 3 illustrate our point that there is no good existing solution to task-level FSL performance estimation.
FSL with Vision & Language Foundation Models In Figure 5 we plot the number of shots versus the MAE and accuracy, measured on logistic regression models trained on features extracted from CLIP, The issues seen in the meta-learning setting are less pronounced here particularly in the Food101 and Flower102 datasots. Mcasuring accuracy can be thought of as sampling from a Bernoulli distribution with the aim of
https://ggthub.com//ll-gg
6

---

Published in Transactions on Machine Learning Research (11/2024)
190
109
100
MAAD Prequetcy
48-
30
Held Ost
CV
CV1O0
Boustrapiing
Hold Out
CV S-Rodd
CVLOD
Bentingppigg
Hold Out
CV STaa
CV LOO
Borhiragping
100
100
Proqueecy
80-
48-
28-
1546 Dut
CVS-add
CY LOO
Dadttripptgg
1846 Ost
CV 5-i--
CV Loo
Bastorapplig
Figure Box plots showing the distribution of absolute differences between the estimated accuracy and oracle accuracy on the meta-test episodes of minilmagenet. Distributions should be ideally concentrated close to possible, but This indicates that many of the performance estimates unreliable.
that substantial proportion of the mass is far away from
Baelia
Beeeltet
Protedle
MAMI
0.5
0.5
0.5
Hold Ou:
0.4
Sesternoptigg
MAX
0.3
MAX
0.3
0.3
Shor. Narcher
Stut Namrber
Slor Nurcher
Stnt Nertor
Figure 3: Dependence of cstimator-oracle error shot number. Estimator error is substantial in the few- shot regime. The error bars indicate the standard error.
measuring the "success" parameter, p. An accurate model will have close to one. The variance of the Bernoulli distribution is p11 (el p), and the variance of the Bernoulli distribution directly determines the value of the MAE, Therefore, estimating performance of models with high accuracy will generally be more reliable. Paradoxically, we mcasure the performance of models to determine if they have high accuracy.
4.2 Performance Estimation on Imbalanced Problems
Standard FSL evaluation protocols assume perfectly balanced class labels, but this is rarely true in practice. We conduct additional experiments on imbalanced data using the linear imbalance protocol of Ochal et al.) (2023), where the support set for each episode contains extreme class imbalance sometimes addressed in the more general machine learning literature; our focus is weakening the assumption of perfect balance. In the linear imbalance setting, each episode is composed of examples from five different classes, but the counts of examples for classes through five are (4, 4, 5, 6, 6). Bascline (i.c., logistic regressions (Chen et al., 2019)) models are trained and the MAE between cach performance estimator and the performance measured on the query set, which has the same class distribution as the support set but contains 100 examples. The results are reported in Table 3 with box plots showing the distribution of absolute differences across episodes in Figure 6 The conclusions remain unchanged from those identified in the balanced classes experiments: 5-fold CV still performs the best, but still has an MAE of at least. 10 for half the episodes.
examples. Note that we do note consider the

---

Published in Transactions on Machine Learning Research (11/2024)
0.5
Average Spearman's Coefficient
Hold Out
5-Fold CV
0.4
LOO-CV
Bootstrapping
0.3
0.2
0.1
0
minilmageNet
CIFAR-FS
MetaAlbum
Figure 4: Mean Spearman correlation between the rankings produced by the oracle and the different, per- formance estimators, computed across all the meta-test episodes in each dataset. A correlation coefficient of indicates the same rankings, -1 indicates the opposite rankings, and 0 indicates that the rankings are unrelated.
FarSAAT
Flower 102
Food-101
Hold Out
0.8
Bootstrapping
0.8
0.8
0.6
MAR
MAB
0.4
0.4
0.4
0.2
0.2
0.2
16
2
16
16
Shoo Number EaroSAT
Shot Namber
Shooe Number
Flower102
Food-101
100
100
80
Accusety
Ancurary
60
Accuney
Ocacle Hold Out
20
5-Frld
20
20
Bootstrapping
2
16
2
4
16
2
16
Shot Numobee
Shot Number
Shot Number
Figure 5: Few-Shot CLIP MAE (top) and accuracy (bottom) of different performance estimators averaged over 10 runs. The error bars indicate the standard error.
4.3 Further Analysis of Cross-Validann Validation
We have shown that 5-fold CV the least bad existing estimator for task-level FSL performance estimation. This might seem surprising, as LOO-CV is generally considered preferred from statistical bias and variance point of view (Witten et. al. 2016). To analyse this in more detail, we plot the error function of folds
8

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Per episode accuracy MAE between accuracy of oracle and accuracy of each estimator for the linear imbalance sctting with 25 support set examples per episode.
Dataset
Hold-Out 5-fold CV
LOO-CV
Bootstrapping
CIFAR-FS
18.14
12.67
14.63
15.99
minilmageNee
16.50
12.40
15.09
19.91
MetaAlbumMini
15.98
9.51
10.47
13.70
minilmageNet
CIFAR-FS
AltuMMmumMini
100
100
80
60
60
60
MAE
8
0-
0-
Hold Out
S-Fold Evaluator
LOO Bootstrapping
Hold Out
5-Fold
CV Bootstrapping
Hold Out
CV S-Fold
LOO
Bootstrapping
Evaluator
Evaluator
Figure 6: Box plots showing the distribution of absolute differences across each episode. Ideally these should be concentrated close to zero, but we observe that there significant probability density far away from zero.
in Figure (left). The results of this show that increasing the number of folds-and therefore reducing the bias introduced by fitting each fold on a smaller training dataset counteracted by another phenomenon that causes LOO-CV to have low quality performance estimates. We find that 5-fold CV presents the ideal trade-off between these two effects.
We hypothesise that the negalive effect that causes CV with a large number folds to become less accurate is related to an induced distribution shift between the training folds and the validation fold. Consider the standard balanced few-shot learning experimental evaluation setting. In the case LOO-CV, the consequence holding out one example for evaluation will mean that the training data contains one class with fewer examples than the her classes. Moreover, the test example will come from this minority class, which will result in LOO-CV evaluating the model in the pathological case where all test examples belong to the minority class encountered during training.
To investigate this hypothesis, we vary the number of ways while keeping the size of the support set fixed. As the number of classes is reduced, the number of training examples per class is increased, and class imbalance is less prevalent. Figure 7 (right) shows the result. Using the number of ways as a proxy for class imbalance in the LOO-CV setting, we see there is an association between class imbalance and MAE of LOO-CV performance estimation.
4.4 Model Selection
An alternative to accurate performance estimates that is acceptable some situations is instead rank set of prospective models and select the one that performs the best according to some performance estimator. In this setting, the estimator can exhibit some types of biases without having an impact on the final rankings. For example, the performance underestimation bias of the estimators may not have significant impact the final rankings, as long as all models are similarly affected by this bias. However, the ranking procedure is still sensitive to variance of the estimators.
To this end, we investigate how well the rankings produced by the different estimators are correlated (in the Spearman Rank corrclation sense) with the rankings produced by the oracle estimator. Rankings are produced for each episode in the meta-test sets, and the mean Spearman rank correlation (w.r.t. the oracle) is computed for each performance estimator.
9

---

Published in Transactions on Machine Learning Research (11/2024)
0.5
0.5
Baseline
Baseline Baseline+ + ProtoNet R2D2
Bascline
0.4
0.4
ProtoNet
R211
0.3
Baseline+++CY
0.3
MAE
MAML
MAE
0.2
0.2
0.1
0.1
0
23
10
25
5
Folds
Ways
Figure 7: Further analysis of the estimation accuracy of cross validation performed on the minilmageNet dataset. Left: MAE wetween k-fold CV estimates and the oracle estimates, as a function of k. Right: MAE of LOO-CV, relative to the oracle, as the number of ways (and therefore class imbalance) is increased. The error bars indicate the standard error.
Table 4: Aggregated Accuracy of the different baseline models. BaselineCV indicates that the ridge regular- isation hyperparameter is tuned on a task-level basis using 5-fold cross validation.
Table 5: Aggregated Accuracy of Task-Level Model
Selection using each of the performance estimators.
Model
CIFAR-FS
minilmageNNtt
Meta-Album
Model
CIFAR-FS
miniImageNet
Meta-A Album
Oracle
80.11 0.495
71.40 + 0.464
63.53 + 0.932
Hold-Out
71.65 = 0.749
62.79:0.710
56.30 = 1.048
Baseline
71.17 + 0.727
39.36 + 0.646
59.96 + 1.688
5-Fold
72.38 0.726
63.73 = 0.717
58.58 1.005
BaselineCV
72.89 + 0.737
62.11 + 0.698
58.46 + 1.745
LOO-CV
73.29
58.53 1.014
MAML
69.51 + 0.762
59.76 à: 0.717
47.10 +: 1.736
Bootstrapping
73.41 0.724
64.05 0.737
58.62 1.011
MAMLCV
40.24 1: 0.840
55.06 1: 0.849
52.77 +: 1.308
The results of this experiment are shown Figure It evident from these plots that no evaluator produces rankings that are highly correlated with the oracle rankings, with a maximum achieved correlation of around only 0.36 for 5-fold CV the meta-album benchmark, and resampling methods with large number of iterations generally being the most reliable approach. However, these results indicate that are unlikely to be able to do elfective model selection using currently available evaluation procedures.
Results
4.5 How can we improve FSL in practice?
We further investigate how existing model selection methods applied at the task-level can be used to im- prove performance. Experiments are conducted on both types of model selection seen in machine learning: hyperparameter optimisation, and algorithm selection.
Hyperparameter Optimisation As a compromise between computational cost and obtaining the best model rank correlation, we propose a new baseline for FSL that 5-fold cross validation to tune the hyperparameters of different learners. We consider the ridge regularisation parameter of Baseline (i.e., logistic regression) model (Chen et al. 2019 and the number of finc-tuning iterations in MAML. The results for this experiment are shown in Table for the meta-learning setting, and Figure X for the CLIP setting, From these results we can see that using 5-fold CV can provide a noticeable improvement over the aggregated accuracy the standard Baseline, but tuning the number of iterations for MAML is unreliable.
Algorithm Selection By using each estimator rank the performance each FSL algorithm on per- episode basis, try to select which algorithm should be used in each episode to maximise performance.
10

---

Published in Transactions on Machine Learning Research (11/2024)
EuroSAT
Flowers
Food101
100
90
100
x x
90
80
x
90
X
70
8D
70
Accuracy
Accuracy
Accuracy
60
70
60
50-
N-102 (all)
30
N-5
N=5
N=5
N=101 (all)
N=10 (all)
N=10
N=10
4.5
8
10
16
20
45
10
1266
20
4.5
10
20
Figure 8: We lune the Ridge regression hyper-paraamter task-level basis using 5-fold cross-validation for different combinations of ways and shots. We report Accuracy vs K for Baseline (without tuning marked with 'o') and Baseline + HPO (after tuning marked with 'x'). is observed that while hyper-paramcter tuning for each fold benefits performance, this effect more pronounced for low values of K.
E.g., MAML might perform best in one episode and Prototypical Networks in another. Table shows aggregated accuracy results when using each estimator to do this type of model selection. These results demonstrate that LOO-CV and Bootstrapping are the best approaches to use for model selection, which is consistent with the rank correlation results from Figure 4
5 Discussion
We investigate the effectivenoss of methods for evaluating and selecting models task-level basis in few- shot learning. The results in Section 4 provide several concrete takeaways that can be used to inform best practices and show the limitations of the state-of-the-art. Revisiting the questions asked in Section we provide three main insights.
Performance Estimation It demonstrated that existing evaluation approaches are not able provide accurate estimates of model performance in the few-shot learning setting. For most combinations learning algorithm and evaluation method, the performance estimates obtained for individual episodes are more than 10% away from the true accuracy, in absolute terms, for the majority of the episodes generate. Even the best combination is only below this threshold approximately 50% of the time.
Recommendations for Practitioners The fact that performance estimates for few-shot learning are unreliable means that there currently no way to validate specific few-shot learning tasks. Practitioners should not deploy any FSL. system that requires being able to estimate in advance whether a particular attempt at lcarning has boen successful there is unacceptable risk of performance being substantially different than estimated by any validation procedure. However, this risk can be mitigated by noting that some performance estimators such LOO-CV and Bootstrapping consistently underestimate the accuracy. Such pessimistic estimators may be acceptable in some scenarios.
Model Selection We investigate how well these inaccurate performance estimates can be used to rank models. Accurate rankings allow practitioners to select the best possible model, even the exact performance is not known. would also enable better degree of hyperparameter optimisation than currently employed in many FSL settings Our findings in this area show that there is only a mild correlation between the best model ranking approaches and the oracle rankings provided by larger held-out datasets that not typically
11

---

Published in Transactions on Machine Learning Research (11/2024)
available in the few-shot setting. Despite the only mild correlation, do achieve small degree of success in using these rankings for hyperparameter tuning.
What makes a good performance estimator? We argue that the stability of the underlying learning algorithm is an important factor when estimating performance. Many of the existing estimators rely on constructing model using different subsets of the available training data and then aggregating several different performance estimates. Algorithmic stability has been shown reduce the dependence on data when fitting models (Bousquet & Elisseeff 2002), and more stable algorithms have also been shown provide more reliable performance estimates when used in conjunction with cross validation (Aghbalou et al. 2022 This line of reasoning is congruent with our empirical results, where we see that the most stable algorithm (Protolypical Networks) consistently has low MAE (relative to other approaches) when coupled with CV estimator.
Recommendations for Future Work Just as few-shot learning requires specialised algorithms that take advantage of learned prior knowledge, we propose that future work should design specialised evaluation procedures. This could take the form of Bayesian estimation procedures that are informed by performance the meta-training and meta-validation episodes, they could leverage other side-information to reduce variance in estimates.
Acknowledgements
This project was supported by the Royal Academy of Engineering under the Research Fellowship programme as well as the São Paulo Research Foundation (FAPESP).
References
Mayank Agarwal, Mikhail Yurochkin, and Yuekai Sun. On sensitivity of meta-learning to support data. In Advances in Newral Information Processing Systems, 2021.
Anass Aghbalou, François Portier, and Anne Sabourin. Consistent cross validation with stable learners. arXiv preprint Xix:22200. 10211, 2022.
Samyadeep Basu, Megan Stanley, John F. Bronskill, Soheil Feizi, and Daniela Massiceti. Hard-Meta- Dataset++: Towards Understanding Few-Shot Performance on Difficult Tasks. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Luca Bertinetto, João F. Henriques, Philip H.S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers In ICLR, 2019.
Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at fow-shot classification. ICLR, 2019.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In International Conference on Learning Representations, 2020.
Bradley Efron. Bootstrap methods: another look at the jackknife. Springer, 1979.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, (49):5514-)) 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332 -338. 2015.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Yann LeCun, Yoshua Bengio, and Geoffre Hinton. Deep learning. nature, 521((553))4366 444, 2015.
Mateusz Ochal, Massimiliano Patacchiola, Jose Vazquez, Amos Storkey, and Sen Wang. Few-shot learning with class imbalance. IEEE Transactions on Artificial Intelligence, 2023.
Alec Radford et al. Learning transferable visual models from natural language supervision, 2021.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning, In International conference on learning representations, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Tarochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations, 2018.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few shot learning. In NIPS, 2017.
Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Mare Brockschmidt. Fs-mol: A few-shot learning dataset of molecules. In Thirty-fith Conference on Neural Information Processing Systems Dalasets and Benchmarks Track (Round 2), 2021.
Mervyn Stone. Cross-validators choice and assessment of statistical predictions. Journal of the royal statis- tical society: Series B (Methodological), 36(2):111-133 1974.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, fferre-Antoin Manzagol, al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
Thsan Ullah, Dustin Carrión-Ojcda, Sergio Escalera, Isabelle Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. Meta-album: Multi-domain mela-datasel for few-shot image classification. Advonces in Neurol Information Processing Systems, 35:3232 0222222-2277, 2022.
Gaël Varoquaux and Veronika Cheplygina. Machine learning for medical imaging: methodological failures and rocommendations for the future. NPJ digital medicine, 5(1):48, 2022.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Ian Witten, Eibe Frank, Mark Hall, and Christopher Pal. Data Mining: Practical Tools and Techniques. Morgan Kaufman, 2016.
Annic Xie, Avi Singh, Sergey Levine, and Chelsea Finn. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning, pp. 40 52. PMLR, 2018.
13


Reviews:
[{"review_id": "M0o1SN5v0v", "reviewer": "Reviewer_sHVT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The central idea in this paper is as follows. A model trained using meta-learning is supposed to perform well on a new task, i.e., predict accurately for test samples drawn from this new tasks. This is what the authors call “task level: validation. The user usually does not care about the average accuracy of this model on a large number of test tasks. This basic point has been made in many papers on few-shot learning over the years. The present paper therefore seeks to develop model selection techniques to select the appropriate model for a test task.\n\nTo this end, the paper conducts a number of experiments to evaluate different model selection techniques. These experiments seem correctly done. And the summary of these experiments is that none of these techniques are useful for the purposes of model selection in few-shot learning. While the authors mention this conclusion in the abstract and the introduction, the tone of the paper is sometimes dissonant with the numerical data. For example, it is said that “5-fold cross-validation is the best of the bad options”. Well that is moot. They are all bad. Or on Page 8, the data says that “ranking models using evaluation procedures” is not useful to perform model selection.\n\nThere seems to be, however, a fallacy prevalent throughout the narrative. It is a fact that few-shot learning is about adapting a model using a small number of support samples. Regardless of whether these samples are used for adaptation, or for model selection. It is also a mathematical fact that model selection is difficult for small sample sizes. Regardless whether one is talking about few-shot learning, or some other statistical model. Concepts like hold-out, k-fold, leave-one-out and bootstrapping-based validation come with well-established mathematical guarantees in the statistics literature. These guarantees are primarily asymptotic. At finite sample sizes, it is difficult (perhaps impossible) to develop a general theory for which estimator would be better for model selection. Certainly this is even more difficult when the sample sizes are as small as practitioners in few-shot learning want to evaluate with. Any theoretical treatment of model selection would have to rely on a specific assumption about the statistical model and the data distribution. To wit, calculating the first two moments using bootstrapping is implicitly an assumption about having a sample size that is large enough for the central limit theorem to kick in.\n\nIn short, I believe the inadequacy of the sample size for model selection is a mathematical problem. Not an empirical one as investigated in this paper. While the authors are correct in that the prevalent methodology in the field is broken, this paper neither (a) gives an example of a well-established few-shot learning-based system that suffered from this issue (this is different from the analysis of some methods on benchmark datasets which the paper does provide), nor (b) does it provide a concrete problem formulation/proposal solution for the issue. This is the beginning of a good paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies different techniques for model selection and model validation for few shot learning. The paper argues that none of the techniques that were evaluated (leave one out cross-validation, k-fold cross-validation, bootstrapping, or hold out) appear to be faithful predictors of the accuracy on the test task (as evaluated using a large number of query samples...this is called “Oracle” in the paper).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would encourage the authors to dig into the mathematical details of what makes different model selection procedures less/more suited for few-shot learning, e.g., does assuming that the data lies on a low-dimensional manifold bring a specific technique into play. Just to give an example, one could imagine transductive inference-based cross-validation which the authors alluded to right at the end of the paper being more useful in this case if support samples span the principal directions of this manifold.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ap8j75f0Pm", "reviewer": "Reviewer_8ey9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Selecting model on the task level is an interesting setting. Estimating model performance using a few examples can be a fundamental ML problem.\n- In general, the empirical evaluation is thorough.\n- Algorithm selection is an interesting application and the initial results delivered by this paper is interesting.\n\nWeaknesses\n- Given the abundant literature in FSL, the paper could discuss more related works. Since the paper introduces a new setting for evaluating task-level performance, I would like to see more FSL methods being evaluated here.\n- The paper could investigate the “optimal” fold between LOO and 5-CV, and the relation wrt. the number of shots/ number of ways.\n- In general, the paper lacks some insights on why different models and different evaluators differ in their performance. Sweeping the number of folds suggested above could help explain part of the difference. Given that no single evaluator is reliable, and there is room for improvement, the paper does not identify concrete directions for improvement.\n- While data imbalance can be an issue for FSL, the experiment setting studied in the paper is hand wavy. First, the number of examples in the imbalance setting is still balanced with the minimum number to be 4 and maximum to be 6. So the minority class still represents 60% of examples compared to the majority class. Second, the paper thinks that the imbalance is the issue that causes CV with a large number of folds (e.g. LOO) to suffer in performance. I don’t think this argument is well backed up. First, the paper could rule out the degenerate case of having only 1 example per class in the support set. Second, the paper should vary the imbalance ratio and report the decrease of performance. More carefully controlled study would seem necessary to make the claim.\n- Figure 3, 5, and 7 should add confidence bounds.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates a few validation methods for probing the few-shot learning task-level error. Due to a lack of support examples, it is often challenging to estimate the query set loss. The paper studies several standard few-shot learning methods with existing methods (e.g. 5-fold CV, LOO, etc.) to estimate the task-level performance. The paper concludes that there is no reliable way to estimate the task-level performance, and there is still a lot of room for improvement. Using task-level performance estimator to choose the best FSL model leads to some improvement in the average FSL performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness section", "broader_impact_concerns": "I do not have broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h3W3Y6LrQS", "reviewer": "Reviewer_pryo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The question about how accurate few-shot learning methods are if validation is done on the query set is a well-justified and interesting question. \n- Well-motivated experimental setup that had many experiments I was hoping to see when reading the paper - comparing various ways to estimate accuracy, correlation with the oracle accuracy, experiments on imbalance classes, and comparing the ranking of models \n\nWeakness\n- The biggest weakness is the question this paper seeks to answer is no longer practical.  The methods/evaluation setup are outdated with foundation models. Couldn’t these tasks just be done zero-shot with multimodal LLMs like GPT-4o? Doing few-shot learning also would not involve the FSL methods tested, but just using in-context learning. Given that this is mainly an empirical paper, it should focus on what is practical.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks at how practical few-shot learning methods are in practice by comparing how well the validation performance (when estimated on the query points) correlates with the oracle performance. This is important because in the few-shot learning setup, we cannot assume we have a large validation set for getting the performance. They look at whether the estimated performance on the query set (using different estimates) correlates with the oracle accuracy and find it doesn’t. There are also additional experiments looking at class imbalance and whether there is correlation between rank of models according to an estimated accuracy and according to the oracle accuracy. Overall, the answer is validation performance cannot be accurately estimated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Critical\n- This does involve a lot of work, but changing the setting to evaluate the few-shot learning performance of LLMs using in-context learning, which has become the dominant paradigm. \n\nNot critical\n- Part of the motivation in the intro is that performance is averaged over a large number of problems instead of considering the worst case (i.e. being robust). As far as I can tell, this scenario is not addressed in the experiments. \n- How many models are used to compare the ranking in fig 4. If it’s not a lot of models, can the spearman correlation be noisy?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciate the contributions of the paper and recommend acceptance. 

This paper provides significant new findings and recommendations for model evaluation and model selection in the few shot setting. This makes the paper a good match with TMLR.
Decision: Accept (poster)