Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning
Authors: Raphael Lafargue
Luke A Smith
Franck VERMET
Matthias Löwe
Ian Reid
Jack Valmadre
Vincent Gripon
Venue: Accepted by TMLR ()

Abstract: The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e. allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again

Published in Transactions on Machine Learning Research (09/2024)
Oops, Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning
Raphael Lafargue
IMT Atlontique, Lab-STICC, UMR CNRS 6285, F-29238, France Australian Institute for Machine Learning, University of Adelaide, Australia CNRS, IRL CROSSING, Adelaide, Australia
Luke Smith
buke.a. ammtth @astlliddddddddeee.. edu
Australian Institute for Machine Learning, University of Adelaide, Australia
Franck Vermet
Vranch.vernee@@unii--beee.
LBMA, CNRS, UMR 6205, Univ Brest, Brest, France
Matthias Löwe
malthhioeeee@@googeeeaa..ccmm
University of Münster, Germany
Ian Reid
redde www.iiddedeedd....
Australian Institute for Machine Learning, University of Adelaide, Australia MBZUAI, Abu Dhabi, United Arah Emirates CNRS, IRL CROSSING, Adelaide, Australia
Vincent Gripon
IMT Atlontique, Lab-STICC, UMR CNRS 6285, F-29238, France
Jack Valmadre
Australian Institude for Machine Learning, University of Adelaide, Australia CNRS, IRL CROSSING, Adelaide, Australia
Reviewed on OpenReview: https: // openreview. net/ met/ forum? id= Jazkk 9yro
Abstract
The predominant method for computing confidence intervals (CT) in few-shot learning (FSL.) is based sampling the tasks with replacement, i.c. allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis betwoen CIs computed with and without replacement. These re- veal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FST. comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, strategically sampling tasks of specilic size. We also introduce new optimized benchmark, which can be accessed at https: //gi
explore methods to further reduce the (size of the) CI by
ber.r///rr/es--eeeberr--------a aagin
1 Introduction
The recent surge interest in few-shot learning (FSL), driven by its potential applications in many real- world scenarios, has led a proliferation of new mothods and novel experimental protocols (Sung et al. 2018 Snell et al. 2017 Bendou al. 2002 Zhang al., 2020). If in conventional machine learning is common to benchmark methods using fixed split into training and validation sets, FSL. presents unique

---

Published in Transactions on Machine Learning Research (09/2024)
challenges due to its reliance on extremely small and, consequently, biased training datasets. In fact, the performance of FSL can dramatically depend on the choice of the given labeled training samples (Arnold et al. 2021).
One question that FSL shares with conventional machine learning is that of the best performing methods. Especially relevant to FSI. the high variance of measured performance based on the choice of labeled data has led practitioners to quickly adopt the standard of aggregating statistics over a large number of artificially generated tasks, stemming from a single (or few distinct) dataset(s). The predominant approach is to generate artificial few-shot tasks by randomly sampling the same dataset with replacement, i.e. permitting the same samples to appear across multiple tasks. The outcome of these numerous tasks is the calculation of an average accuracy and its associated confidence interval (CI) for each method, thereby providing researchers with a statistically relevant basis for comparing the efficacy of different methods.
By allowing the same samples to appear in multiple tasks, the computed CIs account for the randomness of the sampler but not the data itself. In fact, the computed CIs use the usual Lindeberg- Central Limit Theorem (CLT). Hence, for these CIs to be statistically valid, the underlying random variables must be independent and identically distributed (IID). This means that the currently reported Cls should be understood as a likely range of outcomes if the experiment were reproduced using exactly the same data, which we will refer to in the remainder of this paper as Closed CIs (CCIs). This contrasts with what is often of interest in many areas of machine learning: the range of outcomes if the experiment were repeated with data from the same underlying distribution, termed Open CIs (OCIs). Note that the latter could be obtained simply by sampling tasks without replacement, but at the cost of considerably restricting the number of different tasks one can generate for given dataset. This limitation is even more severe if the dataset is small, resulting in potentially larger Cls and inconclusive comparisons between methods.
The purpose of this paper is to highlight this crucial consideration when computing CTs. We propose strategies to address this issue and obtain meaningful comparisons while still accounting for the randomness of the data. These strategies rely on a) Paired Tests (PT), where methods are evaluated on the same set of generated tasks, and b) adequately sizing tasks. Throughout the paper, we focus on the specific case of few-shot classification in vision, the most popular area of research in the field of FSL.
CLIP
DINO
NCC
FT
NCC
FT
NCC
000
0 0++
0 +
CLIP
FT
- -00
0 0++
00+
NCC
+00
+ +
00-
DINO
FT
+00
+
+00
000
Table 1: Comparison between different methods for few- shot classification. Each entry consists of three elements: [With Replacement (Closed), Without Replacement (Open), Paired Tests (PT)). Symbols + and - respoc- lively indicate significant differences (positive and negative) between the row method and the column method while is non-conclusive. Results derive from the DTD test split (bottom left triangle) and Traffic Signs (top-right) split of MetaDataset, with task sampling at 5 shots, 5 ways, and 15 queries. Note the inversion in bold. NCC (Nearest Class Centroid), FT (Fine-tune) with CLIP and DINO as feature extractors.
Our investigation using Open Confidence In- tervals (OCTs) can lead to conclusions that are inconsistent with those obtained using the clas- sical approach in the field of few-shot. learn- ing. In particular, we find that some meth- ods previously reported as statistically signifi- cantly performminng others are actually indis- tinguishable when using OCIs, and vice versa. In addition, we show cases where the use of CCIs lead to (statistically significant) conclu- sions that are diametrically opposed to those obtained using PT. One such example is given in Table D where we compare different methods for few-shot classification depending on their used feature extractor (here CLIP (Radford et al.) 2021) or DINO |Caron et al. 2021)) and their adapting methods (here Logistic Regression (LR), Nearest Class Centroid (NCC) or Fine-Tuning (FT)). In the table, we report three conclusions for each pair of model and method combinations: the first is obtained from the methodology
described in Luo et al. (2023) where the authors used the predominant way to compute CIs, that is CCIs; the second OCIs, and the third is based on PT. A conclusion that the row method is better than the column
2

---

Published in Transactions on Machine Learning Research (09/2024)
one is denoted with +, 0 mcans there is conclusive statement, and that the column method is the most performing of the two. The upper triangular values in blue correspond to the Traffic Sign test split while the rest of the table (in red) refers to the DTD test split in the Metadataset benchmark by 'Triantafillou ct al. (019)). Particularly striking results are found on the Traffic Signs dataset. Indoed, when studied with replacement tests, CLIP with the NCC adapler underperforms DINO with Fine-tuning, yet this outcome is reversed in paíred test assessments. This clearly demonstrates the importance of distinguishing between measurement methods, failing do so can lead significant misinterpretations of results.
The main contributions of this work are:
We highlight the importance of considering data replacement when computing CIs for FSL methods comparison. Our study illustrates the impact CI ranges when transitioning from closed to open Cls standard off-the-shelf vision datasets.
By implementing paired evaluations, wherein multiple methods are compared on identically gener- ated task sets, demonstrate the ability to reach conclusive comparisons more frequently than when relying on simple performances with OCIs.
We investigate how optimize lask generation from a given datasel, laking into account size and number of classes, to reach small ranges of CIs and lead to more conclusive comparisons between methods. The result is benchmark that can be used for few-shot classification of images.
2 Closed Cls vs. Open Cls
In this section, we are interested in better quantifying the difference between CCIs and OCIs. For this purpose, we define notations and outline algorithms for task sampling. We present theoretical analysis of the differences between CCIs and OCIs. Finally, we empirically compare their ranges on real datasets.
2.1 A mathematical description of the problem
2.1.1 Standard Evaluation and Notations
The predominant method evaluation the field of few-shot classification is described Algorithm 1 A of classes K, support set S (Se and a query set {Qe]eck: where S.,, Q, denote the sets of support and query examples for each class K.. Let K | denote the number of ways (Le. classes in few-shot task), S =S.] the number of shots per class, and Q Qcl the number of queries per class (for simplicity, we assume the classes to be balanced). Few-shol evaluation is typically performed by constructing many tasks from a larger evaluation dataset. An evaluation dataset D = (C,X) comprises a set of classes C and examples for all classes X' = (XoJeec- Let C = CC| 2 K denote the number of classes, and N X >> S + Q the number of examples per class. As highlighted the introduction, the rationale followed by the CI computation predominant method is consider D be fixed and non-probabilistic.
few-shot classification task T (K, Q) comprises
3

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm Predominant evaluation algorithm
procedure EVALUATE(T, K, S, Q.C, (Xeleec) tasks, K ways, shots, Q queries, set of classes C, set of data samples (ReJoEc for == 1......, do
3:
K lake(K, shullle(C))
4:
for CE K do
5: Se, 2. + split(S, take(S + Q, shuffle(Xo))
6:
end for
7:
Is -- FRWSHO)LLAARN(S)
D D = (Se)eEk
Ag - R2
KQ LeEk Exeo. [[fs(x) cc
end for
10: A + MEAN(A)
> MEAN(A) AA] 7 Σ 1=1 A
11:
OA + VVA
VAR(A) = T-1
1 Σ 1=1 (A - A)
12:
OA/VT + a
13:
return A + 1.960 A
1.960 r(plamit 95%)
14: end procedure
The standard few-shot (ask sampler constructs T random tasks with K ways, S shots and Q queries from dataset DD as outlined in Algorithm 1 This procedure introduces additional random variables (besides the dataset itself) in the selection of classes and examples. Let Te = SK, 2t) for t = 1,..., T denote the sampling of tasks. The average accuracy on each task is obtained as:
A = 1 E E
E (x) = c).
(1)
KQ CE xE
where f is the evaluated model, conditioned by the support set.
Next, turn to the actually reported metric which is the average accuracy over several tasks.
2.1.2 Computing Confidence Intervals
We compute the accuracy At of the method on each task, and then take the mean across tasks A 7 (A1 +... Ar). As such, we obtain the following formula for the variance:
1
Var[A] Var[Ai] +...+ Var[Ar] Var[A]
(2)
T2
Assuming sufficient sample size and that the mean Á is normally distributed according to the Central Limit Theorem, the 95% confidence interval is obtained as A 1.960 using the formula for standard error (standard deviation of the sample mean):
CI 1.960 = 1.96 AA
VT
Note that in the case of a very small number of lasks, Student's distributions can be used instead. Also, we took the example of 95% Cls, which is arbitrary but very common in the literature. For more generality, we consider a probability Plimit in the following for all theoretical considerations, and stick with the 95% value for experiments.
Note that, as the number of tasks becomes larger, it is inevitable that many tasks will re-use examples, since tasks are constructed independently with replacement. Therefore, as T becomes large, the variance of the sample mean will approach the conditional variance Var[A | D). and the confidence interval will represent the likely range of outcomes if repeat the experiment with random set of tasks the same
4

---

Published in Transactions on Machine Learning Research (09/2024)
dataset. In other words, it provides no insight into how well method would generalize on distribution. On the other hand, if I is chosen small enough, there will be minimal re-use of examples and the assumption of independence may approximately hold, although the confidence interval may be significantly larger.
We will now conduct an empirical evaluation of the disparity between closed (tasks sampled with replacement) and open (tasks sampled without replacement) CTs on real datasets.
2.2 Are OCIs larger than CCIs? An empirical study
In contrast to Algorithm 1 the task sampling without replacement is presented Algorithm 2 (see Ap- pendix). Note that we make an explicit use of a Student's law estimator in this algorithm it expected that for some small datasets it can only generate but few independent tasks. In the latter, the total number of tasks T is determined directly by the sampling process, thanks to specific stopping condition, based the exhaustion of the dataset. This is done in an effort to minimize the obtained CIs' ranges. Since samples cannot be sampled twice, we can consider that the classes and examples are drawn IID from an underlying data distributions pCC) and p(X | C). This is why OCIs also account for the randomness of the data.
In our experiments, we utilize datasots from the Metadatasct Benchmark as referenced in Triantafillou et al.. (019 This benchmark comprises 10 datasets, out of which we employ 9, excluding Imagenet, to focus cross-domain results in line with the recent trend in the literature (Zhou et al. 2022b). These include Omniglot (handwritten characters), Aircraft, CUB (birds), DTD (textures), Fungi, VGG Flowers, Traffic
Signs, Quickdraw (crowd-sourced drawings) and MSCOCO (common objects) [Lake et al. 2015 Maji et al.
2013
Wah et al. 2011: Cimpoi et al., 2014 Schroederr & Cuil 2018: Nilsback & Zisserman 2008
Houben
ct al.
2013 Jongejan et al. 2016 Lin et al. 2014).
Luo et al. (2023) details few-shot accuracies for 2000 tasks with 5-shots, 5 ways, and 15 queries in a com- prehensive table covering various works on the Metadataset datasets. Our study's only difference lies in the adoption of the T = 600 setting, more prevalent choice in existing literature. If CCIs are found to be narrower than OCIs with this smaller T, it will be even starker with T = 2000 tasks as shown in Equation 3 Our primary reference for methods and models is the comprehensive compilation provided by Luo et al. 2023 foundational starting point for our experiments.
Our findings are detailed in Table 2 showcasing results across different few-shot methods and datasets. Firstly, there a noticeable homogeneity in CCIs, arising from the fixed number of tasks set T 600, which contrasts with the variability observed in OCIs. Interestingly, CCIs are substantialy narrower than OCIs for small datasets such as Aircraft and DTD. Conversely, in the case of larger datasets like Quickdraw, CCIs become larger than OCTs due to T 600 being insufficient to deplete the dataset. Indeed, A Aircraft and DTD's test splits contain 1,500 and 840 samples respectively, whereas the test splits for MSCOCO and Quickdraw have much larger sizes of 152,000 and 7.7 million samples respectively. Across various datasets, models and methods, CCIs are on average 3.8 times larger than OCIs. These results highlight the imperative need for accurate interpretation of Confidence Intervals, given the dramatic differences between OCTs and CCIs ranges, that undoubtedly lead to disagreeing conclusions if misinterpreted.
We also notice that for cases where methods reach accuracies near 100%, like adaptation methods using CLIP (unlike those using DINO) on the CUB dataset, both types of CTs become narrower. This is due to accuracy saturation at 100%, which reduces the standard deviation of accuracies.
In the following, we delve into the conclusiveness of comparative studies using CCIs or OCIs.
2.3 Impact on Conclusiveness
First let us recall how confidence intervals are used to draw conclusions when comparing methods. Suppose we have two variables of interest i, and ø2, with their corresponding Plamit confidence intervals (a generalizod version of 95%-CIs) [2 61, 21 + 51] and [ø2 - 62. ã2 + 52]. To draw conclusions about the fact 21 is smaller than Z2, we proceed follows: if the two intervals do not intersect, and x1 + 51 < T 82, then:
5

---

Published in Transactions on Machine Learning Research (09/2024)
Model
CLIP
DINO
Sampling Method
W. Repl.
W/O. Repl.
W. Repl.
W/O. Repl.
Datasel Name
Dataset Size
LR
79.59 + 0.52
84.00 + 4.50
83.29 + 0.51
86.10 + 4.66
DTD
840
FT
76.87 + 0.56
80.76 + 5.60
81.82 + 0.51
84.19 + 5.76
LR
98.30 0.23
99.39
0.84
97.48
0.26
97.58
2.07
VGG Flower
1,425
FT
98.33 0.23
99.27 + 0.93
97.13 + 0.29
97.45 1.98
LR
75.79 + 0.85
69.90 +: 5.59
59.04 + 0.93
53.62 + 5.66
Aircraft.
1,500
FT
74.70 + 0.85
68.95 + 5.82
54.58 = 0.94
49.81 5.17
LR
96.85 + 0.28
97.24 + 1.88
92.06 = 0.43
89.69 + 4.03
CUB
1,770
FT
96.68 + 0.29
97.07 + 1.83
89.16 = 0.52
87.47 + 4.64
LR
90.55 0.49
91.04
0.89
93.67
0.38
94.12 0.79
Omniglot
13,180
FT
92.06 0.48
92.83 + 0.91
94.70 = 0.36
95.09 0.70
LR
70.86 + 0.88
74.78 + 1.89
77.26 + 0.75
81.64 + 1.41
Fungi
13,463
FT
68.03 + 0.95
71.87 = 1.96
74.02 = 0.83
78.88 1.68
LR
82.99 + 0.87
77.02 + 0.92
83.63 = 0.92
75.58 + 1.17
Traffic Signs
39,252
FT
83.64 + 0.85
76.69 + 0.96
84.20 = 0.92
74.93 + 1.23
LR
72.27
0.78
67.97
0.63
76.05 0.72
72.02 0.60
MSCOCO
151,545
FT
70.22 0.79
65.97 = 0.64
75.18 0.75
71.19 0.61
LR
75.79 + 0.66
75.54 + 0.36
74.54 - 0.68
74.07 + 0.38
Quickdraw
7,710,295
FT
76.18 + 0.69
75.93 + 0.38
74.10 = 0.70
73.61 0.39
Table 2: Accuracies and associated CIs of methods different datasets with and without replacement in the sampling of tasks. The dataset size correspond to the number of images in each test split. FT and LR respectively stand for Fine-tune and Logistic Regression.
Pæ < x2) ((x < +8 A 82 52)
Plimit
>
Piiit
(4)
2
where the (1 1 Ptimit)) 3/2 part comes from the symmetry of the Gaussian distribution.
With this in mind, the results listed in Table 2 can lead to different conclusions depending on whether CCIs or OCIs are used. For example, using the DTD dataset, CCIs lead to conclusive comparisons between both backbones and methods, whereas OCTs are inconclusive as they all intersect. Again, this should not be seen as a contradiction, but rather as having different paradigms about the comparisons. CCIs compare methods if they were reused on the same data, whereas OCIs focus on the underlying distribution. Next, we propose methods to improve conclusiveness when comparing methods, namely paired tests in Section 3 and task sizing in Section
3
Paired tests
3.1 Definitions
As an effort to reduce the ranges of Open Confidence Intervals (OCLs), we propose to make use of paired tests. Indeod, as we pointed out in the introduction, FSL tasks have a vast diversity in difficulty, lead- ing to high variance in accuracy across tasks. is noteworthy that task deemed hard for method A often aligns in difficulty for method B. This parallel in task difficulty across different methods

---

Published in Transactions on Machine Learning Research (09/2024)
previously identified by Arnold et al. (2021) and our findings are in agreement, as illustrated in Fig- ure 1 where we plot the accuracies on tasks generated from the Traffic Sign dataset and using two different combinations of feature extractors and adaptation method. In the provided figure, a strong correlation, quantified at 0.675, is evident between two distinct methods. This highlights the poten- tial for reducing the variance in accuracies resulting from task sampling by employing paired lesting. Formally, let us denote A, = Ar-Be B, the accuracy of the method on task relative to a method with ac- curacy Bt, with A Σi= A, the mean difference across tasks. While the mean of differences is sim- ply the difference of the means, ]| = the variance of the difference may be significantly re- duced when the accuracies are positively correlated:
Conriation Coctlident
Accurccy DINO NCC
1 Σ
1 T
Var[A] = Var
A
= Var [A, -,)]
(5)
T t=1
since Var[X- - Y]= Var [X] + Var(Y] Cov(X,)),
The lower variance of A. compared to As results in correspondingly smaller confidence interval, de- tailed in Equation 3 Consequently, this loads to scc- narios where two methods can exhibit significant dif- ferences when analyzed using paired testing despite there being no significant differences when directly comparing accuracics. We performed experiments comparing various methods to fine-tuning (FT). Re- sults are shown in Table [ where each line corre-
70
100
Aucuracy CLIP LR
Figure 1: Scatter plot of task accuracies using two different combinations of feature axtractor/adaptatie methods on the Traffic Signs benchmark.
sponds to specific dataset and feature extractor and each column to combination of an adaptation method and a feature extractor. Two conclusions are depicted, the first being based on directly comparing accuracies, and the second using paired tests instead. Note that in all cases, we rely on OCTs, that is to say sampling without replacements. Let us also notice that paired tests never lead to contradictory con- clusions with direct comparisons of accuracies, a direct consequence of previous remarks about the mean of differences. Their difference lies in the ability to conclude or not.
The fine-tuning adaptation method is selected the baseline in Table 3 primarily due to its substantial compulational cost. The cost of fine-tuning stems from the necessity to updale the weights of the entire feature extractor for each considered task. The objective is to determine if it surpasses more cost-effective methods in performance. In comparative analyses excluding the comparison between CLIP and DINO and locusing only on the same feature extractor, is frequently noted that fine-tuning either underperforms relative to other methods or the results are inconclusive. Indeed, fine-tuning significantly outperforms other methods in only out 36 cases, while underperforming in 14 cases. The outcomes in the remaining instances remain inconclusive. Consequently, overhead, is not particularly ad advantageous to consider. These results should be nuanced by the dependency of fine-tuning on hyperparameters which makes any assertions about this method contingent upon a specific set of typerparameters (Kumar et al. 2022).
conclude that fine-tuning, in light of its considerable computational
For each of the nine considered datasets, we have total of 135 unique comparisons, taking into account only distinct pairs of (model, methods) across two models and three methods. We found that 57 comparisons were conclusive using direct comparison with OCI while 94 were conclusive using paired tests. Figure t in the Appendix illustrates this.
Table I illustrates, on the DTD and Traffic signs datasct, that the three different approaches for computing CIs discussed so far result in varied assessments of the significance of differences between two methods. On all datasets, in 27 instances out of the 114 where the comparison with replacement was conclusive, that is 23% of such cases, a pattern emerges: a comparison initially classified as significantly different becomes non-conclusive under sampling without replacement, and then conclusive again when paired test is applied. On 12 instances, 11% of previously considered cases, conclusiveness is not confirmed by the paired test.
7

---

Published in Transactions on Machine Learning Research (09/2024)
Model
CLIP NCC
DINO
Method
LR
FT
LR
NCC
FT
Model
Dataset
Aircraft
00
00
CUB
00
00
DTD
00
Fungi
0-
CLIP
MSCOCO
0-
Omniglot
00
Quickdraw
0+
00
Traffic Signs
00
00
VGG Flower
00
00
Aireraft
60
CUB
00
DTD
00
80
00
00
00
Fungi
++
0-
0-
DINO
MSCOCO
++
0-
0-
Omniglot
0+
00
Quickdraw
0-
Traffic Signs
P
0-
0+
VGG Flower
00
00
Table 3: Impact of Paired Testing Significance Relative to Simple Comparison. In this analysis, compare the Finetune (FT) method against all other methods across both CLIP and DINO models. The initial character in each pair denotes the significance outcome determined without replacement, while the subsequent character reflects the result. derived with paired lesting. Here, 0 represents a non-significant difference, + indicates a significantly higher accuracy the FT method, and conveys a significantly lower accuracy. The FT columns comparing CLIP and DINO are opposites of cach other.
Particularly striking is instance of inversion, where method previously deemed significantly more accu- rate than another was found to be significantly less accurate using paired test. This reversal was observed in the comparison of Fine-tuning on DINO versus NCC with CLIP features on the Traffic Signs dataset. This implies that a method can significantly outperform another on specifie dataset, yet significantly un- derperform when evaluated across the entire distribution. The dataset thus a particular instance data that favors one method. This example powerfully exemplifies the need for clarity when interpreting Cls.
A natural question that arises from previous considerations is that of how to size tasks when performing sampling without replacement, aiming to reduce the range of obtained Cls. In that matter, consider the size of the support set to be fixed at KS, leaving as the only free variable the number of queries per task and per class Q. Indeed, increasing the number of queries will inevitably reduce the total number of tasks we can construct, as shown the following equation. Assuming a balanced dataset, we can estimate the number of tasks that can sampled by exhausting the full dataset:
D
CN
T
(6)
K(Q
+ 5)
with [T] is the number of samples, accounting for both the support and query sets, in each task.
If increasing Q reduces the number of tasks, it also changes (V) which is proportional to the CL. As such, there exists trade-off between the number of queries and the feasible number of tasks that can be generated to minimize OCIs for any given dataset. Intuitively, measuring A with a small 7 (and consequently high Q) results extensive CI ranges, a phenomenon depicted Equation 3 Conversely, measuring with Q may generate many tasks (large T) with an extremely high variance because the accuracy per class becomes either 0% 100%. In the following, we aim to identify the optimal number of queries, denoted Q",
8

---

Published in Transactions on Machine Learning Research (09/2024)
that effectively minimizes the variance of the average accuracy Var(A) and thus the obtained Cls. We first demonstrate mathematically the existence of such minimum by deriving Var( (4).
4 Sizing tasks to narrow OCIs
We show the Appendix that Var(A) can be written as:
K NC (aQ
Var (A) ( +
+7),
(7)
with 33 and also defined in the Appendix, and B > O.
These parameters are difficult to estimate in particular
VeriA
when dealing with real datasets and methods. If a S O, then Var (A) is decreasing as function of Q since Q N. In the following, we focus only on cases where a 0. This choice is supported by empirical evidence, which we will present later, indicating a U-shaped relationship between the variance of A and Q for a certain range of S. Assuming this, Var (4) reaches its minimum at Q
Dats
Mide
10'
V Next, study what this entails as S and N vary.
Figure 2: Variance of the average accuracy vs. the number of queries with synthetic data. The two classes ()-1,1 and (), The size of the dataset is N = 1000 (500 samples class). Tasks are sampled according to Algorithm 2 The number shots set to 3. We fit this with the model described in Equation I0 and observe a strong fit of the model with our experiment.
4.1 Effect of S and N Q'
represented as 1D Gaussians
Given the definition of and A obtained in Equation TO find that Q' is an increasing function of S and a con- stant function of N. In this section, we show that these results are confirmed empirically on real datasets.
We propose to study the aforementioned variance model of with respect to Q in a simplistic 1D representation of samples. In our model, two class' distribution are repre- sented as two Gaussians (N; NW4,,, o;) with E 2). We then sample artificial balanced dataset of fixed size N. Next, we sample tasks within this artificial dataset until exhaustion with the procedure described in Algo- rithm setting K = C = 2. Using the NCC classifier we obtain a set of accuracies on which we can compute the average accuracy A. This procedure consisting of instantiating the synthetic dataset from Ganssians and measuring is iterated. This yields a set of (A;}y for a given set of parameters (S, Q, N,, from which we compute an empirical variance Var(A).
In Figure 2 we show the measured Var(A) vs Q. We use 1000 samples in the datasets, split between the two classes. We set the number of shots to S 5. The model in Equation 10 is fitted very precisely. The discretisation effect seen at high Q is due to the low number of tasks. Next, we study the effect of S and N and compare it to our experiments with synthetic data.
Increasing S shifts the curve's minimum from Q' = towards Q" - +00 as depicted in Figure 3 This aligns with our modcl's predictions. At 1, opting for Q" effectively has two effects: (a) the high variance of A, due to small support and query sets increases the variance of À and (b) low Q allows significantly larger T, thus reducing the overall variance of A shown in Equation 3 Conversely, for S > 20, the setting boils down to classical transfer learning. Indeed, the narrowest CT is attained with one task with a large support and query set task. Finally, we find third regime where, for a range of values of S, Q" is nontrivial. It corresponds to what is shown in the S regime in Figure
When incrcasing N, Q" should not be affected according to Equation 10 However, we observe a hardly perceptible shift of Q" in Figure when N increased. We consider this effect be sufficiently small and therefore negligible. Next, explore how these results extend to real-world datasets.
9

---

Published in Transactions on Machine Learning Research (09/2024)
N=500
N=500
S=1
S=1
102
N=1000
N=1000
S-5
10a
5
N=m5000
10a
N=5000
S=20
S=21
Var(A)
VarlA)
10
104
10
10
10a
109
10a 10' 107
Q 10
109
101
10'
10
109
10'
10'
Q
Q
Figure 3: Variance of the average accuracy À and number of tasks T across different settings of S and N, derived from synthetic datasets featuring two 1D Gaussian classes, (V(1,,)) and (1, The left pair of graphs display results with fixed number of shots (S = 5), while the right pair of graphs show results for constant sample size in the synthetic dataset (N = 1000).
4.1.1 Real Dataset Experiments
We now shift our focus to the findings derived from real image datasets. These datasets often unbalanced and exhibit variety of class numbers. Our objective is to determine whether the earlier conclusions remain valid in this context.
First, we observe that Q" does not depend on the size of datasets. However the size the dataset scales T which in turn scales C'19s%. Similar to the results with synthetic data, we observe a discretisation of the confidence interval in the high Q (low T) regime.
We also observe a similar phenomenon with different regimes in 1, 5 and 10 shots in Figure 4 Our analysis suggests that for tasks in the shot regime, the best number of queries, Q, should be set to 1. For tasks with S = 5 and S 10, the best values for Q are approximately Q = 5 and Q 7, respectively. Figure [ also clearly shows that the predominent 15 queries is not optimal to narrow the OCI. We also show similar values for Q" using DINO22 instead of CLIP in the Appendix (Figure 5
5 Benchmark Proposal
Building on previously obtained results, we propose simple benchmark where Paired Tests are used and the value of Q is chosen as the minimum found in the previous paragraph. Implicitly, we are assuming that the minimum of A's OCI is also reached at Q*. More precisely, we are assuming the independence of the covariance with respect to Q. These assumptions are backed by the improved number conclusive comparisons Figure g when optimizing Q and using paired tests. Indeed, while PT yieled 94 conclusive comparisions, PT with optimized Q yields little more with 97 conclusive comparisons.
We present our results with the baselines adaptation methods previously studied in Table f using DINOv2 as our baseline model. Our experiments consistently show that, for given model, fine-tuning tends to be less effective than both logistic regression and the nearest class centroid methods. We also observe the choice of model is primordial with a clear advantage of DINOv2 over CLIP and DINO on most datasets. Our benchmark, including the code, seed values, task descriptions, and accuracy results, is available for use.
6 Related Work
Few-Shot Learning Since the seminal works of Vinyals et al. (0016) and Snell et al. 2011 the field of few-shot learning has known many contributions. Most solutions roly on the use a pretrained feature extractor, trained on a generic abundant dataset (Wang al. 2020 Bendou et al. Antoniou al. 2018
The feature extractor then be used is the target problem Wang et al. 2019), adapted before
10

---

Published in Transactions on Machine Learning Research (09/2024)
Class
10
304
CUB
DTD
VOG Hower
MSCOCO
1.bo
101
300
301
10
10
102
10
10
Class
CUR
10P
Pungl
MSCOCO
5-hods
301
100
10
Class
100
Aircrafi
30"
CUB
DTT
Pangl
VOG Flower
10:000
302
MSCOCO
100
309
16
Figure 4: (Left) Confidence Interval ranges (Right) Corresponding number of tasks generated. In all graphs the x-axis the number of queries Q. These results represent. averages from multiple trials, with the number of trials tailored according to the Task Count (T). Some curves are stopped before Q reaches 100 because of the number of samples per class. We do not show Omniglot and Quickdraw for visibility.
classifying (Zhang al. 2021 Most proposed methods differ in the way they combine the pretrained feature extractor and their adaptation to the target problem Luo al. (2023). Over the years, multiple benchmarks have been proposed, including MinilmageNet (Vinyals et al. 2016). Omniglot (Lake et al. 2015) and mieeeet (Ren el. al. 2018). If initially, these benchmarks focused on the in-domain case, where the feature extractor is trained on disjoint classes from the target problem, but all drawn from the same initial dataset, the trend evolved with the introduction of Meta-datasct (Triantafiloou and later COOP (Zhou et al., (022a)), where the feature extractor is trained on large generic dataset and applied to various other domains, including fine-grain problems, embodying cross-domain evaluation.
al.) (019)) (MD)
11

---

Published in Transactions on Machine Learning Research (09/2024)
A few papers focus on the sampling of tasks of targeted difficulty for few-shot learning (Arnold et al. 2021 (2020), the authors claim that model performance can be improved by sampling meta-learning tasks of increasing difficulty. In other works, failed meta-training tasks (deemed hard) are sampled again (Sun et al.
Zhou et al.] 2020). In Zhou et al.
2020 or previously misclassified samples stmples/clane /classcs are more likely to be
sampled in following Lasks Liu et al. (2020) to focus the model on difficult tasks. Estimating task difficulty can itself be difficult task, and several solutions have been proposed Arnold et al. (2021). The idea of difieully-basedd sampling proposed in Arnold et al. (2021) is relevant to this paper since it enables the sampling of groups of tasks with homogeneous difficulty, effectively reducing the confidence interval ranges. In contrast, our research adopts paired tests, method that obviates the need for such dependencies and provides a more universally applicable approach. Paired tests is not a novel contribution of our work. They were introduced over a century ago lo study the evolution of small populations with time (Student) 1908] Hedberg & Ayers 2013). In these seminal studies, the authors shown that individual differences provide more statistical power and insights than changes in averages over the whole population.
Confidence Intervals Confidence intervals were established by Polish mathematician Jerzy Neyman (Ney- man 1937] in the carly 1930's coinciding with Fisher's ideas although supported by different framework. They were more generally used and then required in medical research around the 1980's. They require assuming a specific model for the distribution of the considered data, In contrast, the bootstrap method, introduced in DiCiccio & Efron 1996), offers a distribution-agnosti approach for estimating ranges. We focused on traditional confidence intervals as they are better understood and more often implemented in the literature of few-shot, learning, but similar conclusions could be drawn using Boostrap instead.
Challenges in Statistical Interpretation and Methodological Biases This issue of misleading Cls is not isolated to domain. Belia et al. (2005) have similarly criticized the common misinterpretations surrounding confidence intervals in broader scientific research. Moreover, the propensity to overlook or un- derreport negative or null (non-conclusive) results further exacerbates the problem of biased interpretations. Borji (2017) argues for the importance of acknowledging and analyzing negative results in computer vision. Lastly, the impact of dataset biases on the evaluation metrics has been well-documented by [Torralba & Eiros (2011). In their work, "Unbiased Look at Dataset Biascs", they identify and mcasure several biases such as selection, capture and negative set. It serves as a critical reminder of how dataset-based results can differ from those obtained in real-world distributions.
7 Limitations
A first limitation of our study would like to point out is that for large datasets such as MSCOCO QuickDraw, the predominant method of CI calculation leads to intervals that are actually larger than our proposed OCI. As such, on such large datasets using CCI may not be an unfair approximation.
Furthermore, our mathematical modeling only explains the origin of the minimum of CI with respect to Q but does not provide a way to find it analytically since we cannot easily estimate a, 3 and y in Equation 10
As mentioned the end of paragraph 2.2, saturation at 100% accuracy may negatively impact the compu- tation of confidence intervals and particularly the value of paired tests CI in Equation 5
Finally, we point out that paired tests introduce complexity they require a fixed seed and necessitate saving and publishing individual task accuracies when using the benchmark and comparing methods.
8 Conclusion
In our study, we demonstrated the stark contrast between Open and Closed measurements of method accu- racy in Few-Shot Learuing. Notably, OCIs take inlo account data randonness but are far wider than CCIs. We identified two major approaches that contribute to narrowing the OCIs and subsequently introduced a benchmark which uses these approaches. Our findings underscore the importance of using confidence inter- vals that account for data randomness in evaluations, a practice we advocate extending beyond classification and vision to encompass all domains employing task-based few-shot learning assessments.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Model
CLIP
DINO
DINOv2
Dataset
Method
LR
LR
LR
NCC
1-shot
9.241 + 4.274
25.931 + 3.693
0.000 + 1.161
-0.552 + 1.091
Aircraft
5-shot
2.286 + 7.402
16.143 + 6.311
-22886 = 2.043
-1.286
2.027
10-shot
-0.816 5.803
10.204 5.083
-4.286 = 2.568
-1.224 2.565
1-shot
10.743 + 2.067
25.486 + 2.991
0.000 + 0.000
0.000 + 0.000
CUB
5-shot
2.545 + 1.410
7.515 + 2.150
0.121 + 0.247
-0.121 + 0.247
10-shot
1.008 + 1.464
4.706 = 3.114
0.000 0.000
0.000 = 0.000
1-shot
7.805 + 4.722
5.122 + 4.724
0.976 + 1.941
0.488 + 1.191
DTD
5-shot
8.750 + 5.057
3.500 + 4.109
-0.250 = 1.979
0.000 = 1.348
10-shot
4.762 + 2.905
1.905 + 4.392
-3.492 = 3.253
-1.270 = 1.937
1-shot
15.357 1.262
11.937 1.174
-0.533 = 0.507
0.235
0.438
Fungi
5-shot
10.247 + 1.369
3.115 + 1.225
-3.098 = 0.658
-2417 + 0.677
10-shot
7.316 + 1.286
0.584 + 1.224
-2.879 = 0.890
-1.753 + 0.799
1-shot
12.360 1.013
8,820 + 0.975
0,450 0.377
0,180 - 0.306
MSCOCO
5-shot
9.952 + 0.411
6.406 + 0.389
-0.196 = 0.187
0.172 0.141
10-shot
8.181 + 0.371
3.883 + 0.354
-1.056 = 0.204
-0.202 +-2 0.158
1-shot
-1.898 + 1.227
-5.923 + 1.163
-2.111 = 0.793
-0.850 + 0.755
Omniglot
5-shot
0.760 + 1.016
-1.932 0.954
-1300 = 0.564
-0.973
0.628
10-shot
0.240 + 1.136
-2.246 1.027
-1.788 = 0.669
-0.698 +-t 0.733
1-shot
4.590 + 1.079
6.530 + 1.049
-1.170 = 0.547
1.760 + 0.526
Quickdraw
5-shot
6.016 + 0.429
7.850 = 0.449
-1.028 = 0.210
-0.160 + 0.235
10-shot
5.541 + 0.331
6.454 + 0.354
-1.109 = 0.171
-0.266 + 0.188
1-shot
-1.770 + 1.055
-0.600 + 1.034
-0.920 0.519
0.520 + 0.467
Traffic Signs
5-shot
-4.243 + 0.820
-4.078 + 0.673
-1.537
0.396
-0.367 + 0.395
10-shot
-5.453 + 0.948
-4.919 + 0.781
-2.857 = 0.453
-0.821 + 0.443
1-shot
4.576 + 1.810
12.034 2.243
0.000 0.477
0.000 0.000
VGG Flower
5-shot
0.545 + 0.829
2.182 + 1.521
0.000
0.000
-0.182 +o2 0.378
10-shot
0.000 + 0.000
1.039 + 0.968
0.260 + 0.579
0.260 + 0.579
Table 4: Comparative differences in paired tests. This table contrasts the performance of DINOv2 with Fino-tuning (FT) against DINOv2 combined with Nearest Class Centroid (NCC) or Logistic Regression (LR), as well the performance combinations of CLIP with DINO using LR.
13

---

Published in Transactions on Machine Learning Research (09/2024)
References
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. ar preprint 1xix:1810009502, 2018.
Sébastien Arnold, Guncet Dhillon, Avinash Ravichandran, and Stefano Soatto. Uniform sampling over episode difficulty. Advances in Neural Information Processing Systems, 34:1481 1493, 2021.
Sarah Belia, Fiona Fidler, Jennifer Williams, and Geoff Cumming. Researchers misunderstand confidence intervals and standard error bars. Psychological methods, 10(4):389, 2005.
Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdeloup, Stéphane Pateux, and Vincent Gripon. Easy: Ensemble augmented-sin y-shaped learning: State-of thettf-rrrtttt few-shot classification with simple ingredients. arxiv 2022. arXiv preprint or Xii:2201.00903
Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdeloup, Stéphane Pateux, and Vincent Gripon. ensembeee segmented-shoo-y-shhaee learning: State-of-the-art few-shot classification with simple components. Journal of Imaging, 8(7):179, 2022.
Ali Borji. Negative results in computer vision: A perspective, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.
Mircea. Cimpoi, Subhransu Maji, Tasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Thomas J DiCiccio and Bradley Efron. Bootstrap confidence intervals. Statistical science, 11(3):189-2 1996.
EC Hedberg and Stephanic Ayers. The power of a paired t-test with a covariate. Social science research, 50: 277-291, 2015.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In The 2013 international joint conference on neural networks (IJCNN), pp. 8. leee, 2013.
Jonas Jongejan, Henry Rowley Takashi Kawashima, Jongmin Kim, and Nick Gox-Gieg. The quick, draw!-ai experiment. Mount View, CA, accessed Feb, 17(2018):4, 2016.
Ananya Kumar, Aditi Raghumathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distributtin. arXiv preprint Xiw:2222..005. 2022.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B TTenenbaum. Human-level concept learning through probabilistic program induction. Science, 330(62666: 2015.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Compuler Vision ECCV 2014: 13th European Conference, Zurich, Swilzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755 Springer, 2014.
Chenghao Liu, Zhihao Wang, Doyen Sahoo, Yuan Fang, Kun Zhang, and Steven CH Hoi. Adaptive task sampling for meta-learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 28-28, 2020, Proceedings, Part XVIII 16, pp. 752-769. Springer, 2020.
Xu Luo, Hao Wu, Ji Zhang, Lianli Gao, Jing Xu, and Jingkuan Song. A closer look at few-shot classification again. arXiv preprinl ur 2in:23300 12246, 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. ar Xiv preprint 1rx6:1300..555 2013.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Jerzy Neyman. Outline of theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London. Series A, Mathemalienl and Physical Sciences, 236(767):333 380, 1937.
Maria-Elena Nilsback and Andrew Zissorman. Automated flower classification over a large number of classes. 2008 Sizth Indian Conference on Computer Vision, Graphies & Image Processing, pp. 722 729. IEEE, 2008.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Mengye Ren, Eleni Triantafilou,, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803.006. 00676,, 2018.
Brigit Schroeder and Yin Cui. Fgvex fungi classification challenge 2018. Available online: gillub. com/uie htt///iippedaffgccaec http://ggccffuuii_eeeeee fungi_conngg comp (accessed 14 July 2021), 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Aduances in neural information processing systems, 30, 2017.
Student. The probable error of a mean. Biometrika, 6(1):1-25, 1908.
Qianru Sun, Yaoyao Liu, Zhaozheng Chen, Tat-Seng Chua, and Bernt Schiele. Mela-transfer learning through hard tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1443 1456, 2020.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 1199-120000 2018.
Antonio Torralba and Alexei Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521 1528. IEEE, 2011.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierro-Antoin Manzagol, al. Mota-dataset: A dataset of datasets for learning to learn from few examples. arXie preprinl urXiv:190.003096 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for shot learning. Advances in neural information processing systems, 29, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds- 200-2011 dataset. 2011.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning, arXiv preprint (rXio:1911.04625 2019.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1 34, 2020.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepend: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12203-1222 2020.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-fre -fenn clip-adapter for better lasion-language modeling. orXiv preprint arXiv:211 11.029901. 2021.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 16816 16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))2337-2344 2022b.
Yucan Zhou, Yu Wang, Jianfei Cai, Yu Zhou, Qinghua Hu, and Weiping Wang. Expert training: Task hardness aware meta-learning for few-shot classification. arXiv preprint arXiv:2077.062/ 2020.
16

---

Published in Transactions on Machine Learning Research (09/2024)
A
Mathematical derivation of Var(A)
Supposc tasks are drawn IID without replacement, write the variance of A as:
1 Var (A) T
Var, (At),
(8)
with A the accuracy for an arbitrary task L. By definition of the variance,
Var (A+) [[(A,) -EE)))()))
(9)
For a given support set, the expected accuracy for some class c is denoted Pr.e-
Pt.e
Be, (1[fs, (z) = c) S.).
Then the expectation of A, becomes
E[A[] = lscels
Along the same lines, we can derive B[(),)?,,
[[())) Es, Re, (AA.)
For fixed St, [[fs, (I) c] and [fss (x') = dl are not independent and their distribution will depend on the classes c and d. Using Equation [ we obtain.
B[(A,)²]
1
ΣΣΣΣ
Es, Bo [1fs. (x) (x) c'I]-
(KQ)²
c
We now separate cascs where x = from cases whore c c= d' and finally cases whore both are different.
[((A))
Es, [Me,e]
Q Q- -
Es, (Mrcc)
K2 Σ
Σ c'
Es, [Mr.cMe,c
KQ
KQ
Using Equation find:
1
Q
1 ΣEn
Var (At)
Es. [u,el
Es, (m),))
Es, Es (Es, (k),
KQ
KQ
K2
c'sc
Let us define some parameters,
m 2 E[A]]
K 2
Es, [m,c],
Σ
Es, ((,,,
122
and
m3 4
Es, [Me.)et
K2
d'
We get that
1
m1 Q-1
B[(A.)
m2 m3.
KQ
KQ
This gives
1
Q
--1
1
Var (4,) = KQ
mi +
m2 )
KQ
17

---

Published in Transactions on Machine Learning Research (09/2024)
Then, using Equation 6 (removing the rounding) and approximate:
K
B
Var (A) = NC
(aQ (a + +
(10)
Q
with a 122 + mg - (mi) 3,3 & (mi m2) and 7 mi
ma
+ R + S(ma - (m²))²)
K
K
First, let us notice that B 0 since for μ [0, 11: p² p
B Algorithms
Algorithm 1 samples tasks with replacement and computes CCIs with Equation 3 while Algorithm samples lasks without replacement and uses the student's 's distribution to compute OCIs.
Algorithm 2 Full dataset no-replacement evaluation algorithm
Q,C, (NoJoec) K ways, S shots, Q queries, set of classes C, set of data samples {Xe) Initialize T= 00
procedure EVALUAIEUUTT..DDPEEEEET
D List to store all tasks
3: while There are at least K classes in C with at least S Q examples each do
4: K. - Randomly select classes from C with at least S + Q examples
5: Initialize S = O and Q = 0
for cach K do
S. Randomly select S examples from X.
Q. - Randomly select Q examples from N. excluding Se Remove S. and Q. from Xe
10: Add Se to S
11:
Add Q. to Q
12:
end for Add (S, Q) to T end while
13:
14:
15:
Initialize A= == 0
D List to store all accuracies
16:
for L E T do
17:
Add A, to A end for
Measure the accuracy of task t
18:
19:
(EAN(.A)
20:
605% t(IA - 1, 95%)
Var(A) AA
b t is the critical value for the Student't distribution
21: return A + 695%
22:
end procedure
18

---

Published in Transactions on Machine Learning Research (09/2024)
C
Additional results on the benchmark
These results show the performance differences when taking DINO and CLIP with finetune as basclines. Again finet uning mostly underperforms other adaptation methods.
Method
LR
NCC
Dataset
Sampling
1-shot
-0.690 + 1.790
0.690 + 1.519
Aircraft
5-shot.
-5.286 + 2.802
-2.857 + 2.731
10-shot
-5.510 + 3.391
-0.408
3.290
1-shot
-1.486 + 1.467
0.914 + 1.388
CUB
5-shot
-2.182 + 1.740
-0.970 + 1.738
10-shot.
-4.874 + 2.737
-3.529 2.822
1-shot
-1.220 + 2.111
-0.488 + 1.540
DTD
5-shot
-1.750 + 2.457
-0.250 = 2.518
10-shot
-2.540 + 1.320
-1.587 + 1.937
1-shot
-0.549 + 0.561
1.427 + 0.514
Fungi
5-shot
-3.166 + 0.728
-1.770 0.627
10-shot
-3.658 + 0.920
-3.009 + 0.789
1-shot
0.840 = 0.389
0.000 + 0.380
MSCOCO
5-shot
-0.846 + 0.199
-0.210 0.176
10-shot
-2.488 = 0.224
-0.695 0.193
1-shot
2.976 + 0.675
2.749 + 0.724
Omniglot
5-shot
1.430 + 0.484
-0.183
0.556
10-shot
0.065 + 0.560
-0.458
0.656
1-shot
1.800 + 0.646
3.590 + 0.691
Quickdraw
5-shot
-0.546 + 0.267
-0.680 + 0.303
10-shot
-1.841 + 0.223
-1.331 + 0.260
1-shot
0.880 + 0.429
1.310 + 0.486
Traffic Signs
5-shot
-0.900 + 0.354
0.911 +-2 0.372
10-shot
-2.189 + 0.400
0.725 + 0.412
1-shot
-2.542 + 1.755
1.695 + 1.475
VGG Flower
5-shot
-0.545 + 0.623
-0.909 1.085
10-shot
-0.519 + 0.776
-0.519 + 0.776
Table 5: Paired test difference between DINO with FT and LR and NCC on DINO. FT, NCC and LR respectively stand for Fine-tuning, Nearest. Class Centroid, Logistic Regression.
19

---

Published in Transactions on Machine Learning Research (09/2024)
Method Sampling
LR
NCC
Dataset
1-shot
0.828 + 1.850
4.000 + 2.033
Aircraft
5-shot.
0.000 + 1.934
0.571 + 2.301
10-shot
-3.469 + 2.436
-5.510
2.191
1-shot
-0.229 + 1.500
1.714 + 1.313
CUB
5-shot
-0.364 + 0.962
-0.848 + 1.266
10-shot
-0.044 + 0.934
-0.504 + 1.297
1-shot
-2.195 +'t 2.184
2.195 + 2.584
DTD
5-shot
-2.000 + 3.002
-2.000 = 2.806
10-shot
-5.079 + 2.859
-4.444 + 1.937
1-shot
-2.086 + 0.747
0.439 + 0.703
Fungi
5-shot
-3.217 + 0.792
-2.826 0.868
10-shot.
-4.394 + 0.929
-4.069 + 0.986
1-shot
-0.840 + 0.536
1.670 + 0.528
MSCOCO
5-shot
-2.990 0.241
-0.970 0.232
10-shot
-3.099 + 0.239
-1.503 0.233
1-shot
4.723 + 0.725
4.723 + 0.799
Omniglot
5-shot
1.688 + 0.606
0.532 +'2 0.670
10-shot
0.371 = 0.702
-0.087 0.831
1-shot
0.760 + 0.731
4.940 + 0.759
Quickdraw
5-shot
0.226 + 0.299
-0.284 + 0.327
10-shot
-0.376 + 0.236
-0.754 + 0.268
1-shot
-0.500 + 0.590
0.590 +\t 0.606
Traffic Signs
5-shot
-0.481 + 0.373
-0.745 + 0.449
10-shot
-1.642 + 0.418
-0.458 + 0.478
1-shot
-1.356 + 1.408
0.169 + 1.305
VGG Flower
5-shot
-0.182 + 0.665
0.000 + 0.774
10-shot
-0.519 +'2 0.776
-0.519 0.776
Table 6: Paired test difference between CLIP with FT and LR and NCC on CLIP. FT, NCC and LR respectively stand for Vine-tuning Nearest Class Centroid, Logistic Regression.
20

---

Published in Transactions on Machine Learning Research (09/2024)
D Is Q* Dependent on the Model Used?
We show in Figure 5 that the same values of Q" are found for DINO v2 instead of CLIP.
Clos
10
Aircrat:
CUB
DID
30
Fungl
Chiniglat
VCG Flower
1:bol
301
MSCOCO
307
10
105
10
100
Airczai:
30*
DTD
Fangi
Cmniigott
Flower
300
MSCOCO
10
30'
10
16
10
10
Chos
10F
Aircrat:
102
CUB
DTD
Chnniglot
VGG Flower
10shots
MSCOCO
101
304
16
10F
10
10
107
101
10'
Figure 5: This figure shows that the value of Q" not dependent on the model used. This experiment is conducted using DINO v2 instead of CLIP. (Left) Confidence Interval ranges (Right) Corresponding number of tasks generated. In all graphs the x-axis is the number of queries Q. These results represent averages from multiple trials on DINO v2, with the number of trials tailored according to the Task Count (T)). Some curves are stopped before Q reaches 100 because of the number of samples per class. We do not show Omniglot and Quickdraw for visibility.
21

---

Published in Transactions on Machine Learning Research (09/2024)
E
Statistics on Conclusiveness
Number conclusive comperisonn
800
Total number of comparisons
400
0
no PT no PT Opti Q PT PT + Opti Q
Comparision setting
Figure Effect of using paired tests comparisons between all possible pairs formed from combinations of models and methods summed across datasets (with Omniglol removed for technical reasons) for 1, 5 and 10 shots. The number of conclusive comparisons is counted on the Y-axis. When Q is not optimized, Q 15.
This histogram illustrates that paired test and the optimization of the size of tasks yields the maximum number of conclusive comparisons. Optimizing Q slightly improves the number of conclusive comparisons compared to simple paired tests.
Acknowledgment
This work was performed using HPC resources from GENCI-IDRIS (Grant 202123656B).
The PhD study of Luke Smith is supported by GSK-ple and the Australian Government Research Training Program (RTP) scholarship.
Funded by the Deutsche Forschugggemeinsshaft (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2044 -390685587, Mathematics Münster: Dynamics-Geometryy
F. Vermet conducted this work within the France 2030 framework programme,, the Centre Henri Lebesgue ANR-L1-LABX-0000-1
Avec le soutien de
Région BRETAGnE
Raphael Lafargue supported by Brittany Region, France.
22


Reviews:
[{"review_id": "Mw9U0VFlUN", "reviewer": "Reviewer_ZiuS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper identifies a fundamental flaw in the standard method of computing CIs in FSL, which is crucial for the validity of comparative studies in this field.\n* The authors conduct a thorough comparative analysis between CIs computed with and without replacement, demonstrating the extent of the problem.\n* The proposal of paired tests and optimized task sampling methods offers practical solutions to improve the accuracy and reliability of CI computations in FSL.\n* The paper provides empirical evidence to support the proposed methods, using standard FSL benchmarks to illustrate the improvements.  The paper also develops a new benchmark for FSL.\n\nWeaknesses:\n* The scope of this paper is a bit limited, the empirical validation is primarily focused on vision datasets in FSL. It remains to be seen how well these methods generalize to other types of datasets and tasks.\n* The effectiveness of the proposed methods may depend on specific conditions, such as the size and composition of the datasets, which might limit their applicability in diverse scenarios.\n* Theoretical analysis may be lacking. Almost all results are interpreted via empirical study, the underlying theoretical explanation is not clear.\n* The complexity of implementing the paired test should be discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to address the issue of computing confidence intervals (CIs) in few-shot learning (FSL) by sampling tasks with replacement. This predominant method results in misleading CIs that account for the randomness of the sampler but not the data itself. In particular, this paper first highlights the issue of the current CI computation and shows that computing CIs with replacement would underestimate the true variability. Then, this paper develops a paired test, that is evaluated on the same set of tasks, to address the aforementioned issue. Accordingly, this paper also develops a new benchmark for evaluating the performance of few-shot learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness part.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hOIPfTPID1", "reviewer": "Reviewer_oQ3o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I believe that the paper tackles a crucial problem at the heart of evaluating FSL models, which is typically ignored in this line of work. They provide extensive experimental evaluations supporting the fact that there is a significant discrepancy between closed and open CIs. Indeed, they found that, across various datasets, CCIs can be almost 4 times larger than OCIs. The authors also propose ways to reduce the range of the CI by using i) paired tests, and ii) suitably sizing the tasks through a principled framework. \n\nThe results have a bearing in existing, reported results in prior literature, and highlight an important consideration for the literature going forward. Thus, although I am not an expert in this area, I believe that this paper could have significant impact. I also found the evaluation quite thorough and complete in terms of using the existing available datasets. The writing overall is very clear and well-organized, apart from some minor comments stated below.\n\nOne weakness of the paper is that at times the main body is not self-contained without the appendix, and important aspects are missing; for example, it would be nice if Algorithm 1 was in the main body, and Section 4 can also be improved in that regard, although I understand that the constraint on the page limit could make this difficult.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of computing confidence intervals (CIs) in few-shot learning (FSL). The key observation that drives the paper is that the common way of computing CIs, which is based on sampling with replacement, can lead to misleading results as it does not take sufficiently into account the randomness of the data. To quantify that discrepancy, they compare the results obtained by computing CIs without replacement. They find that many results that reported improvements in the literature are in fact inconclusive when sampling tasks without replacement. Finally, they propose ways of reducing the range of the CI by sampling tasks with a specific size. Notably, they rely on the known method of so-called paired evaluations in order to obtain more conclusive results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some minor comments which are not crucial for my recommendation:\n- The citation style is used incorrectly throughout; for example, \"any assertions about this method contingent upon a specific\nset of hyperparameters Kumar et al. (2022)\" should instead read any assertions about this method contingent upon a specific\nset of hyperparameters (Kumar et al. 2022). That is, use paranethesis when the citation is not syntantically part of the sentence.\n- In the beginning of page 9, there is typo in the sentence \" We consider this effect to be to small enough\"\n- First equation in page 16 is missing a punctation mark.", "broader_impact_concerns": "I do not have any concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8sxTBPbl3z", "reviewer": "Reviewer_pXGx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. **Important findings and well-motivated problems.** This paper finds that the existing evaluation methods for few-shot learning are problematic. Different evaluation methods can lead to different conclusions. Having reliable evaluation metrics is the foundation for developing algorithms. The problems this paper studied are very important for more rigorous research in few-shot learning.\n\n2. **Simple yet intuitive methods.** The paired test is intuitive but effective in narrowing the interval while maintaining the number of tasks. \n\n3. **Comprehensive experiments.** This paper evaluates various datasets, and the comprehensive experiments and ablations support the claims.\n\n4. The paper is, in general, easy to follow. The writing tells the story well.\n\nWeakness\n1. **Application on large-scale datasets.** It looks like the large interval problems only happen in small datasets. For large datasets (shown in Table 2), the OCIs are even smaller than CCIs. In this case, the proposed method may not be necessary for large datasets.\n\n2. **The choice of optimal Q.** If I understand correctly, the empirical choice of Q depends on a specific model, which could be biased. This Q* you find may not be the global optimal size. Given the results in Figure 5, the improvement of using Optimized Q is marginal, making me skeptical about whether this Optimized Q is actually helpful.\n\nTo be honest, I am not very familiar with this paper's background literature, so my comments may not be precise.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the problems of computing confidence intervals for few-shot learning models. Computing confidence intervals is very important as the performance of few-shot models can vary greatly on different sampled tasks. \n\nThe predominant approach allows the same examples to appear in multiple tasks when sampling the tasks (with replacement), named Closed CIs (CCIs) in this paper. Although more tasks can be sampled in this way, CCIs may overfit on specific datasets and not reflect models' performance on real distribution. Open CIs (OCIs), which sample tasks without replacement, are more representative of quantifying models' ability. This paper shows that OCIs are much larger than CCIs, especially in datasets with fewer examples, and their difference is more substantial. Compared to CCIs, using OCIs can lead to completely inverse conclusions when comparing few-shot methods. \n\nMotivated by the correlation of models' accuracies on the same task, this paper proposes the Paired Test (PT), which lets two models evaluate the same task and compute their performance gaps. Pair Tests can reduce the confidence intervals, and the conclusions drawn from PT do not contradict those drawn from OCI. In addition, this paper investigates the influence of query size Q on CI and proposes a way to find the optimal Q. Finally, based on all those findings, this paper introduces a benchmark for future evaluation of few-shot learning methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Have a limitation section to discuss the scope of usage of the proposed method. \n2. The writing could be gentler, making it easier for readers to digest the information. For example, Table 1 contains too much information, and until the very end of the introduction, this table is discussed. Consider moving it to page 2 or adding a paragraph to describe the results earlier.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts an investigation into the evaluation methodology for few-shot classification tasks. Specifically, they identify a flaw in the procedure with which confidence intervals are computed: they take into consideration only the randomness of the sampler but not the data. This is due to how 'tasks' are sampled to evaluate a few-shot learning method, where examples are sampled with replacement (leading to the same samples appearing in multiple 'tasks'), thus producing 'closed CIs'. Instead, one could have computed 'Open CIs' by sampling without replacement, at the cost of having many fewer tasks available (and larger confidence intervals), which is problematic in few-shot learning datasets that are typically small. The authors propose strategies to address this including a Paired Test where different methods are evaluated on the same set of sampled tasks. This allows to perform meaningful comparisons and reach conclusive conclusions easier than in Open CIs due to reducing the confidence intervals.

As all reviewers pointed out, the paper studies an important problem and the claims and findings in this paper are insightful and can be very impactful for how few-shot learning success is measured moving forward, e.g. "Overall, I believe that the paper tackles a crucial problem at the heart of evaluating FSL models, which is typically ignored in this line of work." (Reviewer oQ3o) and "The problems this paper studied are very important for more rigorous research in few-shot learning." (Reviewer pXGx).

The reviewer also appreciated the quality of the writing ("The writing overall is very clear and well-organized", Reviewer oQ3o) and, as discussed above in the "claims and evidence" section, all reviewers found the claims well-supported, and the empirical investigation to be thorough. Overall, I view this paper as a nice contribution for the field of few-shot learning. The findings reveal interesting and previously-unknown conclusions and can inform how few-shot evaluation is carried out moving forward.

A limitation seems to be that the empirical investigation is focused primarily on small datasets (which is justifiable as these are common in few-shot learning benchmarks), but it is not clear if the issues identified here appear in larger datasets as well, and how the proposed methods fare in that case.
Decision: Accept (poster)