On Episodes, Prototypical Networks, and Few-Shot Learning
Authors: Steinar Laenen
Luca Bertinetto
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning
few-shot learning
episodic-learning
analysis
case-study

Abstract: Episodic learning is a popular practice among researchers and practitioners interested in few-shot learning.
It consists of organising training in a series of learning problems (or episodes), each divided into a small training and validation subset to mimic the circumstances encountered during evaluation.
But is this always necessary?
In this paper, we investigate the usefulness of episodic learning in methods which use nonparametric approaches, such as nearest neighbours, at the level of the episode.
For these methods, we not only show how the constraints imposed by episodic learning are not necessary, but that they in fact lead to a data-inefficient way of exploiting training batches.
We conduct a wide range of ablative experiments with Matching and Prototypical Networks, two of the most popular methods that use nonparametric approaches at the level of the episode.
Their "non-episodic'' counterparts are considerably simpler, have less hyperparameters, and improve their performance in multiple few-shot classification datasets.

On Episodes, Prototypical Networks, and Few-Shot Learning
Steinar Laenen School of Informatics University of Edinburgh V.S. E. Laenen ed. ac ac.uk uk
Luca Bertinetto FiveAl luca. buc.bberrinettoofiie... ai
Abstract
Episodic learning is popular practice among researchers and practitioners inter- ested fhot learning. It consists of organising training series of learning problems (or episodes), each divided into small training and validation subset mimic the circumstances encountered during evaluation. But is this always neces- sary? In this paper, investigate the usefulness of episodic learning in methods which use nonparametric approaches, such as nearest neighbours, at the level the episode. For these methods, we not only show how the constraints imposed by episodic learning are not necessary, but that they in fact lead to data iata-inefficient way of exploiting training batches. We conduct wide range of ablative experi- ments with Matching and Prototypical Networks, two of the most popular methods that nonparametric approaches at the level of the episode. Their "non-episodic" counterparts are considerably simpler, have less hyperparameters, and improve their performance in multiple few-shot classification datasets.
1 Introduction
The problem of few-shot learning (FSL) classifying examples from previously unseen classes given only handful of training data has considerably grown in popularity within the machine learning community in the last few years. The reason is likely twofold. First, being able to perform well on FSL. problems is important for several applications, from learning symbols (3 to drug discovery (2). Second, since the aim of researchers interested in meta-learning is to design systems that can quickly learn novel concepts by generalising from previously encountered learning tasks, FSL benchmarks are often adopted as a practical way to empirically validate meta-learning algorithms.
To the best of our knowledge, there is not widely recognised definition of meta-learning, In recent survey, Hospedales et al. [22] informally describe it as "the process of improving a learning algorithm multiple learning episodes". In practical terms, following the compelling rationale that "test and train conditions should match" " [8. [3) several seminal meta-learning papers (e.g. [48] 32 [4] have emphasised the importance of organising training into episodes, i.e. learning problems with a limited amount of "training" (the support set) and "test" examples (the query set) to mimic the test-time scenario presented by FSL benchmarks.
However, several recent works (e.g. 9 49 [I (44) showed that simple baselines can outperform established FSL meta-learning methods by using embeddings pre-trained with the standard cross- entropy loss, thus casting a doubt on the importance of episodes FSL. Inspired by these results, we aim at understanding the practical usefulness of episodic learning in popular FSL methods relying metric- based nonparametric classifiers such as Matching and Prototypical Networks [48] 40]. We chose this family of methods because they do not perform any adaptation test time. This allows us
Work done while research intern FiveAl
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

EPISODES
EPISODES
POSITIVES
NEGATIVES
No EPISODES
[tm +n)"
EPISODES
1omm
PAIRS LOST
(00 +
(w
1)(ma + n²)
Figure 1: Difference in batch exploitation for metric-based methods between adopting or not adopting the concept of episodes during training, on an illustrative few-shot learning problem with 2 ways (classes), and 4 shots (examples) and query per class.
Table I:
The extra number of gradients that, on the same batch, a non-episodie method can exploit with respect to its episodic counterpart grows quadratically as Oww (m² + n² n²))), where is the number of ways, and and are the number of shots and queries per class.
to test the efficacy of episodic training without having to significantly change the baseline algorithms, which could potentially introduce confounding factors.
In this work we perform a case study focussed on Matching Networks (48 and Prototypical Net- works (40), and show that within this family of methods episodic learning a) is detrimental for performance, b) is analogous to randomly discarding examples from a batch and с) introduces a set of superfluous hyperparameters that require careful tuning. Without episodic learning, these methods are closely related to the classic Neighbourhood Component Analysis (NCA) [19 35 on deep embeddings and achieve, without bells and whistles, an accuracy that is competitive with recent methods on multiple FSL benchmarks: minilmaeNeet, CIFAR-FS and tieredImageNet.
PyTorch code is available http:::/gghhub...o//faaaa/o/oopp
2 Background and method
This section is divided follows: Sec. 2.1 introduces episodic learning and illustrates data efficiency issue encountered with nonparametric few-shot learners based on episodes; Sec. [2.2] introduces the losses from Snell et al. [40]. Vinyals al. 48 and Goldberger et al. [19] which
throughout
our experiments; and Sec. 2.3
2.3 explains the three options we explored to perform FSL classification
with previously-trane feature embeddings.
2.1 Episodic learning
A common strategy to train FSL methods is to consider distribution & over possible subsets of labels that is as close as possible to the one encountered during evaluation E 48 Each episodic batch Bg = {S, Q} is obtained by first sampling a subset of labels L from E, and then sampling images constituting both support set S and query
from the set of images with labels in L,,
where S = (s1, y1 y1)..... (Sn, yn)) Q = {(q, y1),.... (qm- Dm)); }. and S₃ and Q₁ denote the sets of images with label y l in the support set and query set respectively.
For most methods, this corresponds to training on a series mini-batches in which each image belongs either the support the query set. Support and query sets are constructed such that they both contain all the classes of L, and fixed number of images per class. Therefore, episodes are defined by three variables: the number of classes
[L] (the "ways"), "), the number of examples per class in the support set n = |.S|] (the "shots"), and the number of examples per class in the query set [Qxl. During evaluation, the set {w, n, m defines the problem setup. Instead, at training time {w, n, m) can be seen as a set of hyperparameters controlling the batch creation, and that (as we will in Sec. 3.2 requires careful tuning.
Note that, in FSL, the sets of classes encountered during training and evaluation disjoint.

---

In Maximum Likelihood Estimation framework, training on these episodes can be written as
arg max E
E SEL
Σ
log Po (vilg,, S.
(I)
0
L
(9ENN))E
For the sake of brevity, with a slight abuse of notation we omit the function fo (e.g. deep neural network) which is used to obtain a representation for the images in S and Q, and whose parameters optimised during the training process. Note that the optimisation of Eq. deppend
optional
set of parameters p. This is obtained by an "inner" optimisation procedure, whose scope is limited to the current episode [22] The idea is that the "outer" optimisation loop. by attending
distribution of episodes, will appropriately shift the inductive bias of the algorithm located in the inner loop, thus learning how learn [77. In recent years, many interesting proposals have been made about what form should have, and how it should be computed. For instance. in MAML [1]] takes the form of an update of the global parameters 0, while Ravi and Larochelle [32] learn to optimise by considering as set of the
of the optimiser's s update rule.
avoid learning a separate set of parameters p altogether, and utilise a nonparametric learner (such as nearest neighbour classifiers) at the inner level. We chose to focus our seminal for the community, but also for ease of experimental design. Having p = Ø considerably reduces the design complexity of the algorithm, thus allowing precise ablations to understand the efficacy of episodic learning without considerably changing the nature of the original algorithms.
Other methods, such Matching and Prototypical Networks 48
40
study these methods not only because they have been
Considerations on data efficiency. The constraints imposed by episodic learning on the role each image has in training batch has subtle but important implications, illustrated in Fig. by highlighting the number of distances contributing to the loss. By dividing batches between support and query set (S and Q) during training, episodes have the side effect of disregarding many of the distances between labelled examples that would constitute useful training signal for nonparametric FSL methods. More specifically, for metric-based nonparametric methods, the number of training distances that are omitted in a batch because of the episodic strategy grows quadratically as (w22mm2 + n²)) (derivation shown in Appendix A). Table breaks down this difference in terms of gradients from positives and negatives distance pairs (which we simply refer to as positives and negatives throughout the rest of the paper). In a typical training batch with 20, m 15 and n 5 [40], ignoring the episodic constraints increases the number of both positives and negatives by more than 150%.
In the remainder of this paper, we conduct case study to illustrate how this issue affects two of the most popular FSL algorithms relying on nonparametric approaches at the inner level: Prototypical Networks 40 and Matching Networks [48].
2.2 Loss functions
are one of the most popular and effective approaches in the few-shot learning literature. They are at the core of several recently proposed FSL methods (e.g. [27 18 11 50 7 and they are used in a number of applied machine learning works (e.g. EEG scan analysis for autism (36) and glaucoma grading [7))
Prototypical Networks (PNs)
40
During training, episodes consisting of a support set S and query set Q are sampled as described in Sec. 2.1 Then, prototype for each class i is computed as the mean embedding of the samples from the support set belonging to that class: C₁ = (1/[Sal) EiSWneS, fe(s)), where fa is a deep neural network with parameters 0 learned via Eq. [
Let C' {(c y1), (Ck, yx)) be the set of prototypes and corresponding labels. The loss can be written as follows:
exp exp=l[fo(c:) - Cy,, 1|²
LPNS
log
Q
Σ -xppexpplfaoo [[f(gg)) C 12
Eq
where k' is an index that goes over all classes.
3

---

Matching Networks (MNs) 48 are closely related to PNs in the multi-shot case and equivalent in the 1-shot case. Rather than aggregating the embeddings of the same class into prototypes, this loss directly computes a softmax over individual embeddings of the support set, as:
Σ exp - -lfo(( fo {s
ES 4
LNNSs
log (9:9) EQ
Σ exp
[56(q:) (o
In their work, Vinyals et al. [48] use the cosine rather than the Euclidean distance. However, (as (40] we observed that the Euclidean distance is better choice for FSL problems, and thus we use it in all the losses of our experiments. Note that Vinyals et [48] also suggest a variant to LMNS (MNs with
"Full Context Embeddings"), where an LSTM (with an extra set of parameters) is used to condition the way the inputs are embedded in the current support set. our experiments, we did not consider this variant as it falls in the category of adaptive episodic learning approaches (p & 2, see Sec. 2.1
Neighbourhood Component Analysis (NCA). LNNs and LPNs sum over the likelihoods that query image belongs to the same class of certain sample (or prototype) from the support set by computing the softmax over the distances between the query and the support samples (or prototypes). This is closely related to the Neighbourhood Component Analysis approach by Goldberger et al. [19 (and expanded to the non-linear case by Salakhutdinov et al. [35] and Frosst al. [5] except for few important differences which we discuss at the end of this section.
Let i [1, be the indices of the images within a batch B. The NCA loss can be written as:
Σ
exp
11, 2
jET.....
j yi
CNCA
Σ log
|B]
exp-1zz: Zk| keT.....
Σ
iE kEl.....
kfi
where z; = [x;) ) an image embedding and y its corresponding label. By minimising this loss, distances between embeddings from the same class will be minimised, while distances between embeddings from different classes will be maximised. Importantly, note how the concepts of support set and query set here do not exist. More simply, the images (and respective labels) constituting the batch = ((x1, v1),.... (xs, }b)} are sampled uniformly.
Given the similarity between these three losses, and considering that PNs and MNs do not perform episode-spede parameter adaptation, {w, m, m,n} can be simply interpreted as the set of hyper- parameters controlling the sampling of and NCA differ three aspects:
during training. More specifically, PNs, MNs
I. First and foremost, due to the nature of episodic learning, PNs and MNs only consider pairwise distances between the query and the support set (Fig. [eett; NCA instead uses all the distances within a batch and treats each example in the same way (Fig. right)..
II. Only PNs rely on the creation of prototypes.
III. Because of how L, and Q are sampled in episodic learning (Eq. D. for PNs and MNs some images might be sampled more frequently than others (sampling "with replacement"). NCA instead visits every image of the dataset once for each epoch (sampling "without replacement").
To investigate the effects of these three differences, in Sec. we conduct wide range of experiments.
2.3 Few-shot classification during evaluation
Once fo has been trained, there are many possible ways to perform few-shot classification during evaluation. In this paper we consider three simple approaches that are particularly intuitive for embeddings learned via metric-based losses like the ones described in Sec. 2.2 Note that, in the 1-shot case, all the evaluation methods considered coincide.

---

k-NN. To classify an image q: E Q we first compute the Euclidean distance to each support point 8j E S: d llf fo(ss))] ||? Then, we simply assign y(q)) to be majority label of the nearest neighbours. A downside here is that k is hyper-parameter that has to be chosen, although reasonable choice the FSI. setup to set it equal the number of "shots" n.
Nearest centroid. Similar to k-NN, we can perform classification by inheriting the label of the closest class centroid, i... y(q.)= arg je...e (x,) - cy This is the approach used by Prototypical Networks [40], SimpleShot [49], and both baselines of Chen et al. [10]
Soft assignments.
To classify an image qi E Q, we compute the values exp(<Il.ooq;) fo(s)))112)
Pij exp(
for all Sj S. which is the probability that inherits its class from j. We then compute the likelihood for each class k: Exyes, Pij. and choose the class with the highest likelihood y(q.)
arg max; Ex,es, Pij. This is the approach for classification adopted by the original NCA paper [9] and Matching Networks [88]
We experiment with all three alternatives and observe that the nearest centroid approach is the most effective (details available in Appendix D). For this reason, unless differently specified, we use it as default in our experiments.
3 Experiments
In the following, Sec. 3.1 describes our experimental setup; Sec. 3.2 shows the important effect the
hyperparameters controlling the creation of episodes; in Sec. 3.3 compare the episodic strategy to randomly discarding pairwise distances within a batch; in Sec. 3.4 we perform a set of ablations to better illustrate the relationship between PNs, MNs and NCA; finally, in Sec. 3.5 version of the NCA to several recent methods.
we compare our
3.1 Experimental setup
We conduct our experiments minilmageNet 48), CIFAR-FS 55 and tieredImageNNet 34] using the popular ResNet-12 variant first adopted by Lee et al. [24] as embedding function fo detailed description of benchmarks, architecture and choice of hyperparameters is deferred to Appendix F, while below we discuss the most important choices the experimental setup.
3
Like Wang et al. [49] for all our experiments (including those with Prototypical and Matching Networks) we centre and normalise the feature embeddings before performing classification, considerably beneficial for performance. After training, we compute the mean feature vectors of all the images in the training set: [etrain ExeDirain X. Then, all feature vectors in the test set are
updated as x; + X, X, and normalised by X; X1
As standard 22 performance assessed on episodes of 5-way, 15-query and 1- or 5-shot. Each model is evaluated on 10,000 episodes sampled from the validation set during training, from the test set during testing. To further reduce the variance, we trained each model three times with three different random seeds, for total of 30,000 episodes per configuration, from which 95% confidence intervals are computed.
3.2 Batch size and episodes
Despite Prototypical and Matching Networks being among the simplest FSL methods, the creation of episodes requires the use of several hyperparameters ((w, m, n... Sec. 2.1 which can significantly affect performance. Snell et al. [40] state that the number of shots between training and testing should match and that one should use higher number of ways w during training. In their experiments, they train 1-shot models with
1, 15 and 5-shot models with = 20, m = 15, with batch sizes of 480 and 400, respectively. Since the corresponding batch sizes of these configurations differ, making direct comparisons between them is difficult.
30,
Note that, since do not final linear layer for classification, backbone is in fact ResNet- 11.
5

---

91
07
1-0
FrandE
Batch 2220
Batch $220
Batch $220
Batch 9U20
Figure 2: 1-shot and 5-shot accuracies CIFAR-FS (val. set) for Prototypical and Matching Networks models trained with different episodic configurations: I-shot with m + n and 5-shot with m + n-8, 16 or 32. NCA models are trained on batches of size 128, 256 and 512 to match the size of the episodes. Reported values correspond to the mean accuracy of three models trained with different random seeds and shaded areas represent 95% confidence intervals. See Sec. 3.2 for details.
Instead, to directly compare configurations across batch sizes, we define episode by its number of shots n, the batch size b and the total number of images per class m + n (the sum of elements across support and query set). For example, if we train a 5-shot model with m +
8 and b 256, its corresponding training episodes will have n = 5, m = 8 n = 3, and w = 256/ [m + n)) = 32. Using this notation, we train configurations of PNs and MNs covering several combinations of these hyperparameters, so that the resulting batch size corresponding to an episode is 128, 256 or 512. Then, train three configurations of the NCA, where the sole hyperparameter is the batch size b.
Results for CIFAR-FS can be found in Fig. where we report results for NCA, PNs and MNs with m + = 8, 16 or 32. Results for minilmageNet observe the same trend and are deferred to Appendix H. For consistency in our comparisons. we evaluate performance using a nearest centroid classifier when comparing against PNs, and soft assignments when comparing against MNs (see Sec. (2.3) Note that PNs and MNs results for 1-shot with m + n = 16 and m + n = 32 are not reported, as they fare significantly worse. The 1-shot m +
16 is 4% worse in the best case compared to the lowest lines in Fig. 2 and the m + n = 32 is 10% worse in the best case. This is likely because these two setups exploit the fewest number of pairs among all the setups, which leads to the least training signal being available. In Appendix E we discuss whether the difference in performance between the different episodic batch setups of Fig. can be solely explained by the differences in the number of distance pairs used in the batch configurations. We indeed find that generally speaking the higher the number of pairs the better. However, one should also consider the positive/negative balance and the number of classes present within a batch.
Several things can be observed from Fig. 2 First, NCA-trained embeddings perform better than all configurations, no matter the batch size. Second, PNs and MNs are very sensitive to different hyperparameter configurations. For instance, with batches of size 128, PNs trained with episodes of 5-shot and n+3 perform worse than a PNs trained with 5-shot episodes and m-n=16 16. Note that, will show in Table 2 the best episodic configurations for PNs and MNs found with this hyperparameter search is superior to the setting used in the original papers.
3.3 Episodic batches random sub-sampling
Despite the inferior performance with respect to the NCA, one might posit that, by training on episodes, PNs and MNs can somehow make better use of a smaller number of distances within a batch. This could be useful, for instance, in situations where it is important to train with very large batches. Given the increased conceptual complexity and the extra hyperparameters, the efficacy of episodic learning (in cases where a smaller number of distances should be considered) should be validated against the much simpler approach of random subsampling. We perform an experiment where we train NCA models by randomly discarding fraction of the total number of distances used in the loss. Then, for comparison, we include different PNs and MNs models, after having computed to which percentage of discarded pairs (in a normal batch) their episodic batches correspond to.
Results can be found in Fig. 3 As expected, we can see how subsampling a fraction of the total available number of pairs within a batch negatively affects performance. More importantly, we can notice that the points representing PNs and MNs models lie very close to the under-sampling version of the NCA. This suggests that the episodic strategy is roughly equivalent, empirically, to only exploiting a fraction of the distances available a batch. Note how, moving along the x-axis of Fig. 3 variants of PNs and MNs exploiting higher number of distances perform better.

---

dec
ma 2004
PMC NUC
18% 28% 42%
5%
16% 48% E25
11
10% 20% 48% Percerage pars eamppe
5% 10% 42% E2N Perceraage of pars tampaed
Perceraage parrs samp Wd
Percertaga of parts samppe
Figure 3: 1-shot and 5-shot accuracies on minilmageNet (val. set) for NCA models trained by only sampling a fraction of the total number of available pairs in the batch (of size 256). Stars and squares represent models trained using the Prototypical or Matching Network loss, and plotted on the x-axis based on the total number of distance pairs exploited in the loss, so that they can be directly compared with this "sub-sampling" version of the NCA. Reported values correspond the mean accuracy of three models trained with different random seeds and shaded areas represent 95% confidence intervals. See Sec. 3.3 for details.
mistinggehe::111 shon
niniknageNor (5
CIRARIFE (1-shen)
CIRAPIFS sho0
NCA (1)
batch baxch size 256 betch size 128
replacement sampling (2)
NCA fixed batch composition (3)
PNs 1-shot (4)
PNs 5-shot
5-shot no proto (8)
PNs 5-shot no S/Q (7)
65%
75%
80%
50%
85%
82%
65%
75%
Figure 4: Ablation experiments on NCA and Prototypical Networks, both on batches (or episodes) of size 128, 256, and 512 on minilmageNet and CIFAR-FS (val. set). Reported values correspond to the mean accuracy of three models trained with different random seeds and error bars represent 95% confidence intervals. See Sec. for details.
3.4 Ablation experiments
To better analyse why NCA performs better, in this section we consider the three key differences discussed at the end of Sec. by performing series of ablations on models trained on batches of size 128, 256 and 512. Results are summarised in Fig. We refer the reader to Appendix obtain detailed steps describing how these ablations affect the losses of Sec. Sec.
First, we compare two variants of the NCA: one in which the sampling of the training batches happens sequentially and without replacement, as is standard in supervised learning, and one where batches are sampled with replacement. This modification (row I and 2 of Fig. 4 has a negligible effect, meaning that the replacement sampling introduced by episodic learning should not interfere with the other ablations. We then perform a series of ablations on episodic batches, i.c. sampled with the method described in Sec. 2. To obtain reasonably perfooomnig model for both 1- and 5-shot models, we use configurations with m + n 8. This means that, for PNs and MNs, models are trained with images per class, and either 16, 32 or 64 classes (batches of size 128, 256 and 512 respectively). The batch size for NCA is also set to either 128, 256, or 512, allowing direct comparison.
The ablations of Fig. compare PNs to NCA. First, we train standard PNs models (row 4 and 5 of Fig. 4 Next, we train a model where "prototypes" are not computed (row 6). This implies that, similar to what happens in MNs, distances are considered between individual points, but a separation between query and support set remains. This ablation allows us investigate if the loss in performance by PNs compared to NCA can be attributed to prototype computation during training (which turned out not to be the case). Then, we perform an ablation where we ignore the separation between support and query set, and compute the NCA on the union of the support and query set, while still computing prototypes for the points that would belong to the support set (row 7). Last, we perform an ablation where we consider all the previous points together: we sample with replacement, we ignore the separation between support and query set and we do not compute prototypes (row 3). This amounts to the NCA loss, except that it is computed on batches with fixed number of classes and fixed number of images per class (row 3). Notice that in Fig. 4 there is only one row dedicated to I-shot models. This is because we cannot generate prototypes from 1-shot models, so we cannot
7

---

CIFAR-FS
BeredlmageNeet
I-shot
S-bbot
I-shot
S-chot
I-shot
S-shot
Eprisodic muchhns
odaRRaNNt F6
56.88 0.62
0.07
TADAM 27]
18.60 0.30
Shon-Fire X3
61.71:
17.642
84.7%
68.124
82.0M4
TEAM 10)
60.07-
75.90+
61.20
75.50
TapNeet
1.1.5
MetnOpeeNttS
0.61
72.0
Variatonal
01:23 0.26
77,00
Simple cromenhhopy boselioes
Trunsdactive finctnning
02
0.56
70.53
70.70 0.74
81.60 0.68
REIC-simpl:
0.50
70.64 0.44
71.5
88.0 =
09.74 0.73
+
Mett-Rasdline
63.17 0.23
78.35 + 0.17
68.02 0.37
63.99 + 0.18
Our inple
[20 opiodes))
58:91 01.12
72.48
00:28
0.13
80.79
66.75 0.13
78.10 = 0.30
PN&
(pisodes)
50:78 01.8::0..2
75.42
0.00
00:04:00 0.12
84.01 1: 0.00
65 06.80-1:::15 0:11
61.20
0.10
MN (our opisodos)
00:17+00.12
73.52
0.00
71 80
0.13
52.41
00.53 0:133
70.06
-..10
PNs (our opisodes)
61.32 0.12
11.77 1 0.09
70:41
0.12
84.61 0.10
08.89 1.1.4
82.20
0.00
SimplesShot
62. 0.12
78.53 0.09
89:08
0.12
$4.40
1 0.08
06.67 0.1.6
81.57
0.10
NCA soft assignment (ours)
82 02.00=22 0.32
76.93
+ 6.11
72449
0.12
83.88
+ 0.09
68.35 0.1.8
81.04 = 01.08
NCA ncarest centroid (ours)
02.05
0.02
78.27
4 0.09
72.49
= 0.12
85.15
0.09
68.35
0.18
83.20 =: 0.30
Table 2: Comparison of methods that use Re as So. on the test set of minil CIFAR-FS, and magredd Values are reported with 95% confidence intervals. For our methods, reported values correspond to the mean accuracy of three models trained with different random seeds.
have a "no proto" ablation. Furthermore, for 1-shot models the "no S/Q" ablation is equivalent to the NCA with a fixed batch composition.
From Fig. 4, we can see that disabling prototypes (row 6) negatively affects the performance of 5-shot (row 5), albeit slightly. Since for PNs the amount of gradient signal is the same with (row 5, Fig.4) or without (row 6, Fig.4) the computation of prototypes, we believe that this could be motivated by the increased misalignment between the training and test setup present in the ablation of row 6. Nonetheless, enabling the computation between all pairs increases the performance (row 7) and, importantly, enabling all the ablations (row 3) completely recovers the performance lost by PNs. Note the meaningful gap in performance between row 1 and 3 in Fig. 4 for batch size 128, which disappears for batch size 512. This is likely due to the number of positives available in an excessively small batch size. Since our vanilla NCA creates batches by simply sampling images randomly from the dataset, there is a limit to how small batch can be (which depends on the number of classes of the dataset). As an example, consider the extreme case of a batch of size 4. For the datasets considered, is very likely that such a batch will contain no positive pairs for some classes. Conversely, the NCA ablation with a fixed batch composition (i.e. with fixed number of images per class) will have a higher number of positive pairs (at the cost of a reduced number of classes per batch). This can explain the difference, positive pairs constitute less frequent (and potentially more informative) training signal. In Appendix E we extend this discussion, commenting on the role of positive and negative distances. In Appendix H we also report the results of second set of ablations to compare NCA and Matching Networks, which are analogous to the ones with Prototypical Networks we just described and lead to the same conclusions.
These experiments highlight that the separation of roles between the images belonging to support and query set, which is typical of episodic learning [8]] is detrimental for the performance of metric-based nonparametric few-shot learners. Instead, using the NCA loss on standard mini-batches allows full exploitation of the training data and significantly improves performance. Moreover, the NCA has the advantage of simplifying the overall training procedure, as the hyperparameters for the creation of episodes {w, n, m) no longer need to be considered.
3.5 Comparison with recent methods
We now evaluate our models on three popular FSL datasets to contextualise their performance with respect to the recent literature. When considering which methods to compare against, we chose those a) which have been recently published, b) that use a ResNet-12 12 architecture [24] (the most commonly
used), and c) with a setup that is not significantly more complicated than ours. For example, we only
report results for the main approach of Tian et al. [44 We omit their self-distillation [16] variant, as
it can be applied to most methods and involves multiple stages of training.
8

---

Results can be found in Table 2 Besides the results for the NCA loss, we also report PNs and MNs results with both the episodic setup from Snell al. 40] and the best one (batch size 512. 5-shot, m + n=16 for both PNs and MNs) found from the experiment of Fig. 2 which brings considerable improvement over the original and other PNs implementations (See Appendix for a comparison of our PNs implementation to other works). Note that our aim is not to improve the state of the art, but rather to shed light on the practice of episodic learning. Nonetheless, our vanilla NCA is competitive and sometimes even superior to recent methods, despite being extremely simple. It fares surprisingly well against methods that use meta-learning (and episodic learning), and also against the performmmn simple baselines based on pre-training with the cross- cross-entropy loss. Moreover, because of the explicit inductive bias that it encodes in terms of relative position in the embedding space of samples from the same class, the NCA loss is useful tool to consider alongside cross-entropy trained baselines.
4 Related work
Bengio et al. 4 and Thrun (43. the general concept of meta-learning is several decades old (for a survey see [47 22 However, in the last few years it has experienced surge in popularity, becoming the most used paradigm for learning from very few examples. Several methods addressing the FSL problem by learning on episodes were proposed. MANN (37 uses Neural Turing Machine [1] to save and access the information useful to meta- learn; Bertinetto et al. [6] and Munkhdalai et al. [25] propose a deep network in which a "teacher" branch is tasked with predicting the parameters of a "student" branch; Matching Networks (48] and Prototypical Networks (40) are two nonparametric methods in which the contributions of different examples in the support set weighted by either LSTM or softmax over the cosine distances for Matching Networks, and a simple average for Prototypical Networks; Ravi and L Larochelle [32) propose instead to use an LSTM to learn the hyperparameters of SGD, while MAML (14] learns to fine-tune an entire deep network by backpropagating through SGD. Despite these works widely differing in nature, they all stress on the importance of organising training in a series of small learning problems (episodes) that are similar to those encountered during inference at test time.
Pioneered by Utgoff [46), Schmidhuber [38 39
In contrast with this trend, handful of papers have recently shown that simple approaches that forego episodes and meta-learning can perform well on FSL benchmarks. These methods all have in common that they pre-train feature extractor with the cross-entropy loss on the "meta-training classes" of the dataset. Then, at test time a classifier is adapted to the support set by weight imprinting (2 [1],
fine-tuning [9. transductive fine-tuning [1]] or logistic regression 4 Wang et al. 44] suggest
performing test-time classification by using the label of the closest centroid to the query image.
Unlike these papers, which propose new methods, we are more focussed on shedding light on the possible causes behind the inefficiency of popular nonparametric few-shot learning algorithms such as Prototypical and Matching Networks.
Despite maintaining a support and a query set, the work of Raghu et al. [31] is similar in spirit to ours, and modifies episodic learning in MAML, showing that performance is almost entirely preserved when only updating the network head during meta-training and meta-testing. In this paper, we focussed on FSL algorithms just as established, and uncovered inefficiencies that not only allow for notable conceptual simplifications, but that also bring significant boost in performance. Two related but different works are the ones of Goldblum et al. [20] and Fei al. [22. The former addresses PNs' poorly representative samples by training on episodic pairs with the same classes (but different instances) and using regularizer enforcing consistency across them. The latter investigates meta- learning methods with parametric base learners, and shows interesting findings on the importance of having tightly clustered classes in feature space, which inspires regularizer that improves non meta-learning models. Bai et al. [3] also show that the episodic strategy in meta-lcarning is inefficient by providing both theoretical and experimental arguments on methods solving a convex optimization problem at the level of the base learner. Similar to us, though via a different analysis, they show that the classic split is inefficient. Chen et al. [ derive a generalisation bound for algorithms with a support/query separation. They do not provide any bounds for methods like NCA, which would be an interesting direction for future work. Triantafillou et al. 45) ignore the query/support separation in order to exploit all the available samples while working in Structured SVM framework. Though the reasoning about batch exploitation is analogous to ours, the scope of the paper is very
different. Finally, two recent meta-learning approaches based on Gaussian Processes 22 41 also
9

---

merge the support and query sets during learning to take full advantage of the available data within each episode.
5 Conclusion
Towards the aim of understanding the reasons behind the poor competitiveness of meta-learning methods with respect simple basclines, in this paper investigate the role of episodes in popular nonparametric few-shot learning methods. We found that their performance highly sensitive to the set of hyperparameters used to sample these episodes. By replacing the Prototypical Networks and Matching Networks losses with the closely related (and non-episodic) Neighbourhood Component Analysis, we were able to ignore these hyperparameters, while improving the few-shot classification accuracy. We found out that the performance discrepancy is in large part caused by the separation between support and query set within each episode. which negatively affects the number of pairwise distances contributing to the loss. Moreover, with nonparametric few-shot approaches, the episodic strategy is almost empirically equivalent randomly discarding portion of the distances available within a batch. Finally, we showed that our variant of the NCA achieves an accuracy on multiple popular FSL benchmarks that competitive with recent methods, making it simple and appealing baseline for future work.
Broader impact. We believe that progress in few-shot learning is important, it significantly impact important problems such as drug discovery and medical imaging. We also recognise that the capability of leveraging very small datasets might constitute threat if deployed for surveillance by authoritarian entities (c.g. by applying to problems such as re-identification and face recognition).
Acknowledgements and Disclosure of Funding
We would like to thank J6ao F. Henriques, Nicholas Lord, John Redford, Stuart Golodetz, Sina Samangooei, and the anonymous reviewers for insightful discussions and helpful suggestions to improve the manuscript and our analysis in general. This work was supported by Five Al Limited.
References
K. Allen, F. Shelhamer, H. Shin, and B. Tenenhaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning, 2019.
[2] H. Altac-Tran, B. Ramsundar, A. S. Pappu, and V. Pande. Low data drug discovery with one-shot lcarning. ACS central science, 2017.
[3] Y. Bai, Chen, P. Zhou, T. Zhao, Lee, Kakade, H. Wang, and C. Xiong. How important is the train-validation split in meta-learning? In International Conference on Machine Learning, 2021.
[4] S. Bengio, Y. Bengio, J. Cloutier, and J. Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks. Univ. of Texas, 1992.
L. Bertinetto, F. Henriques, solvers. In International Conference on Learning Representations, 2019.
Torr, and Vedaldi. Meta-lcarning with differentiable closed-form
[6] L. Bertinetto, J. F. Henriques, J. Valmadre, P. Torr, and A. Vedaldi. Learning feed-forward one-shot learners. In Advances in Neural Information Processing Systems, 2016.
[7] Т. Cao. M. Law,
Fidler. A theoretical analysis of the number of shots in few-shot learning. In International Conference on Learning Representations, 2020.
[8] J. Chen, X.M. Wu, Y. Li, Q. Li, L.-M. Zhan, and F-1. Chung. A closer look at the training strategy for modern meta-lcarning Advances in Neural Information Processing Systems, 2020.
Y. Chen, Y.-C, Liu, Kira, Y.-. F. Wang. and J.-B. Huang. closer look few-shot classification. International Conference on Learning Representations, 2019.
[10] Y. Chen, X. Wang. Z. Liu, H. Xu, and T. Darrell. A new meta- sata-baseline for few-shot learning. arXiv preprint arXiv:2003.00000 04390, 2020.
10

---

[11] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto. A baseline for few-shot image classification. In International Conference Learning Representations, 2020.
[12] N. Fei, Lu, T. Xiang, and S. Huang. Melr: Meta-learing via modeling episode-level relationships for few-shot learning. In International Conference Learning Representations, 2021.
[13] C. Finn. Stanford cs330:
Multi-task and meta-learning, 2019 I lecture 4 non-
parametric meta-lcamers.
https: /www. youtube...b........... www.woom/waaa/eeeee..c bc- bec.bt-tttyYyctisttt>- 2019. Accessed on 26/01/2021.
[14] C. Finn. P. Abbeel, and Levine. Model-agmostic meta-learning for fast adaptation deep networks. In International Conference on Machine Learning, 2017.
[15] N. Frosst, N. Papernot, and Hinton. Analyzing and improving representations with the soft nearest neighbor loss. International Conference on Machine Learning 2019.
[16] T. Furlanello, Z. Lipton, M. Tschannen, L.. Itti, and A. Anandkumar. Born again neural networks. In International Conference on Machine Learning, 2018.
[17] G. Garcfa, R. Del Amor, A. Colomer, R. papillary oct- poc-uoe hybrid learning for glaucoma grading using tailored prototypical neural networks. Artificial Intelligence in Medicine, 1 11: 10212,,,, 2021.
J. Morales Sánchez, and V. Naranjo. Circum-
[18] S. Gidaris, A. Bursuc, N. Komodakis, P. Pérez, and M. Cord. Boosting few-shot visual learning with self-supervision. In IEEE Conference Computer Vision and Puttern Recognition, 2019.
[19] J. Goldberger, G. F. Hinton, S. T. Roweis, and R. R. Salakhutdinov. Neighbourhood components analysis. In Advances in Neural Information Processing Systems, 2005.
[20] M. Goldblum, S. Reich, L. Fowl, R. Ni, V. Cherepanova, and T. Goldstein. Unraveling meta-lcarning: Understanding feature representations for few-shot tasks. In International Conference on Machine Learning. PMLR, 2020.
[21] A. Graves, G. Wayne, and Danihelka. Neural turing machines. arXiv preprint urXiv:1410.5401, 2014.
[22] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-leamning in neural networks: A survey. arXiv preprint arX1v:2004.05339, 2020.
[23] B. M. Lake, Salakhutdinov, and Tenenbaum. Human-level concept leaminng through probabilistic program induction. Science, 2015.
[24] K. Lee. Maji. A. Ravichandran,
Soatto. Meta-learning with differentiable convex optimization. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[25] T. Munkhdalai and H. Yu. Meta networks. International Conference on Machine Learning, 2017.
[26] T. Munkhdalai, Yuan, Mehri, and A. Trischler. Rapid adaptation with conditionally shifted neurons. In International Conference on Machine Learning, 2018
[27] B. Oreshkin, P. R. López, and A. Lacoste. Tadam:
Task dependent adaptive metric for improved few-shot Iearning. In Advances in Neural Information Processing Systems, 2018.
[28] Patacchiola, Turner, F. J. Crowley, M. O' Boyle, and A. Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 2020.
[29] H. Qi, M. Brown, and D. G. Lowe. Low-shot learning with imprinted weights. In IEEE Conference on Computer Vision and Pattern Recognition, 2018
[30] L. Qiao. Y. Shi, J. Li, Y. Wang. T. Huang, and Y. Tian. Transductive episodic-wise adaptive metric for few-shot learning. In IEEE International Conference on Computer Vision, 2019.
[31] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
[32] S. Ravi and H. Larochelle, Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
[33] A. Ravichandran, R. Bhotika, and S. Soatto. Few-shot learning with embedded class models and shot-free meta training. In IEEE International Conference Computer Vision, 2019.
11

---

[34] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations, 2018.
[35] R. Salakhutdinov and G. Hinton. Learning nonlinear embedding by preserving class neighbourhood structure. In Artificial Intelligence and Statistics, 2007.
[36] A. Salekin and N. Russo. Understanding autism: the power of eeg hamnessed by prototypical learning. In Proceedings of the Workshop on Medical Cyber Physical Systems and Internet of Medical Things, pages 12-16, 16, 2021.
[37] A. Santoro. S. Bartunov, M. Botvinick, Wierstra, and Lillicrap. Meta-learning with memory- augmented neural networks. In International Conference on Machine Learning, 2016.
[38] Schmidhuber. Evolutionary principles self-referential learning, or on learning how learn: meta veeta-meta.... hook. PhD thesis, Technische Universität München, 1987.
[39] Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 1992.
[40] Snell. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances Neural Information Processing Systems, 2017.
[41] Snell and R. Zemel. Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes. International Conference on Learning Representations, 2021.
[42] Q Sun, Liu, T.S. Chua, and Schiele. Meta-transfer learning for few-shot learning. IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[43] S. Thrun. Is learning the n-th thing any easier than learning the first? In Advances in Neural Information Processing Systems, 1996.
[44] Y. Tian, Y. Wang. D. Krishnan, J. B. Tenenbaum, and Isola. Rethinking few-shot image classification: a good embedding is all you need? European Conference on Computer Vision, 2020.
[45] F. Triantafillou, R. S. Zemel, and R. Urtasun. Few-shot learning through an information retricval lens. In Advances in Neural Information Processing Systems, 2017.
[46] P. E. Utgoff. Shift of bias for inductive concept learning. Machine learning: An artificial intelligence approach, 1986.
[47] R. Vilalta and Y. Drissi. A perspective view and survey of meta-learning, Artificial Intelligence Review, 2002.
[48] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
[49] Y. Wang, W.-L. Chao, K. Q. Weinberger, and L. van der Maaten. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning. arXiv preprint 1911v:9111.04633 2019.
[50] S. W. Yoon, J. Sco, and J. Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In International Conference on Machine Learning, 2019.
[51] J. Zhang, C. Zhao, B. Ni, M. Xu, and X. Yang. Variational few-shot learning. In IEEE International Conference on Computer Vision, 2019.
12


Reviews:
[{"review_id": "eE1tIxnuqy8", "reviewer": "Reviewer_sPyu", "summary": "This paper studies the question of whether episodes (using a split of support & query sets) are necessary for non-parameteric approaches for few-shot learning (examples include Prototypical Networks & Matching Networks). The authors propose a Neighborhood Component Analysis-based method for few-shot learning, where a mini-batch consists of examples from a subset of base classes, with no support or query split. The NCA loss then involves distances computed across all examples in the mini-batch rather than only using distances computed between support and query examples (as is done in Prototypical Networks & Matching Networks). The authors show that their proposed method is able to achieve better performance for 1-shot & 5-shot classification in miniImageNet, cifar-fs, and tieredImageNet benchmarks. They also speculate that the proposed method performs better because of its use of more distance computations in the loss compared to Prototypical Networks & Matching Networks and confirm this by conducting an experiments where they randomly drop distances in their NCA loss and showing this has a negative impact on performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: to my knowledge an NCA-type loss has not been considered for few-shot learning and the analysis comparing the differences between this loss and Prototypical Networks & Matching Networks loss is also original and interesting.\n\nQuality: submission seems technically sound in terms of the analysis comparing their proposal to existing work. The experiments were also very useful, as they go beyond raw metrics and studied interested aspects of their method that show why it may be working better compared to previous work.\n\nClarity: the paper is very well-written - both description of the method(s) and the experiments are easy to understand and described well.\n\nSignificance: this work (along with other recent work showing that pre-training without any notion of episodes) could be very significant as they're questioning the benefit of more complicated episodic sampling-based approaches to few-shot learning. I do have some minor concerns/questions which could be clarified a bit further and help strengthen the paper.\n\n1. It would be useful to include the results on metrics for Prototypical Networks on ResNet12 from previous work. The authors include results from their own implementation of Proto-Nets but I think including results from previous work is helpful to know that any implementation-specific choices by the authors in their recreation did not affect the metrics negatively.\n2. Based on results of Figure 4, if NCA fixed-batch composition is always strictly better than the default NCA, why not treat fixed-batch composition as the default model? Is it because the default NCA is technically simpler since creating a batch just involves iterating through the data and the benefit only applies to small batches?\f", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "XMRHmas4my5", "reviewer": "Reviewer_XK2J", "summary": "This paper conducts a case study for the non-parametric few-shot classification methods (e.g. Prototypical Networks). \nIt proposes to utilize the classic Neighbourhood Component Analysis (NCA) sampling instead of the original matching or prototypical style episode sampling. \nThe authors conducted ablation experiments to investigate the properties of this new sampling and compare it with the basic Prototypical Networks (PNs) method. \nThe final accuracy is comparable with the recent methods on three benchmark datasets. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Recently, there are several papers focusing on whether meta-learning or episode training is beneficial in few-shot classification, e.g [8,17,27,38]. These papers discuss the necessity of meta-learning and some alternatives such as regularization, pre-training, feature reuse. In this situation, the new findings and overall novelty of this paper are limited. \n\nMoreover, in both the technical and experimental parts, there is a lack of thorough comparison with existing studies on the episode or meta-learning mechanisms. \nMost of the ablation studies are against the PNs method, which is relatively poor. In Table 2, some related studies such as [17] are missing. \n\nin [38], the authors proposed a simple pre-training + classifier learning baseline. The proposed NCA method seems to underperform this simple method, which weakens the significance of proposing such a new sampling strategy. \n\nIn Table 1, an extra number of gradients are required by NCA. It is unclear if this method will be less efficient than PNs. \n\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nAfter Rebuttal. \n\nAfter the rebuttal and discussion with other reviewers, I would like to increase my score to 5. But I still have some concerns about the limited scope and lack of comparisons. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation is the lack of comparison and analysis with related work that also studies the effect of episode training. \nIn Table 2, there is a typo \"MetaOptNet-SVM[21]\". \n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "RsdJOxVIS4e", "reviewer": "Reviewer_GQUE", "summary": "This paper studies the role of the popular episodic training paradigm, in the context of two metric learning-based episodic models: Prototypical Networks (PN) and Matching Networks (MN). They show that these popular methods underperform compared to the closely-related NCA model which is non-episodic, i.e. does not separate the examples sampled in each training batch into disjoint support and query sets. They argue that the superior performance of NCA is because, due to not performing that separation, the total number of pairwise comparisons that are used in the loss computation is larger than those used for PN/MN, making the gradients more informative. Indeed, in episodic models, each query example is only compared to support examples, but not to other query examples, resulting in significantly fewer comparisons. Experimentally, they show that for a fixed batch size (where the ‘batch size’ in the case of the episodes is given by the combined size of the support and query sets), NCA outperforms PN and MN, despite its simplicity in terms of its smaller number of hyperparameters. They also show that randomly discarding comparisons from NCA leads to similar performance to the analogous PN/MN models and perform a set of ablations to “bridge” the gap between PN and NCA, further strengthening their finding that support/query separation hurts performance.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "High-level review\n-----------------------\nThe paper is really well-written and an enjoyable read. Despite the findings not being very surprising to me, the inefficiency with which episodic models exploit their training data compared to non-episodic baselines has not been formally established, and it’s nice to have the empirical evidence shown in the paper to support that intuition. It is also a nice contribution to show that the NCA model can perform well for few-shot classification, and can serve as a strong baseline alongside the simple cross-entropy classifier baselines that are gaining traction. However, the study of this work is limited to only a couple settings for the shot and way (1-shot 5-way and 5-shot 5-way), to simple benchmarks, and only two episodic methods. It is not clear if the same conclusions generalize to other models and to harder settings, e.g. varying shots at test time, and harder generalization problems to classes that are dissimilar to the training set (e.g. coming from different datasets). These aspects limit the impact of the paper in my opinion and make it hard to see what the take-away is. How should few-shot learning researchers change their approach based on this finding? PN/MN are no longer state-of-the-art models, and recent work these days is moving away from (exclusively, at least) relying on episodic training for few-shot learning (e.g. [1,2,3,4,5,6,7,8] to name a few). In some of these works, episodic training is still used for certain components of the network (e.g hypernets that learn to condition a shared network for each task by producing FiLM parameters for instance, which are naturally defined in an episodic manner), or used to fine-tune a pre-trained network, but the representation is pre-trained for a non-episodic objective. For the above reasons, I'm slightly leaning towards rejection of this paper at this stage. Please see below for more detailed comments.\n\nDetailed comments\n--------------------------\n- Based on the observation that PN/MN perform significantly fewer comparisons than NCA, they underperform NCA in the settings considered and the authors conclude that NCA is preferable in those settings. However, one could argue instead that, due to this observation, the episodic models may require a significantly larger batch size to reach their potential (to increase their number of comparisons that contribute to each update), and so perhaps their poor performance is due to that hyperparameter (the batch size) selected inappropriately for these methods. Of course very large batch sizes can be challenging from an implementation standpoint (e.g. due to memory issues etc) but, to reliably explore the potential of these methods, these challenges can be overcome by carefully parallelizing across GPUs for example (or training with a meta-batch of episodes which is actually common practice). A notable example from the recent literature that yields very impressive performance but requires very large batches to do so is SimCLR [9] (it was trained with a batch size >4K). Perhaps “scaling up” episodic methods in this fashion will result in surpassing the (analogously scaled up) NCA method, so in my view the jury is still out on this comparison.\n\n- This paper only focuses on the disadvantages of episodic models and advantages of NCA. It would also be useful to discuss episodic models’ merits. A potential advantage of PN, for example, is that we can modify its training (specifically, through the choice of the training shot hyperparameter), to specialize it to the shot setting of interest. [10] formally shows that the choice of the ‘shot’ hyperparameter during training impacts the properties of the learned embedding space with smaller shots during training focusing on minimizing intra-class variance (pulling examples of the same class together) while larger training shots on maximizing inter-class variance (pushing different classes apart). As a consequence, models with smaller training shots are more appropriate for small-shot test-time tasks, and analogously for larger-shot models. NCA on the other hand does not (explicitly at least) offer that “knob” to tune at training time in order to control the desired properties of the resulting embedding space. Can we theoretically or empirically study whether NCA-learned models are capable of performing well on tasks of a wide range of shots at test time, when using the nearest-centroid classification as the test-time algorithm? The paper only considers 1-shot and 5-shot evaluation settings which are both very small shots.\n\n- An interesting data point to include in the discussion from the gradient-based meta-learning world is that, while MAML [11] uses a support and query separation, its first-order Reptile variant [12] does not. Is there a high-level take-away that connects the relationship of Reptile to MAML and that of PN (or MN) to NCA?\n\n- Related work: [13] should be included in the related work in the context of few-shot learning models that do not require a support and query separation.\n\n- Recently, there have been some theoretical papers that investigate the role of the support/query framework from different perspectives [14,15]. It would be useful to include these in the discussion and consolidate the empirical observations made here with the theoretical findings from those papers.\n\n- In the ablations, what is the PN model where \"prototypes are not computed\"? PN requires prototypes, so omitting their computation is not an ablation of this model; it is a different model. Is this exactly the same as MN in this case? If so, I would just refer to it as MN.\n\n- In the original PN paper they showed empirically that it was beneficial to \"match\" the shot used at training and test times. The results here contradict this, since PN 5-shot is significantly better than PN 1-shot, for both 1-shot and 5-shot evaluation settings, across datasets. Any thoughts on why that might be?\n\n- In Table 2, in the “simple cross-entropy baselines” section, the Classifier-Baseline [5] and Baseline++ [2] should also be included.\n\nMinor comments\n-----------------------\n- Line 55, in the query set definition, in the first tuple, the first label is y_1’. Should it just be y_1 (without the ‘)?\n- Line  87 “the number of training distances that are lost” → “the number of training comparisons that are omitted”? The first phrase seems a little odd.\n- When explaining the “Nearest Centroid” method (line 150): “This is the approach used by Prototypical Networks and SimpleShot”. It also is the approach used by Classifier-Baseline and Meta-Baseline [5].\n\nReferences\n----------------\n- [1] Meta-learning with Latent Embedding Optimization. Rusu et al. ICLR 2019.\n- [2] A closer look at few-shot classification. Chen et al. ICLR 2019.\n- [3] Rethinking few-shot classification: A good embedding is all you need? Tian et al. 2020.\n- [4] TADAM: Task dependent adaptive metric for improved few-shot learning. Oreshkin et al. NeurIPS 2018.\n- [5] A New Meta-Baseline for Few-Shot Learning. Chen et al. 2020.\n- [6] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al. NeurIPS 2019.\n- [7] Improved Few-Shot Visual Classification. Bateni et al. 2020\n- [8] Learning a Universal Template for Few-shot Dataset Generalization. Triantafillou et al. ICML 2021.\n- [9] A Simple Framework for Contrastive Learning of Visual Representations. Chen et al. ICML 2020.\n- [10] A Theoretical Analysis of the Number of Shots in Few-shot Learning. Cao et al. ICLR 2020.\n- [11] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Finn et al. ICML 2017.\n- [12] On First-order Meta-learning Algorithms. Nichol et al. 2018.\n- [13] Few-shot Learning through an Information Retrieval Lens. Triantafillou et al. NeurIPS 2017.\n- [14] How Important is the Train-Validation Split in Meta-Learning? Bai et al. 2021.\n- [15] A Closer Look at the Training Strategy for Modern Meta-Learning. Chen et al. NeurIPS 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I believe they have addressed this adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6ek9uGnl58V", "reviewer": "Reviewer_1YGQ", "summary": "This paper presents a new perspective on the episodic learning of nonparametric few-shot learning methods. The main claim is that current popular nonparametric methods, such as Prototypical Networks (PNs) and Matching Networks (MNs), are not data-efficient because less gradient signal is propagated during training due to the artificial division of the data points to support and query sets. The authors instead propose to use the standard learning protocol with batches and an equivalent loss function based on the Neighbourhood Component Analysis (NCA). This loss function exploits all connections between data points in the batch. The authors then propose three techniques to perform a few-shot classification during evaluation based on k-NN, nearest centroid, and soft assignment. Through extensive experiments, the authors justify their perspective and show comparable results to other recent FSL methods (not necessarily nonparametric ones). ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall it is an interesting paper, and I think it should be accepted.  \n\nStrengths:\n- The claim that the hyperparameters controlling the episode creation can significantly affect the performance makes a lot of sense.  \n- The paper presents extensive empirical justifications for the main claims presented in it.  \n- Despite the method's simplicity it shows good results on relevant benchmarks.  \n- The paper is written clearly and easy to understand.  \n- The results in the paper are easily reproducible. The code is very organized and full experimental details were given for all experiments.  \n\nSome comments:  \n- Although NCA is compared against relatively recent baselines and it is not delivered as a new SoTA, it seems that there is a noticeable gap between its performance and more recent baselines (e.g., [1, 2]). Furthermore, this paper scope is limited only to nonparametric metric-based learners. Therefore, I think the authors should add a discussion on how to extend their approach beyond the current scope and how it can be combined with more recent approaches.  \n- I think that the authors should have addressed other types of non-parametric few-shot learners. For example, recently two approaches based on Gaussian processes were proposed [3, 4]. Although these methods use the standard episodic learning, they ignore the common support/query split (as was done in the ablation study in this paper).  \n- In Table 2 -> CIFAR-FS -> 5-shot, RFIC-simple achieves the highest accuracy and therefore should be in bold.\n\nGeneral wonderings:  \n- Line 175 states \"Each model is evaluated on 10,000 episodes sampled from the validation or the test set\". Does this mean that you didn't use the pre-defined validation set for hyper-parameter selection and early stopping? If so, how did you perform these operations?  Is it consistent with the results reported for the baseline methods in Table 2?\n- When comparing Fig. 3 in the main text with Fig. 1 in the Appendix it seems that using 80% of the pairs yield similar results to those obtained with all the pairs. Is it true? If so, how can you justify this?  \n- Following the previous point, did you try other (more advanced) techniques for pair selection? As a motivation, it may allow achieving similar accuracy while reducing the computational burden required by the denominator of the NCM loss.  \n\n\n[1] Rizve, M. N., Khan, S., Khan, F. S., & Shah, M. (2021). Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10836-10846).  \n[2] Zhen, X., Du, Y., Xiong, H., Qiu, Q., Snoek, C. G., & Shao, L. (2020). Learning to learn variational semantic memory. arXiv preprint arXiv:2010.10341.  \n[3] Patacchiola, M., Turner, J., Crowley, E. J., O'Boyle, M., & Storkey, A. J. (2020). Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels. Advances in Neural Information Processing Systems, 33.  \n[4] Snell, J., & Zemel, R. (2020, September). Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes. In International Conference on Learning Representations.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the potential limitations and negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is currently very borderline with respect to reviewer opinions. The paper is well-written, with a straightforward premise and thorough experiments. The main outstanding concern is the narrow scoping: PN/MN are no longer considered the state of the art in the field, and the narrow focus of the paper makes it unclear how well the findings will generalize to other forms of episodic learning. On the other hand, these are foundational models and are still used in many contexts. The heart of the debate is whether this will help move the field forward. I think that the NCA baseline is simple and effective enough that the answer leans towards “yes.” In particular, if this was published several years ago, I think it is likely that NCA itself could have become a foundational approach to few-shot learning - especially since it requires less tuning.

I think that the relationship between performance, and the number of pairwise distances per batch is compelling. I would like to see the following experiment (which I didn’t find in the current paper): take PN and/or MN and for each episode, randomize the episode several times. That is, take the set of support and query points, compute the loss, shuffle the batch (assigning different query/support points within the batch), and then compute the loss again. Take the loss of the batch to be the average of these, and then take a gradient step. This effectively increases the number of comparisons used in the batch, but in a way that PN/MN can exploit. Based on the hypothesis in the paper, I expect the PN/MN results to improve, but it will be interesting either way.

Decision: Accept (Poster)