On Episodes, Prototypical Networks, and Few-Shot Learning
Authors: Steinar Laenen
Luca Bertinetto
Venue:  (2021)
Keywords: few-shot learning
meta-learning
metric learning
deep learning

Abstract: Episodic learning is a popular practice among researchers and practitioners interested in few-shot learning. It consists of organising training in a series of learning problems, each relying on small “support” and “query” sets to mimic the few-shot circumstances encountered during evaluation.
In this paper, we investigate the usefulness of episodic learning in Prototypical Networks, one of the most popular algorithms making use of this practice.
Surprisingly, in our experiments we found that, for Prototypical Networks, it is detrimental to use the episodic learning strategy of separating training samples between support and query set, as it is
a data-inefficient way to exploit training batches. This “non-episodic” version of Prototypical Networks, which corresponds to the classic Neighbourhood Component Analysis, reliably improves over its episodic counterpart in multiple datasets, achieving an accuracy that is competitive with the state-of-the-art, despite being extremely simple.

Under review as a conference paper at ICLR 2021
ON EPISODES, PROTOTYPICAL NETWORKS, AND FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Episodic learning is popular practice among researchers and practitioners inter- ested in few-shot learning. It consists of organising training in a series of learning problems, each relying on small "support" and "query" sets to mimic the few-shot circumstances encountered during evaluation. In this paper, we investigate the usefulness of episodic learning in Prototypical Networks, one of the most popu- lar algorithms making use of this practice. Surprisingly, in our experiments we found that, for Prototypical Networks, ing strategy of separating training samples between support and query set, as it is a data-ineffieient way to exploit training batches. This "non-cpisodic" version of Prototypical Networks, which corresponds to the classic Neighbourhood Compo- nent Analysis, reliably improves over its episodic counterpart in multiple datasets, achieving an accuracy that is competitive with the state-of of-the-art, despite being extremely simple.
detrimental to use the episodic learn-
INTRODUCTION
The problem of few-shot learning (FSL) classifying examples from previously unseen classes given only a handful of training data has considerably grown in popularity within the machine learning community the last few years. The reason is likely twofold. First, being able to perform well on FSL problems is important for several applications, from learning new characters (Lake) al. 2015) drug discovery (Altae-Trml Tran al., 2017). Second, since the aim of researchers in- terested in meta-learning is to design systems that can quickly learn novel concepts by generalising from previously encountered learning tasks, FSL benchmarks are often adopted as a practical way to empirically validate meta-learning algorithms.
To the best of our knowledge, there is not a widely recognised definition of meta-learning. In recent survey, Hospedales et al. (2020) informally describe it as "the process of improving learning algorithm over multiple learning episodes". Several popular papers in the FSL community (e.g. |Vinyals et al. (2016); Ravi & Larochelle (2017); Finn et al. (017);; [nell et al. (2017)) have emphasised the importance of organising training into episodes, i.e. learning problems with a limited amount of training and (pseudo-)tes, examples that resemble the test-time scenario. This popularity has reached such point that an "episodic" data-loader is often at the core of new FSL algorithms, practice facilitated by frameworks such as Deleu et al. 2019) and Grefenstette al. (2019).
Despite the considerable strides made in FSL over the past few years, several recent works (e.g. Chen al. (2019); Wang al. (2019); Dhillon al. (2020); Tian al. (2220) showed that simple base- lines can outperform established meta-learning methods by using embeddings pre-trained with stan- dard classification losses. These results have cast a doubt in the FSL community on the usefulness of meta-learning and its pervasive episodes. Inspired by these results, we aim at understanding the practical usefulness of episodic learning in arguably the simplest method which makes use of it: Prototypical Networks (Snell et al. (017). We chose to analyse Prototypical Networks not only for their simplicity, but also because they often appear as important building blocks of newly-proposed methods (e.g. Oreshkin al. (2018); Cao et al. (000);; Gidaris al. (2019): Yoon et al. (2019)).
Cheg
With a set of ablative experiments, we show that for Prototypical Networks episodic learning a) is detrimental for performance, b) is analogous to randomly discarding examples from batch and c) it introduces a set of unnecessary syper-parameters that require careful tuning. We also show how, without episodic learning, Prototypical Networks are connected to the classic Neighbourhood

---

Under review as a conference paper at ICLR 2021
Component Analysis (NCA) (Goldberger al. 2005 Salakhutdinov & Hinton 2007 deep em- beddings. Without bells and whistles, our implementation of the NCA loss achieves an accuracy that is competitive with the state-of-the-art on multiple FSL benchmarks; minilmageNet, CIFAR-FS and figred
2 RELATED WORK
1992). Bengio al. (1992) Pioneered by the seminal work of Utgoff 1986), Schmidhuber [1987
and Thrun (1996), the general concept of meta-learning is several decades old (for a survey see Vi- lalta & Drissi (2002) Hospedales et al. (020)). However, in the last few years it has experienced surge in popularity, becoming the most used paradigm for learning from very few examples. Sev- eral methods addressing the FSL problem by learning on episodes were proposed. MANN (Santoro) itt al. 2016) uses a Neural Turing Machine (Graves et al. (014)) to save and access the information useful meta-learn; [Berinetto et al. (0016) propose a deep network in which a "teacher" branch is tasked with predicting the parameters of "student" branch; Matching Networks (Vinyals et al.) 2016) and Prototypical Networks (Snell et al., 2017) are two non non-parametric methods in which the contributions of different examples in the support set are weighted by either LSTM or softmax over the cosine distances for Matching Networks, and a simple average for Prototypical Networks; Ravi & Larochelle| (2017) propose instead use LSTM to learn the aaperparameters of SGD, while MAML (Finn et al. 2017) learns to fine-tune an entire deep network by backpropagating through SGD. Despite these works widely differing in nature, they all stress the importance of organising training in a series of small learning problems (episodes) that are similar to those encoun- tered during inference at test time.
In contrast with this trend, a handful of papers have recently shown that simple approaches that forego episodes and meta-learning can perform well on FSL benchmarks. These methods all have in common that they pre-train a feature extractor with the cross-entropy loss on the "meta-training classes" of the dataset. Then, at test time a classifier is adapted to the support set by weight im- printing (Qi et al., 2018 Dhillon et al., (020). fine-tuning (Chen et al. (2019), transductive fine- tniing (Dhillon et al., 2020) or logistic regression (Tian et al. 2020). Wang et al. (2019) suggest performing test-time classification by using the label of the closest centroid to the query image.
Different from these papers, we try to shed some light on one of the possible causes behind the poor performance of episodic-based algorithms like Prototypical Networks. An analysis similar to ours in spirit the one of |Rghu et al. (2020). After showing that the efficacy of MAML in FSL is due to the adaptation of the final layer and the "reuse" of the features of previous layers, they propose a variant with the same accuracy and computational advantages. In this paper, we focus on an FSL algorithm just as popular and uncover inefficiencies that allow for notable conceptual simplification of Prototypical Networks, which surprisingly also brings a significant boost in performance.
3 BACKGROUND AND METHOD
This section is divided as follows: Sec. 3.1 introduces episodic learning and the formalism used in FSL, Sec. 3.2 reviews Prototypical Networks (often referred to as PNs from now on), Sec 3.3 describes the classic NCA loss and how exactly it relates to PNs, and Sec. 3.4 explains the three options we explored to perform FSL. classification with an NCA-trained feature embedding.
3.1 EPISODIC LEARNING
A common strategy to train few-shot learning algorithms is to consider a distribution & over possible subsets of labels that is close as possible the one encountered during evaluation & D. Each episodic batch Bg {S. Q} is obtained by first sampling subset of labels L from ε, and then sampling images constituting both support set S and query set Q from the set of images with labels in L, where S ((s), 91)..,... ...,, (Sn, yn)); Q = ((q1.1)) ...,..... (qm, ym)), and S₁ and Qx denote the sets of images with label y k in the support set and query set respectively. In Maximum Likelihood Estimation framework. training on these episodes can be written (Vinyals et al. 2016 as
Note that, in FSL, the sets of classes of training and evaluation are disjoint.
2

---

Under review as a conference paper at ICLR 2021
S
Figure 1: Batch exploitation during training for Prototypical Networks (Snell al. 2017 (left) Neigh- borhood Component Analysis Goldherger et al. 2005) (right) on toy few-shot problem of "ways", "shots" and "query" per class. By dividing batches between support set S and query set Q. Prototypical Networks disregards many of the distances between labelled examples that would constitute useful training signal. Details in Sec. 3.3
the optimisation
arg max E LINE ES 0
Σ (q.:))EE
log Po (y (ull S)
In most implementations this corresponds to training on series of mini-batches in which each image belongs either to the support or the query set. Support and query sets are constructed such that they both contain all the classes of L, and a constant number of images per class. Therefore, episodes are characterised by three values: the number of classes w = [L] (the "ways"), the number [Sa] (the "shots"), and the number of examples per class in the query set m = [Qxl- Importantly, during evaluation the triplet {w, n, m) defines problem setup. Usually the triplet remains unchanged across methods, although the benchmark of Triantafiiloo et al. (2019) evaluates on episodes with a variable number of ways and shots. At training time the triplet (w, n. m) be batch creation, and that (as we will see Sec. 4.2) requires careful tuning.
of examples per class in the support set
of hyper-parameters controlling the
3.2 PROTOTYPICAL NETWORKS (PNs)
Prototypical Networks (Snell al. 2017) are one of the most popular and effective approaches in the few-shot learning literature, and they are at the core of many newly proposed methods (e.g. Oreshkin
al. 2018); Gidaris et al. (2019) Allen al. (2019); Yoon al. (2019); Cao et al. (2020).
During training, episodes constituted by support set S and query set Q are sampled as described in Sec. 3.1 Then, prototype for each class k computed the mean embedding of the samples from the support set belonging to that class: C₁ = (1/[Sa]) Eie..ness [o()), where fo is a deep neural networks with parameters 0 learned via Eq.
Let C {(c1, 91), (Ck, }1:) be the set of prototypes and corresponding labels. The loss can be written as follows:
(S Q)
log QI (q.
1
Σ
exp exp-I1.fl (ip-ll.a(g))) - -
(2)
Σx exp ex.f cx2 1|
Here, k an index that goes over all classes. This loss minimised over a number of training episodes. After training. given query image x; from new test episode, classification is ducted by simply consulting the nearost-neighboring prototype computed from the support set the episode, i.e. y(x:) arg min Ilfo (x,) Cyll.
3.3 NEIGHBOURHOOD COMPONENT ANALYSTS (NCA)
Eq. 2 computes the likelihood that query image belongs to the class a certain prototype is represen- tative of by computing the softmax over the distances to all prototypes. This formulation is similar to
3

---

Under review as a conference paper at ICLR 2021
positives
negatives
PNs
w(w - 1)mn
NCA
(m""")
(m + n)²
Table 1: Number gradients from positive and negative distance pairs contributing the loss within batch for NCA and PNs. Respectively, w, n and m represent the number of ways, shots and queries. In Appendix A.8 we show that the extra number of pairs NCA can exploit grows as Ow n²)).
the Neighbourhood Component Analysis approach by Goldberger al. (2005) (and expanded the non non-linear case by Salakhutdinov & Hinton (2007)), except for few important differences which we will now discuss.
Let i E [1,b] b] be the indices of the images within a batch B. The NCA loss can be written as:
Σ
exp
- 22 Zj
jEI....bb
jti
(X) = iEl.....b
1
Σ log
yi
(3)
Σ kEI...., liphi
exp
12 2x
where z; = ff (x:) is an image embedding and yi its corresponding label. By minimising this loss, distances between embeddings from the same class will be minimised, while distances between embeddings from different classes will be maximised. This bears similarities to the (supervised) contrastive loss (Khosla et al., 2020: Chen et al., 2020a), which we discuss in Appendix AA.3 For case of discussion, we refer to the distances between pairs of embeddings from the same class as positives, and to the distances between pairs of embeddings from different classes as negatives.
Importantly, the concepts of support set and query set of Sec. 3.1 and 3.2 here do not apply. More simply. the images (and respective labels) constituting the batch B [(x1, (1x1,8)...... (Xs, yb}} are sampled i.i.d. from the training dataset, as normally happens in standard supervised learning. Con- sidering that in PNs there is no parameter adaptation happening at the level of each episodic batch Bg, S and Q do not have a functional role in the algorithm, and their defining values (w, m, n) can be interpreted as prperpparaeetess controlling the data-loader during training. More specifically, PNs differ from NCA in three key aspects:
1. PNs rely on the creation of prototypes, while NCA does not.
2. Due to the nature of episodic learning, PNs only consider pairwise distances between the query and the support set; the NCA instead uses all the distances within batch and treats each example in the same way.
3. Because of how L, and Q are sampled episodic learning, some images will be in- evitably sampled more frequently than others, and some will likely never be seen during training (this corresponds to sampling "with replacement"). NCA instead visits every im- age of the dataset once and only once within each epoch (sampling "without replacement")
Notes on data-efficiency in batch exploitation. To expand on point 2 above, Fig. I illustrates the difference in batch exploitation. For PNs (left) and training episode of w=3 ways, n=3 shots and m=11 queries, the total number distances contributing to the loss consists of ummn 9 positives and w(w - 1)mnn episodic way, when computing the NCA loss (Fig. right) we have (m-+") "" = 18 positives and (m) + n² = 48 negatives (we summarise these equations in Table This difference in batch exploitation significant; Appendix A.8 we show that it grows exponentially as O(w2(m2² +n²))-
negatives P If we consider
same training batch in non-
To investigate the effect of the three key differences between PNs and NCA illustrated in this sec- tion, in Sec. we conduct a wide range of experiments.
'The number of distances computed in the batch is actually even smaller for PNs: wm-3 positives and w(w m) negatives. However, since gradients are propagated through prototypes, the distances to the individual support points be considered as contributing the loss. This what we do throughout the paper.
4

---

Under review as a conference paper at ICLR 2021
3.4 FEW-SHOT CLASSIFICATION DURING EVALUATION
Once fo has been trained, there are many possible ways to perform few-shot classification during evaluation. In this paper we consider three simple approaches that are particularly aligned for em- beddings learned via metric-based losses such as Eq. 2 or Eq.
k-NN. To classify an image qi E Q we first compute the Euclidean distance to each support point Then, we simply assign y(q:) to be majority label of the nearest neighbours. A downside here is that k is hyper-parameter that has to be chosen, although a reasonable choice in the FSL setup is to set it equal to the number "shots" n.
Sj E S: di3 = [q:) - fa(s)
1-NN with Class Centroids. Similar k-NN, of the closest class centroid, i.e. y(q: = arg min je je k.....)) [x(;,) -aae((x))cc)) - -cyll This is the approach used at test-time by Snell et al. (017) and Wang et al. (2019).
can perform classification by inheriting the label
Soft Assignments. This is what the original NCA paper (Goldberger et al. 2005) used for evaluation. To classify an image qi E Q. we compute the values Pij = fo(s;))11//> Es exp(-Ilfo(e) age is sampled from image We then compute the likelihood for each class k: Σ,,es, Pij. and choose the class with the highest likelihood y(q:) arg max, E., ES, Piy- This approach is the only one closely aligned with the training procedure, and has direct probabilistic interpretation.
(sx))l²) for all 8j S. which is the probability that im-
We also experimented with using the NCA loss on the support set to perform adaptation at test time, which we discuss in Appendix A.I
4 EXPERIMENTS
In the following, Sec. 4.1 describes our experimental setup. Sec. shows the effect of the hyper- parameters controlling the creation of episodes PNs. In Sec. we perform a set of ablation studies to better illustrate the relationship between PNs and the NCA. Finally, in Sec. 4.4 we com- pare our version the NCA to several recent methods on three FSL benchmarks.
4.1 EXPERIMENTAL SETUP
We conduct our experiments on minilmageNeet ( (Vinyals et al, 2016), CIFAR-FS (Bertinetto et al. 2019) and tieredImageNet (Ren al. 2018), using the R variant first adopted by Lee et al. 2019 as embedding function Jo- A detailed description of benchmarks, architecture and imple- mentation details deferred Appendix A.4 while below we discuss the most important choices of the experimental setup.
Like Wang et al. (2019), for all our experiments (including those with Prototypical Networks) centre and normalise the feature embeddings before performing classification, as it is considerably beneficial for performance. After training, we compute the mean feature vectors of all the images in the training set: x = |arrain| ExEDiTain X. Then, all feature vectors in the test set are updated as X; <- x; x, and normalised by x; ||
1
We compared the inference methods discussed in Sec. 3.4 on minilmageNeet and CIFAR-FS. Results can be found in Table We chose to use 1-NN with class centroids in all our experiments, as it per- forms significantly better than k--N or Soft Assignment. This might sound surprising, as the Soft Assignment approach closely reflects the training protocol and outputs class probabilities. We spec- ulate its inferior performance could be caused by poor model calibration (Guo et al.) 2017): since the classes between training and evaluation are disjoint, the model is unlikely to produce calibrated probabilities. As such, within the softmax, outliers behaving as false positives can happen to highly influence the final decision, and those behaving false negatives pletely ignored (their contribution is squashed toward zero). With the nearest centroid classification approach outliers still clearly an issue, but their effect can be less dramatic.
end up being almost com-
As standard, performance is assessed on episodes of 5-way, 15-query and either 1- or 5-shot. Each model is evaluated on 10,000 episodes sampled from the test set (or the validation set, in some
5

---

Under review as a conference paper at ICLR 2021
miniilaageNet
method
5-shot val
5-shot test
Soft Assignment
79.11 4: 0.27
77.16 = 0.10
K-NN
75.82 + 0.21
73.52 + 0.12
1-NN centroid
80.61 + 0.20
78.30 + 0.14
CIFAR-FS
Soft Assignment
76.12 0.32
83.31 = 0.37
I-NN
73.46 0.39
80.94 = 0.38
1-NN centroid
77.80 0.35
85.13 1 0.32
Table 2: Comparison between the different evaluation methods discussed in Sec. 3.4
CIFAR-FS
CIFAR-FS
615
728
20.5
64.
XCA
Prrsozate botett and
Balch
Baich
Figure 2: 1-shot (left) and 5-shot accuracies (right) the val. set of CIFAR-FS, Models trained using NCA, or Proto-nets with different configurations: 1-shot with a=8 and 5-shot with a=8, 16 or 32. Values correspond the mean accuracy of five models trained with different random seeds. Please see Sec. 4.2 for details.
experiments). To further reduce the variance, we trained each model five times with five different random seeds, for a total of 50,000 episodes per configuration, from which error bars are computed.
4.2 CONSIDERATIONS ON EPISODES AND DATA EFFICIENCY
Despite Prototypical Networks being one of the simplest FSL methods, the creation of episodes requires the use of several hyper-parangn parameters (fw, m, n}. Sec. 3.1 which can significantly affect performance. (Snell et al. (2017) state that the number of shots n between training and testing should match and that one should use a higher number of ways w during training time. In their experiments, they train 1-shot models with w 30, n = 1, m 15 and 5-shot models with w 20, n 5, m = 15. This makes the corresponding batch sizes of these episodes 480 and 400, respectively. Since, as seen in Sec. 3.3 the number of positives and negatives grows rapidly for both PNs and NCA (although at different rate), this makes a fair comparison between models trained on different types of episodes difficult.
We investigate the effect of changing these hyper- ppremeearmeters in systematic manner. To compare configurations fairly across episode/batch sizes, we define each configuration by its number of shots n, the batch size b and the total number of images per class (which accounts for the sum between support and query set, a = n + m.) For example, we train a 5-shot model with a = 8 and 6 = 256, means that its corresponding training episodes will have 256/8- = 32. Using this notation, we train configurations of PNS covering several combinations of these hyper-parameters that the resulting batch size corresponds to episode 128, 256 512. Then, we train three configurations of NCA, where the only hyper-paramett parameter is the batch size.
5, q = 5-5 3, and w
Results for CIFAR-FS can be found in Fig. 2 where we report results for NCA and PNs with a = 8, 16 or 32. Results for minilmageNee observe the same trend and are deferred to Appendix A.6] Note that the results of 1-shot with a 16 and a = 32 are not reported, as they fare significantly worse. Several things can be noticed. First, NCA performs better than all PNs configurations, no matter the batch size. Second, PNS is very sensitive to different hyper-parameter configurations. For instance, with batches of size 128, PNs trained with episodes of 5-shot and
32 performs significantly worse than a PNs trained with episodes of 5-shot and a = 16. Finally, we can also notice that, contrary to what has been previously reported (Snell et al. 2017 Cao et al. 2020), the 5-shot model with the best configuration is always strictly better than any I-shot configuration. We speculate that this is probably due to the fact that we compare configurations keeping the batch size constant.

---

Under review as a conference paper at ICLR 2021
minilnnggeeet
miiniImageNeet
1.ahut Anmmey (36)
Aceursyy
NCA
Prits nets siik suc
Piots nots dick s-1l
Plotornnts S-dlet and Piotoretts S-dec anll
Percentugg pairs sumtgped
Pereentinge poins sompued.
Figure 3: I-shot and 5-shot accuracies the minllmageNett test set for models trained with batch size of 256 while only sampling a % of the total number available pairs. Reported values correspond to the mean accuracy of five models trained with different random seeds. Individual points contain models trained using PNs which are plotted on the x-axis based on the relative percentage of distance pairs that are used in their computation compared to NCA on the same batch size. Please see Sec. 4.2 for details.
Episodic batches vs random sub-sampling of standard batches. Despite the NCA outperform- ing all PNs configurations (Fig. P). one might posit that by using more distances within a batch, NCA is more computationally costly and thus the episodic strategy of PNs can sometimes be ad- vantageous (e.g. real-time applications with large batches). To investigate this, we perform an exper- iment where we train NCA models by randomly sampling a fraction of the total number of distances used in the loss. Then, for comparison, we include different PNs models after having computed to which percentage of discarded pairs (in normal batch) their episodic batch corresponds to.
Results can be found in Fig. 3 As expected, we can see how sub-sampling a fraction of the total available number of pairs within batch negatively affects performance. To answer possible concerns on computational efficiency, we can notice that the PNs points lie very close to the under- sampling version of the NCA, which signals that the episodic strategy of PNs roughly equivalent to train with the NCA and only exploiting a fraction of the distances available in a batch.
We see as we move right on the curve in Fig. 3 that the PN performance increases as well. In Appendix A.9 we look more carefully whether the difference in performance between the different episodic batch setups of Fig. can be explained by the differences in the number of distance pairs used in the batch configurations. We indeed find that generally speaking the higher the number of pairs the better, however, one should also consider the positive/negative balance and the number of classes present within a batch.
4.3 ABLATION EXPERIMENTS
To better analyse why NCA performs better than PNs, in this section we consider the three key differences discussed earlier by performing series of ablations
models trained batches of size 256. Results are summarised in Fig. 4 We refer the reader to Appendix A. obtain detailed
steps describing how these ablations affect Eq. 2 and Eq. 3 while Appendix A.7 contains the same analysis also for batches of size 128 and 512.
First, we compare two variants of NCA: one in which the sampling of the training batches happens sequentially and without replacement, as it is standard in supervised learning, and one where the batches are sampled with replacement. Interestingly, this modification (row I and 2 of Fig. 4 has a negligible effect across the two datasets and four splits considered, meaning that the replacement sampling introduced by episodic learning will not interfere with the other ablations.
We then perform a series of ablations on episodic batches, i.c. sampled with the method described in Sec. 3.1 3.1 For each ablation, we perform experiments on PNs trained with 1-shot and 5-shot, both with a 8. This means that both the 1-shot and 5-shot models have 32 classes and 8 images per class, allowing a fair comparison. The batch size is 256 for the NCA too. We first train standard PNs models. Next, we train PNS model where "prototypes" are not computed (point of Sec. 3.3 meaning that distances are considered between individual points, but a separation between query and support set remains. Then, perform ablation where ignore the separation between support and query set (point of Sec. 3.3), and compute the NCA on the union of the support and query set, while still computing prototypes for the points that would belong the support set. Last,
7

---

Under review as a conference paper at ICLR 2021
wiillergeeee (habat)
whilleggeeet (Sabo))
CHRAI PS (14404)
CHARRSS (1osho4)
(1)
NCA replacemett sompling (2)
NCA fioei batch tocppsition
Preconnta 1-shot
Prono-nets Soboo
Proto-nets S-shot proto (6)
Protomota bohot no S/Q (7)
78%
826
Figure 4: Ablation experiments on NCA and Prototypical Networks, both on batches or episodes of size 256 on the validation set of miniImageNet and CIFAR-FS, Please refer to Sec. 4.3 for dctails.
perform an ablation where we consider all the previous points together: we sample with replacement, we ignore the separation between support and query set and we do not compute prototypes. This amounts the NCA loss, except that computed on batches with a fixed number of classes and fixed number of images per class. Notice that in Fig. there is only one row dedicated to 1-shot models. This because we cannot generate prototypes from 1-shot models, so we cannot have "no proto" ablation. Furthermore, for 1-shot models the "no S/Q" ablation is equivalent to the NCA with a fixed batch composition. From Fig. 4 we can see that disabling prototypes (row 6) negatively affects the performance of 5-shot (row 5), albeit slightly. On the other hand, enabling the computation between all pairs increases the performance (last row), Importantly, enabling all the ablations (row 3) completely recovers the performance lost by PNs.
The fact that each single ablation does not have much influence on the performance, but their combi- nation does, could be explained by the number of distance pairs exploited by the individual ablations. Using the formulas described at the end of Section 3.3 we compute the number of positives and neg- atives used for each ablation. For row there 480 positives, and 14,880 negatives. For row there are 576 positives and 20,17 negatives. In both cases, the number is significantly lower than the corresponding NCA, which gets 896 positives and 31,744 negatives, and this could explain the jump in performance from row 6/7 to 3. Moreover, from row 5/6 to 7 we see slight increase in performance, which can also be explained by the (slightly) larger number of distance pairs.
These experiments nonetheless highlight that the separation of roles between the images belonging to support and query set, which is typical of episodic learning (Vinyals al.) (2066). is detrimental for the performance of Prototypical Networks. Instead, using the NCA loss on standard mini-batches allows full exploitation of the training data and significantly improves performance. Moreover, the NCA has the advantage of simplifying the overall training procedure, as the hyper- perem-rrreeeerss for the creation of episodes {w, n, m no longer need to be considered.
Additionally, in Appendix A.7 we repeat the same analysis done here to the simplest variant of Matching Networks (Vinyals et al.) 2016) and obtain the same conclusion; also this case, the separation of roles between support and query samples severely affects performance.
4.4 COMPARISON WITH THE STATE-OF-THE-AA
We now benchmark our models on three FSL datasets, with the purpose of contextualising their performance against the modern literature.
When considering which methods to compare against, we chose those a) which have been recently published, b) that are using a ResNet12 (which we found the most commonly used) and c) with setup that is not significantly more complicated than ours. For example, we only report the results of the main approach proposed by Tian al. (2020) and not their sequential self-distillation (Furlanello) al. 2018 variant, which requires re- re-training multiple times and can be applied to most methods.
Results can be found in Table for miniIaaeeeet and CIFAR-FS and Table 4 for tieredImageNet. In Table 3 we report Prototypical Networks results for both the episodic setup from Snell et al. (017)) and the best one (batch size 512, 5-shot, a=16) found from the experiment of Fig. 2 which brings considerable improvement over the original. We did not optimised for a new setup for Pro- totypical Networks on tieredImageNet, as the larger dataset and the higher number classes would have made the hyper-parameter search too demanding. Notice how our vanilla NCA is competi-
8

---

Under review as a conference paper at ICLR 2021
mindlmageNet
CHEAR-ES
1-shot
5-shot
1-shot
5-shot
Episodic methods
adaReaNet (Munkhdalai et al
2018
56 88 + 0.62
71.91 0.57
TADAM Oreshhkn
0.30
76.70
Shot Free Saaiichnddian di
2019
60.71 n/a
77.64.1 n/a
69.21 n/a
84.71 n/a
TEAM Qiao
2019
00.07 n/a
76.90 n/a
MTL (Sun of 2019)
61.20 1.80
75.50 = 0.80
TapNel Yoor cl al. 2019)
61.63 0.15
76.36 0.10
MctaOpdNGE-SV
of
2019
02 0.61
78.63
72.0 0.7
81.2 + 0.5
Variatonal FSL Zhang et aL
A019
61.23 + 0.26
77.69 = 0.17
Simple baselimes
Transduacive finctuning Dhillon et
2020
62.35 + 0.66
74.53
0.54
70.76 0.74
81.56 + 0.53
RFIC-simple Tian et al.
2020)
62.02 + 0.63
79.64 0.44
71. 0.8
86.0 + 0.5
Meta- Baseline
Clee
2020b
63.17 0.23
79.26 0.17
Our implemeentaiioo::
Proto-nets (nell
setup)
50.93 0.23
75.80
70.20
Proto-nets four sclup)
61.32 1 0.23
77.77
0.15
70.41 + 0.31
84.46 1 0.29
SimpleShox (Wang al. 2019
62.16 1 0.23
78.33 0.17
70.01
1: 0.21
84.50 1 0.11
NCA (ours)
62.52 0.24
18.30..14
72.48 + 0.40
85.13 = 0.29
Table 3: Comparison of methods that ResNet12 on minilmageNet and CIFAR-FS (test set).
fieredImageNet
S-shot
Shot Free (Ravichundran et
2019
63.52 m/a
82.50. n/a
RFIC-shmplz TTan 2020
69.74 + 0.72
84.41 + 0.55
Meta- Bascline Cherreaa
20206
G8 = 0.27
83.29 0.18
MetaOpiNNet-SV Mil 4e
2019
65.90 + 0.72
81.56 0.53
Our implementations:
Proto-nets (Snell et 2017 setup)
65.45 = 0.23
81.14 0.17
SimpleStor
2019
66.17 1: 0.15
80.64 0.20
NCA (ours)
68.36 1: 0.11
83.20 0.18
Table 4: Comparison of methods that use ResNet12 on tieredlmageNet (test set).
tive or superior to recent methods, despite being extremely simple. fairs surprisingly well against methods that use meta-learning (and episodic learning), and also against the high-performing simple baselines based on pre-training with the cross-entropy loss.
CONCLUSION
Towards the aim of understanding the reasons behind the poor competitiveness of meta-learning methods with respect to simple baselines, in this paper we start by investigating the role of episodes in the popular Prototypical Networks. We found that their performance is highly sensitive to the set of hyper-parameters used to sample the episodes. By replacing the Prototypical Networks' loss with the classic Neighbourhood Component Analysis, we are able to ignore these hyper-panne sppeepprraeeters while significantly improving the few-shot classification accuracy. With series of experiments, we found that the performance discrepancy mostly arises from the separation between support and query set within each episode, and that Prototypical Networks' episodic strategy is almost empirically equivalent to randomly discarding large fraction of distances within a standard mini-batch. Finally, show that our variant of the NCA achieves an accuracy multiple popular FSL benchmarks that comparable or superior with state-of- ofte-of-thee-n art methods similar complexity, making it a simple and appealing baseline for future work.
REFERENCES
Kelsey R Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning, 2019.
Han Altae-Tran, Bharath Ramsundar, Aneesh S Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS central science, 2017.

---

Under review as a conference paper at ICLR 2021
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks. Univ. of Texas, 1992.
Luca Bertinetto, João Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. In Advances in Neural Information Processing Systems, 2016.
Luca Bertinetto, João F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differ- entiable closed-form solvers. In International Conference Leurning Reprexentations, 2019.
Tianshi Cao, Mare Law, and Sanja Fidler. theoretical analysis of the number of shots in few-shot learning. In International Conference on Learning Representations, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint 1904. 04232, 2019.
Yinbo Chen, Xiaolong Wang. Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseline for few-shot learning, arXiv preprint arXiv:2003.04330 2020b.
Tristan Deleu, Tobias Würfl, Mandana Samici, Joseph Paul Cohen, and Yoshua Bengio. Torchmeta: A meta-learning library for pytorch. arXiv preprint 1909. 2019.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. International Conference on Learning Representations, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. 2017.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, 2018.
Golnaz Ghiasi, Tsung-Yr Yi Lin, and Quoe V Le. Dropblock: A regularization method for convolu- tional networks. In Advances in Neural Information Processing Systems, 2018
Spyros
Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few- shot visual learning with self-supervision. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In Advances in Neural Information Processing Systems, 2005.
Alex Graves, Greg Wayne, and Ivo Danihelka. arXiv:1410.5401,2011 2014.
Neural turing machines.
arXiv preprint
Edward Grefenstette, Brandon Amos. Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta- learning. arXiv preprint arXiv: 1910v0:110001722 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian q Weinberger. On calibration of modern neural net- works. In International Conference on Machine Learning, 2017.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, arXiv preprint arXiv:2004.05439 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint MrXiv:2004.1136) 2020.
Brenden Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
10

---

Under review as a conference paper at ICLR 2021
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2019.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In International Conference Machine Learning, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, 2018.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Limeng Qiao, Yemin Shi. Jia Li, Yaowei Wang. Tiejun Huang, and Yonghong Tian. Transduc- tive episodic-wise adaptive metrie for few-shot learning. In IEEE International Conference Computer Vision, 2019.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference Learming Rep- resentations, 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representations, 2017.
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In IEEE International Conference on Computer Vision, 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. In International Conference on Learning Representations, 2018.
Ruslan Salakhutdinov and Geoff Hinton. Learning a nonlinear embedding by preserving class neigh- bourhood structure. In Artificial Intelligence and Statistics, 2007.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In International Conference on Machine Learning, 2016.
Jürgen Schmidhuber.
Evolutionary principles in self-referential learning, learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. Learning to control west- memories: An alternative to dynamic recurrent networks. Neural Computation, 1992.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In IEEE Conference Computer Vision and Pattern Recognition, 2019.
Sebastian Thrun. Is learning the n-th thing any casier than learning the first? In Advances in Neural Information Processing Systems, 1996.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: good embedding all you need? arXiv preprint arXiv:2003.11533 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2019.
Manzagol, al. Meta-dataset: A dataset
11

---

Under review as a conference paper at ICLR 2021
Paul E Utgoff. Shift of bias for inductive concept learning. Machine learning: An artificial intelli- gence approach, 1986.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning Artificial Intelligence Review, 2002.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Re- visiting nearest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.04623, 2019.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. In International Conference on Machine Learning, 2019.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In IEEE International Conference on Computer Vision, 2019.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 PERFORMANCE IMPROVEMENTS
Adapting to the support set. Prototypical Networks does not perform any kind of parameter adaptation at test time. On the one hand this is convenient, as allows fast inference; the other hand, useful information from the support set S might remain unexploited.
In the 5-shot case it is possible to minimise the NCA loss since it can directly be computed on the support set: LNCA(S), We tried training a positive semi-defiitte matrix A on the outputs of the trained neural network, which corresponds learning Mahalanobis distance metric as Gold- berger et al.] (2005). However, we found that there was no meaningful increase in performance. Differently, we did find that fine-tuning the whole neural network fe by arg min, LNCA (S) was beneficial (see 5). However, given the computational cost, we opted for non performing adap- tation to the support sets in our experiments of Sec.
Features concatenation. For NCA, we also found that concatenating the output of intermediate layers modestly improves performance (almost) additional cost. We used the output of the average pool layers from all ResNet blocks except the first and we refer to this variant as NCA multi-layer. However, since this is an orthogonal variation that be applied several methods, we do not consider it for our experiments of Sec.
Results on minilmageNet an CIFAR-FS are shown Table
A.2 DETAILS ABOUT THE ABLATION STUDIES OF SECTION
Referring to the three key differences between the Prototypical Networks and the NCA losses listed in Sec. 3.3 in this section we detail how obtain the ablations we used to perform the experiments of Sec. 4.3
We can "disable" the creation of prototypes (point 1), which will change the prototypical loss of Eq. to
Σ
exp - l/q. s, s
C(S Q
Σ
log
(s)
S
Σ exp |1 s
(q.,)EQ
(S..y")
This is similar to LNCA (Eq. 3 where the positives represented by the distances from to Sk. and the negatives by the distances from Q to S Sj. The only difference now is the separation of the batch into query and support set.
Independently, we can "disable" point 2, which gives us
Σ exp
[2]
1
( JEQUC
L(S,Q)
Σ
log
ifj
[Q| ST
Σ
exp - ||
2
(EQCC
(K (RIJ))EQUC JEQUC kif
which essentially combines the prototypes with the query set, and computes the NCA loss on that total set of embeddings.
Finally, "disable" both point and which gives
Σ
exp
||z - 1
LEQUS Ni= (#1
C(S.Q) = 1Q] + |S|
Σ
log
(6)
Zz 12
exp
(24. Kgfi
This almost exactly corresponds to the NCA loss, where the only difference is the construction of batches with fixed number of classes and fixed number of images per class.
13

---

Under review as a conference paper at ICLR 2021
minilmageNet
CIFAR-FS
method
1-shot
5-shot
1-shot
5-shot
NCA
62.52 + 0.24
78.3 0.14
72.48 0.40
85.13 = 0.29
NCA multi-layer
63.21 + 0.08
79.27 = 0.08
0.36
85.42 = 0.29
NCA (ours) multi-
79.79 + 0.08
85.66 + 0.32
Table 5: Comparison hetween vanilla NCA, NCA using multiple evaluation layers and NCA performing optimisation on support set (ss). The NCA can only be optimised in the 5-shot case, since there are not enough positives distances in the 1-shot case. Support set is optimised for 5 epochs using Adam with learning rate 0.0001 and weight decay 0.0005. For details. see Sec. A.]
A.3 DIFFERENCES BETWEEN THE NCA AND CONTRASTIVE LOSSES
Eq. is similar to the contrastive loss functions (Khosla et al. 2020 Chen et al. 2020a) that are used self-supervised learning and representation learning. The main differences are that 1.) In contrastive losses, the denominator only contains negative pairs and 2.) the inner sum the nu- merator is moved outside of the logarithm in the supervised contrastive loss function from Khosla ct al. (2020). We opted to work with the NCA loss because we found it performs better than the supervised constrastive loss in few-shot learning setting. Using the supervised contrastive loss only managed to obtain 51.05% 1-shot and 63.36% 5-shot performance on the minilmagenet test set.
A.4 IMPLEMENTATION DETAILS
Benchmarks. our experiments, we use three popular FSL. benchmarks. minilmageNet (Vinyals ct al. (016)) is subset of ImageNet generated by randomly sampling 100 classes, each with 600 ran- domly sampled images. We adopt the commonly used splits of Ravi & Larochelle (2017) who use 64 classes for meta-training, 16 for meta- vela-vaaiiation and 20 for meta-testing. CIFARE R-FS pro- posed Bertinetto et al. (2019) as an anagolous version of minilmagenet for CIFAR- 100. It uses the same sized splits and same number of images per split as minilmageNet. Iaared (Ren et al., 2018) is also constructed from ImageNet, but contains 608 classes, with 351 training classes, 97 val- idation classes and 160 test classes. The class split have been generated using WordNet (Miller) 1995) to ensure that the training classes are semantically "distant" to the validation and test classes. For all datasets, we use images of size 84x84.
Architecture. In all our experiments, fo is represented by a ResNet12 with widths (64, 160, 320, 640). We chose this architecture, initially introduced by Lee et al. (2019). as it is the one which is most frequently adopted by recent FSL methods. Unlike most methods, do not use a DropBlock regulariser (Ghiasi et al. 2018)), as we did not notice it to meaningfully contribute to performance.
Optimisation. To train all the models used for our experiments, unless differently specified, we used SGD optimiser with Nesterov momentum, weight decay of 0.0005 and initial learning rate of 0.1. For minilmageNet and CIFAR-FS we decrease the learning rate by factor of 10 after 70% of epochs have been trained, and train for a total of 120 epochs. As data augmentations, we use random horizontal flipping and centre cropping.
Only for the experiments of Sec. 4.4 we slightly change our training setup. On CIFAR-FS, we increase the number of training epochs from 120 to 240, which improved accuracy of about 0.5%. For tieredImageNet, we train for 120 epochs and decrease the learning rate by factor of 10 after 50% and 75% of the training progress. For tieredImageNet only we increased the batch size to 1024, as we found it being beneficial. For the other datasets did not improve performance. These changes affect all our methods and baselines: NCA, Prototypical Networks (with both old and new batch setup), and SimpleShot (Wang et al. 2019).
Projection network. Similarly to (Khosla et al. 2020 Chen et al. 2020a), also experimented (for both PNs and NCA) with a projection network (but only for the comparison of Sec. (.4). The projection network is single linear layer A TR x that placed top of fo at training time,
14

---

Under review as a conference paper at ICLR 2021
minilmageNeet
CIFAR-FS
method
1-shot
5-shot
1-shot
5-shot
PNS (SimpleShot)
57.99 + 0.21
74.33 0.16
53.76 = 0.22
68.54 0.19 #.16
PNS (ours)
62.79 + 0.12
78.82 + 0.09
59.60 = 0.13
NCA (SimpleShot)
61.21 + 0.22
76.39 0.16
59.41 + 0.24
73.29 0.19
NCA (ours)
64.94 0.13
80.12 0.09
62.07 = 0.14
76.26 + 0.10
Table 6: Comparison of results on validation set of mineImageNet and CIFAR-FS using the hyperparameters used in SimpleShot/ Wang 2019) and the hyperparameters used in this paper (ours). Results are on batch size 256 (as used in Wang ct al. (209)) with the PNS episodic batch being a=16, as it is the best performing episodic setup we found.
where M is the output dimension of the neural network fo and P is the output dimension of A, which can be considered as a hyper-parammter. The output of A is only used during training. At test time, we do not use the output of A and directly use the output of fo. For CIFAR-F5 and fieredImageNet, we found this did not help performance. For minilmageNet however we found that this improved performance, and we set P 128 (which worked best for both PNS and NCA). Note that this is not an unfair advantage over other methods. Compared to SimpleShot (Wang et al., 2019) and other simple baselines, we actually use fewer parameters without the projection network (effectively making our ResNet12 a ResNetll) since they use an extra fully connected layer to minimise cross entropy during pre- rre-training.
A.5 CHOICE OF HYPER- PARAMETERS
During the experimental design, we wanted to ensure a fair comparison between the NCA and PNs. As a testimony of this effort, we obtained very competitive results for PNs (see for example the comparison to recent papers where architectures of similar capacity were used (Wang et al. 2019 Chen et al. 2019)). particular:
We always use the normalisation strategy of Wang et al. (2019). as it is beneficial also for PNs.
Unless expressively specified, we always used PNs 5-shot model, which in our implemen- tation outperforms the 1-shot model (for both 1-shot and 5-shot evaluation). Instead, (Snell
et al.] 2017) train and tests with the same number of shots.
Apart from the episodes hyper-parametent of PNs, which we did search and optimise over to create the plots of Fig. the only other syper-parameter of PNs are those related the training schedule, which are the same as the NCA. To set them, we started from the simple SGD schedule used by Wang et al. (2019) and only marginally modified it by increasing the number of training epochs to 120 increasing the batch size to 512 and setting weight decay and learning rate to 5e and 0.1, respectively. As a sanity check, we trained both the NCA and PNs with the exact training schedule used by Wang et al. 2019). Results are reported in Table 6 and show that the schedule we used for this paper considerably better for both PNs and NCA. In general, we observed that the modifications were beneficial for both NCA and PNs, and improvements performance in NCA and PNs were highly correlated. This is to be expected given the high similarity between the two methods and losses.
A.6 ADDITIONAL RESULTS FOR SEC. 4.2
Fig.
complements the results of Fig. from Sec.
A.7 ADDITIONAL RESULTS FOR SEC. 4.3
Matching Networks (Vinyals et al. 2016) are closely related to PNs (Snell et al. 2017). 7 and even equivalent in the 1-shot case. The difference is in the use of the support set in the multi-shot case. Whereas PNs generate prototypes by averaging the embedding of the support set, Matching Net- works adopt a weighted neirest-n approach using an attention mechanism. the attention mechanism is softmax over the distances (which the authors suggest in Sec. 2.1.1 of their paper),
15

---

Under review as a conference paper at ICLR 2021
minilnageNet
minilmageNet
1-4bat noTIIRS
5-bott
NCA
Fretocaro Lobott and
Fisteraie S-dket aud
Prett-art balbut aert
Prsto-arts balke ond2)
Batch size
Batch sime
Figure 5: 1-shot (left) and 5-shot accuracies (right) on the validation set of minilmageNet for different batch sizes. Models are trained using NCA Proto-nets with different configurations: 1-shot with = 8 and 5-shot with 16 32. Reported values correspond the accuracy of five models trained with different random seeds. Please see Sec. 4.2 for details.
mindlnagree (b-das)
minlangeeee (baist)
CIRAR:ES (1alan)
CIFAR-PS (belan)
NCA (1)
NCA orphoceent sumpling (2)
NCA lised batch compesition (3)
Mateising-nets L-shot (4)
Matcaing nots S-sbot (5)
005
625
64% 728
003
74%
555 705
7856
00%
023
54%
708
72%
74%
76%
Figure 6:
Ablation experiments on NCA and Matching Networks, both on batches or episodes of size 256 on the validation set of minilmageNet and CIFAR-FS, All methods use soft assignment (Sec. 3.4 test time.
obtain the soft-assignment approach discussed in Sec. of of this paper. The only two differences between Matching Networks with softmax attention mechanism and PNs are the lack of protoypes and the use of the cosine distance, instead of the Euclidean distance (Snell al. (2017) has shown that the Euclidean distance is better choice in FSL).
Given this similarity, and because of the relevance Matching Networks has in the few-shot learning community, we repeated the ablation experiment of Fig. 4 The results can be found in Table 6 In particular, we perform experiments on Matching Networks (without Full Context Embedding) using the softmax attention mechanism, and using Euclidean distance metric instead of cosine distance metric. At training time, Matching Networks corresponds the "no prototype" method in row 6 of Fig. 4 Therefore, the only difference between Matching Networks and NCA during training the separation between the support and query set, leaving us with only one ablation to perform. At test time, evaluating Matching Networks is equivalent to using the soft-assignment approach described in Sec. 3.4 Therefore, for a fair comparison, for both NCA and "NCA fixed- batch composition" methods we also use the soft-assignment evaluation at test time.
As we can see, disregarding the separation between the support and query set also improves the performance of Matching Networks, and significantly This corroborates the findings of Sec. the separation of roles between images in the support and query sets, typical of episodic learning, detrimental the performance of only PN, but also Matching Networks. Instead, using the (closely related) NCA on standard random mini-batches allows for better exploitation of the training data, while simultancously simplifying the training procedure.
Ablation experiments for different batch sizes. We repeated the ablation experiments done for batch size 256 (Fig. 4) also for size 128 and 512. Results can be found in Fig. 7 and Fig. g As we see, the overall trend is maintained. A difference is the meaningful gap in performance between row 1 and 3 in Fig. (size 128), which disappers in Fig. g (batch 512).
This is likely due to the number of positives available in excessively small batch size. Since vanilla NCA relies on using distance pairs and creates batches by simply sampling images randomly from the dataset, there limit to how small batch can be (which depends on the number of classes
16

---

Under review as a conference paper at ICLR 2021
xxiiheg Net (laber)
centrviiii (obba)
CIAIL (labtt)
CIARRIS (Soba)
XCA (1)
NCA applarment sumplng (2)
sompocttion (1]
Pooto soto Lhot (4)
Protosrat S-dek (0)
Profecteda Solett
Profesttt uu S/Q (7)
Figure 7: Ablation experiments on NCA and Prototypical Networks, both on batches or episodes of size 128 on the validation set of mindImageeet and CIFAR-FS, Please refer to Sec. 4.3 for details.
NCA (1)
NDA spplcmmnn wrmpling
NCA food buich compositoon (8)
Photossoa Lobst (4)
Proto-rets
Protocrets Gelot to pecto (6)
Picto odta Soluc S/Q (7)
705 3:5 Figure 8: Ablation experiments on NCA and Prototypical Networks, hoth on batches or episodes of size 512 on the validation set of minilmageNet and CIFAR-FS, Please refer to Sec. 4.3 for details.
995 295 on 055 645 60%
785 300
325
505
300
625 645
205 2008 758 27.5
of the dataset). As example, consider the extreme case of batch of size 4. For the datasets considered, it is very likely that such a batch will contain no positive pairs. For a batch size of 128 and training of 64 classes, with parameter-free sampler the NCA will have in expectation only one positive pair per class. Conversely, the NCA ablation with fixed batch composition (i.e. with set number of images per class) will have a higher number of positive pairs the cost of reduced number of classes per batch). We believe this can explain the difference, as positive pairs constitute less frequent (and potentially more informative) training signal. For the sake of simplicity, and since this only affects smaller batch sizes, opted to vanilla, parameter- free sampler for the NCA the rest of our experiments. Notice that, for batch size 512, there is even a slight (0.2%) decrease in performance using the fixed-batch composition w.r.t. the vanilla NCA.
A.8 DETAILS ABOUT NUMBER OF PAIRS DESCRIPTION OF SECTION
In this section we demonstrate that the total number of training pairs that NCA can exploit within a batch is always strictly superior than the one exploited by the episodic batch strategy used by Prototypical Networks.
To ensure we have "valid" episodic batch with nonzero number of both positive and negative distance pairs, we assume that m 2 1. and w 2 2.
Below, we show that the number of positives for NCA. ie. ("*+") w, is always greater or equal than the one for PNs, which is mnw:
+ n
(m + ni)!
21(m + n - 2)! W
(m + -))(m + 1)w
2
2 (m² + 2mn - m + + n)w
1
2 (m(m 1) + 2mn n(n 1))
2 (2mn)w wmm.
Similarly, show for negative distance pairs that (*) (m n)) > w(w 1) 1)mm:
17

---

Under review as a conference paper at ICLR 2021
rank
method
pos
# neg
total pairs
NCA
1792
129024
130816
2
5-shot am16
1760
54560
56320
3
S-shot a=8
960
60480
61440
S-shot a=32
2160
32400
34560
5 I-shot a=8
448
28224
28672
Table 7: Number of positives and negatives used in the batch size 512 experiments of Fig.
1
(m - =n)²
2!(w 2) (m² + 2mn + n²)
w!
2
1 w(w 1)(m² + 2mn n²)
2
2 w(w 1)(2m
w(w 1mn..
This means that the NCA has at least the same number of positives as Prototypical Networks, and always has strictly more negative distances.
The total number of extra pairs that NCA can rely on is shr (w(mm + n²) - m n).
A.9 DETAILS ABOUT NUMBER OF PAIRS DESCRIPTION OF SECTION 4.2
In Table 7 we plot the number of positives and negatives (gradients contributing to the loss) for the NCA and different episodic configurations of PNs, to see whether the difference in performance can be explained by the difference in the number of distance pairs that can be exploited in certain batch configuration. This is often true, as the ranking can almost be fully explained by the number of total pairs in the right column. However, there are two exceptions to this: 5-shot with a=16 and 5-shot with 8=8.
To understand this, we can see that the number positive pairs is much higher for a=16 than for a=8. Since the positive pairs constitute a less frequent (and potentially more informative) training signal, this can explain the difference. The a=32 variant has an even higher number positives than a=16, but the loss in performance there could be explained by drastically lower number of negatives, and by the fact that the number of ways used during training is lower. So, while indeed generally speaking the higher number of pairs the better (which is also corroborated by Fig. where moving right on the x-axis sees higher performance for both NCA and PNs), one should also consider how this interacts with the positive/negative balance and the number of classes present within batch.
18


Reviews:
[{"review_id": "ugIVEZg1Cpa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe submission investigates the properties of episodic training and its impact on learning using Prototypical Networks as a case study. The paper draws a connection between Prototypical Networks and Neighbourhood Component Analysis (NCA), noting that their loss functions are similar but that NCA is trained non-episodically, which allows it to learn from the relationship between all example pairs in a batch.\n\nWhen controlling for batch size, the paper claims to show that NCA (combined with a nearest-centroid inference strategy) performs better than Prototypical Networks, as evidenced by experiments on CIFAR-FS and mini-ImageNet. Ablation experiments are performed, claiming to show that applying the NCA loss to batches sampled episodically allows Prototypical Networks to bridge the performance gap with NCA, and that the partition of examples within a batch into support and query sets is detrimental to Prototypical Networks training. Finally, NCA is evaluated alongside comparable competing approaches on mini-ImageNet, CIFAR-FS, and tiered-ImageNet, and is claimed to yield results comparable or superior to the state-of-the-art.\n\n#### Strengths and weaknesses\n\n* **+** The value of episodic training is increasingly being questioned, and the submission approaches the topic from a new and interesting perspective.\n* **+** The connection between nearest-centroid few-shot learning approaches and NCA has not been made in the literature to my knowledge and has potential applications beyond the scope of this paper.\n* **+** The paper is well-written, easy to follow, and well-connected to the existing literature.\n* **-** The extent to which the observations presented generalize to few-shot learners beyond Prototypical Networks is not evaluated, which may limit the scope of the submission’s contributions in terms of understanding the properties of episodic training.\n* **-** The Matching Networks / NCA connection makes more sense in my opinion than the Prototypical Networks / NCA connection.\n* **-** A single set of hyperparameters was used across learners for a given benchmark, which can bias the conclusions drawn from the experiments.\n\n#### Recommendation\n\nI’m leaning towards acceptance. I have some issues with the submission that are detailed below, but overall the paper presents an interesting take on a topic that’s currently very relevant to the few-shot learning community, and I feel that the value it brings to the conversation is sufficient to overcome the concerns I have.\n\n#### Detailed justification\n\nThe biggest concern I have with the submission is methodological. One one hand, the authors went beyond the usual practice of reporting accuracies on a single run and instead trained each method with five different random initializations, and this is a practice that I’m happy to see in a few-shot classification paper. On the other hand, the choice to share a single set of hyperparameters across learners for a given benchmark leaves a blind spot in the evaluation. What if Prototypical Networks are more sensitive to the choice of optimizer, learning rate schedule, and weight decay coefficient than NCA? Is it possible that the set of hyperparameters chosen for the experiments happens to work poorly for Prototypical Networks? Would we observe the same trends if we tuned hyperparameters independently for each experimental setting? In its current form the submission shows that Prototypical Networks are sensitive to the hyperparameters used to sample episodes *while keeping other hyperparameters fixed*, but showing the same trend while doing a reasonable effort at tuning other hyperparameters would make for a more convincing argument. This is why I take the claim made in Section 4.2 that \"NCA performs better than all PN configurations, no matter the batch size\" with a grain of salt, for instance.\n\nI also feel that the submission misses out on an opportunity to support a more general statement about episodic training via observations on approaches such as Matching Networks, MAML, etc. I really like the way Figure 1 explains visually how Prototypical Networks miss out on useful relationships between examples in a batch and is therefore data-inefficient. To me, this is one of the submission’s most important contributions: the suggestion that a leave-one-out strategy could allow episodic approaches to achieve the same kind of data efficiency as non-episodic approaches, alleviating the need for a supervised pre-training / episodic fine-tuning strategy. To be clear, I don’t think the missed opportunity would be a reason to reject the paper, but I think that showing empirically that the leave-one-out strategy applies beyond Prototypical Networks would make me lean more strongly towards acceptance.\n\nThe connection drawn between Prototypical Networks and NCA feels forced at times. In the introduction the paper claims to \"show that, without episodic learning, Prototypical Networks correspond to the classic Neighbourhood Component Analysis\", but Section 3.3 lists the creation of prototypes as a key difference between the two which is not resolved by training non-episodically. From my perspective, NCA would be more akin to the non-episodic counterpart to Matching Networks without Full Contextual Embeddings – albeit with a Euclidean metric rather than a cosine similarity metric – since both perform comparisons on example pairs.\n\nThis relationship with Matching Networks could be exploited to improve clarity. For instance, row 6 of Figure 4 can be interpreted as a Matching Networks implementation with a Euclidean distance metric. With this in mind, could the difference in performance between \"*1*-NN with class centroids\" and *k*-NN / Soft Assignment noted in Section 4.1 – as well as the drop in performance observed in Figure 4’s row 6 – be explained by the fact that a (soft) nearest-neighbour approach is more sensitive to outliers?\n\nFinally, I have some issues with how results are reported in Tables 1 and 2. Firstly, we don’t know how competing approaches would perform if we applied the paper’s proposed multi-layer concatenation trick, and the idea itself feels more like a way to give NCA’s performance a small boost and bring it into SOTA-like territory. Comparing NCA without multi-layer against other approaches is therefore more interesting to me. Secondly, 95% confidence intervals are provided, but the absence of identification of the best-performing approach(es) in each setting makes it hard to draw high-level conclusions at a glance. I would suggest bolding the best accuracy in each column along with all other entries for which a 95% confidence interval test on the difference between the means is inconclusive in determining that the difference is significant.\n\n#### Questions\n\n1. In Equation 2, why is the sum normalized by the total number of examples in the episode rather than the number of query examples?\n1. Can the authors comment on the extent to which Figure 2 supports the hypothesis that NCA is better for training because it learns from a larger number of positives and negatives? Assuming this is true, we should see that Prototypical Networks configurations that increase the number of positives and negatives should perform better for a given batch size. Does Figure 2 support this assertion?\n1. Can the authors elaborate on the \"no S/Q\" ablation (Figure 4, row 7)? What is the point of reference when computing distances for support and query examples? Is the loss computed in the same way for support and query examples? The text in Section 4.3 makes it appear like the loss for query examples is the NCA loss, but the loss for support examples is the prototypical loss. Wouldn’t it be conceptually cleaner to compute leave-one-out prototypes, i.e. leave each example out of the computation of its own class’ prototype (resulting in slightly different prototypes for examples of the same class)? In my mind, this would be the best way to remove the support/query partition while maintaining prototype computation, thereby showing that the partition is detrimental to Prototypical Networks training.\n\n#### Additional feedback\n\n1. This is somewhat inconsequential, but across all implementations of episodic training that I have examined I haven’t encountered an implementation that uses a flag to differentiate between support and query examples. Instead, the implementations I have examined explicitly represent support and query examples as separate tensors. I was therefore surprised to read that \"in most implementations [...] each image is characterised by a flag indicating whether it corresponds to the support or the query set [...]\"; can the authors point to the implementations they have in mind when making that assertion?\n1. I would be careful with the assertion that \"during evaluation the triplet {w, n, m} [...] must stay unchanged across methods\". While this is true for the benchmarks considered in this submission, benchmarks like Meta-Dataset evaluate on variable-ways and variable-shots episodes.\n1. I’m not too concerned with the computational efficiency of NCA. The pairwise Euclidean distances can be computed efficiently using the inner- and outer-product of the batch of embeddings with itself.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "oVtw72_1b6S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the usefulness of episodic learning in prototypical learning which is a popular practice in few-shot learning. The authors propose a non-episodic prototypical network which basically corresponds to the classical neighborhood component analysis and they claimed that this network reliably improves over its episodic counterpart in multiple datasets. I have the following comments on the paper:\n\n1. The sections 3.1, 3.2 and 3.3 are not the contributions of the paper. Only section 3.4 can be considered as something new from experimental point of view and not methodologically new. k-NN, 1-NN with class centroids and soft assignments are all some specific experimental settings. Therefore, I do not see the technical contributions of the paper other than the claimed novel experimental settings which is also marginal.\n\n2. The paper shows a robust experimentations and comparisons with prior arts, however, I don't understand how the three settings mentioned in the section 3.4 are evaluated in the tables.\n\n3. I am curious if you have done a comparison with baseline NCA, i.e. equation (3). I have not found the comparison in A.1 which only contains some discussions but no direct comparison.\n\n4. Anyways, The paper is written in good English and I haven't found any typos yet.\n\nBased on my current understandings and above comments, currently I recommend for a weak rejection. However, I would like to follow the discussions on the paper and understand the contributions well.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FgXSA1gvj_B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper's starting point is the question whether the episodic training is beneficial, or not, for FSL / Prototypical Networks. The work can be seen as a follow-up of the recent works showing that simple baselines can outperform rather sophisticated few-shot learning models. Towards answering this question, this paper points out that Prototypical Networks (PN) are related to Neighborhood Component Analysis (NCA), and NCA can be considered as an episodic training-free alternative of PN.\n\nIn more detail, PN aims to learn per-class prototypes based on sample averaging in the feature space. NCA, in contrast, aims to maximize the ratio of total similarity between same-class example pairs to the total similarity between different-class pairs. Due to their similarities in terms of their formulations, the paper claims that NCA loss can be considered as an alternative to PN loss to do non-episodic representation learning for few-shot learning purposes. In addition, the paper has a few strong claims, such as episodic training is “detrimental to learning” and “under no circumstance beneficial to differentiate between support and query set within a training batch\". Clearly, these are intriguing claims.\n \nHowever, there is a gap between the claims and the experimental validation. First, even if ProtoNet loss and NCA loss seem to be similar to each other, they're nevertheless different models, and it takes quite a significant manipulation to convert PN to NCA. Therefore, the fact one particular non-episodic-training based model gives superior results compared to those of PN with episodic-training, does tell us much about detrimental effects of episodic training for PN or in general. Second, while the paper's observations that NCA has the advantage of using more pairwise similarities within a batch compared to PN is indeed insightful, it rather points out to certain weaknesses in the way per-batch / per-episode data is being utilized by the PN formulation, instead of problems about episodic training.\n\nOverall, the paper has interesting observations about PN's weaknesses and shows why one particular simple non-episodic training / non meta-learned approach (NCA) can yield superior results compared to PN, which is a relatively mode sophisticated & well-established approach. However, the paper's (over-strong) claims remain mostly unsupported, which makes the otherwise interesting work poorly framed. The paper, with more water-tight arguments only, could otherwise be a valuable contribution but it requires quite significant & fundamental revisions throughout the paper, therefore, is not ready for publication in its current form.\n\nPost-rebuttal: I would like to thank for the detailed responses and the careful revisions made in the paper. Overall, the paper is now definitely improved in certain ways and initiates an important discussion on the value of episodic training & classifier synthesis for few-shot learning, as opposed to typically-simpler metric-learning based approaches. The paper also approaches this problem from an interesting point of view, by focusing on sample utilization in the episodic training of PN.  \n\nHowever, I still find that the the paper remains somewhat weak in its current form for the following reasons:\n- I maintain my view that NCA vs PN are not direct alternatives to each other, considering that PN allows learning a representation that is optimized for class-average to sample comparisons, whereas, NCA uses a sample-to-sample distance based loss. The fact that the very construction of these two models, despite the similarities pointed out, blurs the strength of the overall NCA vs PN based discussion on the value of episodic training.\n- The claims about the advantages of non-episodic training of NCA is mainly based on the observation that NCA creates more positive/negative labels. However, it is not clear whether it is the more efficient utilization of training samples or just the differences in terms of the predictive model formulations. (Perhaps, averaging based prototype computation is a bad idea, after all, which may not have directly anything to do with episodic training.)\n- To this end, Fig. 3 is indeed interesting, but again the results are not very clear. Here, careful optimizer re-tuning specially for each case can be necessary as subsampling degrades the gradient approximation quality, which creates the question how much fundamentally important efficiency in batch utilization is, as long as one uses a proper optimizer. \nOverall, I think the paper makes a valuable step in an interesting direction but the paper fails to make a strong-enough case. Overall, I  improve my rating by a single level to 4, but find that the paper is not stronger than this in its current form.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3r0gbxU-Dfd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes to use neighborhood component analysis in lieu of prototype loss to train embedding functions of few-shot learning. This method takes full advantage of relations between all sampled points in an episode to facilitate learning, and it removes the distinction between support and query samples during meta-training time.\n\nReason for score: Overall, I lean towards reject. This paper proposes an interesting method that improves upon Prototypical Networks, and performs on-par with other baseline methods. However, the paper does not advance our understanding of how episodic training interacts with few-shot learning beyond the addition of more performance numbers to compare against. \n\nPros: The proposed method is a straight-forward improvement to Prototypical networks. In the wide range of scenarios evaluated in the experiments, the proposed method consistently outperforms ProtoNet. This proposed method should also be easy to implement, making integration with other FSL methods based on ProtoNets feasible. \nWhile not exactly nouveau, using NCA to train embedding functions has not been done before (to my best knowledge) and is a theoretically sound approach.\n\nCons:\n1. Perhaps unsurprisingly, the performance of NCA is lower than some FSL methods that use additional capacity. Even so, I think presenting only favorable comparisons in the performance tables is counter productive as it fails to capture the research context of this work. \n2. The conclusion from the ablation studies on batch size is still unclear to me. The “NCA fixed batch composition” setting seems to perform better than NCA in 3 of the 4 plots in figure 4. This particular setting is interesting as it allows control of the number of classes in each batch. As the batchsize is fixed in the ablation, we won’t know what is the relation between the number of ways and performance of this fixed batch variant of NCA. This ablation is also confusing in that neither “no proto” nor “no S/Q” significantly improves performance, yet their combination performs well. A full ablation that systematically covers all hyperparameters would be helpful in furthering understanding in this direction.\n3. The motivation of the paper feels unclear: on one hand the authors claim that they aim at understanding the (un)usefulness of episodic learning, yet on the other hand this paper doesn’t present any results beyond the final performance number to aid with this understanding. Visualizations and/or theoretical arguments would be greatly appreciated.\n\nMinor points (suggestions, not related to score):\nIntroduction\n“These results legitimately cast a doubt” -> “These results cast a doubt”\nRelated Work\n“Between 2016 and 2017” feels unnecessary\n“Matching and Prototypical…weighted by either an LSTM or a simple average, respectively”: Matching Networks also proposes a sample average variant. Their main difference lies in one uses cosine similarity while the other uses euclidean distance.\n“Differently from these papers” -> Different\nBACKGROUND AND METHOD\nIn 3.3, point 3 is not necessarily correct. Why would some examples be more likely than others in the episodic scheme? All FSL benchmarks (used in this paper) are close to class balanced, and an hierarchical sampling scheme for episodes results in full coverage each epoch just like standard supervised learning.\nIn 3.4 “which is the probability that image i is sampled from image j”. This is under the assumption that each support image defines a Gaussian in the embedding space. This is not true in general. The probabilistic interpretation of NCA should be further explained. \n\n[Post rebuttal]\nI am still leaning against the acceptance of this paper due to concerns about the limited impact of this submission. The topic of \"limitations of episodic training\" has been widely studied and in fact most SoA few-shot learning methods adopt a combination of supervised and episodic training for this precise reason. \n\nThe exploration into the relation between the number of sample pairs and learning performance is indeed correct, but no strong conclusions can be reached due to the logical jumps required. The authors established that the performance is correlated with number of pairs with a log/square-root curve, and that ProtoNet performance is similar to that of subsampled NCA (fig3). The mechanism behind why this is has not been elucidated. I think many questions can be explored to strengthen this paper, for example:\nAre classes embedded tighter together? \nAre hard negatives pushed further apart? \nDoes NCA induce a non-euclidean geometry that is more expressive than the euclidean geometry naturally induced by ProtoNets?\nCan the classification problem be converted into a pair comparison problem, so that PAC learning theory can be used to explain the shape of this curve? \nWhat is the sample complexity of the NCA classifier compared to the Prototypical classifier?\n\nRegarding the proposed method, NCA is certainly an improvement over ProtoNets, but performs worse than existing methods in most experiments. This makes me doubtful of the impact of this work on the methodological front. Better modelling of relationships between few-shot examples has been implicitly and explicitly studied too. For instance, many works adopt sample level set functions in the form of transformers, attention modules, and graph neural networks. Arguably, these additional architectures are more expressive \"deep\" alternatives to NCA, and hence achieves better performance than the proposed method.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is right on the borderline. It questions the utility of episodic training from a novel perspective, driven by a comparison to NCA, with thorough experiments. The hypothesis that more pairwise comparisons per batch/episode benefit learning is also quite interesting, but some reviewers didn’t feel this was convincingly presented.

Prototypical networks are indeed a popular method for FSL, but I do as well think that NCA is more closely related to matching networks, and that it makes more sense for that to be the focus of experimentation. Matching networks involve more direct pairwise comparisons, and so a leave-one-out baseline with this model would probably be a useful comparison.

While I appreciate the desire to focus on a fundamental aspect of FSL and not chase state of the art, I think that it’s important to show where one should go from here. That is, as the reviewers pointed out there are many mechanisms beyond vanilla PNs that have yielded better results than those presented in this paper. I don’t think matching SOTA is necessary here, but it would be nice to show that the insights here complement other mechanisms in FSL.

Decision: Reject