Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the question of whether episodes (using a split of support & query sets) are necessary for non-parameteric approaches for few-shot learning (examples include Prototypical Networks & Matching Networks). The authors propose a Neighborhood Component Analysis-based method for few-shot learning, where a mini-batch consists of examples from a subset of base classes, with no support or query split. The NCA loss then involves distances computed across all examples in the mini-batch rather than only using distances computed between support and query examples (as is done in Prototypical Networks & Matching Networks). The authors show that their proposed method is able to achieve better performance for 1-shot & 5-shot classification in miniImageNet, cifar-fs, and tieredImageNet benchmarks. They also speculate that the proposed method performs better because of its use of more distance computations in the loss compared to Prototypical Networks & Matching Networks and confirm this by conducting an experiments where they randomly drop distances in their NCA loss and showing this has a negative impact on performance.
main_review: Originality: to my knowledge an NCA-type loss has not been considered for few-shot learning and the analysis comparing the differences between this loss and Prototypical Networks & Matching Networks loss is also original and interesting.

Quality: submission seems technically sound in terms of the analysis comparing their proposal to existing work. The experiments were also very useful, as they go beyond raw metrics and studied interested aspects of their method that show why it may be working better compared to previous work.

Clarity: the paper is very well-written - both description of the method(s) and the experiments are easy to understand and described well.

Significance: this work (along with other recent work showing that pre-training without any notion of episodes) could be very significant as they're questioning the benefit of more complicated episodic sampling-based approaches to few-shot learning. I do have some minor concerns/questions which could be clarified a bit further and help strengthen the paper.

1. It would be useful to include the results on metrics for Prototypical Networks on ResNet12 from previous work. The authors include results from their own implementation of Proto-Nets but I think including results from previous work is helpful to know that any implementation-specific choices by the authors in their recreation did not affect the metrics negatively.
2. Based on results of Figure 4, if NCA fixed-batch composition is always strictly better than the default NCA, why not treat fixed-batch composition as the default model? Is it because the default NCA is technically simpler since creating a batch just involves iterating through the data and the benefit only applies to small batches?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies the role of the popular episodic training paradigm, in the context of two metric learning-based episodic models: Prototypical Networks (PN) and Matching Networks (MN). They show that these popular methods underperform compared to the closely-related NCA model which is non-episodic, i.e. does not separate the examples sampled in each training batch into disjoint support and query sets. They argue that the superior performance of NCA is because, due to not performing that separation, the total number of pairwise comparisons that are used in the loss computation is larger than those used for PN/MN, making the gradients more informative. Indeed, in episodic models, each query example is only compared to support examples, but not to other query examples, resulting in significantly fewer comparisons. Experimentally, they show that for a fixed batch size (where the ‘batch size’ in the case of the episodes is given by the combined size of the support and query sets), NCA outperforms PN and MN, despite its simplicity in terms of its smaller number of hyperparameters. They also show that randomly discarding comparisons from NCA leads to similar performance to the analogous PN/MN models and perform a set of ablations to “bridge” the gap between PN and NCA, further strengthening their finding that support/query separation hurts performance.

main_review: High-level review
-----------------------
The paper is really well-written and an enjoyable read. Despite the findings not being very surprising to me, the inefficiency with which episodic models exploit their training data compared to non-episodic baselines has not been formally established, and it’s nice to have the empirical evidence shown in the paper to support that intuition. It is also a nice contribution to show that the NCA model can perform well for few-shot classification, and can serve as a strong baseline alongside the simple cross-entropy classifier baselines that are gaining traction. However, the study of this work is limited to only a couple settings for the shot and way (1-shot 5-way and 5-shot 5-way), to simple benchmarks, and only two episodic methods. It is not clear if the same conclusions generalize to other models and to harder settings, e.g. varying shots at test time, and harder generalization problems to classes that are dissimilar to the training set (e.g. coming from different datasets). These aspects limit the impact of the paper in my opinion and make it hard to see what the take-away is. How should few-shot learning researchers change their approach based on this finding? PN/MN are no longer state-of-the-art models, and recent work these days is moving away from (exclusively, at least) relying on episodic training for few-shot learning (e.g. [1,2,3,4,5,6,7,8] to name a few). In some of these works, episodic training is still used for certain components of the network (e.g hypernets that learn to condition a shared network for each task by producing FiLM parameters for instance, which are naturally defined in an episodic manner), or used to fine-tune a pre-trained network, but the representation is pre-trained for a non-episodic objective. For the above reasons, I'm slightly leaning towards rejection of this paper at this stage. Please see below for more detailed comments.

Detailed comments
--------------------------
- Based on the observation that PN/MN perform significantly fewer comparisons than NCA, they underperform NCA in the settings considered and the authors conclude that NCA is preferable in those settings. However, one could argue instead that, due to this observation, the episodic models may require a significantly larger batch size to reach their potential (to increase their number of comparisons that contribute to each update), and so perhaps their poor performance is due to that hyperparameter (the batch size) selected inappropriately for these methods. Of course very large batch sizes can be challenging from an implementation standpoint (e.g. due to memory issues etc) but, to reliably explore the potential of these methods, these challenges can be overcome by carefully parallelizing across GPUs for example (or training with a meta-batch of episodes which is actually common practice). A notable example from the recent literature that yields very impressive performance but requires very large batches to do so is SimCLR [9] (it was trained with a batch size >4K). Perhaps “scaling up” episodic methods in this fashion will result in surpassing the (analogously scaled up) NCA method, so in my view the jury is still out on this comparison.

- This paper only focuses on the disadvantages of episodic models and advantages of NCA. It would also be useful to discuss episodic models’ merits. A potential advantage of PN, for example, is that we can modify its training (specifically, through the choice of the training shot hyperparameter), to specialize it to the shot setting of interest. [10] formally shows that the choice of the ‘shot’ hyperparameter during training impacts the properties of the learned embedding space with smaller shots during training focusing on minimizing intra-class variance (pulling examples of the same class together) while larger training shots on maximizing inter-class variance (pushing different classes apart). As a consequence, models with smaller training shots are more appropriate for small-shot test-time tasks, and analogously for larger-shot models. NCA on the other hand does not (explicitly at least) offer that “knob” to tune at training time in order to control the desired properties of the resulting embedding space. Can we theoretically or empirically study whether NCA-learned models are capable of performing well on tasks of a wide range of shots at test time, when using the nearest-centroid classification as the test-time algorithm? The paper only considers 1-shot and 5-shot evaluation settings which are both very small shots.

- An interesting data point to include in the discussion from the gradient-based meta-learning world is that, while MAML [11] uses a support and query separation, its first-order Reptile variant [12] does not. Is there a high-level take-away that connects the relationship of Reptile to MAML and that of PN (or MN) to NCA?

- Related work: [13] should be included in the related work in the context of few-shot learning models that do not require a support and query separation.

- Recently, there have been some theoretical papers that investigate the role of the support/query framework from different perspectives [14,15]. It would be useful to include these in the discussion and consolidate the empirical observations made here with the theoretical findings from those papers.

- In the ablations, what is the PN model where "prototypes are not computed"? PN requires prototypes, so omitting their computation is not an ablation of this model; it is a different model. Is this exactly the same as MN in this case? If so, I would just refer to it as MN.

- In the original PN paper they showed empirically that it was beneficial to "match" the shot used at training and test times. The results here contradict this, since PN 5-shot is significantly better than PN 1-shot, for both 1-shot and 5-shot evaluation settings, across datasets. Any thoughts on why that might be?

- In Table 2, in the “simple cross-entropy baselines” section, the Classifier-Baseline [5] and Baseline++ [2] should also be included.

Minor comments
-----------------------
- Line 55, in the query set definition, in the first tuple, the first label is y_1’. Should it just be y_1 (without the ‘)?
- Line  87 “the number of training distances that are lost” → “the number of training comparisons that are omitted”? The first phrase seems a little odd.
- When explaining the “Nearest Centroid” method (line 150): “This is the approach used by Prototypical Networks and SimpleShot”. It also is the approach used by Classifier-Baseline and Meta-Baseline [5].

References
----------------
- [1] Meta-learning with Latent Embedding Optimization. Rusu et al. ICLR 2019.
- [2] A closer look at few-shot classification. Chen et al. ICLR 2019.
- [3] Rethinking few-shot classification: A good embedding is all you need? Tian et al. 2020.
- [4] TADAM: Task dependent adaptive metric for improved few-shot learning. Oreshkin et al. NeurIPS 2018.
- [5] A New Meta-Baseline for Few-Shot Learning. Chen et al. 2020.
- [6] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al. NeurIPS 2019.
- [7] Improved Few-Shot Visual Classification. Bateni et al. 2020
- [8] Learning a Universal Template for Few-shot Dataset Generalization. Triantafillou et al. ICML 2021.
- [9] A Simple Framework for Contrastive Learning of Visual Representations. Chen et al. ICML 2020.
- [10] A Theoretical Analysis of the Number of Shots in Few-shot Learning. Cao et al. ICLR 2020.
- [11] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Finn et al. ICML 2017.
- [12] On First-order Meta-learning Algorithms. Nichol et al. 2018.
- [13] Few-shot Learning through an Information Retrieval Lens. Triantafillou et al. NeurIPS 2017.
- [14] How Important is the Train-Validation Split in Meta-Learning? Bai et al. 2021.
- [15] A Closer Look at the Training Strategy for Modern Meta-Learning. Chen et al. NeurIPS 2020.

limitations_and_societal_impact: Yes, I believe they have addressed this adequately.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper conducts a case study for the non-parametric few-shot classification methods (e.g. Prototypical Networks). 
It proposes to utilize the classic Neighbourhood Component Analysis (NCA) sampling instead of the original matching or prototypical style episode sampling. 
The authors conducted ablation experiments to investigate the properties of this new sampling and compare it with the basic Prototypical Networks (PNs) method. 
The final accuracy is comparable with the recent methods on three benchmark datasets. 
main_review: Recently, there are several papers focusing on whether meta-learning or episode training is beneficial in few-shot classification, e.g [8,17,27,38]. These papers discuss the necessity of meta-learning and some alternatives such as regularization, pre-training, feature reuse. In this situation, the new findings and overall novelty of this paper are limited. 

Moreover, in both the technical and experimental parts, there is a lack of thorough comparison with existing studies on the episode or meta-learning mechanisms. 
Most of the ablation studies are against the PNs method, which is relatively poor. In Table 2, some related studies such as [17] are missing. 

in [38], the authors proposed a simple pre-training + classifier learning baseline. The proposed NCA method seems to underperform this simple method, which weakens the significance of proposing such a new sampling strategy. 

In Table 1, an extra number of gradients are required by NCA. It is unclear if this method will be less efficient than PNs. 

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
After Rebuttal. 

After the rebuttal and discussion with other reviewers, I would like to increase my score to 5. But I still have some concerns about the limited scope and lack of comparisons. 
limitations_and_societal_impact: The main limitation is the lack of comparison and analysis with related work that also studies the effect of episode training. 
In Table 2, there is a typo "MetaOptNet-SVM[21]".
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new perspective on the episodic learning of nonparametric few-shot learning methods. The main claim is that current popular nonparametric methods, such as Prototypical Networks (PNs) and Matching Networks (MNs), are not data-efficient because less gradient signal is propagated during training due to the artificial division of the data points to support and query sets. The authors instead propose to use the standard learning protocol with batches and an equivalent loss function based on the Neighbourhood Component Analysis (NCA). This loss function exploits all connections between data points in the batch. The authors then propose three techniques to perform a few-shot classification during evaluation based on k-NN, nearest centroid, and soft assignment. Through extensive experiments, the authors justify their perspective and show comparable results to other recent FSL methods (not necessarily nonparametric ones). 
main_review: Overall it is an interesting paper, and I think it should be accepted.  

Strengths:
- The claim that the hyperparameters controlling the episode creation can significantly affect the performance makes a lot of sense.  
- The paper presents extensive empirical justifications for the main claims presented in it.  
- Despite the method's simplicity it shows good results on relevant benchmarks.  
- The paper is written clearly and easy to understand.  
- The results in the paper are easily reproducible. The code is very organized and full experimental details were given for all experiments.  

Some comments:  
- Although NCA is compared against relatively recent baselines and it is not delivered as a new SoTA, it seems that there is a noticeable gap between its performance and more recent baselines (e.g., [1, 2]). Furthermore, this paper scope is limited only to nonparametric metric-based learners. Therefore, I think the authors should add a discussion on how to extend their approach beyond the current scope and how it can be combined with more recent approaches.  
- I think that the authors should have addressed other types of non-parametric few-shot learners. For example, recently two approaches based on Gaussian processes were proposed [3, 4]. Although these methods use the standard episodic learning, they ignore the common support/query split (as was done in the ablation study in this paper).  
- In Table 2 -> CIFAR-FS -> 5-shot, RFIC-simple achieves the highest accuracy and therefore should be in bold.

General wonderings:  
- Line 175 states "Each model is evaluated on 10,000 episodes sampled from the validation or the test set". Does this mean that you didn't use the pre-defined validation set for hyper-parameter selection and early stopping? If so, how did you perform these operations?  Is it consistent with the results reported for the baseline methods in Table 2?
- When comparing Fig. 3 in the main text with Fig. 1 in the Appendix it seems that using 80% of the pairs yield similar results to those obtained with all the pairs. Is it true? If so, how can you justify this?  
- Following the previous point, did you try other (more advanced) techniques for pair selection? As a motivation, it may allow achieving similar accuracy while reducing the computational burden required by the denominator of the NCM loss.  


[1] Rizve, M. N., Khan, S., Khan, F. S., & Shah, M. (2021). Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10836-10846).  
[2] Zhen, X., Du, Y., Xiong, H., Qiu, Q., Snoek, C. G., & Shao, L. (2020). Learning to learn variational semantic memory. arXiv preprint arXiv:2010.10341.  
[3] Patacchiola, M., Turner, J., Crowley, E. J., O'Boyle, M., & Storkey, A. J. (2020). Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels. Advances in Neural Information Processing Systems, 33.  
[4] Snell, J., & Zemel, R. (2020, September). Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes. In International Conference on Learning Representations.  
limitations_and_societal_impact: The authors addressed the potential limitations and negative societal impact of their work.
review_title: null

Meta-review: The paper is currently very borderline with respect to reviewer opinions. The paper is well-written, with a straightforward premise and thorough experiments. The main outstanding concern is the narrow scoping: PN/MN are no longer considered the state of the art in the field, and the narrow focus of the paper makes it unclear how well the findings will generalize to other forms of episodic learning. On the other hand, these are foundational models and are still used in many contexts. The heart of the debate is whether this will help move the field forward. I think that the NCA baseline is simple and effective enough that the answer leans towards “yes.” In particular, if this was published several years ago, I think it is likely that NCA itself could have become a foundational approach to few-shot learning - especially since it requires less tuning.

I think that the relationship between performance, and the number of pairwise distances per batch is compelling. I would like to see the following experiment (which I didn’t find in the current paper): take PN and/or MN and for each episode, randomize the episode several times. That is, take the set of support and query points, compute the loss, shuffle the batch (assigning different query/support points within the batch), and then compute the loss again. Take the loss of the batch to be the average of these, and then take a gradient step. This effectively increases the number of comparisons used in the batch, but in a way that PN/MN can exploit. Based on the hypothesis in the paper, I expect the PN/MN results to improve, but it will be interesting either way.

Decision: Accept (Poster)