Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper's starting point is the question whether the episodic training is beneficial, or not, for FSL / Prototypical Networks. The work can be seen as a follow-up of the recent works showing that simple baselines can outperform rather sophisticated few-shot learning models. Towards answering this question, this paper points out that Prototypical Networks (PN) are related to Neighborhood Component Analysis (NCA), and NCA can be considered as an episodic training-free alternative of PN.

In more detail, PN aims to learn per-class prototypes based on sample averaging in the feature space. NCA, in contrast, aims to maximize the ratio of total similarity between same-class example pairs to the total similarity between different-class pairs. Due to their similarities in terms of their formulations, the paper claims that NCA loss can be considered as an alternative to PN loss to do non-episodic representation learning for few-shot learning purposes. In addition, the paper has a few strong claims, such as episodic training is “detrimental to learning” and “under no circumstance beneficial to differentiate between support and query set within a training batch". Clearly, these are intriguing claims.
 
However, there is a gap between the claims and the experimental validation. First, even if ProtoNet loss and NCA loss seem to be similar to each other, they're nevertheless different models, and it takes quite a significant manipulation to convert PN to NCA. Therefore, the fact one particular non-episodic-training based model gives superior results compared to those of PN with episodic-training, does tell us much about detrimental effects of episodic training for PN or in general. Second, while the paper's observations that NCA has the advantage of using more pairwise similarities within a batch compared to PN is indeed insightful, it rather points out to certain weaknesses in the way per-batch / per-episode data is being utilized by the PN formulation, instead of problems about episodic training.

Overall, the paper has interesting observations about PN's weaknesses and shows why one particular simple non-episodic training / non meta-learned approach (NCA) can yield superior results compared to PN, which is a relatively mode sophisticated & well-established approach. However, the paper's (over-strong) claims remain mostly unsupported, which makes the otherwise interesting work poorly framed. The paper, with more water-tight arguments only, could otherwise be a valuable contribution but it requires quite significant & fundamental revisions throughout the paper, therefore, is not ready for publication in its current form.

Post-rebuttal: I would like to thank for the detailed responses and the careful revisions made in the paper. Overall, the paper is now definitely improved in certain ways and initiates an important discussion on the value of episodic training & classifier synthesis for few-shot learning, as opposed to typically-simpler metric-learning based approaches. The paper also approaches this problem from an interesting point of view, by focusing on sample utilization in the episodic training of PN.  

However, I still find that the the paper remains somewhat weak in its current form for the following reasons:
- I maintain my view that NCA vs PN are not direct alternatives to each other, considering that PN allows learning a representation that is optimized for class-average to sample comparisons, whereas, NCA uses a sample-to-sample distance based loss. The fact that the very construction of these two models, despite the similarities pointed out, blurs the strength of the overall NCA vs PN based discussion on the value of episodic training.
- The claims about the advantages of non-episodic training of NCA is mainly based on the observation that NCA creates more positive/negative labels. However, it is not clear whether it is the more efficient utilization of training samples or just the differences in terms of the predictive model formulations. (Perhaps, averaging based prototype computation is a bad idea, after all, which may not have directly anything to do with episodic training.)
- To this end, Fig. 3 is indeed interesting, but again the results are not very clear. Here, careful optimizer re-tuning specially for each case can be necessary as subsampling degrades the gradient approximation quality, which creates the question how much fundamentally important efficiency in batch utilization is, as long as one uses a proper optimizer. 
Overall, I think the paper makes a valuable step in an interesting direction but the paper fails to make a strong-enough case. Overall, I  improve my rating by a single level to 4, but find that the paper is not stronger than this in its current form.
review_title: Interesting work, unsupported claims

final_score: rating: 7: Good paper, accept
review_content: #### Summary

The submission investigates the properties of episodic training and its impact on learning using Prototypical Networks as a case study. The paper draws a connection between Prototypical Networks and Neighbourhood Component Analysis (NCA), noting that their loss functions are similar but that NCA is trained non-episodically, which allows it to learn from the relationship between all example pairs in a batch.

When controlling for batch size, the paper claims to show that NCA (combined with a nearest-centroid inference strategy) performs better than Prototypical Networks, as evidenced by experiments on CIFAR-FS and mini-ImageNet. Ablation experiments are performed, claiming to show that applying the NCA loss to batches sampled episodically allows Prototypical Networks to bridge the performance gap with NCA, and that the partition of examples within a batch into support and query sets is detrimental to Prototypical Networks training. Finally, NCA is evaluated alongside comparable competing approaches on mini-ImageNet, CIFAR-FS, and tiered-ImageNet, and is claimed to yield results comparable or superior to the state-of-the-art.

#### Strengths and weaknesses

* **+** The value of episodic training is increasingly being questioned, and the submission approaches the topic from a new and interesting perspective.
* **+** The connection between nearest-centroid few-shot learning approaches and NCA has not been made in the literature to my knowledge and has potential applications beyond the scope of this paper.
* **+** The paper is well-written, easy to follow, and well-connected to the existing literature.
* **-** The extent to which the observations presented generalize to few-shot learners beyond Prototypical Networks is not evaluated, which may limit the scope of the submission’s contributions in terms of understanding the properties of episodic training.
* **-** The Matching Networks / NCA connection makes more sense in my opinion than the Prototypical Networks / NCA connection.
* **-** A single set of hyperparameters was used across learners for a given benchmark, which can bias the conclusions drawn from the experiments.

#### Recommendation

I’m leaning towards acceptance. I have some issues with the submission that are detailed below, but overall the paper presents an interesting take on a topic that’s currently very relevant to the few-shot learning community, and I feel that the value it brings to the conversation is sufficient to overcome the concerns I have.

#### Detailed justification

The biggest concern I have with the submission is methodological. One one hand, the authors went beyond the usual practice of reporting accuracies on a single run and instead trained each method with five different random initializations, and this is a practice that I’m happy to see in a few-shot classification paper. On the other hand, the choice to share a single set of hyperparameters across learners for a given benchmark leaves a blind spot in the evaluation. What if Prototypical Networks are more sensitive to the choice of optimizer, learning rate schedule, and weight decay coefficient than NCA? Is it possible that the set of hyperparameters chosen for the experiments happens to work poorly for Prototypical Networks? Would we observe the same trends if we tuned hyperparameters independently for each experimental setting? In its current form the submission shows that Prototypical Networks are sensitive to the hyperparameters used to sample episodes *while keeping other hyperparameters fixed*, but showing the same trend while doing a reasonable effort at tuning other hyperparameters would make for a more convincing argument. This is why I take the claim made in Section 4.2 that "NCA performs better than all PN configurations, no matter the batch size" with a grain of salt, for instance.

I also feel that the submission misses out on an opportunity to support a more general statement about episodic training via observations on approaches such as Matching Networks, MAML, etc. I really like the way Figure 1 explains visually how Prototypical Networks miss out on useful relationships between examples in a batch and is therefore data-inefficient. To me, this is one of the submission’s most important contributions: the suggestion that a leave-one-out strategy could allow episodic approaches to achieve the same kind of data efficiency as non-episodic approaches, alleviating the need for a supervised pre-training / episodic fine-tuning strategy. To be clear, I don’t think the missed opportunity would be a reason to reject the paper, but I think that showing empirically that the leave-one-out strategy applies beyond Prototypical Networks would make me lean more strongly towards acceptance.

The connection drawn between Prototypical Networks and NCA feels forced at times. In the introduction the paper claims to "show that, without episodic learning, Prototypical Networks correspond to the classic Neighbourhood Component Analysis", but Section 3.3 lists the creation of prototypes as a key difference between the two which is not resolved by training non-episodically. From my perspective, NCA would be more akin to the non-episodic counterpart to Matching Networks without Full Contextual Embeddings – albeit with a Euclidean metric rather than a cosine similarity metric – since both perform comparisons on example pairs.

This relationship with Matching Networks could be exploited to improve clarity. For instance, row 6 of Figure 4 can be interpreted as a Matching Networks implementation with a Euclidean distance metric. With this in mind, could the difference in performance between "*1*-NN with class centroids" and *k*-NN / Soft Assignment noted in Section 4.1 – as well as the drop in performance observed in Figure 4’s row 6 – be explained by the fact that a (soft) nearest-neighbour approach is more sensitive to outliers?

Finally, I have some issues with how results are reported in Tables 1 and 2. Firstly, we don’t know how competing approaches would perform if we applied the paper’s proposed multi-layer concatenation trick, and the idea itself feels more like a way to give NCA’s performance a small boost and bring it into SOTA-like territory. Comparing NCA without multi-layer against other approaches is therefore more interesting to me. Secondly, 95% confidence intervals are provided, but the absence of identification of the best-performing approach(es) in each setting makes it hard to draw high-level conclusions at a glance. I would suggest bolding the best accuracy in each column along with all other entries for which a 95% confidence interval test on the difference between the means is inconclusive in determining that the difference is significant.

#### Questions

1. In Equation 2, why is the sum normalized by the total number of examples in the episode rather than the number of query examples?
1. Can the authors comment on the extent to which Figure 2 supports the hypothesis that NCA is better for training because it learns from a larger number of positives and negatives? Assuming this is true, we should see that Prototypical Networks configurations that increase the number of positives and negatives should perform better for a given batch size. Does Figure 2 support this assertion?
1. Can the authors elaborate on the "no S/Q" ablation (Figure 4, row 7)? What is the point of reference when computing distances for support and query examples? Is the loss computed in the same way for support and query examples? The text in Section 4.3 makes it appear like the loss for query examples is the NCA loss, but the loss for support examples is the prototypical loss. Wouldn’t it be conceptually cleaner to compute leave-one-out prototypes, i.e. leave each example out of the computation of its own class’ prototype (resulting in slightly different prototypes for examples of the same class)? In my mind, this would be the best way to remove the support/query partition while maintaining prototype computation, thereby showing that the partition is detrimental to Prototypical Networks training.

#### Additional feedback

1. This is somewhat inconsequential, but across all implementations of episodic training that I have examined I haven’t encountered an implementation that uses a flag to differentiate between support and query examples. Instead, the implementations I have examined explicitly represent support and query examples as separate tensors. I was therefore surprised to read that "in most implementations [...] each image is characterised by a flag indicating whether it corresponds to the support or the query set [...]"; can the authors point to the implementations they have in mind when making that assertion?
1. I would be careful with the assertion that "during evaluation the triplet {w, n, m} [...] must stay unchanged across methods". While this is true for the benchmarks considered in this submission, benchmarks like Meta-Dataset evaluate on variable-ways and variable-shots episodes.
1. I’m not too concerned with the computational efficiency of NCA. The pairwise Euclidean distances can be computed efficiently using the inner- and outer-product of the batch of embeddings with itself.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates the usefulness of episodic learning in prototypical learning which is a popular practice in few-shot learning. The authors propose a non-episodic prototypical network which basically corresponds to the classical neighborhood component analysis and they claimed that this network reliably improves over its episodic counterpart in multiple datasets. I have the following comments on the paper:

1. The sections 3.1, 3.2 and 3.3 are not the contributions of the paper. Only section 3.4 can be considered as something new from experimental point of view and not methodologically new. k-NN, 1-NN with class centroids and soft assignments are all some specific experimental settings. Therefore, I do not see the technical contributions of the paper other than the claimed novel experimental settings which is also marginal.

2. The paper shows a robust experimentations and comparisons with prior arts, however, I don't understand how the three settings mentioned in the section 3.4 are evaluated in the tables.

3. I am curious if you have done a comparison with baseline NCA, i.e. equation (3). I have not found the comparison in A.1 which only contains some discussions but no direct comparison.

4. Anyways, The paper is written in good English and I haven't found any typos yet.

Based on my current understandings and above comments, currently I recommend for a weak rejection. However, I would like to follow the discussions on the paper and understand the contributions well.
review_title: Technical contributions are not clear and experiments are generic

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes to use neighborhood component analysis in lieu of prototype loss to train embedding functions of few-shot learning. This method takes full advantage of relations between all sampled points in an episode to facilitate learning, and it removes the distinction between support and query samples during meta-training time.

Reason for score: Overall, I lean towards reject. This paper proposes an interesting method that improves upon Prototypical Networks, and performs on-par with other baseline methods. However, the paper does not advance our understanding of how episodic training interacts with few-shot learning beyond the addition of more performance numbers to compare against. 

Pros: The proposed method is a straight-forward improvement to Prototypical networks. In the wide range of scenarios evaluated in the experiments, the proposed method consistently outperforms ProtoNet. This proposed method should also be easy to implement, making integration with other FSL methods based on ProtoNets feasible. 
While not exactly nouveau, using NCA to train embedding functions has not been done before (to my best knowledge) and is a theoretically sound approach.

Cons:
1. Perhaps unsurprisingly, the performance of NCA is lower than some FSL methods that use additional capacity. Even so, I think presenting only favorable comparisons in the performance tables is counter productive as it fails to capture the research context of this work. 
2. The conclusion from the ablation studies on batch size is still unclear to me. The “NCA fixed batch composition” setting seems to perform better than NCA in 3 of the 4 plots in figure 4. This particular setting is interesting as it allows control of the number of classes in each batch. As the batchsize is fixed in the ablation, we won’t know what is the relation between the number of ways and performance of this fixed batch variant of NCA. This ablation is also confusing in that neither “no proto” nor “no S/Q” significantly improves performance, yet their combination performs well. A full ablation that systematically covers all hyperparameters would be helpful in furthering understanding in this direction.
3. The motivation of the paper feels unclear: on one hand the authors claim that they aim at understanding the (un)usefulness of episodic learning, yet on the other hand this paper doesn’t present any results beyond the final performance number to aid with this understanding. Visualizations and/or theoretical arguments would be greatly appreciated.

Minor points (suggestions, not related to score):
Introduction
“These results legitimately cast a doubt” -> “These results cast a doubt”
Related Work
“Between 2016 and 2017” feels unnecessary
“Matching and Prototypical…weighted by either an LSTM or a simple average, respectively”: Matching Networks also proposes a sample average variant. Their main difference lies in one uses cosine similarity while the other uses euclidean distance.
“Differently from these papers” -> Different
BACKGROUND AND METHOD
In 3.3, point 3 is not necessarily correct. Why would some examples be more likely than others in the episodic scheme? All FSL benchmarks (used in this paper) are close to class balanced, and an hierarchical sampling scheme for episodes results in full coverage each epoch just like standard supervised learning.
In 3.4 “which is the probability that image i is sampled from image j”. This is under the assumption that each support image defines a Gaussian in the embedding space. This is not true in general. The probabilistic interpretation of NCA should be further explained. 

[Post rebuttal]
I am still leaning against the acceptance of this paper due to concerns about the limited impact of this submission. The topic of "limitations of episodic training" has been widely studied and in fact most SoA few-shot learning methods adopt a combination of supervised and episodic training for this precise reason. 

The exploration into the relation between the number of sample pairs and learning performance is indeed correct, but no strong conclusions can be reached due to the logical jumps required. The authors established that the performance is correlated with number of pairs with a log/square-root curve, and that ProtoNet performance is similar to that of subsampled NCA (fig3). The mechanism behind why this is has not been elucidated. I think many questions can be explored to strengthen this paper, for example:
Are classes embedded tighter together? 
Are hard negatives pushed further apart? 
Does NCA induce a non-euclidean geometry that is more expressive than the euclidean geometry naturally induced by ProtoNets?
Can the classification problem be converted into a pair comparison problem, so that PAC learning theory can be used to explain the shape of this curve? 
What is the sample complexity of the NCA classifier compared to the Prototypical classifier?

Regarding the proposed method, NCA is certainly an improvement over ProtoNets, but performs worse than existing methods in most experiments. This makes me doubtful of the impact of this work on the methodological front. Better modelling of relationships between few-shot examples has been implicitly and explicitly studied too. For instance, many works adopt sample level set functions in the form of transformers, attention modules, and graph neural networks. Arguably, these additional architectures are more expressive "deep" alternatives to NCA, and hence achieves better performance than the proposed method.
review_title: Review of On Episodes, Prototypical Networks, and Few-Shot Learning 

Meta-review: This paper is right on the borderline. It questions the utility of episodic training from a novel perspective, driven by a comparison to NCA, with thorough experiments. The hypothesis that more pairwise comparisons per batch/episode benefit learning is also quite interesting, but some reviewers didn’t feel this was convincingly presented.

Prototypical networks are indeed a popular method for FSL, but I do as well think that NCA is more closely related to matching networks, and that it makes more sense for that to be the focus of experimentation. Matching networks involve more direct pairwise comparisons, and so a leave-one-out baseline with this model would probably be a useful comparison.

While I appreciate the desire to focus on a fundamental aspect of FSL and not chase state of the art, I think that it’s important to show where one should go from here. That is, as the reviewers pointed out there are many mechanisms beyond vanilla PNs that have yielded better results than those presented in this paper. I don’t think matching SOTA is necessary here, but it would be nice to show that the insights here complement other mechanisms in FSL.

Decision: Reject