Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Recently, the linear classifier on the top of pre-trained model without meta-learning performs comparably to the prototypical networks for few-shot learning. However, these approaches require re-training every time a new class arrives. This paper focuses on prototype classifier which doesn't require re-training or any meta-learning approach. It founds that the variance of the norm of feature vectors affect the performance of prototype classifiers. It then derives a generalization bound that does not depend on assumptions on the class-conditional distributions. Moreover, the paper theoretically shows that this bound decreases as the variance in norm of the feature vectors decreases, and it empirically investigaes different transformation approaches to lower this variance. Finally, the authors analyze the effectiveness of the feature transformations on multiple datasets that include mini-ImageNet, tiered-ImageNet, CIFAR-FS, CUB, and FC-100.
strengths_and_weaknesses: ### Strengths:

* The paper investigates an interesting problem for few-shot learning by proposing to use prototype classifiers which generally don't require re-training or meta-learning algorithm.

* It theoretically analyze the prototype classifier in terms of variance of the norm of the feature vectors

* It investigates several transformation approaches to minimize this variance that includes L2 normalization, EST,LDA,
EST+L2-norm and LDA+L2-norm

* Experiments on several datasets corresponding to the two different task i.e. i) standard object recognition and ii) cross-domain adaptation.

### Weakness:

* Novelty of generalization bound is limited as it derives a the bound by modifying (Cao et al., 2020)

* 1-shot performance looks good, but for 5-shot experiment, there is negligible performance boost compared to the baseline++ for miniImageNet, tieredImageNet, and CIFAR-FS in Table 1. Could authors provide any explanation or give remarks about it? It is written in the manuscript that all the methods are reimplemented. It could be that the baseline++ was not properly tuned or trained. Even in Table 2, the performance gap is negligible compared to the other methods. Moreover, the same phenomenon can be observed for cross-domain experiments in Table 4 in supplementary materials

* The authors have proposed several feature transformation methods, but there is not a single one which stands out? Can the authors please explain this behavior?
questions: Please see my questions in the weakness section.
limitations: Yes, authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper analyzes which factors lead to better performance of a prototype classifier directly applied on top of a learned representation. This stands in contrast to recent work that has shown that training a linear classifier on top is effective but computationally expensive and requires more hyperparameter tuning. Contributions include a new generalization bound on the performance of a prototype classifier that uses fewer assumptions than previous work and better shines light on the critical factors, including variance of the norm, within-class variance, and between-class variance. Experiments are conducted on several benchmark datasets and shows that with appropriate feature transformation and L2 normalization, a prototype classifier can achieve similar performance to a learned linear classifier.

strengths_and_weaknesses: Strengths
+ The aim is well-motivated. Eliminating the need for retraining a linear classifier and instead simply applying a prototype classifier would make it easier to adapt pre-trained representations to the few-shot setting.
+ The generalization bound removes simplifying assumptions of previous work and in doing so paints a more nuanced picture of the important underlying factors.
+ Experiments are extensive and consist of four datasets and multiple variations of the prototype classifier, including two feature transformation methods both with and without L2 normalization.

Weaknesses
- The clarity could be improved in some places. One is that the discussion of LDA and EST is rather light relative to the important role they play in the experiments. Another is a discussion of how the multiclass bound relate to the binary bound presented in the main paper.
- The discussion of the results could be improved. The important terms are explained at the end of Section 3.3 but it is not very intuitive how future work should go about trying to better control them. I also found Figure 2 somewhat confusing and it was not immediately clear to me how the figure relates to the experimental results.
- The analysis is limited to Euclidean distance, which is a valuable first step, but coupling this with a discussion of e.g. cosine distance would be useful. This is relevant since standard classifiers use inner products to compute logits, which is more closely related to cosine distance than Euclidean.

questions: * As mentioned above, the analysis here is done for a prototype classifier based on Euclidean distance, but cosine distance has also shown to be effective if distances can be scaled appropriately (Oreshkin et al. 2018). To what extent can the current analysis be extended to cosine distance? See also ll. 47-49, where the gap in performance is hypothesized to be due to the different in loss function. Does the gap decrease when cosine distance-based prototype classifiers are used instead?
* The generalization bound presented in Section 3.3 suggests that performance would be improved by controlling the variance of the feature norm. In previous work, hyperspherical prototype networks have been proposed (Mettes et al. 2019), where the norm of feature vectors is enforced to be 1. How does such an approach relate to the  analysis in the paper?
* How do the multiclass results compare to the binary results? Are there any qualitatively different takeaways?
limitations: More discussion about the limitations of the analysis would be useful: e.g. choice of distance and how this could possibly be addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors derive a more relaxed generalization bound for prototypical networks. Using the generalization bound, the authors propose the use of feature transformation to improve generalization of classifiers. The authors experiment with different normalization techniques on multiple few shot learning datasets and find that L2 based normalization can produce similar recognition performance as the one linear discriminant regularization.

strengths_and_weaknesses: — Strengths —

The authors derive a generalization bound that requires fewer assumptions about the data distribution family.

Experimental results on multiple datasets are consistently supported by their developed theorem.

The authors can produce reasonable amount of recognition performance without any fine-tuning and therefore can be useful for devices where backpropagation is not supported.

— Weaknesses —

The novelty of the proposed bound is incremental compared to the generalisation bound Cao et al. It might be good for the authors to elaborate point-wise differences between the bounds

Comparison studies have only been carried out against a few few shot learning methods.

More complex feature transformation methods should also have been explored such as exponential, non-linear kernels, power transforms etc to see whether they can also produce better generalization.

questions: In Figure 1, don’t the shape of the clusters vary because Euclidean distance is used for ProtoNets while dot-product is used for Linear layers ? Is it because of the loss function ? The authors can elaborate on this further.

Can t-SNE feature visualization be shown to see how feature transformation affects discriminability ?

Can this feature transformation approach improve non-prototypical few shot learning methods ?

--------- POST REBUTTAL ----------

After rebuttal I have increased form borderline accept to weak accept.

limitations: No societal impacts
review_title: null

Meta-review: metareview: It has been shown that linear classifier heads on top of pre-trained models can outperform meta-learning approaches. However, this is less adaptable than prototypical classifier heads and requires retraining with each new set of classes. This paper theoretically investigates the generalization of prototypical classifiers and uses this to explore several feature transformations to improve their performance. While there were concerns about the novelty over and above Cao et al., and some minor clarity issues, the reviewers were all generally in agreement that this is a useful contribution to the community and a good starting point for improving prototype classifiers from a theoretical perspective.
confidence: Certain
Decision: Accept