Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission introduces a generalization bound for Prototypical Networks that does not depend on assumptions on the class-conditional distributions. This bound decreases as the variance in norm of the feature vectors decreases, and the authors empirically investigate five feature transformation approaches that are meant to lower this variance: L2-normalization (L2-norm), variance-normalization, Linear Discriminant Analysis (LDA), Embedding Space Transformation (EST), and EST+L2-norm.

Experimental results are reported on the mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100 benchmarks in the 1-shot and 5-shot settings, using ResNet-12 or ResNet-18 as the backbone architecture. The five feature transformation approaches are applied on top of the Baseline and Baseline++ learners before using the extracted features to build a prototypical classifier. The submission compares against the following:
- Prototypical Networks
- Baseline and Baseline++ with a linear classifier on top of the extracted features
- Baseline and Baseline++ with Wang et al.'s centering+L2-norm approach on top of the extracted features.
- Baseline and Baseline++ with no additional feature transformation and a prototypical classifier on top of the extracted features.

The submission concludes that EST+L2-norm performs best, and that its performance-boosting effect decreases as the number of examples per class increases.
main_review: The topic studied in the paper is relevant to the few-shot learning research community, and the prospect of applying prototypical classifiers to a wider range of pre-trained feature extractors is appealing. The related work section is thorough and does a good job of contextualizing the proposed approach.

My expertise is limited in terms of evaluating the correctness of the derived bound; I will leave this to other reviewers and take it at face value in my review. The main issue I have with the submission is clarity. In several instances the notation tripped me up, or the writing was imprecise, or some experimental details were unclear:
- The term "linear classifier with fine-tunining" doesn't make sense to me. I know this formulation has been used in the literature before, but calling it "fine-tuning" is incorrect when the backbone is frozen and a new classifier is learned from scratch.
- Are there typos in the joint probability distributions defined in Section 3.1's third line? In both cases, the index "i" in the product is not used in the expression.
- In Equation 2, should c ~ \tau read as c_1, c_2 ~ \tau?
- Where is Tr(\Sigma_c) defined? It's used in Section 4.4, but the closest definition I could find was the within-class variance \Sigma_\tau discussed in item 2 at the bottom of page 5.
- Since the drawbacks of Cao et al.'s bound are discussed in this work, it would be good to show the bound in the submission. It would avoid having to pull up Cao et al.'s paper and put it side by side with the submission just to compare the bounds.
- Comparing notations with Cao et al., I find their choice of notation and the way they introduce it much clearer, and I wish this submission adopted their notation (with attribution, of course).

I also have some reservations about the experiments:
- Can the authors expand on why the center loss and affinity loss are left out of the comparisons? Is it because they need to be applied during pre-training? If so, it should be made clearer.
- Again, the following is common to a lot of few-shot classification papers, but given the tiered-ImageNet results, what additional evaluation signal is expected from the mini-ImageNet results? Isn't it redundant? The same can be said about FC100 vs CIFAR-FS.
- How is Baseline++ trained in the re-implementation? Section 4.2 mentions the normalization of the projection layer and features as a difference between it and Baseline, but omits the cosine classifier.
- The gap between B+@FT and B@FT is surprisingly small, given the improvements of Baseline++ over Baseline reported by Chen et al. Do the authors have an explanation? Is it possible that Baseline++ was not pre-trained properly in their re-implementation? In particular, Baseline++ scales the normalized weight vectors independently for each class, which is not clearly mentioned in Chen et al.'s paper, but is necessary for good performance. Is that detail also present in the re-implementation?
- The submission goes beyond standard practice and uses a principled criterion for bolding table entries by comparing 95% confidence intervals, which is a good thing that I would like to see in more submissions. I would like to see those 95% confidence intervals in the Appendix, though, so that future papers can also compare to those confidence intervals.

In terms of significance, I would argue that "comparable with ProtoNet and the fine-tuning approach" is not that high a bar. I like the premise of being able to plug a prototypical classifier into any pre-trained model, and I feel that it could have been better exploited in the submission, for instance by using a larger-scale pre-trained classifier out of the box and achieving even better few-shot classification accuracies. I would say however that I am not entirely sold on a prototypical classifier having more desirable properties than a linear classifier or an SVM (like is done with MetaOptNet (Lee et al., 2019)). The paper justifies this through the lens of hyperparameter tuning, but work like Big Transfer (Kolesnikov et al., 2020) shows that heuristics such as BiT-HyperRule can be designed to avoid hyperparameter sweeps.

Additional questions/comments:
- The abstract alternates between present and past verb tense. It would flow better if it was consistent in its verb tense use.
- Since the variance-normalization transformation is described as a variation of the Mahalanobis distance with a diagonal covariance matrix, it would be relevant to cite Bateni et al. (2020)'s Simple CNAPs approach, which uses a Mahalanobis distance-based classification criterion.

References:
- Bateni, P., Goyal, R., Masrani, V., Wood, F., & Sigal, L. (2020). Improved few-shot visual classification. In CVPR.
- Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020). Big Transfer (BiT): General visual representation learning. In ECCV.
- Lee, K., Maji, S., Ravichandran, A., & Soatto, S. (2019). Meta-learning with differentiable convex optimization. In CVPR.
summary_of_the_review: The paper studies an interesting topic but lacks clarity. I have concerns with the experimental setup, and with how the paper justifies the significance of its proposed approach.


---

**POST-REBUTTAL**: Most of my concerns are addressed, but I remain concerned with the submission's significance (see discussion thread).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes how a prototype classifier works well without fine-tuning and meta-learning. The feature is trained with cross-entropy loss with a linear projection layer. In testing, the classifier is a Prototype classifier.

This paper derives a novel generalization bound for the prototypical network and shows that focusing on the variance of the norm of a feature vector can improve performance.
The proposed upper bound is a modification of (Cao et al., 2020), and it does not require the features distributed on any specific distribution. Each covariance matrix does not have to be the same among classes. 

The authors experimentally investigated several normalization methods (L2-N, V-N, LDA, EST, EST-L2-N) for minimizing the variance of the norm.

main_review: [Strengths]
- It is interesting to focus on the different loss functions in feature training (softmax loss) and testing (Prototype classifier). 

- The method which works well without meta-learning and fine-tuning is practical. 

- L2 normalization is well known to improve performance. The paper explains how the variance of the norm is related to the bound of a prototype classifier. 

[Weakness]

The comparison of normalization is incomplete. 
- The L2-N is applied only to EST. Why L2-N is not applied to V-N and LDA. 
- It is not clear if B@CL2 and B+@CL2 use a prototypical classifier, or fine-tuning is conducted. 

The discussion of Sec.4.4 is not completely validated. It claims that  the L2-norm normalization less influences ${\rm Tr}(\Sigma)$ than ${\rm Tr}(\Sigma_c)$. These values before and after the normalization also should be compared. 

There are several unclear or incorrect descriptions. 

- It is claimed that the proposed bound does not assume any specific distribution and the same covariance matrix among classes. However, it seems that the variance and mean of Gaussian distribution are used when expanding the results of one-sided Chebyshev's inequality, eg., Eq.(22). Also, p.15 Eq.(29), the first line to the second line, it seems to be assuming ${\rm Tr}(\Sigma_{c2}) = {\rm Tr}(\Sigma_{c1})$.

- P.5  What means “a secondary expression for the ratio of between-class variance and within-class variance.”?

- P.6  The authors wrote that “Regarding equation 9, the ratio of the Euclidean distance between the class mean vectors and the between-class variance is supported to be constant.” Why can it be supported to be constant? The feature transformation changes the distance of the mean vectors of each class. 

- Abstract; “the same performance can be obtained “. The performance is not completely the same as fine-tuning nor meta-learning.

- Variance-normalization of Eqs.(12) and (13) are not clear. Is the different variance calculated for each class, or is the variance common for all samples? 

- For covariance matrix, $(\phi(x)-\mu_c)^T( \phi(x) - \mu_c )$ seems to be $(\phi(x) - \mu_c) (\phi(x) - \mu_c)^T$ and $(\mu_c - \mu)^T(\mu_c - \mu)$ seems to be $(\mu_c - \mu)(\mu_c - \mu)^T$. 

- p.12 the last row: $\lambda \in R^D$ seems to be $\lambda \in R^1$.


summary_of_the_review: The theoretical analysis of the variance of the norm to impact the bound of the prototypical classifier is interesting. However, the descriptions seem to contain errors, and experimental validation is not satisfactory. 

==
Post rebuttal
==

My concerns have been solved. I have slightly increased the score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors have derived a generalization bound for the prototypical networks, which provides some insights towards improving protonets without finetuning using simply normalizing the feature vectors and reducing the variance of the norm. The derivation has relaxed the previous assumption of considering feature distribution and class covariance matrix to be a particular form. To justify the claim, the authors experimented on several feature transformation methods on standard few-shot benchmarks.


main_review: The interesting aspects of the paper are as follows:
1. Theoretical analysis of protonets with relaxed constraints is interesting.
2. Evaluation on benchmarks and experimental results seems to be consistent.

However, some concerns are as follows:
1. The theoretical analysis just relax the assumption of having particular class distribution (in both class distribution and class covariance matrix), but the overall approach is marginally novel compared to Cao et al.
2. The theoretical results derived in this paper are quite intuitive though, e.g., L2 normalization of features is effective. These are quite well-known tricks. I was looking for more interesting insights which are non-trivial.
3.  Variance normalization performs worse - can you connect this with the theoretical analysis performed here?
4. How this paper addressed the second drawback, i.e., what is the optimal feature transformation, that is not clear to me. Several feature transformation methods are evaluated, however, there is no guarantee that the chosen feature transformation method is optimal. 
5. The flow of the paper might be improved, specially in the experimental results section.
summary_of_the_review: Overall the theoretical analysis is interesting, however the novelty seems to be marginal. The insights provided in the paper are quite intuitive  and proper connection with theory and experiments need to be explained better. I would like to see how the authors response the raised issues.
review_title: null

Meta-review: This paper explores prototype vs linear classifiers for few-shot learning. It has been found that pre-training a classifier network, followed by training a linear head can produce competitive results to meta-trained prototypical networks. A natural question therefore is whether one can directly derive prototypical classifiers from pre-trained classifiers. Naively applying this idea doesn’t work well in practice though, and this paper performs a theoretical investigation to determine why. The theory is a generalization of Cao et al., 2020, that doesn’t require assumptions on the class-conditional distributions. The theory suggests that the variance of the norm of the feature vectors plays a role, so the paper explores a few feature transformations to reduce this. It demonstrates on a few benchmark datasets that transforming the feature vectors can indeed allow us to create prototypical classifiers from pre-trained networks. As a minor quibble, the paper twice refers to “direction of the norm of class mean vectors” - This should just be the direction of the class mean vectors, right? Norm is not a direction, it’s a magnitude.

During the discussion phase, a number of questions arose, mainly around the clarity of the presentation and a request for a few additional baselines (e.g., L2 normalization combined with LDA/V-N). These points were resolved by the authors. The main outstanding issue is whether there is enough novelty/significance in the paper to merit acceptance, and on that point, the reviewers felt this is borderline. On the one hand, the theory is more general and does directly point to aspects of the feature space that can yield better generalization results. On the other hand, tricks like L2 normalization are already known, and the utility of prototype classifiers over linear classifiers like SVMs is unclear.

After careful consideration, further discussion with the reviewers, as well as the program committee, it was generally agreed that this paper does not quite meet the bar in terms of the novelty or significance of its contribution. The authors mentioned time and space benefits relative to fine-tuned classifiers in their response, and I think one way to improve the paper would be to demonstrate this advantage in a real-world application or challenging scenario.
Decision: Reject