Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This works deal with the image-level annotation problem in few-shot classification. During training time, given support and query images, the authors first split images into patches, use a transformer to extract feature tokens, and learn which tokens are related to the label more. During test time, it self-learned within support images to find which tokens are more important. The experiment shows a consistent improvement.
strengths_and_weaknesses: Strength:
The transformer-only architecture makes it very clean and thus makes it potentially be a new baseline of the problem. Reweighting features to focus on task-specific information is also a reasonable idea. The experiment shows a consistent improvement.

Weakness:
It is better to discuss other feature re-weighting-based methods in few-shot classification. For example, for fine-grained few-shot classification, Lee et al. [A] shows a consistent improvement in all previous methods by adding an attention-based feature re-weighting module.

[A] Lee et al., "Task Discrepancy Maximization for Fine-Grained Few-Shot Classification.", CVPR 2022.
questions: It is unclear to me what "meta fine-tuning" means. Does it mean the step of fine-tuning the model with support data at interference time?

limitations: Yes, the author addressed the potential problem when training data is highly limited.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Motivated by supervision collapse caused by standard few-shot training on weak image-level labels, authors introduce a token-based approach based on unsupervised vision transformers that reweights tokens in an inner loop based on their discriminative power. The model achieves strong results and demonstrates the viability of vision transformer models on few-shot tasks without extra pre-training. 
strengths_and_weaknesses: STRENGTHS:

•	Demonstrated use of vision transformers for few-shot learning is on its own a neat contribution

•	Demonstrated use of purely unsupervised pre-training for few-shot learning is also a neat contribution, if not quite as novel

•	Results are impressive and span multiple benchmarks and architectures

•	Straightforward and sensible approach to token aggregation and reweighting

WEAKNESSES: 

•	The strong results come with a major caveat: the ViT-small and Swin-tiny architectures have 22M and 29M parameters respectively, while the compared baselines are almost entirely based on ResNet12, which by my recollection has only 12M parameters. While this is still less than the widely-used WRN-28-10 backbone (36.5M params), I worry that the comparisons presented in the paper are apples-to-oranges. The difference in model size should be discussed and addressed. 

•	Use of vision transformers for few shot classification deserves an empirical study all on its own. Understandably this is not provided here, but because of this it is unclear to what degree improvement is coming from the token reweighting scheme (in theory compatible with existing convolutional architectures) vs the vision transformer backbone (in theory compatible with existing few-shot classifiers). For example, how does the token reweighting scheme compare to simply training a linear classifier head on the support features from a vision transformer? Admittedly, a full comparison along both these axes would be clearly out of scope here. 

•	Similarly, there is no ablation study provided for the impact of token reweighting vs the logsumexp aggregation scheme. How much better is logsumexp aggregation than direct addition, for example, which would correspond to basic prototype comparison with reweighted averages on each prototype? More broadly, it appears that the token reweighting scheme is broadly compatible with many existing token-to-token classifiers such as CTX and FRN, and it is not clear how the logsumexp aggregator compares. 

•	More generally, the approach, while straightforward and sensible, does contain a few design choices that are not fully explained or empirically justified (for example, in addition to above, the choice of token similarity metric). 

•	A slightly relevant omitted citation: the masked inner token reweighting scheme might possibly owe some conceptual debt to Batch Folding from [Few Shot Learning with Localization in Realistic Settings, CVPR2019], which also models a support-to-support classification task with an identical image-masked leave-one-out scheme (though admittedly implemented quite differently).
questions: My main questions involve the two broad weaknesses outlined above. 

1. How should readers interpret the difference in model backbone sizes, and how does this impact the presented results?

2. Can authors provide or elaborate on the missing ablations – what is the performance contribution of the vision transformer backbone vs token reweighting vs logsumexp aggregation? And less importantly, how significant was the choice of similarity metric?

Less impactful questions/comments:

3. Is the inner-loop token-reweighting scheme compatible with more sophisticated patch-to-patch techniques, such as CTX or FRN? Or is this (understandably) left for future work?

4. Given a limited computation budget this may not be feasible, but it would be interesting to see if the results from Fig4 and Sec3.2 hold for a more standard meta-fine-tuning technique (i.e. a basic Prototypical setup). Authors appear to have produced such a model in at least one setting for Fig5 (“average”). What does the performance look like? Can authors elaborate or speculate on this? 

Some small typos:

•	Pg2 line 58: extend  extent

•	Pg2 line 66: class is in  class in

•	Pg3 footnote: in generally  is generally

•	Pg4 line 121: The in this way introduced  In this way, introduced / The introduced

•	Pg4 line 158: device  devise

•	Pg7 fig7 caption: SDG  SGD
limitations: The analysis of 1-shot effectiveness and discussion of smaller training datasets is insightful. The entanglement of vision transformer benefits with token reweighting benefits in presented results is not discussed. Societal impacts are not discussed, though do not extend beyond those of few-shot learning in general.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses the problem of few-shot classification. The main idea is to establish semantic correspondences between the patches from the support and the query images. These correspondences are then used to reason which class a query image belongs to. In order to down-weight the impact of background patches when performing classification, the authors also introduce an online optimization strategy to determine which patches in the support images are most informative when performing few-shot classification. The method uses the Vision Transformer to encode the patches in the support and query images. In order to learn strong generic features, the Vision Transformer is trained in an unsupervised manner using the masked image modelling task. The self-supervised pre-training is shown to provide better results then the supervised counterpart. The proposed method obtains state-of-the-art results on four few-shot classification benchmarks. 
strengths_and_weaknesses: ## Strengths
**S1**: The paper is well written and easy to read.

**S2**: The proposed few-shot classifier using patch-wise correspondences is novel and interesting. The online optimization allows determining which regions are most crucial to perform classification and can be helpful specially in case of clutter in the support set images. The use of patch-wise correspondences allows determining the class of the query image by jointly reasoning over the support set as well as the query.

**S3**: The self-supervised pre-training of the Vision Transformer makes sense, specially in the context of few-shot learning, to learn generic feature representation. 

**S4**: The proposed approach is shown to obtain state-of-the-art results on 4 standard benchmarks (mini ImageNet, Tiered ImageNet, CIFAR-FS, FC100).

**S5**: The authors provide helpful analysis and ablation studies, showing the impact of the major contributions.


## Weaknesses
I do not have any major issues with the paper. Some minor issues which could be addresses

**W1**: As shown in Fig. 4, the use of self-supervised pre-training for Vision Transformer provides a significant improvement, compared to supervised training. It would be interesting to see what performance existing classifiers (e.g. learning a linear classifier, prototype) obtain when using the same backbone network. This would help evaluate the benefits of proposed embedding similarity based classifier.

**W2**: In the sota comparison, most previous methods use ResNet-12 backbone, while the authors employ ViT-small and Swin-Tiny. It will be helpful if the authors include the model sizes (number of parameters) for each of these backbones for comparison.

**W3**: The approach computes patch-wise correspondence between all the support and query images. I wonder if this could become computationally expensive when dealing which large number of number classes, or when each class has more samples (e.g. 100 way - 30 shot classification). A discussion on this would be beneficial.
questions: Please check the comments listed under Weaknesses.
limitations: The authors discuss the limitations of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper adopts self-supervised trained Vision Transformer (ViT) architecture as the feature extractor, deriving patch-level representations for few-shot classification problems. To exploit the relation across patches, the authors propose a token importance reweighting mechanism (which is required to perform during both training and testing stages). The experimental results show satisfactory performance in several commonly-used FSL benchmarks to verify the effectiveness of this method.
strengths_and_weaknesses: The overall paper is easy to follow. The idea of utilizing the self-supervised trained ViT architecture to derive patch-level representations is interesting and seems effective for few-shot learning tasks. Since self-supervised model pre-training is agnostic to the image-level class labels, the trained model is more generalizable for downstream tasks. Also, applying ViT to extract patch-level features allows the model to produce more fine-grained information. However, I have the following concerns about this work:

1. In Fig. 4, the authors show that self-supervised pre-training performs significantly better than the supervised-pre-training counterpart. However, in prior SSL literature (e.g., [16, A]), SSL pre-training only slightly outperforms supervised pre-training (sometimes or even worse than it). A proper explanation (or insight) that the proposed SSL pre-training surpasses a large margin over the supervised counterpart reported in Fig. 4 is needed.

2. The t-SNE visualization in Fig. 5, only verifies that the patch-level embeddings derived from the same “instance” are clustered together, and those from different instances are separated from each other. However, this figure only explains/visualizes separation between different instances, but not the discrimination between different classes (which is much more important for FSL). It is desirable to see whether the embeddings extracted from the same “class” are gathered, while the embedding of different classes would separate far from each other.

3. Since the title of this paper emphasizes the aspect of “generalization” in few-shot learning, one would expect learning strategies with results/comparisons with recent cross-domain few-shot learning works (e.g., [B]). In other words, cross-domain FSL aims to transfer the learned knowledge to the novel classes in unseen target domains (showing generalization ability).

4. Although this paper applies Masked Image Modeling (MIM) as the pretext task for pre-training ViT, use of other self-supervised pre-training approaches like contrastive learning (e.g., DINO [C], MoCo v3 [A]) would be possible. It will be also good if the authors provide some insights or comparisons about the choice of the self-supervised pre-training approach. If MIM is desirable for this task, more explanations and supports would be needed.

5. The need to perform additional learning for test data (few-shot instances from novel classes) is needed for the proposed work (but not necessarily for a number of SOTAs). I'd like to see how the authors would elaborate on this issue.

[16] He et al. “Masked autoencoders are scalable vision learners.” CVPR 2022
[A] Chen et al. “An Empirical Study of Training Self-Supervised Vision Transformers.” ICCV 2021
[B] Chen et al. “A Closer Look at Few-shot Classification.” ICLR 2019
[C] Caron et al. “Emerging Properties in Self-Supervised Vision Transformers.” ICCV 2021




questions: Please see my questions raised in the above weakness part.
limitations: The authors did provide discussions on the limitation of the proposed work.
review_title: null

Meta-review: metareview: 
 This paper tackles few-shot learning with a transformer architecture and, inspired by the intuition that fine-grained information is ignored in existing methods, uses an inner-loop token re-weighting method to improve results. Overall the reviewers appreciated the use of modern architectures (Vision Transformers), the reasonableness of the re-weighting intuition, and experimental results. Concerns were raised about comparison to existing methods with similar intuitions (e.g. [A] mentioned by eF5W), fairness of the comparison with respect to model capacity and in general ablations demonstrating that it's the method (not transformers by themselves) leading to improved results, and lack of principled explanations for the design choices, and computational complexity. 

  The authors provided strong rebuttals, including new experiments using linear classifiers and prototypical approaches, use of smaller models, and a demonstration of potential pruning methods to address computational complexity. The reviewers were overall receptive to the rebuttal, and all recommended acceptance of this paper after some back-and-forth. The paper provides both a nice benchmark applying Vision Transformers to few-shot learning as well as a method that is demonstrably better through ablation studies. Therefore, this paper provides several nice contributions to the community, and I recommend acceptance. 
confidence: Certain
Decision: Accept