Rethinking Generalization in Few-Shot Classification
Authors: Markus Hiller
Rongkai Ma
Mehrtash Harandi
Tom Drummond
Venue: NeurIPS 2022 Accept (2022)
Keywords: Few-shot learning
Vision Transformer
Transformer
Classification

Abstract: Single image-level annotations only correctly describe an often small subset of an image’s content, particularly when complex real-world scenes are depicted. While this might be acceptable in many classification scenarios, it poses a significant challenge for applications where the set of classes differs significantly between training and test time. In this paper, we take a closer look at the implications in the context of few-shot learning. Splitting the input samples into patches and encoding these via the help of Vision Transformers allows us to establish semantic correspondences between local regions across images and independent of their respective class. The most informative patch embeddings for the task at hand are then determined as a function of the support set via online optimization at inference time, additionally providing visual interpretability of ‘what matters most’ in the image. We build on recent advances in unsupervised training of networks via masked image modelling to overcome the lack of fine-grained labels and learn the more general statistical structure of the data while avoiding negative image-level annotation influence, aka supervision collapse. Experimental results show the competitiveness of our approach, achieving new state-of-the-art results on four popular few-shot classification benchmarks for 5-shot and 1-shot scenarios.

Rethinking Generalization in Few-Shot Classification
Markus Hiller*
Rongkai Ma Mehrtash Harandi²
Tom Drummond'
School of Computing and Information Systems, The University of Melbourne
Depparmment of Electrical and Computer Systems Engineering, Monash University markus hi hillerdstudent wnwmelb.m...mm edu.au frongkai ma, mehrtash. harandi @mmonash edu tom. drunmond@umimell edu
Abstract
Single image-level annotations only correctly describe an often small subset of an image's 's content, particularly when complex real-world scenes are depicted. While this might be acceptable in many classification scenarios, poses significant challenge for applications where the set of classes differs significantly between training and test time. In this paper, we take closer look at the implications in the context of few-shot learning. Splitting the input samples into patches and encoding these via the help of Vision Transformers allows us to establish semantic correspondences between local regions across images and independent of their respective class. The most informative patch embeddings for the task at hand are then determined function of the support set via online optimization at inference time, additionally providing visual interpretability of 'what matters most' in the image. We build recent advances in unsupervised training of networks via masked image modelling to overcome the lack of fine-gra ome-grained labels and learn the more general statistical structure of the data while avoiding negative image-level annotation influence, aka supervision collapse. Experimental results show the competitiveness of our approach, achieving new state-of-the-ar results on four popular few-shot classification benchmarks for 5-shot and 1-shot scenarios.
1
Introduction
Images depicting real-world scenes are usually comprised of several different entities, e.g., family walking their dog in a park surrounded by trees, person patting their dog (Figure Nevertheless, popular computer vision datasets like ImageNet (39) assign a single image-level annotation to classify their entire content. Hence, such a label only correctly applies an often small subset of the actual image. As result, models trained on such data via srudient-based methods learn to ignore all seemingly irrelevant information, particularly entities that occur across differently labelled images. While this might be acceptable for conventional classification methods that encounter a diverse number of training examples for all classes they are expected to distinguish, it poses a major but often overlooked challenge for applications where the set of classes differs between training and test time. One such affected area is few-shot learning (FSL) where approaches are expected to correctly classify entirely new classes at test time that have never been encountered during training, just by being provided with a few (e.g., one or five) samples for each of these new categories. During test time, entities that have not been part of the set of training classes and have possibly been perceived as irrelevant might very well be part of the set test classes - yet, the method was taught to ignore these. Similarly, method might overemphasize the importance of certain image patterns learned during training that are however of no relevance for the test classes, resulting in supervision collapse (9].
Joint first authorship
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Images
Image
Support
Pasch Emportance Weighting
Query Image
Class
? Class
Class
Class
Class
?
Class
Class
Classification can be ambiguous!
(1) Task-spec fic intra-class vs.
(2) Patch Importance Weighting via determined similarities
(3) Clessify using relevant petches
inter-class smilarities
Tackling classification ambiguity by interpreting images in context. (Left): Labels assigned to real-world images with multiple entities only correctly describe a subset of the depicted content, leading to ambiguous classification results. (Right): Leveraging intra- and inter-class similarities and differences across the support set allows our method to determine the importance of each individual patch at inference time, i... to find out 'what matters most' in each image. This information is then used to reweight support-query similarities and resolve ambiguity.
Figure I:
Few previous works [ 19 partially tackle the above challenges. CTX (9) proposes to learn the spatial and semantic alignment between CNN-extracted query and support features using Transformer-style attention mechanism. The authors further show that self-supervised learning tasks (i.e., SimCLR) can be integrated into episodic training along with normal supervised tasks to learn more generalized features, which benefits solving unseen tasks and mitigates supervision collapse. CAN [19] achieves this in a similar manner by performing cross-attention between class prototypes and query feature maps, highlighting the region of feature map important for classification during inference. While both methods propose important contributions towards tackling supervision collapse, there exist important drawbacks. Firstly, both methods build their ideas around aligning prototypes based on each query. Such prototypes are merely class-aware and ignore all inter-class information present in the support set part that has however been shown to be crucial for few-shot learning [35] 49]. Furthermore, learning query-aligned class representations requires performing the same operation for each query, rendering such approaches rather inefficient at inference time.
Summarizing our and previous works' observations, we aim to address what we see as the two main criteria: 1) building an understanding about an image's 's structure and content that generalizes towards new classes, and 2) providing the ability to interpret the provided samples in context, i.e., finding the intra-class similarities and inter-class differences while jointly considering all available information.
Our work F To alleviate the negative influence of image-level annotations and avoid supervision collapse, we decompose the images into patches representing local regions, each having higher likelihood of being dominated by only one entity. To overcome the lack of such fine-grained annotations we employ self-superviser training with Masked Image Modelling as pretext task [60] and use a Vision Transformer architecture [0] as encoder due to its patch-based nature. We build our classification around the concept of learning task-specific similarity between local regions as function of the support set at inference time. To this extent, we first create a prior similarity map by establishing semantic patch correspondences between all support set samples irrespective of their class, i.e., also between entities that might not be relevant or potentially even harmful for correct classification (Figure [T] step (1)). Consider the depicted support set with only two classes: 'person" and 'cat'. The lower-right image is part of our support set for 'cat' - and the dog just happens to be in the image. Now in the query sample that shall be classified, the image depicts a person patting their dog. We will thus correctly detect a correspondence of the two dogs across those two images, as well as between the person patches and the other samples of the person support set class. While the correspondences between the person regions are helpful, there is no 'dog' class in the actual support set (i.e., 'dog' is out-of-tar task information). rendering this correspondence harmful for classification since it would indicate that the query is connected to the image with the 'cat' label. This is where our token importance weighting comes into play. We infer importance weight for each token based on its contribution towards correct classification of the other support set samples, actively strengthening intra-class similarities and inter-class differences by jointly considering all available information -n in other words, we learn which tokens 'help' or 'harm' our classification objective (Figure [ step (2)). These importance-reweightee. support set embeddings are then used as basis for our similarity-bas query sample classification (step (3)). Our main contributions include the following:
Our code is publicly available at htthtts:: http://wwwb.oommkkkaaaaa////uuu

---

XS
Ps
Token importance weights generator
N-KXL
N-KxLXL
2
N-KxCxHxW N-KxMxC
Palr-wise similarity
fo
Broadcasting & Reweighting
Logits aggregation
X
N*KxLxD
S
low -.- high
Pg
Zg
CxHxW
MxC
1xLxD
N-KxLxL
Class probabilities
Illustration of the proposed method FewTURE. Support and query set images split into patches and encoded by our Transformer backbone. Classification of query set images is performed by using the reweighted similarity of the encoded patches w.r.t the support set tokens.
Figure 2:
1. We demonstrate that Transformer-oni architectures in conjunction with self-superviee pretraining can be successfully used few-shot settings without the need of convolutional backbones any additional data.
2. We show that meta tnn-ttnnng of Vision Transformers combined with our inner loop token importance reweighting can successfully use the supervision signal of provided support set labels while avoiding supervision collapse.
3. We provide insights into how establishing general similarities across images independent of classes followed by our optimization-bnn based selection at inference time can boost general- ization while allowing visual interpretability at the same time, and show the efficacy of our method by achieving new state-of-the of-hh-a-----tt results on four popular public benchmarks.
2 Few-shot classification via reweighted embedding similarity
We start this section by briefly introducing the problem setting tackling in this work: inductive few-shot classification. We then provide an overview of our proposed method FewTURE Few- shot classification with Transformers Using Reweighted Embedding similarity (Figure before elaborating on the main elements in more detail.
Problem definition. Inductive N-way K-shot few-shot classification aims to generalize knowledge
learned during training on Dirain to unseen test data Drest, with classes Crrain NCrest = 0, using only few labelled samples. We follow the meta-learning protocol of previous works [49] to formulate the few-shot classification problem with episodic training and testing. An cpisode is composed of a support set X, {(>** v 1....y N; 1,..., . K; y E Ctrain), }. where x: denotes the k-th sample of class with label yok and query set Xr = {xx) y²) 1, I,..,, N). where æy
denotes query sample class with label v"
2.1 Overview of FewTURE
As depicted in Figure 2 encode the image patches P, of the support set samples along with the query sample patches Pa via fo and obtain corresponding sets of tokens 2, and zg. respectively be noted that while choose illustrate our method via the use of one single query sample, the classification of all query samples is computed at the same time in one single pass in practice. We retrieve our 'prior' correspondence map 5 expressing the token-wise similarity between the encoded semantic content of the local regions in the query sample and all patches of all support samples, allowing us consider all available information jointly without incurring information loss due to averaging or similar operations. This 'prior' similarity map represents correspondences between regions of samples irrespective of their individual class, i.c. also between entities that might be
Withoutt loss generality, we present our method for case of one query sample per class to improve case understanding. The exact number of query samples per class is generally unknown in practice. "Note that some tensor shapes in the illustrations might differ from the equations for of visualization.
3

---

relevant or potentially even harmful for correct classification. Using the annotated support set samples, we now infer a task-specific importance weight factor v for each support token 2 representing its contribution to correctly classify other samples in the support set via online optimization at inference time (Section 2.4). We then reweight the prior similarities to obtain our classification result for the query sample, jointly considering all available information.
2.2 Sell-supervised pretraining against supervision collapse
To overcome the problem of supervision collapse induced by image-level annotations, split the input images into smaller parts where each region has a higher likelihood of only containing one major entity and hence a more distinct semantic meaning. Since labels are available for this more fine-grained data, we encode the information of each local region via an unsupervised method.
We build our approach around the recently introduced idea of using Masked Image Modeling (MIM) 3 60) as a pretext task for self-supprvised training of Vision Transformers. In contrast to previous unsupervised approaches [5] which focused mainly global image- representations, MIM randomly masks a number of patch embeddings (tokens) and aims to reconstruct them given the remaining information of the image. The introduced token constraints help our Transformer backbone to learn an embedding space that yields semantically meaningful representations for each individual image patch. We then leverage the information of the provided labels through fine-tuning the pretrained backbone in conjunction with our inner loop token importance weighting described in the following sections while successfully avoiding supervision collapse (see experimental results in Section 3.2
2.3 Classification through reweighted token similarity
As illustrated in Figure
2 split each input image E R into sequence of M = pa patches p = {p') M with each patch p¹ IR PC We then flatten and pass all patches of the support and query images as input to the Transformer architecture, obtaining the set of support
tokens Z, - fo (P)) with Z, {2 1n-1,.... k, K} gnk (==k(|=1,.... L; 2x
nkl
RD) and query tokens 24 = fo with 24 = 1=,1-1,11-01 L: 24 R. Vision Transformers
like ViT [10] 47) satisfy L = MM whereas hierarchical Transformers like Swin (28) generally emit
reduced number of tokens L < .M due to internal merging strategies.
Having obtained all patch embeddings, we establish semantic correspondences by computing the pair-wise patch similarity matrix between the set of support tokens Z, and query tokens 2y as S I,,,, = sim(= 2/7). where 1, -,=1, 1......., T. and 4 11 ... L. Note that local image regions representing similar entities exhibit higher scores. While any distance metric can be used to compute the similarity (sim), we found cosine similarity to work particularly well for this task. We then use the task task-specific token importance weights v E R Lx1 inferred via online optimization based on the annotated support set samples (see Section 2.4 to reweight the similarities through column-wiss addition and obtain our task- specific similarity matrix
N---KLXLL where each element in S is obtained by nn
as = S + [o 11 11xL], with elements glad's Note that this addition of our reweighting logits corresponds multiplicative reweighting in probability space. We temperature-scale the adapted similarity logits with 1/rs and aggregate the token similarity values across all elements belonging the same support set class via LogSumExp operation, i.e. aggregating K L logits per class followed by softmax - resulting in the final class prediction Dg for the query sample B.g as
Fuk
9g = softmax (187) N n=1)= softmax (1 DII
N
K
L
Σ
exp (2l4
(1)
k= -=11,-11=1
2.4 Learning token importance inference time
We use all samples of a task's 's support set together with their annotations to learn the importance for each individual patch token via online optimization inference time. As depicted in Figure we formulate the same classification objective in the previous section but aim to classify the support set samples instead of query sample. In other words, we use two versions of the support set samples: one with labels as 'support' Z, and one without as 'pseudo-query' Zeq. and obtain the similarity

---

Forward
Update
Class probability
Class label
Backpropagate
Masking
Logits aggregation
Z
N-K-LxN-KKL
Broadcast
NxXxXxD
Pair-wise similarity
Loss
addition
Z
N-K-LXN-KLL
N-KXLXD
N-KxL
N-KXL
Figure 3: Inner loop token importance weight generator. The most helpful tokens for the task hand are determined as function of the support set via inner-loop optimization at inference time by reweighting all token similarities based their contribution towards correct classification result.
matrix S. L-N-K- The token importance weights are initialized to v" = 0 RN-K-LXI and column-wiie added to form S, = S. - ' [100. 1IxN-K-L].
The goal is now to determine which tokens of Z, most helpful in contributing towards correctly classifying Zxg. and which ones negatively affect this objective. To prevent tokens from simply devise the following strategy for our N-way K-shot tasks. For scenarios with > samples per class, we apply block-diagonal masking with blocks of size I. x T. to the similarity matrix S, mcaning that we enforce classification of each token in 2. exclusively based on information from other images. Since there are no other in-class examples available in 1-shot scenarios, we slightly weaken the constraint and apply local masking in an m X m window around each patch, forcing the token be classified based on the remaining information in the image. We found local window of m = to work well throughout our experiments for both architectures.
classifying themselves,
We temperature-scaal scaaling and aggregate all modified similarity logits across all elements belonging to the same support set class of the annotated 2, for each element 2xq zn apply a softmax operation and obtain the predicted class probabilities g"* for each support sample (ef. Equation DD. Given that the prediction is now dependent on the initialized token similarity weights v, we can Tormulate an online optimization objective by using the support labels y as
N
K
arg mii
LCE
y
V (v)
(2)
R=1
is to be noted that by using column-wise addition of v. we share the importance weights of each support token across all pssuud-query' tokens and thus constrain the optimization to jointly learn token importance with respect to all available information. In other words, task-specific intra-class similarities will be strengthened while inter-class ones will be penalized accordingly. We further do not require any second order derivatives like other methods during meta fine-tuning, since the optimization of e is decoupled from the network's parameters.
3 Experiments and discussion
3.1 Implementation details
Our training strategy is divided into two parts: self-supervised pretraining followed by meta fine- tuning. It is to be noted that each architecture exclusively trained on the training set data of the respective dataset that is to be evaluated, and additional data is used.
Datasets. We train and evaluate our methods using four popular few-shot classification benchmarks, namely minilmageNet 449, fieredImageNet 338 CIFAR-FS and FC-100 [35].
Architectures. We compare two different Transformer architectures in this work, the monolithic ViT architecture [10] 47) in its 'small' form (ViT-small aka DeiT-small) and the multi-scale Swin architecture 28 in its "tiny" version (Swin-tiny).
Self-supervised pretraining. We employ the strategy proposed by 660 to pretrain our Transformer backbones and mostly stick the hyperparameter settings reported in their work. Our VIT and Swin
5

---

Superised Pre re-TTannng
Bot-Supervised Pre-Traning
Superiied Pro Pre-Tranng
Seft-Si Surviisedd Pre-Training
90.00
66
Shht Tost Accurayy
58hel Teal Accureey
80.00
05.00
20.29
70.00
$7.35
$5.00
69.00
45.5
47.56
45.00
60.00
ViT-Small W/0 M-FT
ViT-Small wy M-FT
Swin-Tiny w/o M-FT
Swin-Tiny M.FT
VIT-Small w/o M-FT
ViT-Small Wy M-FT
Swin-Tiny wio M.FT
Swin-Tin M-FT
Figure 4: Supervised self-supervised pretraining. Average test accuracies minilmageNet for our method with different pretraining methods, with (w/) and without (w/o) meta fine-tuning (M-FT).
Patch embeddings of a support class
Patch embeddings of the entire support set
trained w/o V
w/
w/ U. stop
w/ 8t step
Figure 5: Instance and class embeddings. Visualised are the projected tokens of 5 instances of the same novel support set class (left) and of the entire support set (right). From left to right: Instance embeddings meta-trained using our classifier without task task-specific token reweighting ('w/o v') d') vs. trained with 15 reweighting steps ('w/ v'); Embeddings of entire 5-way 5-shot support set obtained by our approach trained with 15 steps, displayed reweighting step vs. step 15. (PCA projection)
architectures are trained with batch size of 512 for 1600 and 800 epochs, respectively, We use Nvidia A A100 GPUs with 40GB each for our ViT and 8 such GPUs for our Swin models.
Meta fine-tuning. We use meta fine-tuning to further refine our embedding space by using the available image-level labels conjunction with our roken-reweighting method. We generally train for up to 200 epochs but find most architectures to converge earlier. We evaluate at each epoch on 600 randomly sampled episodes from the respective validation set to select the best of parameters. During test time, we randomly sample 600 episodes from the test set to evaluate our model.
Token importance reweighting and classifier. We cosine similarity compute S. While the temperature Ts used to scale the logits can be learnt during meta fine-tuning, we found Ts = 1/va to be good default value (or starting point if learnt, see supplementary material). We SGD as optimizer with a learning rate of 0.1 for the token importance weight generation.
Please refer to supplementary material of this paper for more detailed discussion regarding datasets, implementation and hyperparameters.
3.2 Self-supervised pretraining and token-reweighted fine-tuning improve generalization
In this section, we investigate the influence of self-supervised pretraining compared to its supervised counterpart using the provided image-level labels (de-facto standard in most current state-of-the-ar methods). We only vary the training strategy of our backhone and use our introduced token similarity- based classifier with 15 inner-loop reweighting steps for all experiments. Figure illustrates that both ViT-small and Swin-tiny with self-supervised pretraining alone (w/o meta fine-tuning (ne-ttnnng) learn more generalizable features than their respective supervised versions for both 1-shot (left-hand side) and 5-shot finht-hand side) scenarios. In fact, across all cases w/o meta fine-tuning, the self-supervised pretrained Transformers outperform their supervised counterparts by more than 10% and up significant 19% in the case of Swin-tiny. This clearly demonstrates that our self-supervised pretraining strategy explores information that is richer and beyond the labels. Another interesting insight is that the meta fine-tuning stage does not improve supervised backbones much as their unsupervised counterparts. Specifically after unsupervised pretraining, meta fine-tuning of the Transformers in conjunction with our reweihheeiggtting strategy is able to boost the performance by

---

Figure 6: Learning token importance at inference time. Visualized importance weights learnt via online optimization for support set samples in a 5-way 5-shot task on the miniImageNet test
6.77% (ViT-small) and 13.01% (Swin-tiny) for 1-shot, and 9.94% (ViT-small) and 10.10% (Swin- tiny) for 5-shot. While such significant improvements cannot be observed across the supervised to generally start off lower after pretraining but benefit more from networks, the Swin versions the fine-tuning than ViT. The observed results clearly indicate that our token-reweighted fine-tuning strategy is able to further improve the generalization of self-supervised Transformers, thus performing better on the novel tasks of the unseen test set. Figure aadiiionally depicts projected views of the tokens of 5 instances from novel class as well as the entire novel support set embedding space. Representations obtained with our classifier seem to retain the instance information ('w/o v') and separation is improved when using token importance reweighting ('w/ v'). While the projected tokens of the entire support set show partial overlap between classes as is expected due to commonalities like e.g. similar background, our reweighting clearly determines the flass-characterisse in their original dlass-respectiv color). These results indicate that our similarity- -based classifier coupled with task-specific token reweighting is able to better disentangle the embeddings different instances from the same class well as other classes, which prevents the network from supervision collapse and achieves the higher performance observed on the benchmarks. They further show that self-supervised pretraining is helpful but not sufficient to achieve well-separated representations without supervision collapse that are suitable for few-shot classification.
tokens (displayed
3.3 Selecting helpful patches at inference time
Higure[]shows visualization of the patch importance weights e that learned at inference time during the inner loop adaptation for the support set images. Brighter regions represent higher importance weight, meaning that these patches will contribute most to the classification of query samples if matches with high similarity can be found. Judging from the visualized weights, FeRTURE seems to consistently select characteristic regions the depicted objects, e.g., the rim the bowls, strings of the guitar or the dogs' facial area, and to exclude unimportant out-of-task information.
We further investigate the influence of the number of optimization steps in our inner loop token importance weighting method using ViT-small on minilmageNet. The results in Figure indicate that increasing the steps up to 20 aligns with increased performance, both during validation and testing. While the initial increase in test accuracy when using our token reweighting (steps >> 0) is rather significant with 1.15%, the contribution of higher step numbers comes the cost of higher
84.00
84.50
$4.51
Valicaton Accuracy
83.00
Acouraoy
84.00
81.85
86.05
82.00
steps 15 steps steps
81.00
83.00
steps
steps
82
80.00
30
40
50
60
70
80
0 stop
steps
10 staps
15 steps
20 stops
Training Iterations
Token Re-weighting Steps
(a)
Figure 7: Inner loop token reweighting. Average classification accuracies on the minilmageNet validation set (a), and test set (b) for varying inner loop optimization steps, evaluated with ViT-small backbone and SGD with 0.1 as learning rate.
7

---

Table 1: Average classification accuracy for 5-way 1-shot and 5-way 5-shot scenarios. Reported are the mean and 95% confidence interval on the unseen test sets of minilmageNeet 49 Negeel
and
38 using the established evaluation protocols.
magele
fieredImageNee
Model
Backbone
Params
1-shot
5-shot
1-shot
5-shot
ProtoNet
ResNet-12
12.4 M
62. 20.33
79.46220.4
68. 28..5520.23
84.01 =0.56 56
FEAT
ResNet-12
12.4 M
06.78:0.20
82.05=20144
70.8020.23
84.79=0 16
DeeDEMD 55
ResNet-12
12.4 M
65. 10.82
82 82.41:0.56
71.1620.87
86.03:0 58
IEPT 57
ResNet-12
12.4
67.05++0.44
82.90=2.30
72.24=0..0
86.73=11
MELR
ResNet-12
12.4 M
67.40+0.43
83.40=2.22
72. 440..51
87.01=0.33
FRN 150
ResNet-12
12.4 M
66.45-t..1
82.83=00113
72.06:20.22
86.89=0.11
CG
ResNet-12
12.4 M
67.02 00:02
82.32200.1
71 21.66:20.23
85.50=0 15
DMF 5
ResNet-12
12.4
67.76+0.46
82.71#0.31
71.89=0.52
85.96=0.35
InfoPatch
ResNet-12
12.4 M
67. +0.44
82.44=0.31
BMI.
ResNet-12
12.4 M
67.0440.63
83.63=0.29
68.99=20.00
85. $5.19=0.8
CNI.
ResNet-12
12.4 M
67.96:20.9
83.3620051
73.42200.5
87.72=0.31 75
Meta-NVG
ResNet-12
12.4 M
0.0
83.82
44.58
86.73#0.6
ResNet-12
12.4 M
69.37+
84.40=0.44
72.25=t0.72
86.95=0.4r
COSOC 20
ResNet-12
12.4 M
69.28++0.40
85.16=0.42
73.57=0043
87.57=0.10
Meta DeepBDC 52
ResNet-12
12.4 M
67.3 03.443
84.46=2022
72.3440.49
87.31
LRO [40
WRN-28-10
36.5 M
61.76:20.08
77.59=20112
66.33-20005
81.4420.00
CC+ror 15
WRN-28-10
36.5 M
62. 20.44
79.87=2.58
70.53:20.5
84.98=2 36
FEAT
WRN-28-10
36.5 M
65. 1000.0
81.11 11.11
70:41:0:23
84.38:00.16
PSST 8
WRN-28-10
36.5 M
64. 10.444
80 30.64
MetaQDA 58
WRN-28-10
36.5 M
67.83++0.64
84.28=2060
74.33=20.5
89.56220.7
OM [37
WRN-28-10
36.5 M
06.78+0..0
85.29:2.41
71.54:0.29
87.79=0.0m
Few TURE (ours)
ViT-Small
22 M
62.02e
84.31
72.96:4..92
86.43=0007
FewTURE (ours)
Swin-Tiny
29 M
72.40+a.75
86.38=0.4p
76.32=0.87
89. 89.96+0.00
Table 2: Average classification accuracy for 5-way 1-shot and 5-way 5-shot scenarios. Reported are the mean and 95% confidence interval on the unseen test sets of CIFAR-FS [4] and FC-100 [35] using the established evaluation protocols.
CIFAR-FS
FC100
Model
Backbone
# Params
1-shot
5-shot
1-shot
5-shot
ProtoNet 42
ResNet-12
12.4
41.54
57. 57.08++.76
MetaOpt
22
ResNet-12
12.4
72.00=0.70
84.20+0.50
41.1020.60
55.50+0.00
MABAS
21
ResNet-12
12.4 M
73.51
85.65440.6n
42.31
1640677
RFS [46
ResNet-12
12.4
73.90
86.90+0.50
44. 44.60=a.70
60.90+v.oz
BML [6
ResNet-12
12.4
73.45=0047
88.0440.32
CG [4
ResNet-12
12.4
73.00=0:70
85.80+0.50
Meta-NVG
556
ResNet-12
12.4
74.63=0.91
86.45+0.50
46. 46.40=2051
61.33+.77
RENet 20
ResNet-12
12.4 M
74.51=0.46
06.60
TPMN 51
ResNet-12
12.4
75.50=0.90
87.20+0.60
46.93
63. 03.7644.74
MixFSL 1
ResNet-12
12.4
$1.89=0..8
60.70+0.01
CC+rot
WRN-28-10
36.5
73.62=0.31
86.05+0.22
PSST g
WRN-28-10
36.5
77.02=0.38
88.4540.35
Meta-QDA 58
WRN-28-10
36.5
75.83
88.7940.75
Few TURE (ours)
ViT-Small
22
76. 16.10=2088
86.1440..4
46. 26.20=a.mv
63.1440.78
FewTURE (ours)
Swin- Swin-Tiny
29
77.7610..1
88.9010.8 59
47.6810.78
13.81
computational complexity, and we generally found anything between 5 and 15 steps to be a good performance inference-time trade-off (see supplementary material for further details).
3.4 Evaluation on few-shot classification benchmarks
We conduct experiments using the few-shot settings established in the community, namely 5-way 1-shot and 5-way 5-shot meaning the network has to distinguish samples from 5 novel classes based on a provided number of or 5 images per class. We evaluate our method FewTURE using two different Transformer backbones and compare our results against the current state of the art in Table for the minilmageNet and tiered/mageNeet, and in Table for the CIFAR-FS and FC100 datasets. is to be noted that in contrast to previous works, we do not employ the help of any convolutional

---

backbone but instead (and as far as we are aware for the first time) use a Transformer backbone together with our previously introduced token importance reweighting method achieve these results. We are able to set new state of the art results across all four datasets in both 5-shot and 1-shot settings, improving particularly the 1-shot results on manelmageNet and faered by significant margins of 3.03% and 1.74%, respectively.
3.5 Increasing efficiency by pruning the token sequence
To further improve the computational efficiency of our method, we investigate its behaviour when limiting the number of tokens that are considered to establish patch-wise correspondences to only a subset allowing FewTURE to scale to potentially large many-way many-shot settings. We use the attention maps inherent in our approach (averaged over all heads) to prune the number of patch tokens by only using the ones within the top-k attention values to compute the similarity matrix S. The results obtained with our ViT-small backbone for pruming the number of tokens to 75%, 50%, 25% and 10% of the original token number indicate that such pruning might be an interesting avenue for future work, with our method still achieving 96.4% of its original performance when only retaining 10% of the number of tokens.
Table Pruning the number of
tokens. Test accuracy for 5-way 5-shot on minllmaeeNet [49]
tokens
Test Acc.
100%
84.05 + 0.53
75%
83.15 + 0.57
50%
83.81 + 0.59
25%
81.79 + 0.37
10%
81.05 + 0.62
3.6 Ablation on the type of classifier
Table 4: Changing the classifier. Test accuracy for 5-way 5-shot on minilmageNet 49
We train a linear classifier as well prototypical approach using our pre-trained ViT-small backbone for a 5-way 5-shot setting on minilmagenet [49] to investigate the influence of the choice of clas- sifier (Table 4 We were able obtain test ac- curacy of 82.80% for the prototypical network af- ter optimising the pre-trained backbone with meta- finetuning, which is competitive to the results we obtain for our method without reweighting ("O step' in Figure ZbD.. but is clearly outperformed by our reweighting based approach. To provide a fair com- parison, we optimize the linear classifier at inference time to adapt to the support set and obtained a maximum test accuracy of 82.37%. Both results indicate the quality of embedding our backbone is able to produce but also demonstrate the importance of our task-specific reweighting- brased approach. For further ablation studies, please refer to the supplementary material.
Classifier
Test Acc.
Prototyp. w/ Euclid. Dist.
82.80 + 0.59
Prototyp. w/ Cosine. Dist.
79.90 + 0.65
Linear (optimized online)
82.37 + 0.57
FeWTURE O rew. steps)
82.68 + 0.55
FewTURE (15 rew. steps)
84.05 0.53
3.7 Limitations
Our introduced method is arguably more powerful for cases where multiple examples of the same class are provided, i.c. in N-way K-shot settings with K I. While FewTURE works well in 1-shot settings (Table and (2). the inner loop adaptation procedure still aims to exclude cross-class similarities that hurt classification performance, but has less diverse information for selecting the most helpful regions due to the lack of other in-class comparison samples which might thus yield slightly less-refined token selections compared to multi-shot scenarios (supplementary material).
While we did not face significant problems regarding the comparably very small size of our datasets (e.g. minilmageNet with 38K compared to the usually used ImageNet with 1.28M training images), more specialized applications with highly limited training data might be negatively impacted and successfully training our method due to the reduced inductive bias present in the Transformer architecture could prove challenging.
4
Related work
Over the past few years, the family of few-shot learning (FSL.) has grown diverse and broad. Those closely related to this work can be categorized into two groups: metric-based methods [3] 32
42

---

49 54, 55
Metric-based methods, such as ProtoNet DeepEMD [55]. and RelationNet 44 aim learn class representation (prototype) by averaging the embeddings belonging to the same class and employ predefined (ProtoNet and DeepEMD) or learned (RelationNet) metric to perform prototype-query matching. FEAT (54] and TDM [23] take this step further and use attention mechanisms to adapt the extracted features to the novel tasks. Our method instead fully utilizes the embeddings of local image regions (patch tokens), preventing loss of information and supervision collapse occurring in the aforementioned prototype-based approaches (see supplementary material).
and optimisation based methods [3 18 22
34
40,
62
Optimisation- based methods such as MAML [13 and Reptile 34 propose to learn a set of initial model parameters that can quickly adapt to a novel task. However, updating all model parameters is often not feasible given large backbones and only few labeled samples during inference. To alleviate
this so-called meta-overfitting problem, CAVIA 62] and I.FO [40] propose to learn and adapt a lower dimensional representation that is mapped onto the network. Our method is inspired by such lower-dimensional adaptation strategies and learns a tiny set of context-aware re-weighting factors online for each novel task without requiring higher-order gradients, resulting in flexible and efficient framework.
Self-supervieed learning for FSL. Although self-supervised learning is underrepresented in the context of few-shot learning. some recent works [T5
B3 4 have shown that self sulf-vuueriiiion via
pretext tasks can be beneficial when integrated as auxiliary loss. S2M2 (33 employs rotation [[6] and exemplars [T] along with common supervised learning during the pre-training stage. CTX demonstrates that SimCLR
6 can be combined with supervised-learning tasks in an episodic- training manner to learn a more generalized model. In contrast, FeWTURE demonstrates that the
60) can be used in a standalone manner to learn more generalized features for few-shot learning on small-scale datasets.
self-supervised pretext task (i.e., Masked Image Modelling [2 3 17
25
36
45
Vision Transformers in FSL. Transformers have been immensely successful in the field of Natural Language Processing. Recent studies 100 2 28 48) suggest that encoding the long-range depen- dency of data via self-attention also yields promising results for vision tasks (e.g., image classification, joint lision-languag modeling, etc.). However, there is an important trade-off between leveraging the rich representation capacity and the lack of inductive bias. Transformers have gained reputation to generally require significantly more training data compared to convolutional neural networks (CNNs), since properties like translation invariance, locality and hierarchical structure of visual data have to be inferred from the data (27]. While this data-hungry nature largely prevented the use of Transformers in problems with scarce data like few-shot learning, some recent works demonstrate that a single Transformer head can be successfully adopted to perform feature adaptation (9 [4]. To the best of our knowledge, FewTURE is the first approach that uses fully Transformer-basee architecture to obtain representative embeddings while being trained exclusively on training data of the respective few-shot dataset.
5 Conclusion
In this paper, we presented a novel approach to tackle the challenge of supervision collapse introduced by one-hot image-level labels in few-shot learning. We split the input images into smaller parts with higher probability of being dominated by only one entity and encode these local regions using a Vision Transformer architecture pretrained via Masked Image Modeling (MIM) in sulf-super way to learn a representative embedding space beyond the pure label information. We devise classifier based patch embedding similarities and propose a token importance reweighting mechanism to refine the contribution of each local patch towards the overall classification result based around intra-class similarities and inter-class differences as function of the support set information. Our obtained results demonstrate that our proposed method alleviates the problem of supervision collapse by learning more generalized features, achieving new state-of-the-as results on four popular few-shot classification datasets.
The authors would like thank Zhou al. [60] for sharing their insights and code regarding self-super pretraining, as well as Dosovitskiy et al. [10], Touvron et al. [47 and Liu el al. (88 for sharing details of the ViT and Swin architectures.
Acknowledgements.
Parts of this research were undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility established with the assistance of LIEF Grant LE170100200.
10

---

References
[1] Arman Afrasiyabi,
Lalonde, and Christian Gagné, Mixture- feature space learning for few-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9041-9051, 2021.
[2] Sara Atito, Muhammad Awais, and Josef Kittler. Sit: Self-supervised vision transformer. arXiv preprint
arXiv:2104.0 000440060.2220 2021.
[3] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In International Conference on Learning Representations. 2022.
[4] Luca Bertinetto, Joao F Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-fone solvers. In International Conference on Learning Representations, 2019.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Rojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[6] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages
1597-16077. PMLR, 2020.
[7] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
9640-9649) 2021.
[8] Zhengyu Chen, Jixic Ge, Heshen Zhan, Siteng Huang, and Donglin Wang. Pareto self-supervised training for few-shot learning. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 13663-13672, 2021.
[9] Carl Doersch, Ankush Gupta, and Andrew Zisserman. CrossTransformers: spatially- few-shot transfer. In Advances in Neural Information Processing Systems, 2020.
[10] Alexey Dosovitskiy. Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth
16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020. [11] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. Advances in Neural Information
Processing Systems, 27, 2014.
[12] Nanyi Fei. Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. Meir: Meta-learning via modeling episode-level
relationships for few-shot learning, In International Conference on Learning Representations, 2020. [13] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep
networks. In International Conference on Machine Learning, pages 1126-1 1136-1135. PMLR. 2017.
[14] Zhi Gao, Yuwei Wu, Yunde Jia, and Mchrtash Harandi. Curvature generation in curved spaces for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8691 -8700, 2021.
[15] Spyros Gidaris. Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with superviiiv... In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8059-8065 2019.
[16] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation lcarning by predicting image rotations. In International Conference on Learning Representations, 2018.
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders
are scalable vision learners. arXiv preprint arXiv:2111 10..2111.66377 2021.
[18] Markus Hiller, Mchrtash Harandi, and Tom Drummond. On enforcing better conditioned meta-lcarning for rapid few-shot adaptation. In Advances in Neural Information Processing Systems. 2022.
[19] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for
few-shot classification. Advances in Neural Information Processing Systems, 32, 2019.
[20] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[21] Jackycom Kim, Hyoungseok Kim, and Gunhee Kim. Model-agnostic boundary-adversarial sampling for test-time generalization in few-shot learning. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedingy, Part 16, pages 599-617. Springer, 2020.
[22] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-lcarning with differen- tiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10657-10665, 2019.
[23] SuBeen Lee, WonJun Moon, and Jac-Pil Pil Heo. Task discrepancy maximization for fine-grained few-shot classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 5331-5340, 2022.
[24] Chumyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation lcarning. International Conference on Learning Representations (ICLR), 2022.
[25] Zhaowen Li, Zhiyang Chen, Fan Yang. Wei Li. Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34, 2021.

---

[26]
Chen Liu, Yanwei Fu. Chengming Xu, Siqian Yang, Jilin Li, Chengjie Wang, and Li Zhang. Learning few-shot embedding model with contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8635-8643. 2021.
[27] Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems, 34, 2021.
[28] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 10012-10022 2021.
[29] Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shortcut learning of background for few-shot learning. Advances in Neural Information Processing
Systems, 34, 2021.
[30] Jiawei Ma. Hanchen Xie, Guangxing Han, Shih-Fu Chang, Aram Galstyan, and Wael Abd Almageede Partner-assisted learning for few-shot image classification. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 16573-10522 2021.
[31] Rongkai Ma. Penglei Fang, Gil Avraham, Yan Zuo, Tom Drummond, and Mehrtash Harandi. Learning instance and task-aware dynamic kernels for few shot learning. In European Conference on Computer Vision, pages 257-274, Springer, 2022.
[32] Rongkai Ma, Pengfei Fang, Tom Drummond, and Mehrtash Harandi. Adaptive poincaré point to set distance for few-shot classification. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1926 1934, 2022.
[33] Puncet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEF/CVP Winter Conference on Applications of Computer Vision, pages 2218-2227, 2020.
[34] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint
1r0iv:1833. 02999, 2018.
[35] Boris Oreshkin, Pau Rodríguez. López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for
improved few-shot learning. Advances in Neural Information Processing Systems, 31, 2018.
[36] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexii A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544 2016.
[37] Guodong Qi. Huimin Yu, Zhaohui Lu, and Shuzhao Li. Transductive few-shot classification on the oblique manifold. In Proceedingy of the IEEE/CVF International Conference on Computer Vision, pages 8412 8422. 2021.
[38] Mengye Ren, Fleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenhaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint (t0::/803.0670 2018.
[39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, ct al. Imagenet large scale visual recognition challenge.
International Journal of Computer Vision, 115(3):211-22 1-252, 2015.
[40] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2018
[41] Christian Simon, Piotr Koniusz, Richard Nock, and Mchrtash Harandi. Adaptive subspaces for fow-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4136-4145, 2020.
[42] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Newral Information Processing Systems, 30. 2017.
[43] Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few-shot
learning? In European Conference on Computer Vision. pages 645-666. Springer. 2020.
[44] Flood Sung. Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition. pages 1199-122 1008, 2018.
[45] Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. Vimpac: Video pre-training via masked token prediction and contrastive learning. arXiv preprint arXiv:2106.11255
2021.
[46] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, pages 266-282. Springer, 2020.
[47] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference on Machine Learning, pages
PMLR, 2021.
[48] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.
Maxvit: Multi-axis vision transformer. In European Conference on Computer Vision, pages 459-479. 2022.
[49] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 39:3630-3633 2016.
[50] Davis Wertheimer, Luming Tang, and Bharath Hariharan. Few-shot classification with feature map reconstruction networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
12

---

Recognition (CVPR), pages 8012-8021, 2021.
[51]
Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. aware part mining network for few-shot lcarning. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 8433 8442, 2021.
[52] Jiangtao Xie, Fei Long. Jiaming Lv. Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance few-shot classification. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[53] Chengming Xu, Yanwei Fu, Chen Liu, Chengjie Wang. Jilin Li, Feiyuc Huang, Li Zhang, and Xiangyang Xue. Learning dynamic alignment via meta-filter for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5 5182-5191, 2021.
[54] Han-Jia Ye, Hexiang Hu. De- Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
8808-8817, 2020.
[55]
Chi Zhang. Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd:
Few-shot image classification with differentiable earth mover's 's distance and structured classifiers. In IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), 2020.
[56] Chi Zhang, Henghui Ding, Guosheng Lin, Ruibo Li, Changhu Wang, and Chunhua Shen. Meta navigator: Search for a good adaptation policy for few-shot learning. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 9435-9444, 2021.
[57]
Manli Zhang, Jianhong Zhang. Zhiwu Lu, Tao Xiang, Mingyu Ding, and Songfang Huang. lept: Instance- level and episode-level pretext tasks for few-shot learning. In International Conference on Learning Representations, 2020.
[58] Xueting Zhang, Debin Meng, Henry Gouk, and Timothy M Hospedales. Shallow bayesian meta learning for real-world few-shol recognition. Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 651-660, 2021.
[59] Jiabao Zhao, Yifan Yang, Xin Lin, Jing Yang, and Liang He. Looking wider for better adaptive repre- sentation in few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence pages 10981-10989, 2021.
[60] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
[61] Ziqi Zhou, Xi Qiu, Jiangtao Xie, Jianan Wu, and Chi Zhang. Binocular mutual learning for improving few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8402 8411, 2021.
[62] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning In International Conference on Machine Learning, pages 7693-7702. PMLR, 2019.
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.7
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section 3.]
and supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 3.1 and supplementary material
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [Yes] See Section 3.1 and supplementary material
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3.]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 3.1 and supplementary material
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
We make code publicly available and included the url to our github repository, footnote 2 on page 2
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review
Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "v6SSYysrYfX", "reviewer": "Reviewer_erRy", "summary": "The paper addresses the problem of few-shot classification. The main idea is to establish semantic correspondences between the patches from the support and the query images. These correspondences are then used to reason which class a query image belongs to. In order to down-weight the impact of background patches when performing classification, the authors also introduce an online optimization strategy to determine which patches in the support images are most informative when performing few-shot classification. The method uses the Vision Transformer to encode the patches in the support and query images. In order to learn strong generic features, the Vision Transformer is trained in an unsupervised manner using the masked image modelling task. The self-supervised pre-training is shown to provide better results then the supervised counterpart. The proposed method obtains state-of-the-art results on four few-shot classification benchmarks. ", "questions": "Please check the comments listed under Weaknesses.", "limitations": "The authors discuss the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n**S1**: The paper is well written and easy to read.\n\n**S2**: The proposed few-shot classifier using patch-wise correspondences is novel and interesting. The online optimization allows determining which regions are most crucial to perform classification and can be helpful specially in case of clutter in the support set images. The use of patch-wise correspondences allows determining the class of the query image by jointly reasoning over the support set as well as the query.\n\n**S3**: The self-supervised pre-training of the Vision Transformer makes sense, specially in the context of few-shot learning, to learn generic feature representation. \n\n**S4**: The proposed approach is shown to obtain state-of-the-art results on 4 standard benchmarks (mini ImageNet, Tiered ImageNet, CIFAR-FS, FC100).\n\n**S5**: The authors provide helpful analysis and ablation studies, showing the impact of the major contributions.\n\n\n## Weaknesses\nI do not have any major issues with the paper. Some minor issues which could be addresses\n\n**W1**: As shown in Fig. 4, the use of self-supervised pre-training for Vision Transformer provides a significant improvement, compared to supervised training. It would be interesting to see what performance existing classifiers (e.g. learning a linear classifier, prototype) obtain when using the same backbone network. This would help evaluate the benefits of proposed embedding similarity based classifier.\n\n**W2**: In the sota comparison, most previous methods use ResNet-12 backbone, while the authors employ ViT-small and Swin-Tiny. It will be helpful if the authors include the model sizes (number of parameters) for each of these backbones for comparison.\n\n**W3**: The approach computes patch-wise correspondence between all the support and query images. I wonder if this could become computationally expensive when dealing which large number of number classes, or when each class has more samples (e.g. 100 way - 30 shot classification). A discussion on this would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dDhOVIcpwNF", "reviewer": "Reviewer_eF5W", "summary": "This works deal with the image-level annotation problem in few-shot classification. During training time, given support and query images, the authors first split images into patches, use a transformer to extract feature tokens, and learn which tokens are related to the label more. During test time, it self-learned within support images to find which tokens are more important. The experiment shows a consistent improvement.", "questions": "It is unclear to me what \"meta fine-tuning\" means. Does it mean the step of fine-tuning the model with support data at interference time?\n", "limitations": "Yes, the author addressed the potential problem when training data is highly limited.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe transformer-only architecture makes it very clean and thus makes it potentially be a new baseline of the problem. Reweighting features to focus on task-specific information is also a reasonable idea. The experiment shows a consistent improvement.\n\nWeakness:\nIt is better to discuss other feature re-weighting-based methods in few-shot classification. For example, for fine-grained few-shot classification, Lee et al. [A] shows a consistent improvement in all previous methods by adding an attention-based feature re-weighting module.\n\n[A] Lee et al., \"Task Discrepancy Maximization for Fine-Grained Few-Shot Classification.\", CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KCdhzm_i_c", "reviewer": "Reviewer_rwKM", "summary": "This paper adopts self-supervised trained Vision Transformer (ViT) architecture as the feature extractor, deriving patch-level representations for few-shot classification problems. To exploit the relation across patches, the authors propose a token importance reweighting mechanism (which is required to perform during both training and testing stages). The experimental results show satisfactory performance in several commonly-used FSL benchmarks to verify the effectiveness of this method.", "questions": "Please see my questions raised in the above weakness part.", "limitations": "The authors did provide discussions on the limitation of the proposed work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall paper is easy to follow. The idea of utilizing the self-supervised trained ViT architecture to derive patch-level representations is interesting and seems effective for few-shot learning tasks. Since self-supervised model pre-training is agnostic to the image-level class labels, the trained model is more generalizable for downstream tasks. Also, applying ViT to extract patch-level features allows the model to produce more fine-grained information. However, I have the following concerns about this work:\n\n1. In Fig. 4, the authors show that self-supervised pre-training performs significantly better than the supervised-pre-training counterpart. However, in prior SSL literature (e.g., [16, A]), SSL pre-training only slightly outperforms supervised pre-training (sometimes or even worse than it). A proper explanation (or insight) that the proposed SSL pre-training surpasses a large margin over the supervised counterpart reported in Fig. 4 is needed.\n\n2. The t-SNE visualization in Fig. 5, only verifies that the patch-level embeddings derived from the same “instance” are clustered together, and those from different instances are separated from each other. However, this figure only explains/visualizes separation between different instances, but not the discrimination between different classes (which is much more important for FSL). It is desirable to see whether the embeddings extracted from the same “class” are gathered, while the embedding of different classes would separate far from each other.\n\n3. Since the title of this paper emphasizes the aspect of “generalization” in few-shot learning, one would expect learning strategies with results/comparisons with recent cross-domain few-shot learning works (e.g., [B]). In other words, cross-domain FSL aims to transfer the learned knowledge to the novel classes in unseen target domains (showing generalization ability).\n\n4. Although this paper applies Masked Image Modeling (MIM) as the pretext task for pre-training ViT, use of other self-supervised pre-training approaches like contrastive learning (e.g., DINO [C], MoCo v3 [A]) would be possible. It will be also good if the authors provide some insights or comparisons about the choice of the self-supervised pre-training approach. If MIM is desirable for this task, more explanations and supports would be needed.\n\n5. The need to perform additional learning for test data (few-shot instances from novel classes) is needed for the proposed work (but not necessarily for a number of SOTAs). I'd like to see how the authors would elaborate on this issue.\n\n[16] He et al. “Masked autoencoders are scalable vision learners.” CVPR 2022\n[A] Chen et al. “An Empirical Study of Training Self-Supervised Vision Transformers.” ICCV 2021\n[B] Chen et al. “A Closer Look at Few-shot Classification.” ICLR 2019\n[C] Caron et al. “Emerging Properties in Self-Supervised Vision Transformers.” ICCV 2021\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A9HFNo0-M3c", "reviewer": "Reviewer_xXQn", "summary": "Motivated by supervision collapse caused by standard few-shot training on weak image-level labels, authors introduce a token-based approach based on unsupervised vision transformers that reweights tokens in an inner loop based on their discriminative power. The model achieves strong results and demonstrates the viability of vision transformer models on few-shot tasks without extra pre-training. ", "questions": "My main questions involve the two broad weaknesses outlined above. \n\n1. How should readers interpret the difference in model backbone sizes, and how does this impact the presented results?\n\n2. Can authors provide or elaborate on the missing ablations – what is the performance contribution of the vision transformer backbone vs token reweighting vs logsumexp aggregation? And less importantly, how significant was the choice of similarity metric?\n\nLess impactful questions/comments:\n\n3. Is the inner-loop token-reweighting scheme compatible with more sophisticated patch-to-patch techniques, such as CTX or FRN? Or is this (understandably) left for future work?\n\n4. Given a limited computation budget this may not be feasible, but it would be interesting to see if the results from Fig4 and Sec3.2 hold for a more standard meta-fine-tuning technique (i.e. a basic Prototypical setup). Authors appear to have produced such a model in at least one setting for Fig5 (“average”). What does the performance look like? Can authors elaborate or speculate on this? \n\nSome small typos:\n\n•\tPg2 line 58: extend  extent\n\n•\tPg2 line 66: class is in  class in\n\n•\tPg3 footnote: in generally  is generally\n\n•\tPg4 line 121: The in this way introduced  In this way, introduced / The introduced\n\n•\tPg4 line 158: device  devise\n\n•\tPg7 fig7 caption: SDG  SGD", "limitations": "The analysis of 1-shot effectiveness and discussion of smaller training datasets is insightful. The entanglement of vision transformer benefits with token reweighting benefits in presented results is not discussed. Societal impacts are not discussed, though do not extend beyond those of few-shot learning in general. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\n•\tDemonstrated use of vision transformers for few-shot learning is on its own a neat contribution\n\n•\tDemonstrated use of purely unsupervised pre-training for few-shot learning is also a neat contribution, if not quite as novel\n\n•\tResults are impressive and span multiple benchmarks and architectures\n\n•\tStraightforward and sensible approach to token aggregation and reweighting\n\nWEAKNESSES: \n\n•\tThe strong results come with a major caveat: the ViT-small and Swin-tiny architectures have 22M and 29M parameters respectively, while the compared baselines are almost entirely based on ResNet12, which by my recollection has only 12M parameters. While this is still less than the widely-used WRN-28-10 backbone (36.5M params), I worry that the comparisons presented in the paper are apples-to-oranges. The difference in model size should be discussed and addressed. \n\n•\tUse of vision transformers for few shot classification deserves an empirical study all on its own. Understandably this is not provided here, but because of this it is unclear to what degree improvement is coming from the token reweighting scheme (in theory compatible with existing convolutional architectures) vs the vision transformer backbone (in theory compatible with existing few-shot classifiers). For example, how does the token reweighting scheme compare to simply training a linear classifier head on the support features from a vision transformer? Admittedly, a full comparison along both these axes would be clearly out of scope here. \n\n•\tSimilarly, there is no ablation study provided for the impact of token reweighting vs the logsumexp aggregation scheme. How much better is logsumexp aggregation than direct addition, for example, which would correspond to basic prototype comparison with reweighted averages on each prototype? More broadly, it appears that the token reweighting scheme is broadly compatible with many existing token-to-token classifiers such as CTX and FRN, and it is not clear how the logsumexp aggregator compares. \n\n•\tMore generally, the approach, while straightforward and sensible, does contain a few design choices that are not fully explained or empirically justified (for example, in addition to above, the choice of token similarity metric). \n\n•\tA slightly relevant omitted citation: the masked inner token reweighting scheme might possibly owe some conceptual debt to Batch Folding from [Few Shot Learning with Localization in Realistic Settings, CVPR2019], which also models a support-to-support classification task with an identical image-masked leave-one-out scheme (though admittedly implemented quite differently).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept