Exploring Efficient Few-shot Adaptation for Vision Transformers
Authors: Chengming Xu
Siqian Yang
Yabiao Wang
Zhanxiong Wang
Yanwei Fu
Xiangyang Xue
Venue: Accepted by TMLR ()

Abstract: The task of Few-shot Learning (FSL) aims to do the inference on novel categories containing only few labeled examples, with the help of knowledge learned from base categories containing abundant labeled training samples. While there are numerous works into FSL task, Vision Transformers (ViTs) have rarely been taken as the backbone to FSL with few trials focusing on naive finetuning of whole backbone or classification layer. Essentially, despite ViTs have been shown to enjoy comparable or even better performance on other vision tasks, it is still very nontrivial to efficiently finetune the ViTs in real-world FSL scenarios. To this end, we propose a novel efficient Transformer Tuning (eTT) method that facilitates finetuning ViTs in the FSL tasks. The key novelties come from the newly presented Attentive Prefix Tuning (APT) and Domain Residual Adapter (DRA) for the task and backbone finetuning, individually. Specifically, in APT, the prefix is projected to new key and value pairs that are attached to each self-attention layer to provide the model with task-specific information. Moreover, we design the DRA in the form of learnable offset vectors to handle the potential domain gaps between base and novel data. To ensure the APT would not deviate from the initial task-specific information much, we further propose a novel prototypical regularization, which minimizes the similarity between the projected distribution of prefix and initial prototypes, regularizing the update procedure. Our method receives outstanding performance on the challenging Meta-Dataset. We conduct extensive experiments to show the efficacy of our model. Our model and codes will be released.

Published in Transactions on Machine Learning Research (08/2022)
Exploring Efficient Few-shot Adaptation for Vision Trans- formers
Chengming Xu
cmmu18@fudan.edue edw..
School of Data Science, Fudan University
Siqian Yang Yabiao Wang
seusonsyyeng@@eeennnn.ccmm com caseyyoengd@lennenl.comm
Youtu Lab, Tencent
Zhanxiong Wang Tencent
maczcoung@@tencnnt.ccom
Yanwei Fu" Xiangyang Xue
yonweifu edu.co ziangyangrue @f ffddnn edu..on
School of Data Science, Padan University
Reviewed OpenReview: https: o/per//vee./......... net/ forum? id=n3gLz4el.11
Abstract
The task of Few-shot Learning (FSL) aims to do the inference on novel categorics containing only few labeled examples, with the help of knowledge learned from base categories containing abundant labeled training samples. While there are numerous works into FSL task, Vision Transformers (ViTs) have rarely been taken the backbone to FSL with few trials (Hu et al. 2022 Evci et al. 2022. Abnar et al.) focusing naäve finetuning of whole backbone or classification layer. Essentially, despite ViTs have been shown to enjoy comparable or even better performance on other vision tasks, it is still very nontrivial to efficiently finetune the ViTs in real-world FSL scenarios. To this end, we propose a novel efficient Transformer Tuning (eTT) method that facilitates finetuning ViIs in the FSL tasks. The key noveltics come from the newly presented Attentive Prefix Tuning (APT) and Domain Residual Adapter (DRA) for the task and backbone luning, individually. Specifically, in APT, the prefix is projected to new key and value pairs that are attached to each self-attention layer to provide the model with task-specific information. Morcover, we design the DRA in the form of learnable offset vectors to handle the potential domain gaps between base and novel data. To ensure the APT would not deviate from the initial task-specific information much, we further propose novel prototypical regularization, which minimizes the similarity between the projected distribution of prefix and initial prototypes, regularizing the update procedure. Our method receives outstanding performance on the challenging Meta- Satta-Dt We conduct extensive experiments to show the efficacy of our model. Our code is available at httth::///// segttuub.cmm/ooadee//11TT.. TMLR202MR
1 Introduction
Modern computer vision models such as ResNet (He et al. 2016 and Faster R-CNN Ren et al. 2015 are trained on large-scale training sets, and not well generalize to handle the long tail categorics with few
*This paper supported by the project NSFC(62076067).

---

Published in Transactions on Machine Learning Research (08/2022)
labeled samples. Few-shot Learning (FSL) has thus been studied to make inference insuffiieently-lablle novel categories typically with the transferable knowledge learned from base categories which are provided with abundant labeled training samples. Essentially, the FSL can be taken as representation learning, as its backbones should idcally extract features representative and gencralizable to various novel tasks. Currently Convolutional Neural Networks (CNNs), especially ResNet, are the predominant backbone and widely utilized
in most existing FSL works (Ravi & Larochelle 2017 Sun al. 2019).
Finn et al. 2017 Nichol et al.] 2018 Li et al. 2017
Recently, by taking the merits of Multi-headed Self-Attention (MSA) mechanism and Feed Forward Net- work (FFN), the transformers have been widely used in the recognition (Alexey ct al.) Liu et al. 2021b), detection (Beal et al. 2020) and image editing (Cao et al. 2021p. The general pipeline of Pretrain- (Meta- train) (rain)-Finntuua has been explored in few ViT' on FSL (Hu et al., 2022 Evei et al.. 2022 Abnar et al.,, recently. Particularly, the ViT models are first pretrained waweaannddmmttttraaainn /meta-trained on a large-scale dataset. Then a test-time finetune procodure is set up for cach target task on novel data. The finctuning strategy can be generally calegorized into linear classifier probing and backbone tuning: the former one optimizes the reasonable decision boundaries by the fixed embeddings, while the latter one considers the adaptation of both embedding space and classifier.
In this paper we focus on the backbone tuning method. (Hu et al. 2022 shows that the naive Pretrain-Meta- train-Finetune (P>M>F) baseline can generally have satisfactory performance in FSL. Unfortunately, it involves heavy computations and potential overfitting in FSL setting. Particularly, (1) It typically demands extraordinary computing power to formulate episodes from a large number of support classes to update the whole network parameters. Thus it is less efficient in many real-case applications. For example, the edge devices such as mobiles donot have enough computational power to adapt all model parameters by cetpualiize/seciiaaizeeee data collected on these devices. (2) It is very subtle and difficult to directly fine-tune trained deep models on one or two labeled instances per class, as such few-shot models will suffer from severe overfitting (Snell et al.) 2017: Fei-Fei et al. 2006 Brian et al.). By contrast, humans have the ability of conducting few-shol recognition from even single example of unseen novel calegory with very high confidence.
Such problems may be the culprit of the phenomenon that their proposed finetune strategy only works on part of datasets and has less effect to the others. This suggests their limited usage of VIT backbone for any potential FSL applications. An alternative choice is to finetune specific layers in a ViT model with much smaller tunable parameters (ViT-s block in Fig. [a). Such a strategy nevertheless can only finetune either low-level or high-level features, leading to inferior performance in many cases. Therefore it is desirable to have an efficient and weighte ViT tuning method that shall not only avoid overfitting to small training samples, but also achieve high performance of FSL.
In this paper, we present a novel efficient Transformer Tuning (eTT) for few-shot learning task, which adopts gretrain-finetune pipeline. To pretrain our transformer, we advocate utilizing the recent sele-super method - DINO /Caron et al. 2021). ). Our key novelties are in the finetuning stage. As illustrated in Fig. [b), we propose Attentive Prefix Tuning (APT) and Domain Residual Adapter (DRA) as the key components to our eTT, to elficiently learn the newly-introduced tunable parameters over novel support sets. Specifically, we formulate the attentive prototypes by aggregating patch embeddings with the corresponding attention weights of the class token for cach image, so as to provide the model with abundant task-spocific information and guide each self-attention layer aggregate more class-related features. To encourage the prefix to keep the prior knowledge from initial prototypes, we further propose novel prototypical regularization which restricts the relationship between the prefix and prototypes by optimizing the similarity of their projected distributions. Morcover, we propose to additionally adopt a light-weighted domain residual adapter in the form of learnable offset to deal with the potential failure of APT on large domain gaps. Extensive experiments are conducted to evaluate our eTTT: we use the ViT-tiny and ViT-small backbones on the large-scale Meta-Datase et al. 2019) consisting of ten sub-datasets from different domains; and the results show that our model can achieve outstanding performance with comparable or even much fewer model parameters. Thus our eTT is a promising method on efficiently finetuning ViTs on the FSL tasks.
(Triantafiloou
Our paper has the following contributions.
1. In order to solve the problem of inefficiency and make better use of ViT in FSL, we propose novel
2

---

Published in Transactions on Machine Learning Research (08/2022)
Domain residual adapter
20
Params size (M)
Key/value
Flug
VIT block
pairs
15
project
Domain residual adapter
10
Visual
VIT block
prefix
5
mimllize
Attentive
prototypes
0
Full VIT-s
VIT-s block block
Prefix
DRA
Few-shot
episodes
Support images (b) Attentive Prefix Tuning in Task Tuning
(a) Tunable parameters in Backbone Finetuning
Figure 1: (a) Comparing with other backbones, we propose the Domain Residual Adapter (DRA) to lune much less parameters in our efficient Transformer Tuning (eTT); and effective for large-scale FSL. (b) The few-shot support samples are first processed into attentive prototypes which are used to initialize the task-specific visual prefix. Then the prefix together with the domain adapter are attached to each layer of the ViT to finetune our ViTs.
finetuning method named efficient Transformer Tuning (cTT).
2. Inspired by recent advance in language model, a novel attentive prefix tuning is presented utilizing the attentive prototypes to embed the task-specific knowledge into pretrained ViT model. Particularly, we propose a new initialization strategy tailored for FSL by leveraging prototypical information from the self-attention layers. Moreover, novel domain residual adapter is repurposed to handle the various domain gaps between training and testing data.
3. We introduce prototypical regularization term which can constrain the update procedure of prefix during finetuning to maintain the initial task-specific knowledge.
4. By utilizing the proposed eTT, our ViT models receive remarkable performance on Meta-Dataset, overpassing the cxisting ResNet-based methods without using additional training data. More importantly, both of the model scale and efficiency of our method are comparable with the other competitors, indicating the promising application of ViTs in FSL.
2 Related Works
Few-shot recognition. FSL learns transferable knowledge from base classes and adapt it a disjoint set (novel classos) with limited training data. Among those FSL tasks, few-shot image recognition is the with most focus and rescarches. Existing works can be grouped into two main categories. One is optimization-base
methods (Ravi & Larochelle 2017 Finn et al. 2017 Nichol et al. 2018 Li et al. 2017 Sun et al. 2019). which learn parameters that can be better finetuned on few-shot support sets. The other is metric- based
methods such as ProtoNet (Snell et al.] 2017). RelationNet (Sung et al. 2018 CAN (Hou et al.)
2019).
DMF (Xu et al. 2021), COSOC (Luo et al.] 2021) and CTX (Doersch et al.; 2020), which solve FST. by applying an existing or learned metric on the extracted features images. Particularly, CTX (Doersch et al., 2020) builds up cross altention module which interacts between query and support images to adaptively aggregate better prototypes than simply averaging all support features. While these methods perform well on classical few-shot learning settings, most of them adopt convnet as backbone, especially ResNet (He et al. 2016). We, on the opposite, try to make full use of another widely-applied structure, i.e. ViT, in PSL, which requires extra design for training and finetuning strategy.
Transformer in vision tasks. Transformers widely utilize the self-attention mochanism which originally are employed process the feature sequence in Vaswani et al. (2017). Then large scale transformers become increasingly popular in NLP tasks to build complex language models, and also extend to vision tasks (Alexey
3

---

Published in Transactions on Machine Learning Research (08/2022)
2021 Liu et al. 2021b) by formulating the token sequence with image patches processed with position embedding. It has been shown the efficacy in various applications, such as (Liu et. al. 202Ta] for image caption, (Sun et al. 2020) for multiple object tracking and (Esser et al. 2021 Cao et al. 2021 for image inpainting and oditing. Critically, ViTs is typically trained by very large-scale dataset, and few effort has been dedicated in training or fineluming on few-shot supervision. We follow the mataraai-tta--ai----iiiee
et al. Yuan et al.
pipeline (Hu et al., 2022), while their method finetune the whole ViTs on few-shot examples, and thus has less efficiency and can easily overfit. In contrast, our proposed eTT has the key components of DRA and APT, demanding much less tunable parameters with much better performance.
Finctuning algorithm for VIT. The idea of finctuning ViTs on small-scale datascts has been partly investigated in Natural Language Processing (NLP) communities. Houusby et al. (2019) proposed to attach two learnable bottleneck adapters to each transformer layer. Other works (Xiang & Percy] Brian et al.) make use of the prompt which trains a small task-specific prompt for each task so that the prompt can guide the model with knowledge corresponding to the task. Such prompting idea from NLP inherited and repurposed to finetune learnable prefix for each novel episode in this paper. However, these works (Xiang & Percy Brian et al. Houlsby et al. 2019) initialize the prefix or prompt with word embeddings which is not available in our problem. Instead, we propose an attentive prototype with regularization initializing the visual prefix with object-centric embeddings. Additionally, we notice that very good concurent technical report (Jia et al. 2022) also studies finetuning visual prompt for pretrained ViTs in downstream tasks. We highlight the two key differences from our TTT. The first is about the initialization. While initialization strategy does not matter in their method and the corresponding tasks, we show in our experiments that randomly initializing prefix does lead to sub-optimal performance in FSL, which leads to the necessity of a well-designed initialization. The sccond is that we further propose regularization term to restrict the prefix, which has never been studied in existing works.
Task-specific Adapter. The idea of task-specific adapter has been explored in several works like Li ct al. 2022 Rebuffi et al. 2017) to adapt CNNs to learn the whole information from support sot. Besides, Requeima el al. 2019 Baleni et al. 2020) adopt Feature-wise Linear Modulation (FiLM) layers (Perez el al., 2018) to adapt speciic information into networks. In contrast, we repurpose the adapter as the domain rosidual to update transformer blocks in more light-weighted way with less learnable paramcters. Beyond different structures, our proposed DRA intrinsically serves the domain adapter rather than meta-learner for the FST. in Rusu et al. (2018); Sun et al. (2019); Li et al. 2002); Requeima et al. (2019). While these previous works require meta-training to optimize their adaptation modules, our method simply utilizes the novel support data to learn the DRA, thus reducing the training cost. Furthermore, our DRA is mostly tuned to bridge the visual domain gap between base and novel categories, thus improving the learning of APT on each episode task.
3 Methodology
3.1
Problem Setup
We formulate shw- learning in the meta-learning paradigm. In general, we have two sets of data, namely meta-train set D. = ((I, y/) We E C,} and meta-test set D, = ((I, ys E C.) which contain the base and novel data respectively and are possibly collected from different domains. C. and C, (C, n = 0) denote base and novel category sets. FSL aims to train a model on D, which is generalizable enough on D.. In the testing phase, the model can learn from few labelled data from each category of C..
While most previous FSL works (nell et al. 2017 Sung et al. 2018) utilize the setting of N-way K-shot in mini-ImageNet, ie., K training samples from N class, we follow CTX (Doersch et al. 2020) to adopt the setting on the large-scale Meta-Dataset (Triantafillou et al. 2019). In each episode T N is first uniformly sampled from [5, Nmaal where Nmax cquals to (50, [c,l) or min [C.]) on training or testing stage, accordingly. N is supposed to be accessible knowledge during both training and testing. In the most naïve case, one can get N by directly counting the number of support classes. From each of the sampled category, M query samples per category are randomly selected, and thus constructing the query set Q = ((T). v() Na A fter that random amount of samples are taken from the rest of samples belonging to these categories to
4

---

Published in Transactions on Machine Learning Research (08/2022)
Patch embedding
transformer layer
transformer layer
Linear
ProtoNet
LN
MSA
1
attention
layer
FFN
0
K
6
V
LN
Ag A
Patch
embeddings
MSA
projector
8
Pp
Image embedding
aggregate
Figure 2: Schematic illustration of our proposed model. For each image, we first fetch its patch embedding sequence and the atfention score with regard to the last. layer's class token, from which the image embedding can be computed. Then the visual prefix is initialized as the attentive prototypes image embeddings. The prefix, together with the proposed domain residual adapter are attached to the model. The final features are processed with an extra linear transformation layer and predicted with ProtoNet. Dashed arrows denote forward propagation before test-time finetuning.
form the support set S {[T."MP , "i } Note that compared to the classical N-way K-shot setting, such a setting generates dass-imbalanced support sets, and different episodes contain different numbers of support samples. This is much more challenging to the model and learning algorithms, they shall handle both extremely large and small support sets.
3.2 Overview of Our Method
To handle the optimization of various episodes large-scale dataset, present novel finetuning model elficient Transformer Tuning (eTT) as shown in Fig. 2 Our eTT follows the pipeline in Hu et al. (002)), and has key stages of the pretraining and finetuning. We employ DINO as pretraining, and conduct the task tuning by attentive prefix tuning (Sec. 3.4), and backbone tuning with domain residual adapter (Sec. 3.5)
Pre-training. As previous work (Hu et al. 2022) shows the importance of self-supervised pre-training to learning vision transformer models, we adopt the same principle and introduce the self-supervised learning model pre-train our VIT backbone on base data. Specifically, we utilize the recent State-of-the-art self-supervised ViT models DINO (Caron et al.] 2021 to pretrain model. DINO builds up supervision based on a self-distillation framework by using the multi-crop strategy (Caron et al. 2020). As we will show in our experiments, such a pre-trained model shall have good cluster property even among cross domain images, polentially benefiting our following FSL stages. Note that different from (Hu et al. takes an ofe-the- model pretrained with DINO on full ImageNet, we strictly follow the FSL protocols to retrain the DINO models on the meta-train split in the target dataset to avoid the abuse of extra data.
2022) which
One would ask whether it necessary to make of the annotations for base data, since supervised pretrain has been proven to be effective in many previous FSL works (Ye et al. 2020 Hou et al. 2019). As we will show in the experiments, an additional finetuning with image labels on basc data cannot bring consistent improvement and even makes it worse on most datasets, which may be caused by the overfitting on the image labels leads to less generalization ability across different domains. Moreover, compared with vanilla supervised training, the attention maps for models trained by DINO contain more semantic information, which we will utilize in the following context.
3.3 Preliminary: Vanilla Test-time Finetuning
Before fully developing our fine-tuning contributions, we review the simple and effective finctuning method named LT+NCC [Li al. 2021). The novel modules proposed by us in the following context are all adopted together with this simple baseline method. Given ViT backbone So that is parameterized by 0 and
5

---

Published in Transactions on Machine Learning Research (08/2022)
episode T, the support features {a; (x;">")) 1 where z;'r fo," are extracted from the support set (I;""") Mizl. Then, a learnable linear transformation results in the final support fealures used for classification { )= Ji= where 2.uPp = ((x;"p)). The prediction of these support images can thus be calculated based on the similarity between the transformed features and the aggregated prototypes as,
added the model to realize the adaptation, which
cxp(d(0, .F))
Ee
PPPP=C
( (c)
E 1
81
Ee exp(d(E; .5,))
where d denotes cosine similarity, i..., d(a,b) = bbb and adopt the cross entropy loss optimize the transformation Φ. Precisely speaking, for each support image puup together with its annotation youpp, the objective function is as following:
||l|i We fix all of the parameters in the original backbone,
fC -y "Pp log "upp
(2)
After finetuning, & is applied to query features and the same procedure above is performed between the processed query features { and protolypes (Te) for the inference of each episode.
3.4 Task Tuning by Attentive Prefix Tuning
We finetune the pre-trained VIT with support set via an attentive prefix luning strategy. Specifically, prefix matrix Op E Nexd is first initialized, where Np denotes the number of prefix. Then a bottleneck g is added upon Op to produce 0p E (ENex()Ld) where L denotes the number of backbone layers. The plays the same role as the projector in each self-attention layer, except that all layers share the same module. The produced 0r can be reshaped and seen as I, value and key pairs (0), Of 1 By 0g E Ne The MSA block in the L-th layer can then be reformed by attaching these new pairs to the original key and value sequences:
A' = Att [K:O]
output = A [V:0]]
where [: denotes concatenation, Attn denotes the calculation of MSA matrices. In this way, the prefix can affect the attention matrix A' and result in different output features from the original ones.
Remark. Compared with the naive strategy that finctunes specific layers in VIT (VIT-s block in Fig. aa) which can only adjust part of blocks, the prefix can evenly adapt each layer's image embedding with almost the same parameter size as one transformer layer, as shown in Tab. [[a. By fixing the model parameters and optimizing the prefix Op and the transformation module g: the support knowledge can be smoothly embedded into the prefix, which further helps the task adaptation.
Attentive Prototype. The initialization of the prefix is very important to APT, as it greatly boosts the performance. Critically, quite different from the prefix or prompt tuning in NLP and visual-context tasks that have task-specific instructions explicitly as word embedding sequences, each episode in our FSL only has the few support images and their labels. Thus, rather than steering the model with 'what should be done' as in Xiang& & Percy] our APT shall provide the model with 'what we have globally' by leveraging the class-specific information. Thus, the attentive prototype is presented to aggregate the image embeddings with object-centric attention, as the initialization of the prefix. Particularly, each support image suup first transformed to patch embebdding sequence { m== with the starting patch embedding layer,
(4) where m = 1,...,,"* is the patch index; fez denotes the patch embedding layer which is typically a convolutional layer whose kernel size equals to patch size; and E indicates the position embedding. Meanwhile, we can get unnormalized attention score A E g between the class token and image patches from the last MSA layer, where denotes number of heads in each MSA module. Such an attention voctor can focus on the foreground in the image, especially for models trained with DINO (Caron et al. 2021). p, with each head indicating a particular part or an object. We can thus get the initial image-level representation
goupp fop, (I + Epos
h
p2
À A=
soup
Anm Cammssup ssupp
(5)
12
6

---

Published in Transactions on Machine Learning Research (08/2022)
where is softmax function. Compared with simply averaging all patch embeddings, the attentive embeddings can highlight the objects of interest and suppress the background information. Then the prototypes I can be calculated by averaging the attentive image embeddings belonging to each support category. We set the number of prefix as N, which is available during testing for cach episode, and initialize the prefx with F.
Remark. In this way, commonly-used prototypes can provide the model with comprehensive information about the episode. Also such first-order statistics is comparable with the normal patch features among the layers. This benefit the training with more stability. When N is large, more prefix are required to fully learn the information included by each episode. On the other hand, when N is small that the episode is relatively easy, fewer prefix can handle the support knowledge without trouble while decrcasing the computing debt.
3.5 Backbone Tuning by Domain Residual Adapter
Finetuning few-shot tasks by APT will make a good balance between performance and efficiency. To further improve the model generalization ability on different domains, we further propose the backbone tuning by leveraging the Domain Residual Adapters (DRA), as illustrated in Fig. 2 Specifically, for the I-th transformer layer, we attach two learnable offset vectors 8,, 6/ E Rd the MSA and FFN, After features are processed with MSA and FFN, the corresponding offsets added to them so that the extreme domain gap can be neutralized. These offsets are expected to represent the gap between source and target domains, and transfer the original manifold to a more appropriate one.
3.6 Loss Functions
Prototypical Regularization. In addition to the cross entropy loss in Eq. 2 we propose a novel prototypical regularization to ensure the class-specific information, which is embedded in the prefix via initialization, can be maintained during update. The knowledge in attentive prototypes is distilled to the prefix during finetuning. Concretely, in each iteration, the prototypes E and prefix 0p are first projected to latent space via projector module v which produces H' and 0% respectively. Then the distillation loss is computed using these two embeddings as,
N Σ (1=m,0pp)
1
Edistt
N
n=1
where (a, looob.. The above objective function can ensure the prototype of each category and the corresponding prefix contain consistent information, which is indicated by the similarity between distributions after projection. To make training more stable and avoid collapse, for each episode we maintain exponential moving average (EMA) of म' as the center variable Ceenter. Before calculating ldist, we standardize V' as o 2 (X'=teenter) where denotes softmax function and is the temperature typically set 0.04.
Once having both of the above losses calculated, we can optimize the model parameters including the DRA, the prefix logether with the transformation and the projector v, with the following objective function:
L= Aldist
where the scalar weight À controls the strength of the regularization.
Remarks. For a ViT with L layers, na heads and d feature dimension, the size of trainable parameters is (N + + dpros + d)d + 2(d" + Ld, where d' is the hidden dimension for transformation module g and doroj denotes output dimension for the projector vb, which much smaller than that of the whole backbone model. Specifically, the learnable modules during finetuning have only about 9% parameters with regard the whole transformer model when using small and ViT-tiny.
7

---

Published in Transactions on Machine Learning Research (08/2022)
4 Experiments
4.1 Experimental Setup
Datasets. We use Meta-Dataset (Triantafilouu et al. 2019) the most comprehensive and challenging large-scale FSL benchmark. It has 10 sub-datasets such as ImageNet (Deng et al., 2009) and Omniglot (Lake) et al. 2013), with various domain gaps. Our experiments are conducted under the single training source setting, i.. only ImageNet used for training, and the meta-test split of all ten datasets for evaluation. Some of the (est datasets such CUB share similar or highly-related calegories with ImageNet, while the others have greater domain gaps. Note that Hu et al. (2022) claims pretraining all images in the training set of ImageNet is reasonable for introducing extra data and boosting the performance. However, such a strategy utilizes much more training samples (1.28M images, 1000 classes in ImageNet v.s. 0.9M images, 712 classes in meta-train split of ImageNet). Empirically so many additional images can greatly benefit generalization ability of self-supervised learning methods. Therefore to make a more fair comparison, we strictly follow the experiment protocol used in CTX (Doersch et al.) 2020) and shall not use any extra data even in the unsupervised pretraining stage. We resize all images to 224 224 for ViT-small and 84 x 84 for ViT-Liny.
Implementation details. We set the patch size as 8 for ViT-tiny (as it has small input image size), and keep the other feature dimension as 384 and patch size as 16. We strictly follow the hyper- parameere setting and data augmentation in DINO (Caron et al., 2021) for pretraining. In test-time finetuning, we empirically set the hidden dimension of the transformation module as d/2, and output dimension dproj of the projector as 64 for all datasets. We utilize AdamW optimizer finetuning, with learning rate set as 1e-3 3 for TraffieSign and 5e 4 for other datasets. X is set 0.1. For simplicity, the selection of hyper-paramettrss is conducted on the meta-validation set of ImageNet, which is the only within-domain sctting in Meta-Dataset.
as default. We adopt standard ViT-small with 12 layers, 6 attention heads,
Evaluation benchmark. We report the accuracy of randomly sampled 600 episodes for each dataset and the average accuracy when comparing with the existing methods. The comprehensive comparison of both accuracy and 95% confidence interval is in Appendix.
Backbone
Image size
Params(M)
FLOPs(G)
Res18
84x84
11.69
1.82
ViT-tiny
84x81
5.38
0.72
Res34
224x224
21.80
3.68
ViT-small
224x224
21.97
4.61
Table 1:
Comparison of parameter size and FLOPs between different backbones.
4.2 Comparison with State-of-the-art Methods
Before the comprehensive comparison, it is necessary show the comparison between different backbone is fair enough since our backbone model is not the same as the existing method. Therefore we present the comparison of size of model parameters and FLOPs in Tab. I in which the FLOPs of all models are computed by [vcoreF] The results show that (1) compared with Res18, ViT-tiny is much smaller and efficient model, and (2) ViT-small is approximately comparable to Res34. In this way, the comparison of our proposed method with sfate-of-the-art methods is reasonable and fair.
We compare our model with ProtoNet (Snell et al. 2017), CTX Dooessh el al. 2020 TSA (Li el al.
2022)
etc. These methods take the backbones of 18 or ResNet34. Also, the pretrain-mett--raan--ii baseline (P>M>F) (Hu et al. (2022) is not considered in computing average rank since extra data is used. As in Tab. 2 when using ViT-small as backbone whose parameter size is comparable to that of ResNet34, our model receives 1.6 average rank on all dataset. Specifically, on Texture and Fungi, our model outperforms the strongest competitors CTX and TSA by about 8% and 10,, while on other datasets the performance of our model is still comparable with or slight better than that of the existing methods. We notice that our model
8

---

Published in Transactions on Machine Learning Research (08/2022)
Model
Backbone
ILSVRC
Omni
Acraft
CUB
DTD
QDraw
Fungi
Flower
Sign
COCO
Avg
Rank
Finetune
45.78
60.85
68.69
57.31
69.05
42.60
38.20
85.51
66.79
34.86
56.96
10.2
Proto
50.50
59.98
53.10
68.79
66.56
48.96
39.71
85.27
47.12
41.00
56.10 16.10
10.5
Relation
34.69
45.35
40.73
49.51
52.97
43.30
30.55
68.76
33.67
29.15
42.87
14.6
Res18
P-MAML
49.53
63.37
55.95
68.66
66.49
51.52
39.96
87.15
48.83
43.74
57.52
9.2
BOHB
51.92
67.57
54.12
70.69
68.34
50.33
41.38
87.34
51.80
48.03
59.15
8.2
TSA
59.50
78.20
72.20
74.90
77.30
67.60
44.70
90.90
82.50
59.00
70.68
4.3
Ours
VIT-t
56.40
72.52
72.84
73.79
77.57
67.97
51.23
93.30
84.09
55.68
70.54
4.1
Proto
53.70
68.50
58.00
74.10
68.80
53.30
40.70
87.00
58.10
41.70
60.39
7.4
CTX
Res34
62.76
82.21
79.19
80.63
75.57
72.68
51.58
95.31
82.65
59.90
71.28
2.8
TSA
63.73
82.58
80.13
83.39
79.61
71.03
51.38
94.05
81.71
61.67
74.93
2.5
P>M>F*
74.69
80.68
76.78
85.04
86.63
71.25
54.78
94.57
88.33
62.57
77.53
VIT-s
Ours
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
1.6
Table 2: Test accuracies and average rank on Meta-Datasct. Note that different backbones are adopted by these moth denotes using extra data for training. The bolded items are the best ones with highest. accuracies.
is inferior to the best ones in Omniglot, while this is reasonable. Since Omniglot images represent simple characters with monotone color patterns, each image patches contain less information than images in other datascts. Vanilla ViTs have less efficiency in dealing with these image patches due to limited interaction among patch embeddings. This problem can be solved with much sophisticated variants of ViT like Swin (Liu et al., 2021b), and will be taken as future works. Moreover, our proposed method is better than P>M>F, which not only utilizes extra data for training but also finetunes all model parameters during testing, more than half of the datasets, which strongly indicates the effectiveness of the proposed finetuning strategy in this paper. As for using ViT-tiny which has much less parameter than Res18, our model is still comparable to the state-of-the-art methods and outperforms many popular baselines. Particularly, compared with ProtoNet which is one of the most famous and efficient methods in FSL, our eTT shows significant boost on Aireraft by 19.74% and TrafficSign by 36.97%. The reason of the inferior results on several datasets against TSA can be two folds. Firstly, the ViT-tiny intrinsically has smaller capacity than Res18. On the other hand, while it is common to train ViT with large scale images and patches so that the images are splitted into abundant patches and each level- token can receive enough information. In contrast, we adopt 84 x 84 images with 8 x 8 patch size for ViT-tiny so that the comparison with Res18 is fair, which lead to less patches with smaller size and may have negative influence on the performance. In general, the results indicate that our proposed eT can make ViT models a desirable choice for large scale FSL problems.
4.3 Model Analysis
To further validate the effectiveness of our method, we conduct series of ablation studies on using ViT-small below.
4.3.1 Design of Each Module
Can finetuning on meta-train set boost the performance? One would ask whether it is necessary to make use of base annotations, as supervised pretraining is also effective in many FSL works ( YYe al. 2020 Hou et al.) 2019). To verify it, we finetune DINO-pre-trained ViT-small meta-train split of ImageNet, in which the options of all typer-parameters and data augmentations follow DeiT (Tuvron et al. either way of class token features or averaged patch features as image representations. After such a supervisod finetuning, we Lest the models both with the basic test-time finetuning method as in Sec. 3.3 which we denote as LT+NCC, and with our proposed eTT. The results are shown in Fig. 3 from which we find out that (1) Supervisod finetuning does improve test accuracies on ImageNet, CUB and MSCOCO. Particularly, the token finetune model receives 89.83% accuracy on CUB when testing with our eTT, which is remarkably better than any other models. This is reasonable as similar images between ImageNet and these datasets. By
2021 using
9

---

Published in Transactions on Machine Learning Research (08/2022)
Model
ILSVRC
Omni
Acraft
CUB
DTD
QDraw
Fungi
Flower
Sign
COCO
Avg
Proto
63.37
65.86
45.11
72.01
83.50
60.88
51.02
92.39
49.23
54.99
63.84
LT+NCC
65.96
67.62
64.03
77.10
83.46
63.88
57.79
93.13
66.91
56.04
69.59
Last
66.32
71.04
78.04
86.25
86.67
64.22
55.69
94.44
65.55
55.94
72.42
First
61.54
50.46
69.23
79.17
83.10
68.69
49.93
93.50
54.28
58.45
66.84
LN
66.22
70.45
69.41
81.29
86.37
66.28
58.38
96.25
71.09
59.57
72.53
APT
66.75
75.16
75.41
84.25
86.47
69.55
60.03
96.38
78.20
61.10
75.33
Adapter
66.53
72.31
73.75
83.73
86.86
66.74
58.49
96.15
82.65
62.40
74.93
eTT
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
Random
66.12
76.33
78.35
84.77
86.78
70.13
59.25
96.00
82.28
59.59
75.96
Avg
66.11
75.06
77.07
85.16
87.35
70.72
61.79
96.54
84.28
62.18
76.73
Sampling
67.81
76.72
77.96
85.79
87.25
70.19
60.73
96.27
83.72
62.17
76.86
Full
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
Linear
66.35
74.26
79.42
83.65
86.02
71.11
55.73
95.89
82.73
59.90
75.51
Bottleneck
67.29
76.06
79.72
85.60
87.21
70.59
61.59
96.15
85.00
62.02
77.12
FiLM
66.91
75.32
78.26
85.78
86.83
70.29
61.65
96.50
84.48
61.75
76.78
Offset
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
w/o PR
66.72
74.20
78.42
85.06
87.01
70.34
61.64
96.51
84.23
61.08
76.52
w PR
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
w/o Stand
67.09
76.42
78.87
83,10
86.50
70.09
61.02
96.33
82,88
61.33
76.36
w Stand
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
Table 3: Test accuracies on Meta-Dataset of different variants of our proposed method. The bolded items are the best ones with highest accuracies.
training on the image annotations of ImageNet, the model learns class-specific knowledge which cannot be obtained during self-supervised learning. Since the categories are highly correlated and overlapped among these datasets, the learned knowledge can also benefit the recognition on these novel datasets even though the specific novel classes do not appear in the meta-train set. (2) Despite the improvement on the three datasets, models with supervised finetuning degrade the other datasets, especially on Traffic Sign and VGG Flower. This is due to fitting class labels weakens the effect of these features and makes it harder to generalize to novel domains. When taking into account the performance of all datasets, pretraining with DINO is generally the much more desirable choice for better generalization over different domains. (3) The improvement of our propose method against. the basic LT+NCC not consistent. among three different kinds of pretraining strategy. For example, while our method can boost the performance of DINO pre-trained model by 9.47% on Aircraft and 4.83% on CUB, it can only bring much less advantage on models with supervised finctuning.
Effectiveness of APT and DRA. We test the DINO pre-trained model with different kinds of testing strategies including (1) Proto: Directly generating predictions based on ProtoNet. The prototypes are computed using averaged class token features from each category.
(2) LT+NCC: The basic test-time finetuning method in Sec. 3.3 (3) Last: Finetuning the last transformer layer during testing, together with LT+NCC. which has similar parameter size our method. (4) First: Finctuning the first transformer layer during testing, together with LT+NCC. which has similar parameter size to our method. (5) LN: We try to finetune the affinity parameter in each layer normalization as alternative finetune strategy, which is utilized in many cross-domain FSL works [Tengg et al. Tsutsui et al. 2022). (6) APT: The model is finetuned using APT together with LCCNCC, using cross entropy loss and the proposed prototypical regularization. (7) Adapter: The model is finetured using DRA logether with LT+NCC, using cross entropy loss. (8) eTT: The model is finetuned using our proposed APT, DRA and LT+NCC. The results in Tab. 3 show that while LC+NCC can fundamentally improve the model which indicates the importance of test-time finetuning, adding our proposed modules to the finetuning procedure can consistently bring higher performance. Also, finetuning specific transformer layer can only bring limited improvement few datasets: finetuning the last.
10

---

Published in Transactions on Machine Learning Research (08/2022)
taken
token
90
pool
DINO
DINO
Test Accuracy
Test Accuracy
50
50
BURCC
Orni
Acafft
CUB
DTD
ODraw
Fungi
fooer
Sinn
COCO
SSVRC
Onnis
Acraff
CUB
DTD
ODraw
Fungi
Power
Sign
aoco
Test Accuracy when using NCC
(b)
Test Accuracy when using eTT
Figure 3: Test accuracy of different training strategy if testing with (a) LT+NCC (b) eTT.
Before finetune
LT+NCC
Ours
Figure 4: Visualization of feature embeddings from randomly sampled episode of TrafficSign.
layer leads to good performance Aireraft, CUB and Texture, while updating the first layer leads to good performance on Quickdraw and MSCOCO. However, this simple finetuning strategy cannot bring consistent improvement on all datascts. This indicates that diffcrent data requires different levels adaptation, and the improvement is much smaller than that of our method. Moreover, we give the (SNE visualization of feature embeddings randomly sampled episode from TrafficSign in Fig. g which demonstrates that utilizing our proposed method can better regulate the feature embeddings into proper clusters.
Is prototypical initialization necessary? One of the most important parts of our APT is the attentive prototypical initialization in which we use attentively aggregated patch embeddings to initialize the prefix matrix. To verify this strategy, we compare several different choices of initialization, including (1) Random: random initialization from normal distribution. (2) Avg: simply averaging all patch embeddings from each category. (3) Sampling: randomly sampling one image for each category, and then initializing the prefix matrix with the averagod patch embeddings of cach image. (4) Full: computing prototypes with our proposod attentive prototype. Results in Tab. 3 show that random initialization performs the worst, which can be resulted from insufficient task-specific information provided by the prefix in this way. Meanwhile, among all other strategies, using the attention map to aggregate patch embeddings as in Eq. 5 is better than simply averaging, leading to about 1% improvement on average.
Do we need a more complex adapter structure? One would argue that our DRA structure is too simple learn the complex knowledge from support images. In Tab. 3 we compare three different instantiations of adapters including (1) Linear: As in Li et al. (2022), we use linear layer for each adapter, whose output are than added to the original features in the MSA and FFN. (2) Bottleneck: We expand the linear layer
11

---

Published in Transactions on Machine Learning Research (08/2022)
to bottleneck structure where two linear layers are used for each adapters. (3) FiLM: We compare DRA with a FiLM--like variant, in which we add a scaling vector for each adapter as in FiLM layer Perez et al. (2018). Note that such a method is similar to MTL Sun et al. 2019) in FSL. The difference lies in that we still use the original way to directly tune the parameters on the novel support sets, instead of using another meta-trained module to generale the parameters. (4) Offset: Only an offset vector adopted for each adapter. The results reveals that the linear adapter performs the worst, which means such a structure improper for VIT in finetuning. Moreover, we also find that using the bottleneck adapter will result in a dilemma. If we use small initial value for the adapter, the weights of each layer can only achieve gradient with extremely small values. As the result, these weights, except the bias term of the last layer, can hardly be updated based on the support knowledge, which means such an architecture almost equals to our design where only an offset vector is utilized. On the other hand, if large initial values are adopted to avoid gradient diminishing, then the output features from the adapters can make the predictions severely deviate from those without adapters, thus leading to worse performance. As for the FiLM-like DRA, it is worse than offsct DRA by about 0.8% on average, while it doubles the parameter size based on offset DRA, leading to no significant additional elficacy.
Effectiveness of prototypical regularization. We also validate this regularization. In Tab. we present the test accuracy when finctuning with and without this loss function. We can find that by applying this objective function, the model can have higher results on most datasets. Besides, as described in Sec 3.6 we use a standardization technique when computing the prototypical regularization. To verify its efficacy, we compare the model with and without such a standardization. The results are shown in Tab. 3 When not using standardization, the results are generally worse given comparable confidence intervals (Tab. 11 The results verify that this strategy can help the model with more stable finetuning procedure.
dpros
ILSVRC
Omni
Acraft
CUB
DTD
QDraw
Fungi
Flower
Sign
COCO
Avg
64
67.18
75.30
78,88
86.20
87.09
69.82
61.61
96.31
82.24
62.14
76,68
96
66.23
75.69
78.26
85.67
87.28
70.25
61.97
96.59
84.10
62.17
76.82
128
67.31
76.83
78.81
85.77
87.36
70.16
60.81
96.53
84.29
62.12
77.00
256
66.83
78.04
78.38
84.60
86.68
70.43
61.03
96.23
85.33
62.10
76.97
192
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
Table 4:
Test accuracies on Meta-Datase of different variants of our proposed method. The bolded items are the best ones with highest accuracies.
4.3.2 Comparison among Different Hyper- paperrpprrametr Settings
In additional to the ablation study about the proposed module, We further verify different choices of hyper- parameters in our model. Especially, dyroj for the transformation module in APT and X for the prototypical regularization are tested in Tab. 4 and Tab. 5 in the Appendix. In general, the improvement is not consistent. For dproj; we can find that using 192-d hidden dimension can get globally better results, which indicates that such a choice can make good balance between the model capacity and scale so that the finetuning can be conducted both efficiently and effectively. As for X, 0.1 seems to be a desirable choice. Intuitively, smaller À leads to less control of the prefix from the proposed prototypical regularization. Therefore, the prefix may lose the desired information during the optimization on the support set. On the other hand, when A is too large, the regularization overwhelms the label supervision, and thus the model can hardly adapt to the support knowledge, leading to worse performance especially on Omniglot and Aircraft.
5 Conclusion
We propose a novel finetuning method named efficient Transformer Tuning (eTT) for few-shot learning with ViT as our backbone. By fixing the parameters in the backbone and utilizing attentive prefix tuning and domain residual adapter, our method can guide the ViT model with comprehensive task-specific information, which leads to better representations and performance. This is demonstrated by the fact that we establish new state-of-the-arts on the large-scale benchmark Meta-Dataset.
12

---

Published in Transactions on Machine Learning Research (08/2022)
References
Samira Abnar, Mostafa Debghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training, In The Tenth International Conference on Learning Representations, ICLR 2022.
Dosovitskiy Alexey, Beyer Lucas, Kolesnikov Alexander, Weissenborn Dirk, Zhai Xisohua, Unterthiner Thomas, Dehghani Mostafa, Minderer Matthias, Heigold Georg, Gelly Sylvain, Uszkoreit Jakob, and Houlsby Neil. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14493 14502, 2020.
Josh Beal, Eric Kim, Eric Tzeng Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer- based object detection. arXiv preprint arxiv:2012. 09958, 2020.
Lester Brian, Al-Rfou Rami, and Noah Constant. The power of scale for parancter-cffiient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021.
Chenjic Cao, Yuxin Hong, Xiang Li, Chengrong Wang, Chengming Xu, Yanwei Fu, and Xiangyang Xue. The image local autoregressive transformer. Advances in Neural Information Processing Systems, 34, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 99244,4,,, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on compuler vision and pallern recogrition, pp. 248 255. leee, 2009.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosttransformers: spatially-aware few-shot transfer. Advances in Neural Information Processing Systems, 11-21981-21993 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-12883, 2021.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing intermediate representations for better transfer learning. In International Conference on Machine Learning, pp. 6009 6033.
PMLR, 2022.
Li Fei-Fei, Rob Fergus, and Pictro Perona. One-shot learning of object categories. 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFE Conf. Comput. Vis. Pattern Recog., 2016.
Ruibing Hou, Hong Chang, MA Bingpeng, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In Adv. Neural Inform. Process. Syst., 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799) PMLR, 2019.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9068 9077, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint crXim:200..12119 2022.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332-3333) 2015.
Hacbeom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minscop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks. In 8th Interna- tional Conference on Learning Representations, ICLR 2020.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few-shot classification. In SEE//CV International Conference on Computer Vision (ICCV), pp. 9526-9535, October 2021.
Weihong Li, Xialei Liu, and Hakan Bilen. Cross-domain few-shot learning with task-specific adapters. In LEEE/CVF Conference on Computer Vision and Pattern Recognition 2022, 2022.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arxiv: 707.09935, 2017.
Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing Liu. Cptr: Full transformer network for image captioning. arXiv preprint Xrx:221101. 10804, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 10012-10022, 2021b.
Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shortcut learning of background for few-shot learning. Advonces in Neural Information Processing Systems, 34, 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order mela-learning algorithms. arXie preprinl 0rXiv:1803.022 020999 2018
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Int. Conf. Learn. Represent., 2017.
Sylvestre-Alvis Rebulli, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 32, 2019.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascamu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.055: 05960, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Adv. Neural Inform. Process. Syst., 2017.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. arXiv preprint orXin:2012. 15460, 2020.
Qjanru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog,, 2018.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347- 10357. PMLR, 2021.
Eleni Triantafiloou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A cataset of datasets for learning to learn from few examples. arXie preprint XxXx:1903.0000 2019.
Hung-Yu Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming- Yang- Cross-domain few-shot classification via learned feature-wise transformation. In 8th International Conference on Learning Representations, ICLR 2020.
Satoshi Tsutsui, Yanwei Fu, and David Crandall. Reinforcing generated images via meta-learning for one-shot fine-grained visual recognition. TEEE Transactions on Pultern Analysis and Machine Intelligence, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Lisa Li Xiang and Liang Percy. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021.
Chengming Xu, Yanwei Fu, Chen Liu, Chengjie Wang, Jilin Li, Feiyue Huang, Li Zhang, and Xiangyang Xue. Learning dynamic alignment via meta-filter for few-shot learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pp. 5182 5191, 2021.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8808 8817, 2020.
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 558-567, 2021.
A Appendix
A.1 Limitations and Future Work
Despite the marginal effectiveness and efficiency of our proposed eTT, we mainly notice two points that should be explored in the future: (1) The plain ViT backbone utilized in this paper, may not be the best choice to the simple dataset, e.g., Omniglot, while well-designed VIT backbone may potentially better improve the efficacy of our method on such datasct. (2) A flexible finetuning algorithm such as (Lee et al.) may have belter generalization ability when facing episodes with various shots and ways, than the commonly-used methods that adopt fixed test-time finetuning iterations. However, it is non-trivial to directly merge such mothods with our proposed cTT due to different network structures and tuning strategies. It can be taken as the future work to properly utilize these flexible finetuning algorithm to further improve the performance of VIT in FST..
15

---

Published in Transactions on Machine Learning Research (08/2022)
A.2 Additional Experiment Results
A.2.1 Full Comparison with state-of-the-ark methods
We show the accuracies together with confidence interval in Tab. G Beyond the accuracies which is analyzed in the main context, the confidence interval of our eTTTT on both ViT-tiny and ViT--small is comparable with the other competitors, which reflects that our method is stable and robust enough among different testing episodes.
A.2.2 Influence of Training set
As we have stated in the main context, our eT is trained on the meta-train split of ImageNet to make fair comparison with other methods. To show to what extent training on the full ImageNet training set instead of the meta-train set can itppact the performance, we test our eTT using off-the-shelf DINO ViT-s model. The results are shown in Tab. 7 We can find that (1) For those datasets on which DINO meta-train performs better than P>M>F, using full ImageNet to train DINO can bring further improvement.
(2) With the help of more data, our eTT overpasses P>M>F on ILSVRC and MSCOCO. (3) While more data does improve the results on Omniglot and TralficSign, the final results are still worse than those of P>M>F, which we think may be correlated with the limitations of our method as analyzed above. Given all these results, as a lighter model in that no meta-training phase is utilized and only few paramoters are engaged in the test-time tuning, our method can still enjoy comparable performance with P>M>F when training on same amount of data.
A.2.3 Influence of DINO
The DINO pretrain procedure is an important part of our method. To verify the effectiveness of DINO pretrain so that the comparison with other methods is fair enough, we present in Tab. S the results of TSA using DINO-pretrained ResNet-34 and eTT using supervised pretrained ViT-s. We can find that (1) The effect of DINO is not consistent on two backbones. While DINO benifits our eTT with about 5% accuracy on average, it severely weakens the performance of TSA with a large margin. It means that for FSI., different. backbones require different pretrain strategy respectively. (2) While supervised pretrained ViT-s performs worse most datasets, it is better on CUB and COCO, which indicates learning label information from ImageNet can help the model understanding novel knowledge from these two datasets.
A.2.4 Verification of potential overfitting in finetuning
As we have stated in Sec. D finetuning the whole backbone model with few support data will meet potential overfitting problem. To revcal if such problem exists in Meta-Dataset, we conduct an experiment as follow: during a normal lesting phase, we select all episodes whose minimum shot (minimum number of support images for each class) is no larger than 2 (extremely small number of labelled instances). and compare the average accuracies of eTT and simple finetuning based on these episodes. Tab. g and Tab, I0 show the accuracies on support sets and query sets respectively. We can find that most of the datasets both methods receive nearly 100% accuracy, which means these two methods can well learn the support data. Given this fact, finetuning is much worse than eT in terms of query accuracies. The overfitting can be reflected given high training accuracies and worse testing performance, and to some extent our proposed eTT can fix this problem.
A.2.5 More Visualization
We visualize the self-attention map from models with and without DRA on ILSVRC and TraffignSign in Fig. 5] Specifically, we randomly sample an episode from each datasct and use our eTT to tune the model based on the support samples. Then we calculate the self-allention map of the last. layer's class loken and highlight the areas with top 20% attention scores. We can find that for the in-domain ILSVRC episode, the model can attend to similar regions no matter whether DRA is used. In contrast, the model without DRA can easily attend to background regions with less valuable information, which reveals a potential reason that these two models has similar accuracies on ILSVRC but large performance gap on TrafficSign.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Futhermore, visualize more episodes from Aircraft, TrafficSign and MSCOCO in Fig. Fig. and Fig. which shows that our porposed eTT can remarkably improve the embedding space after lest-time finetuning.
A.3 Broader Impact
Our paper presents more efficient and practical FSL pipeline utilizing VIT. We hope this work can shed light the broader usage of ViT in FSL tasks. On the other hand, the proposed method can provide researchers with alternative choice for FSL applications in real-case scenarios with large-scale meta-train set and challenging various test episodes.
dpros
ILSVRC
Omni
Acraft
CUB
DTD
QDraw
Fungi
Flower
Sign
COCO
Avg
0.01
67.01
76.56
78.34
85.53
86.96
70.03
61.20
96.17
85.00
62.67
76.95
0.05
66.49
77.40
78.92
85.80
87.54
70.23
60.78
96.28
84.95
62.38
77.08
0.5
66.88
77.73
78.65
86.00
87.15
70.48
61.64
96.23
84.39
63.39
77.25
0.9
67.03
76.55
77.89
85.78
87.04
70.08
62.45
96.20
84.44
62.83
77.03
0.1
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
Table 5: Test accuracies on
of different variants of our proposed method. The bolded items
are the best ones with highest accuracies.
Baokbone
ILVVR
Ounii
Acrah
CUB
QDrow
Fungt
Flower
COCO
Finctune
46.751
60.861,66 06
68.091.20 26
67.311.26
09.05g
42.601
38.201,
85.510,05
66.791
54.59g.g 97
10.2
Proto
50.301
59.981.30
55.101
79:
86.56g
48.901
89, 25,270,77
47.321.10
91.091
10.5
P-MAME
Reiis
34.091.01
49.63] 05
63.37)
45.301.30 36
65.050.90
s0
68.099.90
19.511.05
52.07g. .9
66.49g
51.62)
39.961.11
50
87.156.00
08.766.85
33.071.00 0.05
48.831.00
48.741.15 12
01
0.2
BOBB
51.921
67
84.120.90
70.690.90 90
08.34g.77
50.331
41.381.12
87.390.0
b1.8911
46.03g
8.2
10
20
72:201.00
74.90p.99
17.30p 70
60g
20.301 .00
90.900
62.50p 68
.o0
Oure
V
58.401
72.322,30 36
72.84;.01
7S.79
67 67.979 TT.570.84
51.23
95.3300 57
84.00M.07
55.65
4.1
Proto
58.701 07
68.00l.00
74.190.02
06.50g.77 77
54,301
40.701.15
78
68.
08
CTX
Rea4
22.70g.000 99
82.21
00
7M 1w.496..0 69
80.630..5
15.57p.64
T2.88g
51.581
95.390
57
82.6500.7 71
59.500 022
2.8
TWA
53.73g
82.581
80. 80.18:.00
63. 63.39p.40
76.61p
91 63g
51 31.381.17
94.050,
61.71p.9t
61.67p
2.5
w7 78.11,.22
79.91,.00
85.930.91
87.022.57
81.34.87
63.89;
65.0%p.9m
Table 6:
Test accuracies, confidence interval and average rank on
Note that different
backbones are adopted by these mothods. The bolded items are the best ones with highest accuracios.
Model
Train Set:
ILSVRC
Omni
Acraft
CUB
DTD
QDraw
Fungi
Flower
Sign
COCO
Avg
eTT
teta-train
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
eTT
full
74.76
78.73
80.10
86.99
87.72
71.20
61.95
96.66
85.83
64.25
78.82
P>M>F
full
74.69
80.68
76.78
85.04
86.63
71.25
54.78
94.57
88.33
62.57
77.53
Table Test accuracies on Meta-Dataset of different variants of our proposed method. The bolded items are the best ones with highest accuracies. The highlighted rows denote the final model in our main paper.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Model
Pretrain
ILSVRC
Ommi
Acraft
CUB
DTD
Qraaw
Fungi
Plower
Sign
COCO
Avg
TSA
Sup.
59.50
78.20
72.20
74.90
77.30
67.60
44.70
90.90
82.50
59.00
70.68
TSA
DINO
48.18
64.94
56.74
45,49
69.06
59.61
31.13
81.01
48.70
26.18
53.09
eTT
Sup.
65.17
67.47
73.30
87.71
84.50
67.46
55.51
92.55
64.08
63.68
72.14
eTT
DINO
67.37
78.11
79.94
85.93
87.62
71.34
61.80
96.57
85.09
62.33
77.61
Table 8: Test accuracies on Meta-Dataset of different variants of our proposed method. The bolded items are the best ones with highest accuracies. The highlighted rows denote the final model our main paper.
Model
ILSVRC
Omni
Acraft
CUB
DTD
QDraw
Fungi
Flower
Sign
COCO
Avg
eTT
100.00
99.99
100.00
100.00
100.00
100.00
99.79
100.00
100.00
99.15
99.89
FT
100.00
99.87
100.00
100.00
100.00
100.00
96.95
100.00
100.00
95.20
99.20
Table 9:
Support set accuracies cTT and Finetune on testing episodes whose minimum shots no larger
than
Model
ILSVRC
Omni
Acraft
CUB
DTD
Qrraw
Fungi
Flower
Sign
COCO
Avg
FT
29.19
54.54
35.10
41.54
53.66
43.37
38.53
76.76
72.90
41.21
48.68
eTT
40.22
64.79
41.33
55.11
66.20
49.14
56.33
85.03
75.29
56.19
58.96
Table 10:
Query sot accuracies of eTT and Finetune on testing episodes whose minimum shots no larger than The bolded items are the best ones with highest accuracies.
Model
ILSVRC
Omni
Acraft
CUB
DTD
QDraw
Fungi
Flower
Sign
COCO
w/o stand
1.06
1.25
1.05
0.89
0.61
0.92
1.05
0.38
0.96
0.96
stand
0.97
1.22
1.06
0.91
0.57
0.87
1.06
0.46
0.90
0.99
Table 11:
The corresponding confidence intervals of models in ablation study on standardization.
18

---

Published in Transactions on Machine Learning Research (08/2022)
W DRA
W/ DRA
W DRA
w/o DRA
E
+
60
STOP
Figure 5: Visualization of self-attention from model with and without DRA on ILSVRC (left) and TrafficSign
(right). The white regions are those with top 20% attention scores
19

---

Published in Transactions on Machine Learning Research (08/2022)
Before finetune
LT+NCC
Ours
Before finetune
LT+NCC
Ours
Before finetune
LT+NCC
Ours
Before finetune
LT+NCC
Ours
Figure 6: More visualization of feature embeddings from a randomly sampled episode of TrafficSign.
20

---

Published in Transactions on Machine Learning Research (08/2022)
Before finetune
LT+NCC
Ours
Before finetune
LTCCCC
Ours
Before finetune
NCC
Ours
Before finetune
LT+NCC
Ours
Figure 7: More visualization of feature embeddings from a randomly sampled episode of Aircraft.
21

---

Published in Transactions on Machine Learning Research (08/2022)
Before finetune
LT+NCC
Before finetune
LT+NCC
Ours
Before finetune
NCC
Ours
Before finetune
LT+NCC
Ours
Figure 8: More visualization of feature embeddings from randomly sampled episode of MSCOCO.
22


Reviews:
[{"review_id": "uUcSGg1U5Q", "reviewer": "Reviewer_3uwk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. This work proposed an APT module that includes a well-designed initialization strategy for FSL.\n2. It proposed an effective DRA module with only two learnable offset vectors for each transformer layer, which is low-cost and efficient.\n3. It proposed a novel prototypical regularization for prefix distillation.\n\nWeaknesses:\n1. The proposed APT is based on the PT (P-tuning), which can thus hardly be considered an innovation module. Please highlight only the novel part, i.e., the initialization of the prefix, and compare it to using other initialization strategies (including naive methods such as the random initialization that has been used). \n2. The illustration in Figure 1b is not easy to understand, even with the corresponding texts given in the Introduction section. \n3. The quantitative comparison in Table 2 is confusing, is it highlighting the higher performance? In the column of ILSVRC, the 74.69 in row P>M>F* is higher than 67.37 in row Ours, while the 67.37 is highlighted. Similar confusion cases appear in Table 3.\n3. It is more intuitive if add an average performance comparison in the quantitative evaluation. \n4. Other details: In eq. (4), the m =1, …, P^2, while it is denoted in the line above eq. (4) as {xxx}_{m=1}^P, P or P^2, please keep consistent.\n5. For the references, as the paper's main contribution is a finetuning method for FSL, it is helpful to cite and compare it to these two works: Meta-Learning with Latent Embedding Optimization (LEO, ICLR 2019); and Meta Transfer Learning for Few-shot Learning (MTL, CVPR 2019) which is also based on learning adaptive (scaling and shifting) weights for few-shot tasks while fixing pre-trained parameters. These methods used ResNet-12 (the most popular arch). It would be helpful if the proposed method can show more effectiveness than this popular architecture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a finetuning method for FSL, namely eTT, with two key components APT and DRA. By using eTT, finetuned ViT achieved consistently good performances on meta datasets, compared to ResNet-18 and -34 models.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Need to solve the \"Weaknesses\", including the novelty, detail, and reference related issues.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t8VXlkeF9r", "reviewer": "Reviewer_EfBc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n\n1. The idea of only fine-tuning the prefix parameters is interesting, and optimized to train.\n2. The performance of the method overall is quite strong, and the ablations further provide usefulness of each module.\n3. The problem setting and the proposed modules have been well-explained. \n\n- Weaknesses\n\n1. While the idea of attaching prefixes to the transformer has been explained and ablated well, the DRA module analysis is lacking. It is claimed that the DRA module is used to bridge the domain shifts between base and few shot classes which have considerable domain gap. This is not clear from the experiments Table 3, and it is not clear that the benefits of the proposed DRA are especially observable in cases of domain shift.\n\n2. Pretraining on DINO is a choice that is not present in prior works, but only present in this work. So I am curious to know how much of performance improvement is due to this pretraining strategy? From Fig 3, LT+NCC also benefits heavily from DINO pre-training, so what would be the performance of methods in Table 2 like TSA with this pre-training?\n\n3. It is also argued that PMF uses a lot more data, but can you also perform a like-to-like comparison by adding that data to your model as well? When presented with equal data, does your method do better or still worse than PMF?\n\n4. In general, it is easier for readers to have an additional column in Tables 2 and 3 stating the average accuracy along with rank.\n\n5. The improvements provided by prototypical initialization seems to be marginal. Rando already seems to be a competitive baseline, and avg is almost equal to the heuristic proposed on most datasets.\n\n6. You mention that you set the number of prototypes is equal to N, but if I understand correctly, isn't N sampled uniformly from [5, N_max] as stated in sec 3.1?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a transformer architecture for few-shot adaptation. They first train the transformer backbone on self-supervised data using DINO, and then fine-tune the architecture using the provided base-classes. However, instead of fine-tuning the entire network, they only place \"prefixes\" to each transformer layer before key and query modules and only fine-tune corresponding parameters, which are light-weight and small. They also design domain residual adapter and prototype regularization to improve training. Prefixes are initialized using class-conditioned embedding of the samples. Experiments show competitive results on Meta-dataset.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Add more analysis on the domain residual adapter.\n\n2. If possible, comparisons with prior works also using DINO SSL.\n\n3. Comparison with PMF under equal amounts of data.", "broader_impact_concerns": "I do not see any significant ethical concerns specific to this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QiXbol380i", "reviewer": "Reviewer_hyDZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n========\n[+] Exploring adaptation algorithms for few-shot learning with ViT backbones is interesting given the recent popularity of this architecture, and relatively underexplored\n\n[+] The proposed model with ViT-small outperforms the ResNet34 variant approaches (ResNet34 has roughly the same number of parameters as ViT-small) on 7 out of 10 datasets\n\n[+] The authors perform a thorough ablation study, and I also like the fact that they report sensitivity to hyperparameters (Table 4).\n\nWeaknesses\n==========\n[-] While for the most part the paper is well-written, there are some clarity issues, one of which is particularly severe (see detailed comments below)\n\n[-] Some related work is missing\n\n[-] The proposed method does not improve upon previous works on all datasets. More research is needed to understand those limitations and tradeoffs.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an approach for adapting a pretrained ViT model for few-shot learning tasks. In contrast to prior work, their finetuning approach has a small number of trainable parameters. \n\nIn more detail, they introduce a learnable “prefix” (drawing inspiration from prefix tuning in NLP). In their case, this “prefix” is a matrix containing N_p d-dimensional vectors, where d is the dimensionality of the transformer. Then, a learnable layer maps those to a higher dimensionality, in order to yield N_p d-dimensional vectors for each key and each value matrix of each layer which are concatenated with the usual keys and values of that layer. This structure allows the task-specific information (captured in the prefix, which is learned for each new task) to affect all layers of the network. In practice N_p is set to N which is the number of classes in the given task, and the d-dimensional vectors are initialized in a specific way, from what they refer to as “attentive prototypes”. Specifically, the attention mask is computed between the class token (which contains global class information) and the final-layer patch embeddings from the transformer, for each support set image. This attention mask is then applied over image-level representations. A prototype per class is computed from these masked image-level representations by averaging over representations of (masked patches of images of) each class. The concatenation of these prototypes is the initialization to the prefix matrix. During training on each task, this prefix is learned, but they include a regularizer that incentivizes not to move too much from that initialization. Finally, they introduce a module called Domain Residual Adapter (DRA) which is comprised of learnable offset parameters for the MSA and FFN sub-layers of each transformer layer. This allows more flexibility to change the backbone, which they hypothesize is useful for tasks that are substantially different from the pretraining data.\n\nEmpirically, they use a pretrained DINO model with ViT-tiny and -small, and show improved results over baselines on Meta-Dataset (setting of training only on ImageNet). They also run a number of ablations and analyses to verify the usefulness of the different components of their method.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "# Critical changes\n\n## Add missing related work\n\nThe authors erroneously claim that ViTs haven’t been used for few-shot learning, citing a single exception. A couple other papers that have used them are [1] and [2] (see References at the bottom). \n\nThe “Remark” paragraph in section 3.5 would be more appropriate to move into the related work section. Also, there are some related works missing from that context too. CNAPs [4], and FLUTE [5] also use a form a adapters (FiLM conditioning) and the latter is especially related as there is no meta-learning of those adapters and instead scaling and shifting coefficients are learned to account for domain discrepancy, like in DRA.\n\nSection A.1 (in the Appendix) states that a flexible finetuning algorithm that adapts test-time finetuning based on number of shots and ways has not been explored. In fact, though, it has been explored. [6]\n\n\n## Address clarity issues (ordered in decreasing importance, roughly)\n\nThe authors claim that their approach follows the pretraining - meta-training - finetuning pipeline (e.g. in the introduction they say “In this paper, following the pretrain-meta-train-finetune pipeline [...]” but nowhere in the paper is a meta-training phase mentioned. Their entire model section is dedicated to the finetuning part. There could be a meta-training phase where episodes are used and the same finetuning procedure is applied in each episode, and perhaps some of the learnable parameters can be trained in this fashion, but the authors don’t mention anything to that effect. This was a serious clarity issue for me as I kept wondering what was meta-learned while reading the paper, and it’s still not clear to me (I assume there is no meta-learning).\n\nFrom the experimental section, I understand that the authors re-trained DINO, to ensure it is only trained on the 712 classes of ImageNet that are in Meta-Datet’s meta-training set (which is great!). I would update the text accordingly, as phrases like “off-the-shelf” (e.g. in the Pre-training paragraph on page 5) make it sound like an already-trained one was used which would make for unfair comparisons.\n\nWhat is “normalized cosine similarity” ? Cosine similarity already involves some normalization, is some additional normalization going on here?\n\nIn Equation 5, shouldn’t the upper limit of the inner sum (the one iterating over m) be P^2 instead of P since according to the notation used there should be P^2 patches?\n\nIn section 3.6, the authors state “[...] the model parameters, which include the adapter, the prefix together with the transformation module and the projector”. Not clear what is meant by “adapter” here. I’m assuming it’s referring to the learnable vectors of DRA, but should be more precise, as DRA had not been referred to as the “adapter” before. Also not clear what is the “transformation module”. Is this referring to module g? Should use consistent names to refer to modules throughout the paper.\n\nThe authors say in the text that Figure 4 shows an Omniglot episode, but the caption of Figure 4 says the episode is from Traffic Signs. Please correct.\n\nIn section 3.6, the authors state “we further standardize x’ with a centering statistics” - are the EMAs used for this calculated anew using the support set of each episode? Please clarify.\n\nThe approach described in section 3.3 should not be referred to as finetuning, since no parameters are finetuned. The backbone is frozen and the new transformation \\phi is learned from scratch (not finetuned). The same is true of the proposed ‘task finetuning’ which again should not be called finetuning (the prefix parameters and g are trained from scratch). Task adaptation is a more general and appropriate term.\n\n\n## Address minor soundness issues\n“Given extremely small number of labelled images, the strategy of finetuning all parameters will be prone to overiffing the data”. Is there some empirical evidence that the authors had in mind here? If so, please include a citation (or otherwise weaken this claim / re-think the motivation for this). In practice it turns out that finetuning is powerful and less prone to overfitting than originally thought. The fact that naive finetuning is SotA on difficult benchmarks like [3] is a testament to this.\n\n“DINO builds up supervision based on a self-distillation framework. [...] Accordingly, such a pre-trained model shall have good cluster property even among cross domain images” - why is this the case? I understand the motivation behind self-supervised learning producing rich features but I’m not aware of any guarantee for generalizing well to (all) new domains? Please add a citation or revise.\n\nThe authors don’t say how validation was done for hyperparameters. Table 4 shows the results of different hyperparameters on the *test* set, but how did the authors pick the hyperparameters with which they computed the main results for their model? This is particularly relevant and important to comment on for generalization to new datasets and transparency here is important.\n\n## Additional ablations\n\nIn section 3.6, the authors state “we further standardize x’ with a centering statistics” - it would be nice to also have an ablation that removes this component. \n\nDespite the thorough ablations in this paper, there is still a confounding factor between the proposed method and a lot of the previous baselines: pre-training is done in a self-supervised manner here using DINO whereas traditionally this is done supervisedly (with a few exceptions, e.g. CTX uses a mixture of both, jointly). It is unclear how big of a role this difference plays to the success of this method. Ideally, the authors would re-run some of the previous methods starting from a self-supervised trained backbone, even for resnets. But perhaps the easiest way to bridge the gap is to train ViT-t/-s in a *supervised* only way (this is different from their experiments where supervised finetuning was attempted). This will be a useful datapoint to report and can give more insights into comparisons with previous work.\n\n\nNon-critical changes that would strengthen the paper\n========================================\n\nThough I really appreciate the authors re-training DINO on the meta-training set of ImageNet only for the main results, an additional experiment that shouldn’t be too hard to perform is using the pre-trained DINO off-the-shelf (trained on the entire ImageNet), in order to get apples-to-apples comparisons with P>M>F which is the most closely-related out of the methods in Table 2, since it also uses ViT-s. This would allow to better gauge the merit of the proposed adaptation approach in a more meaningful way.\n\nIn the set of experiments for the question: “Do we need a more complex adapter structure?” - another variant to explore here is learning both additive and multiplicative parameters (instead of additive ones only like the proposed method). This would be more akin to FiLM conditioning which learns an affine transformation.\n\nThe combined Meta-Dataset+VTAB benchmark [3] is arguably a more appropriate choice for evaluating this work. One reason for this is that the entire ImageNet is fair game for meta-training in that case, so different off-the-shelf models can actually be used. Further, it opens the door to comparisons against larger transfer learning models that were evaluated there, which is very relevant for this research direction.\n\n\nSome typos and wording suggestions (there are more, please proof read the paper)\n====================================================================\n‘predominate’ → ‘predominant’ (in the introduction)\n\n‘much less model parameters’ → ‘much fewer model parameters’ (in several places)\n\n‘fineuning’ → ‘finetuning’ (in related work)\n\n\n‘based data’ → ‘base data?\n\n‘analogous information’ → ‘consistent information’ (in section 3.6)\n\n\nReferences\n==========\n[1] Exploring the Limits of Large Scale Pre-training. Abnar et al.\n\n[2] Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning. Evci et al.\n\n[3] Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark. Dumoulin et al.\n\n[4] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al.\n\n[5] Learning a Universal Template for Few-shot Dataset Generalization. Triantafillou et al.\n\n[6] Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks\n", "broader_impact_concerns": "I don’t have any specific concerns on the ethical implications of this work. \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was overall positively received by reviewers and received two "Learning Accept" and one "Accept" recommendation after the author response. The authors addressed most smaller and larger concerns of the reviewers in the revised manuscript, including clarifications and additional ablations and results.

Overall I recommend Accept with minor revision as the paper clearly validates experimentally its claims:
* Proposes novel and efficient model for few-shot on Vision Transformer (ViT)
* Provides solid experimental evaluation and good performance for the proposed model
* Presents clear ablation study

## Required changes for minor revision
* Please make sure that all promised changes in the author response are incorporated in the final version.
* Fix claim in abstract: The author acknowledged missing citations and included them in the main part of the paper “The general pipeline of Pretrain-(Meta- train)-Finetune has been explored in few ViTs on FSL (Hu et al., 2022; Evci et al., 2022; Abnar et al., 2021), recently.”, but missed to fix the abstract “with the only exception (Hu et al., 2022).”, which is now in contradiction.
* The author refer to (Hu et al., 2022) as “preliminary study” (page 5), which seems misleading given it is [now] a CVPR 2022 paper [this might not have been clear to the authors at submission time], and it should also be cited as such. Furthermore, the authors should check if other arxiv papers are published papers.
* Please use consistent bolding in tables and note it in the table caption, e.g. clarify why the top of table 6 is not bolded in the caption.
* Table 6: It is confusing why the numbers for the line “DINO meta-train” and “VIT-s+DINO” have identical numbers, but different names in the model column. Please revise and/or clarify in the caption.
* I am missing the fix of reviewer EfBc’s comment “You mention that you set the number of prototypes is equal to N, but if I understand correctly, isn't N sampled uniformly from [5, N_max] as stated in sec 3.1?”, the revised manuscript still states “N is first uniformly sampled from [5, Nmax] … on training or testing stage”. Please revise/clarify.
Decision: Accept (poster)