Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
Authors: Cheng Cheng
Lin Song
Ruoyi Xue
Hang Wang
Hongbin Sun
Yixiao Ge
Ying Shan
Venue: NeurIPS 2023 poster (2023)
Keywords: Few-shot Learning; Vision-Language Model Adaption

Abstract: The contrastive vision-language pre-training, known as CLIP, demonstrates remarkable potential in perceiving open-world visual concepts, enabling effective zero-shot image recognition.
  Nevertheless, few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters on few-shot samples, resulting in longer inference time and the risk of overfitting in certain domains.
  To tackle these challenges, we propose the Meta-Adapter, a lightweight residual-style adapter, to refine the CLIP features guided by the few-shot samples in an online manner.
  With a few training samples, our method can enable effective few-shot learning capabilities and generalize to unseen data or tasks without additional fine-tuning, achieving competitive performance and high efficiency.
  Without bells and whistles, our approach outperforms the state-of-the-art online few-shot learning method by an average of 3.6\% on eight image classification datasets with higher inference speed.
  Furthermore, our model is simple and flexible, serving as a plug-and-play module directly applicable to downstream tasks.
  Without further fine-tuning, Meta-Adapter obtains notable performance improvements in open-vocabulary object detection and segmentation tasks.

Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
Cheng Cheng' Lin Song"* Ruoyi Xue
Hang Wang'
Hongbin Sun it Yixiao Ge
Ying Shan
Xi'an an JinoTong University Tencent Al Lab
cheng2016 xjt..edu.unn ronnysong www.oontteeennnnccommmmmm
Abstract
The contrastive vision-language pre-training, known as CLIP, demonstrates re- markable potential in perceiving open-world visual concepts, enabling effective zero-shot image recognition. Nevertheless, few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters few-shot samples, resulting in longer inference time and the risk To tackle these challenges, style adapter, to refine the CLIP features guided by the few-shot samples in an online manner. With few training samples, our method can enable effective few-shot learning capabilities and generalize to unseen data or tasks without addi- tional fine-tuning, achieving competitive performance and high efficiency. Without bells and whistles, our approach outperforms the state-of-the- online few-shot learning method by an average of 3.6% eight image classification datasets with higher inference speed. Furthermore, our model is simple and flexible, serv- ing as plug-and- and-play module directly applicable to downstream tasks. Without further fine-tuning Meta-Adapter obtains notable performance improvements in open-vocabulary object detection and segmentation tasks.
weer-fitting in certain domains. propose the Meta- Meta-Adapter, lightweight residual-
Introduction
The contrastive vision-language pre-training 115), known CLIP [6]. has shown the impressive potential modeling open-world visual concepts 77177. which benefits multiple vision tasks including image recognition and open-vocabulary perception [0] It can be mainly attributed to the large-scale datasets [2] and the advanced pre-learning techniques 18] By constructing prompts based visual categories, CL.IP shows effective zero-shot image classification capabilities and generalization abilities for unseen data [19]. Recently, few-shot learning based on CLIP has garnered increasing research attention. Motivated by the success feature adapters 20 and prompt tuning are
21 for natural language processing, a wide range of few-shot approaches for CLIP 22
proposed and studied.
Few-shot learning methods for CLIP can be categorized into offline 22 g and online approaches 77. according to whether the
is required for the few-shot samples of unseen categories. Offline methods extract the knowledge from few-shot samples via parameter optimization. Notable examples include CoOp [ and CoCoOp (22), which replace the hand-crafted templates in CLIP with learnable continuous tokens by fine-tuning on few-shot samples. Additionally, CIIP-Adapter [ introduces feature adapters into CLIP by learning task-specific knowledge from few-shot samples. Although the extra components yield promising few-shot learning capabilities, they also incur
'quual contribution. Corresppnning author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Iragshet
Inference Time for Meta
40
EursSAT
OofordPats
Inference Tip
30
Accuracy/NS
38
200
36
Zero-shor CLIP
10
Tip-Adapter
chr
Meta-Adapter
Tip Aoppte
10
20
30
Number of image shots
(a) Top-1 accuracy with 16 shots.
(b) Top-1 accuracy on ImageNet
29 Novel.
Figure I: Comparison of different few-shot learning techniques. The models are trained on the set of base classes and evaluated on the novel classes. The time is measured on a Tesla V100 GPU.
additional training costs and suffer from considerable over-fitting in a certain data distribution [22 233. To climinate training expenses, online method called Tip-Adapter 7 is proposed. This method proposes a hand-crafted modulation function that adjusts the ratio between category embeddings and few-shot visual embeddings. It obtains the knowledge from few-shot samples without fine-tuning and shows a notable improvement against the zero-shot manner. However, due to its sophisticated hyper-parameter search scheme, we find that Tip-Adapter still tends over-fit in the distribution of seen data (details referring to Sec. [) resulting in limited generalization capability. Different from previous methods, we attempt to explore new perceptive: learning an online few-shot learner for CLIP via meta-learning.
To achieve it, propose the Meta-Adapter that replaces the hand-crafted modulation function and searching scheme in Tip-Adapter with lightweight residual-style network. The offline few-shot learning methods require additional fine-tuning for few-shot samples from unseen categories. In contrast, our approach employs the meta-testing mechanism (24 thereby the categories of training and testing data of our model can be different. By using limited number of few-shot data, the Meta-Adapter can be trained to enable few-shot learning capability. Without additional fine-tuning, it can further generalize other unseen data and extract knowledge from the few-shot samples in an online manner. To achieve high efficiency, the Meta-Adapter is constructed by lightweight network based on the gated multi-heaa attention mechanism (28). which bridges the gap between few-shot image features and textual features for each category. This procedure can be considered as a learnable filter to refine the category embeddings guided by the few-shot images. Since the Meta-Adapter does not require additional fine-tuning, it only has a slight computational overhead over the zero manner. Compared with the Tip-Adapter, it alleviates the over-fitting problem and demonstrates superior generalization across datasets. Furthermore, the Meta-Adapter is methods, making it
simple and can be applied plug-and and-play module to various versatile solution for many open- vocebulayyy downstream tasks.
The extensive experiments demonstrate the effectiveness and efficiency of the Meta-Adapter image classification, object detection, and segmentation. To verify the generalizability Meta-Adapter, we conduct series of ablation studies, including cross-category generalization within certain dataset, cross-de cross-dataset generalization. and cross-task generalization which explores the potential of Meta-Adapter in downstream tasks. As shown in Figure a). by training on the data of base classes, the Meta-Adapte achieves an average of 3.6% absolute gains over Tip-Adapter across the novel classes of eight image classification datasets under the 16-shot setting. With larger number of image shots, our method achieves increased performance gain over the Tip-Adapter, which is illustrated in Figure (bb). Besides, through directly evaluating the ImageNet (29 pre-trained model on other seven classification datasets, our method obtains an average of 4.9% improvements against Tip-Adapter. In addition, Meta-Adee Adaptrr shows the potential to improve other tasks, such as open-vocabulary object detection, which leads consistent improvements in both object detection and instance segmentation.

---

By integrating the ImageNet-pretraing Meta-Adapter with the framework, ViLD (T], our method achieves 1.0% absolute gains on the average precision of rare categories AP, without bells and whistles.
open- open-vocabulary object detection
2 Related Work
2.1 Vision-Language Pretrained Models
Inspired by the success of pre-trained models in the field of CV and NLP, many works are proposed to pre-train large-scale models to process both vision and language modalities. A typical vision-language model consists of four key components, i.e., vision encoder, language encoder, fusion encoder, and
loss function. Recently, following the success of the base models both CV and NLP 30
6
36],
the community of multi-modal learning can take advantage of these large-scale base models to better elevate the performance. VisualBERT 37). OSCAR [38. Uniter [39] utilize BERT 30 to preprocess the raw text and demonstrate impressive results in multimodal tasks, c.g., visual question answering (VQA). Besides, these methods require a well-designed fusion encoder to integrate the cross-modal interaction. Recently, CLIP 22, DeCLIP [40] and ALIGN [] demonstrate that vision- language contrastive learning is capable of generating transferable features to downstream tasks and the multimodal interaction can be well interpreted by simply calculating the dot product between vision and language embeddings. Without additional self-attention or cross-attention modules, the multimodal embeddings can be pre-computed and stored, which is more efficient and can be easily adapted to other tasks.
2.2 Vision- Model Adaption
Many recent works focus on exploring effective and efficient approaches for adapting lisgun-language
15 43 which are prompt-tuning methods, and feature adapters methods, e.g., Tip-Adapter 77 [3]. Inspired by the success of prompt learning 21 44),
models to downstream tasks [8 22 7 9
41
42
14
[3
e.g., Context- Optimiittttiiiiiiionn (CoOp)
CoOp proposes to replace the hand-crafi templates (2] with continuous tokens that can be optimized in traditional fine-tuning fashion. Besides, to mitigate the woeful over-fitting issue, CoCoOp further introduces integrate image-specific tokens learned by a shallow MLP. Compared with the manually designed prompt templates, CoOp and CoCoOp achieve impressive performance on few-shot image classification. Different from these prompt tuning methods, CLIP-Adapter and Tip- Aip-Adapter conduct residual feature blending to integrate few-shot knowledge with CLIP's zero-shot knowledge. They keep the whole CLIP's parameters frozen and fine-tune an acceptable small number of additional weights and show impressive results in few-shot image classification. Besides, by initializing the linear weights with few-shot knowledge (i.c., cache model in this context). Tip- Tip-Adapter can further pose a training- -free free fashion with preferable performance. Nevertheless, these methods suffer from over-fitting, especially when the domain gap between the source and target datasets large.
2.3 Meta-Learning
simple interpretation of meta-learning 24 is "learning-to-leam" 45 which corresponds to improving the generalization by searching for the algorithm (inductive bias) that is best suited for a given task family. On the contrary, traditional machine learning algorithms [46 47 are expected to be improved as more data from certain single task. Usually, meta-learning is conducted on learning instances sampled from a task family, which is expected to simulate base learning algorithm that performs well on new tasks sampled from this family. Besides, as mentioned in [24]. all training
instances can be sampled from a single task in a special case. In the context of adapting the vision- language model to downstream tasks, meta- meta-learmig can be viewed as learning the general fine-tuning algorithms which bring consistent gains over different tasks or datasets. Current methods 22 g focus mainly on improving the performance of certain tasks or datasets. To the best of our knowledge, this paper is the first one that studies the potential of meta-learning in the field of vision-language model adaption.
3

---

Table 1: Comparison of cross-datasel generalization based on ImageNet [29] pre-training- The Tip-Adapter
and Meta-Adapter tuned ImageNet and frozen for other datasets. A reflects the generalization ability across datasets.
Method
FGVC
OxfordPets
SUN397
UCF101
Caltech101
DTD
EuroSAT
Avg.
A
Zero-shot CLIP
0.42
56.25
28.96
21.05
60.62
10.00
4.17
25.92
Tip-Adapter*
13.96
68.75
45.16
40.09
68.33
42.92
56.25
47.92
Tip-Adapter 13.96
67.19
43.80
39.47
67.08
40.00
56.25
46.82
Meta- eeta-ed
19.58
72.66
51.25
52.28
71,46
49.17
64.58
54.43
+6.51
Meta- Meta-er
15.21
72.66
48.54
47.54
67.92
48.33
62.50
51.81
+4.99
indicates searching hyper-parameter training for each evaluation dataset individually.
Method
In this section, introduce the proposed Meta-Adapter. In Section 3.1 we first revisit CLIP and Tip-Adapter. In (3.2 we elaborate on the implementation of the proposed Meta-Adapter. In discuss the difference with other related works.
Section 3.3
3.1 Revisiting CLIP and Tip-Adapter
As vision-language pre-training model, CLIP has shown impressive zero-shot learning potential in modeling open-world visual representation 10 by exploiting contrastive learning with large- scale noisy image-text pairs. To achieve zero-shot image classification, CLIP computes classification scores by measuring the cosine distance between the image features and per-class textual features. Specifically, given an image y, let f Dx be the feature of the query image and [ W/ E RDx1 be a set of category embeddings generated by the text encoder. The indicates the dimension of embedding space and N denotes the number of total categories. The textual feature w,, for each class derived from hand-crafted templates that one of typical form is "a photo of [CLASS]". The class token is then replaced by specific category name, such as "Alp" or "Lemon", as shown in Figure 2 The predicted logits of the given image y belonging to the i-th class can be formulated as:
w
logits(ye = =) = Wrr |
Tip-edapter 7 further proposes an online method learn knowledge from few-shot samples. This method employs a straightforward modulation function with a stochastic hyper-parameter search strategy, achieving impressive few-shot performance for a certain domain. Concretely, two hyper- parameters, i.c., and B, are introduced to adjust the ratio between visual and textual features for different datasets. Given a set of support images x = {x,) in N ways and shots, the predicted logits of Tip-Adaptes can be formulated as
FJS |E II 1),
W
logits(ye = i|x, a, 3)
a expe
Mt |IS
where F, E RD is support embeddings of few-shot samples, and L; R NxK is the corresponding one-hot labels of i-th class. Tip-Adapter obtains remarkable few-shot performance without additional training for sho- samples. However, Tip-Adapter relies heavily the typer-parameter search strategy on the target dataset, rendering it susceptible to over-fitting within a certain data distribution and limiting its distrf-disttibution generalization capabilities. As presented in Table T we fix the hyper-pan prpee-pprameters searched on ImageNet (29) and directly evaluate the performance of Tip-Adapter on seven other datasets. It demonstrates poor generalization performance of the Tip-Adapter across different distributions, Compared with the searching scheme for each dataset individually, the Tip-Adapter exhibits notable decline in performance.
3.2 Meta-Adapter
To address the issue of poor generalization, we propose a learnable Meta-Adapter to replace the handcraft modulation function and searching strategy in Tip-Adapter. Different from the Tip- Adapter,

---

Support Images
Categories
Meto-AAaaper
Alp, Lemon, ",, Viaduct
F
[CLS]
photo
the Alp
Visual Encoder
Textual
Q
K
Encoder
Multi-head Attention
F
Meta-Adapter
@
Support Embeddings
Category Embeddings
Gate
Visual Encoder
Softmax
| FlI
Alp Lemon Dog
Vicstber
Query Imoge
MLP
Diagram of the proposed Meta-Adapte which employs learnable network to refine the category embeddings guided by few-shot images.
Figure 2:
we model the few-shot learning in a visual- -anguage scheme learnable filter on textual features under the guidance of few-shot image samples, to obtain more discriminative category embeddings. Motivated by the non-local filters [8 50] in computer vision, we propose a Meta-Adapter based on the gated multi-head attention mechanism.
As shown in Figure through CLIP encoders, we first extract the support embeddings of input few-shot images and category embedding. The Meta-Adapter then extracts and transfers the few- shot knowledge from visual features into textual features obtain refined category embeddings. Specifically, we apply the original category embeddings as query and the support embeddings as both key and value into a multi-head attention block. Unlike the standard transformer encoder (28). our approach only introduces Multilayer Perceptron (MLP) layers for query and key. This strategy is crucial since no feature transformations are performed on values, the zero-shot capability is generally not changed after training. The predicted logits with the proposed Meta-Adapte can be formulated as:
wi!
logits(ye- i[x)
where u MetaAdapterrw, F).
| [ISII
The w is the refined category embeddings. In the Meta-Adapter, as shown in the right-hand of Figure 2 the proposed method adaptively aggregate the support embeddings according to the affinity between categories and few-shot images. The aforementioned procedure can be implemented by cross-attention mechanism:
F F F=
where W1 and W2 indicate the weights of MLP layers. The a denotes the Softmax function, and F represents the aggregated support features. Intuitively, similar to the non-local filters, Meta-Adapter could disregard some outlier samples while paying more attention to the samples that are more related the category description
42 resulting robust feature representations.
Besides, the importance textual and visual features for few-shot learning varies across different data distributions (9] Therefore, propose learnable gating block g(-), generating modulation scalar, to adaptively control the ratio between category embeddings and aggregated support embeddings. Accordingly, the refined category embedding can be obtained by:
+ g(w) F.
(5)
where @ denotes Hadamard product. Through training on the few-shot samples, the gating block could adjust the ratio according to the category descriptions. It enables the proposed method to effectively integrated few-shot knowledge with zero-shot knowledge.
5

---

Table 2: Quantitative results of in-domain generalization setting on UCFI01, Caltech101, DTD, and FGVCAireraft datasets between Meta-Adapter and other methods.
UCF101
Caltech101
DTD
FGVCAircraft
Model
Base
Novel
Base
Novel
Base
Novel
Base
Novel
Zero-shot CLIP
79.42
21.05
93.39
60.62
59.38
10.00
23.84
0.42
Tip-Adapter
85.17
40.09
95.09
68.33
68.36
42.92
30.27
13,96
Meta-Adapter
82.44
52.28
93.39
71.46
64.26
49.17
27.32
19.58
3.3 Comparison with Counterparts
Compared the offline methods, e-g., CLIP-Adapter [9] and CoOp R our Meta-Adapter does not require additional fine-tuning for target samples, significantly reducing the computational costs during inference. In addition, compared with the online methods, e-g., Tip-Adapter (7.. the proposed technique replaces the handcrafted hyper-paramett support samples. As shown in Table [ the Meta-Adapter can better alleviates the over-fitting problem and demonstrates the generalization across datasets without further fine-tuning, Moreover, as the Meta-Adapte refines the textual embedding features directly without altering their dimensions, can naturally be applied to a variety of downstream tasks based on CLIP.
search process with learnable network on
4 Experiments
It should be noted that the distributions of training and testing sets can be identical dissimilar and it is crucial that Meta-Adapter can well perform both scenarios. Besides, the potential of Meta-Adapter in downstream tasks is also of great importance. We refer these three situations as "cross-category generalization', "'roos-dataser generalization", and "cross-task generalization" respectively. Specifically, for "cross-category generalization", we split the full categories of each dataset into base and novel sets according to the per-category accuracy predicted by Zero-shot CLIP, that is, the base set contains easy samples and the novel set contains hard samples. This dataset split strategy simulates a rather difficult situation to verify whether Meta- Adapter is able to learn the dataset-irrelevant approach, especially for hard samples. We provide details of dataset splits in the supplementary material. Before diving into experimental analysis, we first give the details of the experimental setup.
Datasets For cross-category generalization experiments, we use 8 representative image classification datasets: ImageNet [29], FGVCAircraft [51], OxfordPets [52], SUN397 (53) UCF101 [44] Cal- tech101 [55]. DTD (66, and EUroSAT (57) which cover a diverse set of classification tasks. As
for the cross-dataset generalization experiment, ImageNet is further utilized the source dataset
and its three variants are treated as target datasets, i.e., ImageNet-A (58), ImageNet-R (59), and
ImageNet-Sketch [60]. Moreover, explore the potential of Meta-Adapter on open-vocabulary detection, we conduct experiments on LVIS [6]]
Baselines We compare Meta-Adapter with two training- free methods: Zero-shot CLIP and
Tip-Adapter 7
Training Details As for the CLIP backbone, we choose ResNet50 46 as the visual encoder in most experiments and transformer 6 as the textual encoder. We adopt the prompt ensemble strategy 777 which inputs 7 templates into the CLIP textual encoder and then averages them the final prompt embeddings. We optimize the Meta-Adas Adapter on the base set with a batch size 64 and use AdamW optimizer
62 with learning rate of 0.0001 and cosine scheduler for epochs.
4.1 Cross- Generalization
Based on empirical studies, it can be observed that Tip-Adapter often requires large hyper- parameters (ar and 3) when applied to specific datasets. These hyper-parameters are used to smooth the classifica- tion distribution and give significant weight to few-shot knowledge. This phenomenon indicates that Tip- Wip-Adapter heavily relies on few-shot knowledge, even for relatively general datasets like ImageNet. As result, over-fitting issues arise, which hinder its generalization ability. Table demonstrates that

---

Table 3:
Quantitative results on ImageNet of different models utilized various vision backbones.
Model
RN50
RN101
ViT-B/32
ViT-B/16
RN50X16
RN50x64
Zero-shot CLIP
32.82
39.22
40.10
45.77
50.10
54.67
Tip-Adapter
36.51
42.42
43.71
49.84
53.08
57.99
Meta- Meta-Adapter
40.19
47.01
46.91
52.60
55.51
60.41
Tip- Adapter achieves slightly higher classification accuracy than Meta-Adapter on the training set for datasets like UCFI01 and 11 However, when it comes to novel samples, Tip-Adapter falls behind Meta-Adapter with significant gaps, such as 40.26% versus 47.72% on UCF101. This suggests that Tip-Adapter becomes excessively tailored to specific localized distribution due to its excessive pyper-parameter search strategy.
On the contrary, thanks to the ingenious and generic ensemble approach, Meta-Adapter enjoys comparable performances on the base set and superior performances on the novel set. As shown in Figure [(b) Meta-Adapter shows superior performances over other methods on the ImageNet dataset. Compared to sero-shot CLIP, Meta-Adapter consistently surpasses it on all different few- shot settings. In comparison to Tip-Adapter, the classification accuracy of both methods witnesses relatively steady improvements. When shots are less than 4, Tip-Adapter slightly outperforms Meta- Adapter. The reason is two-fold. First, Tip-Adapter directly calculates classification logits given both few-shot features and their corresponding one-ho hot labels, which can be viewed as a shortcut solution compared to the generic ensemble approach of Meta-An Meta-Adapter. Second, Tip-Adapter exploits the potential of few-shot knowledge via hyper-parameter searching strategy which intends to find out the highest accuracy Aip-Ad by a clear margin and classification accuracy goes up consistently while Tip-Adapter witnesses performance drops when shots are 32, indicating a possible performance limitation for Tip-Adapter And we present the quantitative comparison on the other 7 datasets under 16 shots setting in Figure [(a) It can be observed that Meta-Adapter significantly boosts the classification accuracy over Zero-shot CLIP and surpasses Tip-Adapter with gains of up to +7.46%. Due to space limitations, we present the comparison of Meta-Adapter and other methods on the remaining seven datasets under different few-shot settings in the supplementary material.
certain datasets. Nevertheless, shots get larger, Meta-Adapter outperforms
To further verify the effectiveness of Meta-Adapter, we apply different visual encoders for all methods and conduct experiments on ImageNet. The quantitative comparison is shown in Table 3 Undoubtedly, Meta-Adapter maintains its leading position over Tip-Adapter regardless of the choice of visual encoders. As a more advanced backbone, such as ViT-B/16 22, is utilized, the classification accuracy of Meta-Adapter further increases. This suggests that the learning potential of Meta-Adapter can be enhanced by employing a more powerful vision-language model. In summary, compared to previous training-free methods, Meta-Adapter only effectively mitigates the issue of over-fitting but also retains superior generalization ability. As result, it achieves state-of-the -hht--rr- classification accuracy on the novel set.
4.2 Cross-Dataset Generalization
is of great importance that a learned classifier maintains comparable performance when handling different datasets with diverse distributions. It is more challenging because the appearances and shapes can be totally dissimilar across datasets (e.g., from object recognition of ImageNet to textual classification of DTD). Besides, are interested know whether Meta-Adapter can learn dataset- irrelevant discriminative ensemble approach. To this end, we transfer the optimal hyper-parameters of Tip-Adapter searched on the source dataset and evaluate the performance on target datasets. And Meta-Adapter is first trained on the source dataset, i.c., base set, and evaluated on the target dataset, i.c., novel set, with all learnable parameters frozen. Both experiments are conducted under the 16-shot setting. We report the relative accuracy improvements when transferring from ImageNet (with all categories utilized) to the other datasets, as shown Figure ([a] Tip-Adapter base results are set as the baseline in Figure 3(a), thus the corresponding relative accuracy improvements are always 1.0. Considering that ImageNet contains a variety of classes, e.g., different animal breeds and different kinds of vehicles, it is not surprising that both models retain comparable accuracy compared with their counterparts. Nevertheless, Tip-Adapter poses rather clear performance drops compared with Meta-Adapll Moreover, transferring from ImageNet SUN397, UCF101, and EuroSAT
7

---

1.10
Relative Accuracy Inprovamentse
Rolative Accursyy Ingoreettteeee
1.1
1.05
1.0
1.00
0.9
Tip-Adapter base results
Tip Adapter base results
0.8
Tip-Adapter transfer results
Tip Adapter transfer results
Meta- Adapter base
Metavid apter base results
Adapter
Meta Adattt transfer results
FGVCAircraft
OrfordPets
SUN397
U0310l
Caltec1101
DTD
EuroSAT
0.90
FGVCMrcraft
OxtordPets
SUNB97
uchaal
EuroSAT
Sottcha
(a) ImageNet to Others
(b) Others to ImageNet
Figure 3: Relative accuracy improvements of Tip-Adapter and Meta-Adapter cross-dataset general- ization experiments.
Table Quantitative results of domain generalization experiments between Tip-Adapter and Meta- Adapter. The data in parentheses records the changes brought by comparing with Zero-shot CLIP.
Target Datasets
CLIP Backbone
Model
ImageNet-A
ImageNet-R
ImageNet-Sketch
zero-shot CLIP
23.88
60.54
35.45
RN50
Tip-Adapter
23.25(-0.63)
58.73(-1.81)
34.77(-0.68)
Meta-Adapter
23.71(-0.17)
59.96(-0.58)
35.54(+0.09)
Zero-shot CLIP 50.65
77.82
48.42
VIT-B/16
Tip-Adapter 49.89(-0.76)
76.94(-0.88)
48.13(-0.29)
Meta-Adapter
51.12(+0.47)
77.54(00.28)
48.76 48.76(+0.34)
results in surpassing their baselines, i.e., directly trained on the target datasets, which suggests that the learning potential of Meta- Adapter can benefit from a generalized dataset. Besides, we report the relative accuracy improvements of Tip-Adapter and Meta-Adapter by transferring from 7 small classification datasets (with all available categories utilized) to ImageNet, as shown Figure (bb) Surprisingly, Meta-Adapter maintains comparable results while Tip-Adapter poses clear performance drops, particularly on OxfordPets and Caltech 101. Thus, it can be concluded that Meta-Adapter retains better transferability against diverse distributions and domain shifts.
Furthermore, conduct domain generalization experiments as in to generalize to out-of-de of-distribution data, which raises the question of whether Meta-Adapper possesses the same advantage. To this end, we transfer Tip-Adapter's optimal hyper-paramenn Meta-Adapter's weights searched and optimized on ImageNet to its three variants, i.e., ImageNet-A [88] IeageNet-R [59], and ImageNet-Sketch (0). Besides, we report the performance of Zero-shot CLIP on these three datasets as the baseline. The quantitative results are presented in Table It is clear that directly transferring Adapterss'''s optimal hyper--prrmee parameters searched on ImageNet to its variants leads performance drops, even worse than Zero-shot CLIP. This phenomenon accords with the previous statement that Tip-Adapter is sensitive to hyper-parameter settings, or to say Tip-Adapter suffers from severe wettffttting issues. On the contrary, Meta-Adapter can better adapt to domain shifts, demonstrating comparable performances with Zero-shot CLIP.
Humans have natural ability
(a and and
4.3 Cross-Task Generalization
It is major concern whether few-shot learning methods can benefit downstream tasks. To this end, we integrate Meta-Adapter and Tip-Adapter with apen-vocabulary object detection method ViLDD m. ViLD utilizes CLIP to explore the open-vocabular potential, it replaces the typical classifier in

---

Table 5: Comparison of different few-shot methods on LVIS 61 val set.
Object Detection
Instance Segmentation
Methods
AP,
APc
APr
AP,
APc
APr
ViLD
18.1
27.3
32.0
17.4
25.5
28.5
ViLD Tip-Adapter
11.3
23.3
26.7
10.8
21.8
24.2
ViLD Meta- Metater
19.1
27.3
31.7
18.3
25.5
28.3
66 with textual features generated by CLIP's text encoder and aligns the textual features and ROI features by knowledge distillation.
object detection framework 63
47
65
17
We first generate pre-processed region features of LVIS [1] given its annotations as the few-shot samples. Following ViLD, object categories in the LVIS dataset are split into "frequent", "common", and "rare" according to their frequency in the training set. Among them, 866 frequent and common categories of LVIS are taken the base set, and 337 rare categories are held out as the novel set. For Tip-Adapter, the prediction logits of ViLD is modified by adding an additional few-shot term.
logits(f) f(F)o
((((F) L, where (p(x) exp(-3(1 -))
(6)
where f(f) represents the region features of proposal f;; ip(.) is the modulation function introduced in Tip-Adapter; F and L denotes few-shot region features and their corresponding one-hot labels. It should be noted that Equation [ applied to both ViLD-text and ViL.D-image introduced in ViLD. As for the ryper-paramete settings (a: and 3) of Tip-Adapter, we observe that setting them to be the ones that searched ImageNet will cause ViLD collapses. Thus, to I, which encourages ViLD to rely more on CLIP knowledge to avoid collapse. Besides, we utilize the ImageNet pre-trained Meta-Adapter
0.05 and
integrate the pre-processed LVIS few-shot knowledge into the original textual features. And for ViLD, we utilize the re-implement version as suggested in DetPro [0] which replaces the pre-trained ResNet-50 46 with self superviee pre-trained SoCo [70]. We report the average precision in Table From Table 5] it is clear that Meta- Alttter can boost the detection ability on rare categories by clear margin while Tip-Adapter corrupts the detection performances due to its poor transferability and its necessity to modify the original prediction score of ViLD. Besides, considering that part of the LVIS annotations contains rather small bounding boxes (usually around 10 x 10), the quality of the LVIS few-shot dataset may not be good as the image classification counterparts. As mentioned before, Tip-Adapter naturally and heavily relies on few-shot knowledge, which may be one of the main causes of poor performance in open-vocabulary object detection. On the contrary, Meta- Adapter learns the generic ensemble approach and can be easily integrated into open-vocabulary object detection methods without changing their formulation of prediction scores.
Table 6: Comparison of Zero-Shot CLIP, Tip-Adapter, and Meta- Adapter on UCF101 and Caltech 101 datasets in in-domain generalization setting. H: Harmonic mean (to highlight the generalization trade-off (22))
Dataset
ImageNet Novel
UCF101
Caltech101 Novel
Model
Base
H
Base
Novel
H
Base
H
Zero-shot CLIP
71.9
32.8
45.0
79.4
21.1
33.4
95,4
60.6
74.1
CLIP-Adapter
76.3
15.1
25.3
89.4
5.4
10.2
97.3
39.3
54.0
CoOp
75.3
2.7
5.2
89.3
1.0
2.0
97.2
31.3
47.4
CoCoOp
75.5
33.9
46.8
86.5
9.1
16.5
96.8
60.9
74.8
Meta- Aetta-Adapter
76.3
40.8
53.2
82.4
47.7
0.4
94.9
76.1
84.4
4.4 Comparison with Offline Methods
As shown Table we provide the ablation studies between our meta-adapter and other offline methods 22 g 9 For fair comparison, similar to CoCoOp, the experiments adopt base-to- novel generalization setting. The results demonstrate that our method has significant advantages in generalization novel classes, e.g. improving over CoCoOp by 6.9% ImageNet. Additionally,

---

as in previous works, we introduce Harmonic Mean to measure overall performance on base and novel classes. It can be observed that our approach also shows clear superiority terms of overall performance. More importantly, finetuning is not needed for our method when applying to new datasets or tasks.
4.5
More ablation studies of the Meta-Adapter
A2: Thanks for this insightful suggestion. Accordingly. we further conduct more ablation studies to demonstrate the advantages of our design. As shown in Table the results demonstrate that multi- head attention contributes most significantly to improving accuracy. The proposed learnable gating block can further enhance the performance while introducing value projection leads to decreased generalization capability. Besides, as shown in Table g we increase the model scale of the meta- adapter by widening the projection layers (Wider) or cascading multiple modules (Deeper). The results show that increasing the number of modules can improve the parameter size and accuracy slightly, but brings a significant efficiency decrease.
Table 7: Ablation study on different components, The 'VP', 'MHA', and 'LGB' indicate the value projection layer, multi-head attention block, and learnable gating block, respectively.
Method
ImageNet
SUN397
UCF101
DTD
Meta-Adapter w/ VP
25.6
18.2
5.9
5.0
Meta-Adapter w/o MHA
32.9
28.9
21.4
10.0
Meta-Adapte w/o LGB
39.6
51.3
49.8
53.8
Meta-Adapter
40.2
52.7
51.4
54.6
Table 8: Comparison of different model scales on several datasets.
Method
ImageNet
SUN397
UCF101
DTD
#Param
Latency
Meta-Adapter
40.8
52.6
51.0
55.8
2.1M
3ms
Wider (x2)
40.1
51.4
50.0
55.0
4.2M
5ms
+ Wider ( (x4)
40.5
51.3
50.4
55.0
8.4M
9ms
+ Deeper (x2)
40.4
52.9
52.2
56.7
4.2M
6ms
+ Deeper (x4)
38.9
52.7
51.4
56.3
8.4M
1lms
5 Limitations and Conclusion
Limitations Based on the findings presented in Table it can be observed that the learning potential of Meta-Adapter may encounter limitations when the classification accuracy of Zero-shot CLIP is high, as seen in the cases of UCF101 and Caltech101. The underlying reason behind this could be the combination of image-image similarity scores with text-image similarity scores, which might impede the potential of few-shot learning. In other words, in such scenarios, MetaeAd appears to favor zero-shot knowledge over few-shot knowledge, resulting in an imbalance between the two. Furthermore, empirical studies indicate that Meta-Adapter struggles to be applied open- open-vocabulary semantic segmentation tasks where obtaining high-quality few-shot datasets is challenging. It is possible that incorporating external data could help alleviate this issue. These aforementioned challenges are left for future research and investigation.
Conclusion This paper demonstrates the sigmificant potential of Meta-Adaptes, new few-shot learning method for CLIP, which is designed to overcome the limitations of previous methods in terms of poor generalization ability and low efficiency. The Meta-Adapter, employing meta-testing mechanism and a lightweight residual-style network, extracts knowledge from few-shot samples without the need for additional fine-tur munittnnng, thus alleviating the over-fitting issue while maintaining high efficiency. Our findings highlight the impressive performance of the Meta-Adapter in various tasks, including image classification, object detection, and segmentation, indicating its superior generalization capabilities across datasets and tasks. Future work could focus on further refining the Adaperr and exploring its potential applications in other vision tasks, advancing the capabilities of few-shot learning techniques in visual concept modeling.
10

---

6 Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 62204200).
References
[1] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Lc, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning, pages 4904-4916. PMLR, 2021.
[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[3] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694-9700 99055, 2021.
[4] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-128 12900. PMLR, 2022.
[5] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosscini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv:2205.0/3: 2022.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 /929, 2020.
[7] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03950, 2021.
[8] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. International Journal of Computer Vision, 13009):2337-2348, 2022.
19] Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 2021.
[10] Yu Du. Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084-14093, 2022.
[11] Xiuye Gu, Tsung-Yr Lin, Weicheng Kuo, and Yin Cui. Open- vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. /3921, 2021.
[12] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, and Peng Gao. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. arXiv preprint arXin:2003.02/5// 2023.
[13] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama addpter Efficient finc-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[14] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
[15] Xiangyang Zhu, Renrui Zhang, Bowei Hc, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint arXiv:2304.01199, 2023.
[16] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixioo Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self ielfrinsttuction. arXiv preprint arXiv:2305 187.52 2023.
11

---

[17] Rui Yang. Lin Song. Yixiao Ge. and Xiu Li. Boxsnake: Polygonal instance segmentation with box supervision. arXiv preprint arXiv:2303. 11630, 2023.
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[19] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Komblith, Rehecca Roclofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[20] Neil Houlsby, Andrei Giurgu,, Stanislaw Jastrzehski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficieet transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMIR, 2019.
[21] Taylor Shin, Yasaman Razeghi, Robert L Logan IV. Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
[22] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16816 16816-16825, 2022.
[23] Chengeheng Ma, Yang Liu, Jiankang Deng, Lingxi Xie, Weiming Dong, and Changsheng Xu. Understand- ing and mitigating overfitting in prompt tuning for vision-language models. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[24] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5149-511 2021.
[25] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. International Conference on Machine Learning, pages 1126 1135. PMLR, 2017.
[26] Yoonho Lee Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning. pages 2927-2902290 2936. PMLR, 2018.
[27] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford. and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in Neural Information Processing Systems, 29, 2016.
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention all you need. Advances neural information processing systems, 30, 2017.
[29] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A large-scale hicrarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[30] Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018
[31] Lin Song, Songyang Zhang, Songtao Liu, Zeming Li. Xuming He, Hongbin Sun, Jian Sun, and Nanning Zheng. Dynamic grained encoder for vision transformers. NIPS, 2021.
[32] Lin Song, Yanwei Li, Zhengkai Jiang, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng. Fine- grained dynamic head for object detection. NIPS, 2020.
[33] Lin Song, Yanwei Ii, Zeming Li, Gang Yu, Honghin Sun, Jian Sun, and Nanning Zheng. Learnable tree filter for structure-preet preeeurvii- feature transform. NIPS, 2019.
[34] Lin Song, Shiwei Zhang, Gang Yu, and Hongbin Sun. Tacnet: Transition-awar context network for spatio-temporal action detection. In CVPR, 2019.
[35] Lin Song. Yanwei Li, Zhengkai Jiang, Zeming Li, Xiangyu Zhang, Hongbin Sun, Jian Sun, and Nanning Zheng. Rethinking learnable tree filter for generic feature transform. NIPS, 2020.
12

---

[36] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. In CVPR, 2020.
[37] Liunian Harold Li. Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 11t.:i::100000000 909..00..55.. 2019.
[38]
Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, al. Oscar: object-semantic aligned pee-training for vision-language tasks. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Springer, 2020.
Part XXX 16, pages
[39] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation leamiig. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[40] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. 2021.
arXiv preprint
[41] Yuan Yao, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Pevl: Position- enhanced pre-training and prompt tuning for vision-language models. arXiv preprint 111.: 2022.
[42] Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Class-aware visual prompt tuning for vision-language pre-trained model. arXiv preprint arXiv:2208. 08340, 2022.
[43] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong. Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023.
[44] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423 438, 2020.
[45] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pages 3-17, 1998.
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[47] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961- 22011299922220 2017.
[48] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794-7803, 2018.
[49] Qingsen Yan, Lei Zhang, Yu Liu, Yu Zhu, Jinqiu Sun, Qinfeng Shi, and Yanning Zhang. Deep hdr imaging via a non-local network. IEF. Transactions on Image Processing, 29:4308-43223 2020.
[50] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeeze- excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[51] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- tine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515. 2013.
[52] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3498-3343 3505. IEEE, 2012.
[53] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to z0o. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3485 3492. IEEE, 2010.
[54] Khurram Soomro, Amir Roshan Zamir. and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 121.0422, 2012.
[55] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshop, pages 178-178. IEEE, 2004.
13

---

[56] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 3606-3613, 2014.
[57] Patrick Helber, Benjamin Bischke. Andreas Dengel. and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-22 1226 2019.
[58] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[59] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340 8349, 2021.
[60] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advonces in Neural Information Processing Systems, 32, 2019.
[61] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5356-
5364, 2019.
[62] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1711:1711..0100,, 2017.
[63] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-emn: Towards real time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[64] Jianfeng Wang. Lin Song, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng. End-to-end object detection with fully convolutional network. In CVPR, 2021.
[65] Jinrong Yang, Lin Song, Songtao Liu, Zeming Li. Xiaoping Li, Hongbin Sun, Jian Sun, and Nanning Zheng. Dbq-ssd: Dynamic ball query for efficient 3d object detection. arXiv preprint arXiv:2207. 10909, 2022.
[66] Weixin Mao, Jinrong Yang. Zheng Ge, Lin Song, Hongyu Zhou, Tiezheng Mao, Zeming Li, and Osamu Yoshie. Gmm: Delving into gradient aware and model perceive depth mining for monocular 3d detection. arXiv preprint ar31v:2.366. / /7500, 2023.
[67] Songyang Zhang, Lin Song, Songtao Liu, Zheng Ge, Zeming Li, Xuming Hc, and Jian Sun. Workshop on autonomous driving cvpr 2021: Technical report for streaming perception challenge. arXiv preprint arXiv:2108.0m 04230, 2021.
[68] Jianwen Jiang, Yu Cao, Lin Song, Shiwei Zhang, Yunkai Li, Ziyao Xu, Qian Wu, Chuang Gan, Chi Zhang, and Gang Yu. Human centric spatio-temporal action localization. In ActivityNer Workshop on CVPR, 2018.
[69] Shiwei Zhang, Lin Song. Changxin Gao, and Nong Sang. Glnet: Global local network for weakly supervised action localization. IEEE Transactions on Multimedio, 2210022610-2 2019.
[70] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via oeject-level contrastive learning. Advances in Neural Information Processing Systems, 34:22682-22666 2021.
14


Reviews:
[{"review_id": "NInxLC6Gwp", "reviewer": "Reviewer_deHD", "summary": "This paper proposes an online adaptation method for CLIP (no fine-tuning of few-shot samples of unseen categories are required, unlike CoOp, CoCoOp, and CLIP-Adapter), called Meta-Adapter. The main claim seems to be that Meta-Adapter is more robust than the most related approach Tip-Adapter, which relies heavily on the hyperparameter search strategy on the target dataset – alpha and beta in Eq (2) which help adjust the weight between category embeddings and few-shot visual embeddings.\n\nMore specially, the high-level idea of Meta-Adapter can be found in Figure 2 and Eq. (3). One can make a comparison between Eq. (2) and Eq. (3) to see how Meta-Adapter simplifies the online adaptation. The architecture of Meta-Adapter is based on the gated multi-head attention mechanism [26].", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1: Even though the design choice for Meta-Adapter seems arbitrary, it is simple and sound. \n\nS2: Experimental results show the superiority of Meta-Adapter to Tip-Adapter across tasks/settings and architectures (Table 1-5).", "weaknesses": "W1: Experimental settings seem to deviate from those in Tip-Adapter, making them less convincing. Examples: (i) This paper considers 8 image classification datasets instead of 11. (ii) The efficiency analysis is lacking. (iii) Related to (ii), there are no CoOp, CoCoOp, or CLIP-Adapter baselines. (I understand that these are online approaches but it would be nice to discuss, for example, an effectiveness/efficiency tradeoff).\n\nW2: Clarity. This point is minor but overall the paper seems to jump into details too quickly and it would benefit from more coarse-to-fine writing. For example, Section 3.2 could benefit form describing different components before jumping into details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Experiments on more image classification datasets.**  \n**A1:**\nThanks for the valuable suggestion. As shown in Table 7, we conduct the experiments in the other 3 datasets as the reviewer mentioned. Similar to the reported results in the main paper, our method also achieves consistent gains over the Tip-Adapter. We will add this table to the revision.\n\n\n**Table 7. Quantitative results of Meta-Adapter and other methods on Food101, Stanford Cars, and Oxford Flowers datasets.**\n\n\n|Method          | Food101       | Stanford Cars | Oxford Flowers |\n| -----------    | :-----------: | :-----------: | :-----------:  | \n|  Zero-Shot CLIP| 77.4          |  55.7         | 66.0           |\n|  Tip-Adapter   | 77.8          | 66.7          | 89.9           |\n|  Meta-Adapter  | **79.0** | **67.3**| **93.5** |\n\n\n**Q2: Efficiency analysis is lacking.**  \n**A2:**\nWe have reported the comparison of inference time for online methods in Figure 1(b). The results show that the inference time of Tip-Adapter increases linearly with the number of input shots, due to its individual encoding for each shot. In contrast, our method can maintain a much lower and constant time consumption, while achieving a higher accuracy.\n\n**Q3: Comparison with offline methods.**  \n**A3:**\nAs shown in Table 8, we provide the ablation studies between our meta-adapter and other offline methods. For a fair comparison, similar to CoCoOp, the experiments adopt a base-to-novel generalization setting. The results demonstrate that our method has significant advantages in generalization on novel classes, e.g. improving over CoCoOp by 6.9% on ImageNet. Additionally, as in previous works, we introduce Harmonic Mean to measure overall performance on base and novel classes. It can be observed that our approach also shows clear superiority in terms of overall performance. Moreover, as shown in Table 9, we provide training time comparisons between different methods, where the duration is based on default settings in the original papers. The results show that our method not only surpasses these offline approaches in generalization performance but also has noticeable advantages in training speed. More importantly, finetuning is not needed for our method when applying to new datasets or tasks. We will add these tables to the revision.\n\n**Table 8. Comparison of Zero-Shot CLIP, CLIP-Adapter, CoOp, CoCoOp, and Meta-Adapter on ImageNet, UCF101, Caltech101, DTD, and FGVCAircraft datasets in the in-domain generalization setting. H: Harmonic mean.**\n\n| Dataset        | ImageNet | ImageNet | ImageNet | UCF101 | UCF101 | UCF101 | Caltech101 | Caltech101 | Caltech101 | DTD  | DTD   | DTD  | FGVCAircraft | FGVCAircraft | FGVCAircraft |\n|----------------|----------|----------|----------|--------|--------|--------|------------|------------|------------|------|-------|------|--------------|--------------|--------------|\n| Model          | Base     | Novel    | **H**        | Base   | Novel  | **H**      | Base       | Novel      | **H**          | Base | Novel | **H**    | Base         | Novel        | **H**            |\n| Zero-shot CLIP | 71.9     | 32.8     | 45.0     | 79.4   | 21.1   | 33.4   | 95.4       | 60.6       | 74.1       | 59.3 | 8.2   | 14.3 | 23.9         | 0.6          | 1.2          |\n| CLIP-Adapter   | 76.3     | 15.1     | 25.3     | 89.4   | 5.4    | 10.2   | 97.3       | 39.3       | 54.0       | 70.2 | 2.0   | 3.9  | 32.1         | 0.3          | 0.6          |\n| CoOp           | 75.3     | 2.7      | 5.2      | 89.3   | 1.0    | 2.0    | 97.2       | 31.3       | 47.4       | 71.6 | 1.5   | 2.9  | 32.7         | 0.3          | 0.6          |\n| CoCoOp         | 75.5     | 33.9     | 46.8     | 86.5   | 9.1    | 16.5   | 96.8       | 60.9       | 74.8       | 69.1 | 3.0   | 5.8  | 30.0         | 0.8          | 1.6          |\n| Meta-Adapter   | 76.3     | 40.8     | **53.2**     | 82.4   | 47.7   | **60.4**    | 94.9       | 76.1       | **84.4**       | 64.1 | 49.1  | **55.6** | 30.8         | 17.1         | **21.9**         |\n\n**Table 9. Training time on ImageNet of Meta-Adapter and other offline methods on a single GeForce RTX 3090.**\n\n|CLIP-Adapter (200 epochs)        | CoOp (200 epochs)      | CoCoOp (10 epochs) | Meta-Adapter (10 epochs)  |\n| -----------    | :-----------: | :-----------: | :-----------:  | \n|  17h 30min | 15h 10min          |  23h 30min         | **20min**           |\n\n**Q4: Minor issues about clarity.**  \n**A4:**\nThanks for the constructive comments. Accordingly, we will improve the presentation by adding more high-level descriptions and backgrounds before Section 3.2."}, {"review_id": "3MWibFjbm0", "reviewer": "Reviewer_JEGz", "summary": "This paper proposes Meta-Adapter which can refine the CLIP features guided by the few-shot samples in an online manner. The major challenge of adapting CLIP with few-shot samples is over-fitting. Compared with offline approaches CoOp or online approaches TIP-Adapter, Meta-Adapter alleviates the over-fitting problem and demonstrates superior generalization across datasets. The author further adapts the Meta-Adapter to an open-vocabulary object detector, ViLD, and also finds decent improvements", "questions": "- The captions of each figure are short. I recommend adding more detailed explanations for each figure.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ This paper is generally well-written and easy to follow.\n+ The proposed Meta-Adapter shows improved generality over different datasets compared with the Tip-adapter baseline.\n+ Meta-Adapter can be served as a plugin module for open-vocabulary models, such as ViLD.", "weaknesses": "- Limited comparisons with existing approaches. Even though there are few online approaches like Tip-adapter, it is also encouraged to include more competitors, such as offline approaches in the experimental discussions.\n- The authors claim that Meta-Adapter uses 'a lightweight residual style adapter'. However, what is the optimal size of the adapter is not justified in ablation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Limited comparisons with existing approaches.**  \n**A1:** \nThanks for the suggestions. As shown in Table 7, we provide the ablation studies between our meta-adapter and other offline methods. For a fair comparison, similar to CoCoOp, the experiments adopt a base-to-novel generalization setting. The results demonstrate that our method has significant advantages in generalization on novel classes, e.g. improving over CoCoOp by 6.9% on ImageNet. Additionally, as in previous works, we introduce Harmonic Mean to measure overall performance on base and novel classes. It can be observed that our approach also shows clear superiority in terms of overall performance. Moreover, as shown in Table 8, we provide training time comparisons between different methods, where the duration is based on default settings in the original papers. The results show that our method not only surpasses these offline approaches in generalization performance but also has noticeable advantages in training speed. More importantly, finetuning is not needed for our method when applying to new datasets or tasks. We will add these tables to the revision.\n\n**Table 7. Comparison of Zero-Shot CLIP, CLIP-Adapter, CoOp, CoCoOp, and Meta-Adapter on ImageNet, UCF101, Caltech101, DTD, and FGVCAircraft datasets in the in-domain generalization setting. H: Harmonic mean.**\n\n| Dataset        | ImageNet | ImageNet | ImageNet | UCF101 | UCF101 | UCF101 | Caltech101 | Caltech101 | Caltech101 | DTD  | DTD   | DTD  | FGVCAircraft | FGVCAircraft | FGVCAircraft |\n|----------------|----------|----------|----------|--------|--------|--------|------------|------------|------------|------|-------|------|--------------|--------------|--------------|\n| Model          | Base     | Novel    | **H**        | Base   | Novel  | **H**      | Base       | Novel      | **H**          | Base | Novel | **H**    | Base         | Novel        | **H**            |\n| Zero-shot CLIP | 71.9     | 32.8     | 45.0     | 79.4   | 21.1   | 33.4   | 95.4       | 60.6       | 74.1       | 59.3 | 8.2   | 14.3 | 23.9         | 0.6          | 1.2          |\n| CLIP-Adapter   | 76.3     | 15.1     | 25.3     | 89.4   | 5.4    | 10.2   | 97.3       | 39.3       | 54.0       | 70.2 | 2.0   | 3.9  | 32.1         | 0.3          | 0.6          |\n| CoOp           | 75.3     | 2.7      | 5.2      | 89.3   | 1.0    | 2.0    | 97.2       | 31.3       | 47.4       | 71.6 | 1.5   | 2.9  | 32.7         | 0.3          | 0.6          |\n| CoCoOp         | 75.5     | 33.9     | 46.8     | 86.5   | 9.1    | 16.5   | 96.8       | 60.9       | 74.8       | 69.1 | 3.0   | 5.8  | 30.0         | 0.8          | 1.6          |\n| Meta-Adapter   | 76.3     | 40.8     | **53.2**     | 82.4   | 47.7   | **60.4**    | 94.9       | 76.1       | **84.4**       | 64.1 | 49.1  | **55.6** | 30.8         | 17.1         | **21.9**         |\n\n\n**Table 8. Training time on ImageNet of Meta-Adapter and other offline methods on a single GeForce RTX 3090.**\n\n\n\n|CLIP-Adapter (200 epochs)        | CoOp (200 epochs)      | CoCoOp (10 epochs) | Meta-Adapter (10 epochs)  |\n| -----------    | :-----------: | :-----------: | :-----------:  | \n|  17h 30min | 15h 10min          |  23h 30min         | **20min**           |\n\n\n**Q2: The captions of each figure are short.**  \n**A2:**\nMany Thanks. We will add more details to the captions according to the reviewer's suggestion.\n\n**Q3: What is the optimal size of the adapter is not justified in ablation.**  \n**A3:**\nWe appreciate the comments. Since we pursue online few-shot learning, our method needs to balance both accuracy and efficiency. As shown in Table 9, we increase the model scale of the meta-adapter by widening the projection layers (Wider) or cascading multiple modules (Deeper). The results show that increasing the number of modules can improve the parameter size and accuracy slightly, but brings a significant efficiency decrease. We will clarify it and add this table to the revision.\n\n**Table 9. Quantitative results of Meta-Adapter's variants on several datasets.**\n\n\n|Method       | ImageNet       | SUN397       | UCF101     |  DTD       |  #Param  | Latency |\n| ----------- | :-----------:  | :-----------:| :---------:|  :-------: | :-------:|  :----: | \n|Meta-Adapter | 40.8           |  52.6        | 51.0       | 55.8       | 2.1M     |  3ms    |\n|Wider (X2)   | 40.1           |  51.4        | 50.0       | 55.0       | 4.2M     |  5ms    |\n|Wider (X4)   | 40.5           |  51.3        | 50.4       | 55.0       | 8.4M     |  9ms    |\n|Deeper (X2)  | 40.4           |  52.9        | 52.2       | 56.7       | 4.2M     |  6ms    |\n|Deeper (X4)  | 38.9           |  52.7        | 51.4       | 56.3       | 8.4M     |  11ms   |\n"}, {"review_id": "tXgq6Zbs2W", "reviewer": "Reviewer_iC61", "summary": "The main goal of the paper is to explore an approach that is light-weight to allow a CLIP-pretained model to perform well in few-shot settings. The proposed approach (called Meta-Adapter) essentially learns an additional multi-head attention network with an additional gating function. The approach is simple and seems to achieve interesting results. ", "questions": "see weaknesses section above\n\nThe authors argue that the work is doing some sort of \"meta-learning\" (lines 116-118) - however, section 3 (method) does not talk about any meta-learning setting as far as I can tell - can you please give an explanation what the meta-learning part is here?", "limitations": "ok for me", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The addressed problem is of great practical relevance and pretraining with CLIP is highly popular these days\n\nOverall the idea is quite simple (which is positive) and intuitive \n\nThe experiments give a variety of comparisons to zero-shot CLIP and TIP-Adapter", "weaknesses": "The paper mentions on line 61 that the paper performs abalation studies - but honestly the reported experiments in the main paper are not real proper ablations - in particular the paper should have ablated the design choices of the approach where I could not find any experiment - since the approach is so simple that would have been very easy to do and would have helped to understand the approach better\n\nThe paper reports slightly lower \"base class\" performance (e.g. table) 2 for most datasets - which is then \"compensated\" for by higher \"novel class\" performance. This is a classic trade-off many works have - but I would have liked to not just see a single pair of base/novel class performance but rather a curve/set of novel/base class performances - that would be again more interesting and telling \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: More ablation studies of the Meta-Adapter.**  \n**A1:** \nThanks for this insightful suggestion. Accordingly, we further conduct more ablation studies to demonstrate the advantages of our design. As shown in Table 7, the results demonstrate that multi-head attention contributes most significantly to improving accuracy. The proposed learnable gating block can further enhance the performance while introducing value projection leads to decreased generalization capability. Besides, as shown in Table 8, we increase the model scale of the meta-adapter by widening the projection layers (Wider) or cascading multiple modules (Deeper). The results show that increasing the number of modules can improve the parameter size and accuracy slightly, but brings a significant efficiency decrease. We will add these tables to the revision.\n\n**Table 7. Ablation study on different components. The LGB, VP, and MHA indicate the learnable gating block, value projection layer, and multi-head attention block, respectively.**\n\n\n|Method              | ImageNet      | SUN397       | UCF101        |  DTD          |\n|-----------         | :------:      | :---:        | :----:        |:----:         | \n|Meta-Adapter w/  VP | 25.6          |18.2          | 5.9           | 5.0           |\n|Meta-Adapter w/o MHA| 32.9          |28.9          | 21.4          | 10.0          |\n|Meta-Adapter w/o LGB| 40.2          |51.3          | 50.2          | 55.0          |\n|Meta-Adapter        | **40.8**          |**52.6**          |**51.0**          | **55.8**          |\n\n\n**Table 8. Quantitative results of Meta-Adapter's variants on several datasets.**\n\n\n|Method       | ImageNet       | SUN397       | UCF101     |  DTD       |  #Param  | Latency |\n| ----------- | :-----------:  | :-----------:| :---------:|  :-------: | :-------:|  :----: | \n|Meta-Adapter | 40.8           |  52.6        | 51.0       | 55.8       | 2.1M     |  3ms    |\n|Wider (X2)   | 40.1           |  51.4        | 50.0       | 55.0       | 4.2M     |  5ms    |\n|Wider (X4)   | 40.5           |  51.3        | 50.4       | 55.0       | 8.4M     |  9ms    |\n|Deeper (X2)  | 40.4           |  52.9        | 52.2       | 56.7       | 4.2M     |  6ms    |\n|Deeper (X4)  | 38.9           |  52.7        | 51.4       | 56.3       | 8.4M     |  11ms   |\n\n\n\n**Q2: The performance trade-off between base and novel classes.**  \n**A2:** \nThanks for this valuable suggestion. To demonstrate the generalization ability of Meta-Adapter, similar to CoCoOp [22], we introduce the harmonic mean as the evaluation criteria, which is used to reflect the overall performance on both seen and unseen sets [57]. As shown in Table 9, although Tip-Adapter can overfit the seen data on some datasets, its overall performance is significantly inferior to Meta-Adapter. We will update the table in the revision.\n\n**Table 9. Comparison of Zero-Shot CLIP, Tip-Adapter, and Meta-Adapter on ImageNet, UCF101, Caltech101, DTD, and FGVCAircraft datasets in the in-domain generalization setting. H: Harmonic mean.**\n\n| Dataset        | ImageNet | ImageNet | ImageNet | UCF101 | UCF101 | UCF101 | Caltech101 | Caltech101 | Caltech101 | DTD  | DTD   | DTD  | FGVCAircraft | FGVCAircraft | FGVCAircraft |\n|----------------|----------|----------|----------|--------|--------|--------|------------|------------|------------|------|-------|------|--------------|--------------|--------------|\n| Model          | Base     | Novel    | **H**        | Base   | Novel  | **H**      | Base       | Novel      | **H**          | Base | Novel | **H**    | Base         | Novel        | **H**            |\n| Zero-shot CLIP | 71.9     | 3.8     | 45.0     | 79.4   | 21.1   | 33.4   | 95.4       | 60.6       | 74.1       | 59.3 | 8.2   | 14.3 | 23.9         | 0.6          | 1.2          |\n|Tip-Adapter  |       73.3         | 36.5          | 48.7     | 85.2     | 40.3      | 54.7      | 96.3   | 73.3    | 83.2    | 66.8       | 41.7       | 51.3       | 30.7 | 14.9   | 20.0  | \n| Meta-Adapter   | 76.3     | 40.8     | **53.2**     | 82.4   | 47.7   | **60.4**    | 94.9       | 76.1       | **84.4**       | 64.1 | 49.1  | **55.6** | 30.8         | 17.1         | **21.9**         |\n\n[22] Zhou K, Yang J, Loy C C, et al. Conditional prompt learning for vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16816-16825.\n\n[57] Xian, Y., Schiele, B. and Akata, Z., 2017. Zero-shot learning-the good, the bad and the ugly. In\nProceedings of the IEEE conference on computer vision and pattern recognition (pp. 4582-4591).\n\n\n\n**Q3: Which part explains the meta-learning settings?**  \n**A3:**\nSorry for the confusion. We have presented the meta-learning strategy of our method in Line 46-50. Specifically, we split the data into seen and unseen sets according to category, domain, or task. During training, we randomly sample few-shot image/text pairs from the seen set to optimize the parameters of the meta-adapter, enabling it with general few-shot learning ability. During testing, we first sample some few-shot image/text pairs per category from the unseen set, then fed to the frozen meta-adapter to generate embeddings for each category, and finally predict classification scores by calculating the similarity between the embeddings and features of other images in the unseen set. We will clarify it in the revision."}, {"review_id": "UELT8Cq2Xn", "reviewer": "Reviewer_uypu", "summary": "This paper proposes a meta-adapter structure for CLIP like vision-language backbone. Specifically, a cross-attention with a gate mechanism are used to construct the meta-adapter. It aims to improve the few-show learning ability for current CLIP backbone. Compared with other baselines such as CLIP-adapter and Tip-adapter, the proposed meta-adapter obtains improvements on several experimental settings.", "questions": "Please check the weaknesses.", "limitations": "Discussed by the authors.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies a popular question about CLIP few-shot learning.\n2. The motivation and idea are well presented.\n3. Meta-adapter is straightforward and easy to follow.", "weaknesses": "1. The meta-adapter uses a cross-attention and gate module, which are commonly used for this field resulting in the limited technical contribution. Simply adding these modules to improve the final performance is not very surprising.\n2. The proposed module is general for vision-language model, but the evaluation is only for classification. Supplementing retrieval evaluation will further support this method.\n3. Some figures are relatively misleading the readers. For example, in figure1, please unify the number scales in the performance axis. Current version is a little confusing to show the performance gain.\n4. As shown in Tab1 and the limitation section discussed by the authors, the general performance gain is not very significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Limited technical contribution of the meta-adapter.**  \n**A1:** \nThe main contribution of this paper is to introduce meta-learning into clip adapters for the first time, to achieve online few-shot learning for visual-language models. We mainly pursue building a new framework for clip adapters, rather than the network architecture. Although the structure of the Meta-Adapter is simple, as shown in Figure 1, it not only has higher efficiency but also stronger generalization performance than the Tip-Adapter. Besides, as shown in Table 7, compared to other sophisticated offline methods, our method demonstrates clear advantages in generalization capability and overall performance.\n\n**Table 7. Comparison of Zero-Shot CLIP, CLIP-Adapter, CoOp, CoCoOp, and Meta-Adapter on ImageNet, UCF101, Caltech101, DTD, and FGVCAircraft datasets in the in-domain generalization setting. H: Harmonic mean.**\n\n| Dataset        | ImageNet | ImageNet | ImageNet | UCF101 | UCF101 | UCF101 | Caltech101 | Caltech101 | Caltech101 | DTD  | DTD   | DTD  | FGVCAircraft | FGVCAircraft | FGVCAircraft |\n|----------------|----------|----------|----------|--------|--------|--------|------------|------------|------------|------|-------|------|--------------|--------------|--------------|\n| Model          | Base     | Novel    | **H**        | Base   | Novel  | **H**      | Base       | Novel      | **H**          | Base | Novel | **H**    | Base         | Novel        | **H**            |\n| Zero-shot CLIP | 71.9     | 32.8     | 45.0     | 79.4   | 21.1   | 33.4   | 95.4       | 60.6       | 74.1       | 59.3 | 8.2   | 14.3 | 23.9         | 0.6          | 1.2          |\n| CLIP-Adapter   | 76.3     | 15.1     | 25.3     | 89.4   | 5.4    | 10.2   | 97.3       | 39.3       | 54.0       | 70.2 | 2.0   | 3.9  | 32.1         | 0.3          | 0.6          |\n| CoOp           | 75.3     | 2.7      | 5.2      | 89.3   | 1.0    | 2.0    | 97.2       | 31.3       | 47.4       | 71.6 | 1.5   | 2.9  | 32.7         | 0.3          | 0.6          |\n| CoCoOp         | 75.5     | 33.9     | 46.8     | 86.5   | 9.1    | 16.5   | 96.8       | 60.9       | 74.8       | 69.1 | 3.0   | 5.8  | 30.0         | 0.8          | 1.6          |\n| Meta-Adapter   | 76.3     | 40.8     | **53.2**     | 82.4   | 47.7   | **60.4**    | 94.9       | 76.1       | **84.4**       | 64.1 | 49.1  | **55.6** | 30.8         | 17.1         | **21.9**         |\n\n\n**Q2: The evaluation is only for classification.**  \n**A2:**\nAs shown in Table 5 and Section 4.3 in the manuscript, we evaluate our Meta-Adapter for object detection and segmentation tasks. Without further fine-tuning, our method demonstrates superior generalization capabilities, achieving much higher performance than the Tip-Adapter and zero-shot baseline. Additionally, in Table 8 we provide the quantitative results (recall@10) of retrieval evaluation on several classification datasets. The results show that our Meta-Adapter obtains consistent and considerable improvements over Tip-Adapter across datasets.\n\n\n**Table 8. Average Recall of Zero-Shot CLIP and Meta-Adapter on several datasets for retrieval evaluation.**\n\n\n\n|Recall@10       | ImageNet      | FGVC | Oxford Pets |  SUN397 | UCF101 | DTD |\n| -----------    | :-----------: | :-----------: | :-----------: |  :-----------: | :-----------: |  :-----------: | \n|  Zero-shot CLIP |57.5          | 15.1  | 77.0        | 51.9     |  52.4     |   29.8  |\n| Tip-Adapter     |62.5          | 18.8  |77.3        | 57.8     |  59.0     |   39.3  |\n| Meta-Adapter    |**64.2**| **22.4**     | **77.6**          | **61.5**      |  **64.5**    |**46.0**  |\n\n\n\n\n**Q3: Some figures are relatively misleading to the readers.**  \n**A3:** \nThanks for the suggestion. We will unify the number scales of Figure 1 in the revision.\n\n**Q4: The general performance gain is not very significant in Table 1.**  \n**A4:**\nPlease note that our Meta-Adapter improves over Tip-Adapter by 4.96% on average, while having lower inference cost. Additionally, $\\Delta$ represents the gap between optimization on individual datasets and cross-dataset generalization performance. The results reflect that our algorithm has strong generalization capabilities, enabling the cross-dataset performance to be comparable to individually optimized performance. In contrast, Tip-Adapter not only has inferior absolute performance but also notably weaker generalization abilities."}, {"review_id": "kESGqV1LG4", "reviewer": "Reviewer_Qojd", "summary": "For the Vision-Language Model task, which usually requires a small number of samples for fine-tuning, this work proposes a Meta-Adapter method. The Meta-Adapter method is based on the gated multi-head attention mechanism, and can be generalized to unseen categories without additional fine-tuning after a small amount of training by Few-Shot. The effectiveness of this work is experimentally demonstrated.", "questions": "1. Why is it called Meta-Adapter? In my understanding, the method of this work is to generalize to new samples by using a small number of training samples, and this method is not called meta-learning.\n2. The motivation for this work is that \"few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters on few-shot samples \", and the same motivation for Tip-Adapter. It seems to me that Meta-Adapter is an improved approach based on Tip-Adapter, is there a new improvement in the motivation for the approach to this work?\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. From the experimental results, this work performs better in corss-dataset generalization and has good performance on downstream tasks.\n2.\tThis work is well-written, the related works are well summarized and the contributions are clearly demonstrated. The empirical performance is promising.\n", "weaknesses": "1. The explanation of the formulas in this work could be more clear to help the reader understand the content.\n2. The connection between the meta-learning method and Meta-Adapter is not well explained.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The explanation of the formulas could be more clear.**  \n**A1:** \nMany thanks. Accordingly, we will provide a more comprehensive explanation of the formulas in Section 3.2, including additional descriptions and analyses of the symbols and their functions.\n\n**Q2: Why is it called Meta-Adapter? The connection between the meta-learning method and Meta-Adapter.**  \n**A2:**\nThe Meta-Adapter aims to build a new clip adapter with general few-shot learning abilities instead of specializing in certain tasks and domains. To achieve it, as stated in Line 46-50, we adopt the meta-testing mechanism. Specifically, we split the data into seen and unseen sets according to category, domain, or task. During training, we randomly sample few-shot image/text pairs from the seen set to optimize the parameters of the meta-adapter, enabling it with general few-shot learning ability. During testing, we first sample some few-shot image/text pairs per category from the unseen set, then fed to the frozen meta-adapter to generate embeddings for each category, and finally predict classification scores by calculating the similarity between the embeddings and features of other images in the unseen set. We will clarify it in the revision.\n\n**Q3: The improvement in the motivation against Tip-Adapter.**  \n**A3:**\nThe motivation of the Meta-adapter is to learn how to learn from few-shot samples. It has the more general ability to learn in a few-shot setting rather than domain-specific capabilities. Previous methods, including Clip-Adapter and Tip-Adapter, require fine-tuning or hyper-parameter search tailored to the target domain in order to achieve good performance. Our method, on the other hand, can be applied to other domains or tasks seamlessly and demonstrates significant efficacy and performance advantages."}]

Author rebuttal:

Meta-review: This paper receives three borderline accepts and two weak accepts. Initially, the reviewers have common concerns for writing clarity, insufficient experiments (e.g., efficiency analysis, the adapter size ablation, limited comparisons with existing approaches), the connection with meta-learning, etc. The rebuttal answers the questions from the reviewers, and the reviewers' concerns are resolved after the rebuttal. AC gives the recommendation of accept (poster) based on the authors' and reviewers' response, and encourages the authors to include these additional results and improve the paper writing in the revised version.
Decision: Accept (poster)