GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph
Authors: Xin Li
Dongze Lian
Zhihe Lu
Jiawang Bai
Zhibo Chen
Xinchao Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Efficient transfer learning
vision-language model
adapter-style tuning
TL;DR: We propose a brand-new adapter-style tuning strategy, i.e., GraphAdapter,  for the Vision-Language Models.

Abstract: Adapter-style efficient transfer learning (ETL) has shown excellent performance in the tuning of vision-language models (VLMs) under the low-data regime, where only a few additional parameters are introduced to excavate the task-specific knowledge based on the general and powerful representation of VLMs. However, most adapter-style works face two limitations: (i) modeling task-specific knowledge with a single modality only; and (ii) overlooking the exploitation of the inter-class relationships in downstream tasks, thereby leading to sub-optimal solutions. To mitigate that, we propose an effective adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual adapter by explicitly modeling the dual-modality structure knowledge (i.e., the correlation of different semantics/classes in textual and visual modalities) with a dual knowledge graph. In particular, the dual knowledge graph is established with two sub-graphs, i.e., a textual knowledge sub-graph, and a visual knowledge sub-graph, where the nodes and edges represent the semantics/classes and their correlations in two modalities, respectively. This enables the textual feature of each prompt to leverage the task-specific structure knowledge from both textual and visual modalities, yielding a more effective classifier for downstream tasks. Extensive experimental results on 11 benchmark datasets reveal that our GraphAdapter significantly outperforms the previous adapter-based methods.

GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph
Xin Li Dongze L Lu²; Jiawang Bai Zhibo Chen ; and Xinchao Wang"t
University of Science and Technology of China
2 National University of Singapore
Tsinghua University
1ixin6666mail.. uutc.. edu. cn, www.hhlleeluueeeeee .dd. ag, bju19@mails, bjv19@mails.taiinguuaa.... edu.
chenzhibo@ustc.co edu. en, xinchao@nus. eetaaaoonuaeeeuu..... edu.sg
Abstract
Adaptcr-style efficient transfer learning (ETL) has shown excellent performance in the tuning vision-language models (VLMs) under the low-data regime, where only a few additional parameters are introduced to excavate the task-specific knowledge based on the general and powerful representation of VLMs. However, most adapter-style works face two limitations: (i) modeling task-specife knowledge with single modality only; and (ii) overlooking the exploitation of the inter- class relationships in downstream tasks, thereby leading sub-optimal solutions. To mitigate that, we propose an effective adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual adapter by explicitly modeling the dual- modality structure knowledge (i.e., the correlation of different semantics/classes in textual and visual modalities) with dual knowledge graph. In particular, the dual knowledge graph is established with two sub-graphs, i.e., a textual knowledge sub- sub-graph, and visual knowledge sub-graph, where the nodes and edges represent the semantics/classes and their correlations two modalities, respectively. This enables the textual feature of each prompt leverage the task-specific structure knowledge from both textual and visual modalities, yielding a more effective classifier for downstream tasks. Extensive experimental results on 11 benchmark datasets reveal that our based methods. The code will be released at https://gtt giggttu....... www./o/o//littar.cooooooocoooooe GraphAdapter
significantly outperforms previous adapter-
Introduction
Recent large-scale Dision-language models (VLMs), such as CLIP 55 and ALIGN 27 have shown their promising representation capability for a series of downstream vision tasks
such as
classification 44
18 86). generation 61 31 32 60,, and recognition 7 73 Different from previous pre- pre-training models with ImageNet 20 B based on large-scale parameters,
85
VLMs can learn more prosperous, fine- fine-graine grained, and consistent semantics from billions of text-image
pairs with contrastive learning. This enables the VLMs with two advantages: 1) performing superior
zero-shot generalization capability (89] with simple hand-craft prompts like "a photo of [class]",
and 2) possessing impressive transferability for downstream tasks (67 69 57 65] 86 by finetuning
the VLMs in low-data regime. Notably, traditional fine-tuning methods typically involve tuning all parameters of the model to adapt to downstream tasks. However, this approach may not be well-suited for fine-tuning large VLMs in resource-constrainee scenarios, and result in overfitting due to the
'Equal Contribution Corresponding Authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

"A photo [dog]"
Text
"A of (dog)'
Text Encoder
(Sepanate
baselleggr)
car
Aleerr
periee
Virual Encoder
Visual kincoder
(a) Zero-shol CLIP
(b) CLIP-Adapter
Loaraabic residaals
Tesnnal knowiedgel
(Pased iemesl& strectoal koowlegge
Dasi hasnledge griph
"A photo (dog)"
"A (shog]"
Enunder
Steen
Guappan sonn
Testual mislr Vicust wodes
perion
Visual Encoder
Visual
Encoder
TaskRes
GraphAddaper
Figure I:
The comparison between (a) Zero-shot CLIP (55). (b) CLIP-Adapter [8] (c) TaskRes 82
and (d) our proposed GraphAdapter. We can observe that previous works model task-specifie knowledge with single modality and lacks the exploitation of structure knowledge, In contrast, our GraphAdapter aims to exploit the fused vision and language structure knowledge in data (i.e., the inter-class relationships in dual modalities) for textual feature adapter with graph learning.
limited number of available samples. To mitigate this, efficient transfer learning (ETL) proposed to transfer the task-relevant knowledge from VLMs to downstream tasks by tuning a few parameters.
There two popular ETL approaches for VLMs, including prompt tuning 89 88
10
62 25
[2. and adapter-style tuning [18] [2 63
86 85). In particular, prompt tuning aims to adjust the textual classifier adaptively toward downstream tasks by adding learnable prompts the input side, which outperforms zero-shot CLIP by large margin with few-shot samples, such as CoOp 89 and CoCoOp 88) Despite that, there is limitation in the prompt tuning of VLMs X 82 needs to pass the data through the textual encoder every iteration during training, yielding higher contrast, by using the textual encoder once only, adapter-style works tend to refine the textual classifier or visual features with simple but efficient feature modulation for specific task on the output side, For instance, CLIP-Adapter [8] exploits one simple bottleneck layer to adjust the textual and visual embeddings of VLMs, which exceeds the zero-shot CLIP by 3.02% on ImageNet with the one-shot setting. TaskRes [82] utilizes learnable task-specific parameters as pripr- indeeinndeeedennt residuals to adjust the textual embeddings. Another popular line [86] 85 (00]
90
demand for resources.
seeks to augment the prior knowledge for downstream tasks with the cooperation CLIP and other pre-trained large vision or language models, such as DINO (5] and GPT [
However, there are two limitations in most adapter-style works on ETL: 1) only modeling task- specific knowledge from single modality perspective, such as CLIP-adapter [18), TaskRes [82) and Tip-Adapter [86), where the adaptation achieved based on the independent visual or textual feature. 2) overlooking the explicit exploitation of the structure knowledge (i.e., the relation between different semantics/classes) downstream tasks. small number of samples in low-data regime are hard to guide the model to sufficiently excavate the structure knowledge in downstream tasks, leading to the bias for partial attributes in the data, such as color and shape, and causing the sub-optimal transferability and generalization capability 33 49 9 45 46). It is vital to model the multi-modality structure knowledge of downstream tasks the tuning of VLMs in the low-data regime.
To mitigate the above limitations, we propose a brand-new adapter-style tuning strategy, dubbed GraphAdapter, which aims to model task-specific knowledge for downstream tasks with the fused textual and visual structure knowledge. We achieve this by solving two crucial challenges: I) how to model the structure knowledge (i.e., the inter-class relationship) for downstream tasks, and 2) how to learn the task-specific knowledge by introducing two- wwo-modality (i.e., visual and textual) structure knowledge. Recently, graph learning 449 on modeling data knowledge structure. However, the potential of graphs in improving the efficient transfer learning of VLMs has not yet been fully explored. Inspired by this, for the first challenge, we aim to exploit graph learning to model the structure knowledge for downstream tasks when tuning VL.Ss. Consequently, we propose the dual knowledge graph, which is composed of textual sub- sub-graph and visual sub-graph. Particularly, establish the relationship between different
7 36] has shown the prevailing performance

---

semantics/classes in textual space, we regard the textual feature of a specific class as one node of the textual graph and measure inter-class relationships by measuring their corresponding distance of features as edges of the graph. Similarly, the visual knowledge graph aims to model the relationship between different semantics/classes from the visual perspective, where the nodes are constructed with mean features of the training samples from the same class. In this way. our dual knowledge graph contains two-modality structure knowledge from downstream tasks, which enables the feature adapter with sufficient and reliable perception for downstream tasks.
For the second challenge, one intuitive strategy is to introduce the textual/visual structure knowledge for the textual/ /viuall feature adapter, separately. However, this strategy still models the task tusk-specific knowledge with a single modality for each textual/visual adapter (e.g., the textual adapter is only aware of textual structure knowledge). To mitigate this limitation, we introduce the dual knowledge graph for textual/visual adapters, and let each adapter be aware of the structure knowledge in the same modality and cross-modality. (Notably, we only exploit the textual adapter in our paper, since the limited gain with the combination of textual and visual adapters. Please see Table ( To exploit the dual knowledge graph effectively, we introduce graph learning to the textual adapter, where each feature of the prompt warps the textual and visual structure knowledge through the graph convolution network (GCN), and then fuse two warped knowledge to modulate the original feature for classification in a residual way. Based on the above strategies, we propose an effective adapter-style tuning strategy, i.e., GraphAdapter, which achieves superior performance in tuning the vision- lanoun-a models under a low-data regime.
The contributions of this paper are summarized as follows:
We pinpoint that the dual-modality structure knowledge (i.e., the inter-class relationship in textual and visual space) is vital for the efficient transfer learning (ETL) of VLMs in the low-data regime.
Based the above analysis, we propose brand-new adapter-style tuning strategy, i.e., GraphAdapter, for the ETL. of VLMs, which models the dual-modality structure knowledge from the textual and visual space with our proposed dual knowledge graph and enables the feature adapter to leverage the fused visual and language knowledge for better learning of task-specific knowledge from downstream tasks, yielding an effective tuning of VLMs.
We evaluate our Graphadapter on 11 popular benchmarks on few-shot classification. The ex- periments demonstrated that our Graphadapter significantly outperforms previous prompt-base based or adapter-style works, even on the challenging fine-grained image classification task, such as FGVCAireraft [8]
2 Related Work
Vision-Language Pre-training (VLP) aims to learn the universal textual and visual representation simultancously with amounts of text-image pairs. A series of works have revealed that pre-trained vision-language representations can help lots of downstream tasks to improve their performances, such as few-shot classification
cross-modality generation (50] 56, (4). and visual recognition [69]. The methods of VLP can be roughly divided into three types: 1) dual-encoder architecture, which aligns the textual and visual representation with text-image matching or text-image contrastive learning. One typical work is CLIP [55), where the promising visual representation is learned by matching the visual concepts with the corresponding textual representation with contrastive learning. 2) The second type [66 43 64 34
[89 18 82],
84 722 is fusion-encoder architecture, where the textual and visual representation are fused with cross-modal attention. In general, two independent feature extractors are exploited to extract the modal-specific representations, respectively, and the cross- modal transformer is used for learning the text-image shared representation. 3) The third type is a combination of dual-branches and fusion-branch architectures. For instance, VLMo [I] proposes the multiway transformer, which consists of the visual-specific branch, text-specific branch, and text-visual branch. Recently, amounts of studies are devoted to exploring how to tune the large vision-language model to downstream tasks with few parameters and data, i.e., efficient transfer leamingg This will be clarified in the next section.
Efficient Transfer Learning (ETL) is proposed to transfer the task task-specific knowledge to down- stream tasks by tuning the partial parameters 24 There are two prominent directions for the ETI., iee. prompt tuning 889
18
37
80
79
39
40
16
26 76 70)
88
and adapter-style
tuning [18 82 85 86). Prompt engineering stems from natural language process, (NLP), which aims
3

---

to formalize various NLP tasks with different prompt templates [8 47 Based on this development, prompt engineering introduced to the visual-language task. For instance, Zero-shot CLIP [55] introduces the simple manual prompt template, like "a photo of a [class]", and outperforms the
linear probe by 1.9% on ImageNet [2]. As the pioneering work, CoOp [89] for the first time introduces the learnable prompt to transfer the task-specific knowledge to few-shot tasks. Following this, amounts of work [88] 87
73 6 improve the prompt tuning from multiple perspectives, such as generalization 888 knowledge prototype (87), augmentation (85] and
85
44
28
57
62 10 25 42
diversity 44 In contrast, adapter-style works tune the VLMs to downstream tasks by adjusting the textual and visual features. Early work CLIP-Adapter [8]] only exploits one bottleneck layer to improve the performance of few-shot classification. TaskRes [2] introduces the task cask-independent adapter to decouple the prior knowledge of the pre- pre-trained models and task task-specific knowledge. There are also some works 86 85] that seek extra prior knowledge from other pre-trained large models, such as DINO [ which also yield great performance on adapter-style works. In principle, the above adapter-style methods achieve efficient transfer learning for VLMs from two perspectives: 1) increasing instance-we wise task-wise adaptability based on downstream tasks [88 82]. 2) taking into account the relation between training and test samples. [86]. However, the above works only model task-specific knowledge with a single modality, and lacks the utilization of structure knowledge in downstream tasks, leading to the sub-optimal solution. In this paper, we aim to excavate the structure knowledge of data with graphs and further improve the performance of adapter-style tuning.
performs excellent capability in formalizing the topologiaa structure of data with the nodes and edges, which has been broadly applied in various fields, such
Graph learning 77 [4] [29] 33 81
as biology [55] and social networks [59]. With the emergence of GCN [29]. some works take
step forward to introduce structure graphs into transfer learning 99] g
33
15] The core
challenge for graph learning is how to construct the nodes and edges. Previous works usually construct the nodes from three perspectives, including objects (9) semantic features (8), or model parameters 77) There are also some works that utilize the hypergraph to achieve better learning of structure knowledge [13 (8). In contrast, in this paper, we are the first work introduce graph learning into the efficient transfer learning (ETL) of vision- langua-anggugg tasks.
3 Approach
3.1 Preliminaries
Contrastive Language-Image Pre-training (CLIP) is designed to learn continuous visual represen- tation based on 0.4 million text-image pairs. Different from the pretraining with ImageNet, where the labels are discrete and cannot delicately describe the attributes of images and the relation between different classes, CLIP learns the visual representation with the supervision of textual embedding by contrastive loss. There are two feature extractors in CLIP, i.e., the textual encoder E, and visual encoder F.,, which are used to warp the prompts and image into the textual embedding 2 and visual embedding 2.- In the test process, K prompts like "a photo of a [class]" are inputted to the textual feature extractor to obtain the textual embeddings {at Jk=1 each image is achieved with:
LK as the labels of K classes. The class of
P(y clzz) exp(sim( 2f)T))
E exp(ssim &e, //T)
where c, sim, and T denote the class, cosine similarity, and learned temperature of CLIP, respectively, Despite CLIP has performed excellently on zero-shot image classification, the efficient transfer leaminng of CLIP is still required to adapt the pre-trained CLIP to downstream tasks.
3.2 Our Approach: GraphAdapter
To introduce the dual-modality structure knowledge of the downstream tasks to the feature adapter, we introduce dual knowledge graph 9 (9,, G. which is composed of the textual knowledge sub-graph G. and visual knowledge sub-grap -ab-graph Go to obtain and store the textual structure knowledge of prompts and visual structure knowledge of training samples, respectively. Then the feature of the sample/prompt can warp the structure knowledge of downstream tasks in the two modalities to refine themselves, thereby leveraging the dual structure knowledge for better modeling for downstream tasks. The whole framework of our GraphAdapte is depicted in Fig. 3 which is composed of two parts, including the construction of dual knowledge graph and the adapter-style CL-IP-based classification. Particularly, the dual knowledge graph is established only once for the whole training

---

Textual-subgraph
photo of [dog]"
GraphAdapter
photo of [eat]"
Text
Encoder
"A photo a [car]]
Shared
x
CDr
GCN
flwess
"A photo of [class]'
Encoder
posson
GCN
dog
Visual
Encoder
Visual
Encoder
Visual-subgraph
Shirred
Kxa
Dual knowledge graph
The pipeline of GraphAdapter, which is composed of the dual knowledge graph and CLIP- based image classification pipeline. Here, the dual knowledge graph contains the delicately designed sub-graphs for textual and visual knowledge, respectively, where the textual knowledge sub-graph is established with the textual feature from the prompts of the downstream task, and the visual knowledge sub-graph is constructed with the visual features of training samples from the downstream task. In the optimization process, the feature of each prompt used for classification will seek the fused visual and textual structure knowledge from the dual knowledge graph to adjust themselves for better classification. and d are the number of classes and the dimension of textual/visual features.
Figure 2:
process, and the textual encoder is also only run once to obtain the textual embedding 2; of each prompt. In the optimization process, given textual feature 2,, the relation between it and the nodes of the dual knowledge graph will be computed as the condition to warp the knowledge from the dual knowledge graph adjust the textual embedding in the residual form. We will clarify GraphAdapter as follows:
Textual knowledge sub-graph. To excavate the structure knowledge (i.e., the relation between different semantics) of the downstream tasks in textual space, we construct the textual knowledge sub-graph G, {C,, }. where the nodes C, aims capture the semanties of different classes and the edges { } is used to measure the relation different nodes. Notably, the classifier of CLIP is based the textual features of the prompts from different classes, where the textual feature of a within-class prompt can represent the semantics of one class. Therefore, given one downstream task with K classes, the nodes set C, {c]
*x) obtained with the mean feature of the prompts from the different class, where ci and d denotes the textual feature of the 7th class and the dimension of the textual feature. And the edges & between different nodes are computed with the cosine similarity between different nodes since the classification is achieved by computing the cosine similarity between the features in CLIP.
dd²
|cl i,j [1,
where (e)d) denotes the edge between ith and jth nodes.
Visual knowledge sub-graphe Different from the textual knowledge sub-graph. where the structure knowledge only from the textual concept of a downstream task, the visual knowledge sub-gra sib-graph measure more fine-grained relations of different semantics in visual space, since the diverse visual feature of different samples. As shown Fig. 3 to construct the visual knowledge sub-graph G. (C., E pass the augmented image group from the same class into visual encoder obtain their visual features, and then compute the mean features of them as the nodes C, {ce 1K E K xxd of for visual knowledge graph. The edges ε, = (e5. j j [1, K]} are computed with the cosine similarity between different nodes in the visual knowledge sub-graph.

---

Adapting with the dual knowledge graph. After constructing the dual knowledge graph G (. we can achieve the feature adapter by introducing the dual- modality structure knowledge adaprively from the text and image sub-graphs. Notably, previous works on adapter-style tuning only model the task-specific knowledge with a single modality, which lacks the exploitation of cross- modality knowledge. In contrast, our GraphAdapter utilizes both inner-modality and cross-modality structure knowledge for feature adapters. Concretely, given the textual feature 2; from the textual encoder of CLIP, we aim to warp the žn into the textual and visual knowledge sub-graphs to extract the textual modality and cross-modality structure knowledge, respectively. One simple strategy is to regard the textual/visual features 24 and 2y as the query, and then warp the knowledge from the graph nodes C₁ and C, based on similarity. However, this ignores that the structure knowledge is also required to be optimized to suit the downstream tasks. To achieve this, in the adapting process, regard the textual feature 2n as one node and then warp it to the visual and textual sub-graphs, where the textual features obtain dual-modality structure knowledge by interacting with two sub-graphs in the same graph space with the graph convolutional networks (GCN).
Specifically, in the optimization process, we expand the textual knowledge sub-graph and visual knowledge sub-graph by concatenating the nodes C//C, of the textual/visual knowledge sub-graph with the textual features 2t, and then compute their edges (i.e., the correlation between different nodes) as:
C₁ = Col,
Cvt = 122, Col,
sim Ci) Ext = sim 22) E
1
sim((e,Co))
Eu sim 2c)
E,
In this way, we can warp the textual feature to the dual knowledge graph space, which provides the basis for interaction and knowledge transfer between the textual feature and the dual knowledge graph. Then, the Graph Convolution Network (GCN) gte and Jot are utilized to excavate the knowledge for adapting the texture feature from each node textual and visual knowledge sub-graphs:
C+e = gee(Cue Ers = WWnCCWWu). C. (Cre Evs) Ce
where &.. Der and E.t- Dur are the adjacent matrix for graph learning. o is an activation function. And the Drc and Dut are the matrices used for Laplace normalization [29 for the edges as:
K (Evt + D))
Du diag( ((E
(En + E(n++D)) Dp), D., diag >
The meaning of Eq. 4 is seeking the prior knowledge from other nodes in one graph refine itself based on the correlation (i.e., the adjacent matrix). And thus, we can obtain the adapted/refined textual feature Zи Cū, [ and Zut [.. To fuse the inner-modality and cross-modality structure knowledge, we introduce one hyper-parameter 3 to fuse these two features as 2 = 3*2ee+ +(1 (1-9) -- +z Then the feature adapter is achieved in residual form, where the hyper-parameter is used to adjust the weights of prior knowledge from the dual knowledge graph and original feature: 27 Q2 a)
In the whole training process, only the parameters of GCN Iu and Jve are trainable under the constraint of the cross entropy function between the label of samples and the predicted label Eq.
4 Experiments
4.1 Datasets and Implementation Details
Datasets. Following previous adapter-style studies 118 82 86). we validate our GraphAdapter
on 11 few-shot classification tasks, including ImageNet 12 StandfordCars (30). UCF101 [63].
Caltech101 177. Flowers 102 [5T]. SUN397 (75). DTD M. EuroSAT 21. FGVCAircraft 48].
OxfordPets 53 and Food101 3 Among them, OxfordPets, Food101, StanfordCars, Flowers 102,
and FGVCAircraft belong to fine-graine grainer classification tasks, EuroSAT is for remote sensing classifi- cation, and DTD is the dataset of texture classification. To investigate the generalization capability of our GraphAdapter, we follow the CoOp [89 and conduct experiments on ImageNetV2 [58], ImageNet-Sketch (68), ImageNet-A 233 and ImageNet-R (22..
Implementation details. If not mentioned, we set the pre-trained backbone ResNet-50 of CLIP [9] as the base layer to produce the visual feature. To estimate the applicability of our method, we also

---

Average over Betosets
Catech30t
743-
193
673-
623
Onty
hecdar
632
NGVCArcuuf
Finweedi
Scae
R
12
Sritfe 35-
Tokmes Tip
broaed engouuppi
Bubber usaed www.eeareeeeeoecooommmmmm clss
Surrer tradiin cur
Fos1301
Irageder:
Oxfordeess
Sea
04
Tp Tp-Adapton"
Humber lracted ming coumple StanfordCars
Sunder trinnng aua
ever taindd
SINDO
UCF301
70-
Tack
Seiffes
Trrdee
Curf
Numbac chibe o0 ligg cumpla par cias
Humbor Vincee wng.ourck
Handerr vann esampias cass
Figure 3: The performance comparison of our Graph with the state-of-the-an methods few-shot learning, including 1-12-44/8-/16-shooss on 11 benchmark datasets. We provide all numerical results of this figure in the Supplementary.
apply our GraphAdapter to other CLIP visual encoders, including ResNet- 101 20), VIT-B/32 [4]. and ViT-B/16 [14,, and validate effectiveness on ImageNet. We optimize our model for 100 epochs for 2, 4, 8, and 16-shots. In the training process, we utilize the Adam optimizer with an initial learning rate of -3, which drops with the cosine learning rate decay schedule. Notably, to achieve stable training, where the small learning rate Le - with the few-shot training samples. Before the training, we exploit the visual encoder of pre-trained CLIP to extract the visual features for these few-shot samples from the same class and average them the node of this class. The number of images used for each class is decided by the number of shots. For example, for the 2-shot task, two images are used for each node. The data augmentation strategy only contains "random resized cropping" and "random flipping".
follow previous works [82 89] and utilize the warmup strategy for the training, - is applied at the first epoch. The visual sub-graph is constructed
4.2 Comparisons with State-of-the-arts
Few-shot learning. We compare our proposed GraphAdapter with several state-of-the-ar works on ETL, including Zero-shot CLIP (55). CoOp 189), clip-adapter (18). Tip-Adapter-F [86), and
7

---

Table 1: The performance comparison in terms of generalization capability on four CLIP visual backbones. The ETL methods are optimized with the ImageNet dataset on 16-shot setting and tested on cross-domain datasets, including ImageNet- V2, -Sketch, -A, and
Source
Target
Method
Backbone
ImageNet
-V2
-Sketch
-A
-R
Average
Zero-shot CLIP 555
58,18
51.34
33.32
21.63
56.00
40,58
Linear Probe CLIP [51
55.87
45.97
19.07
12.74
28.16
28.16
CoOp [89
62.95
55.11
32.74
22.12
54.96
41,23
ResNet-50
TaskRes 82
64.75
56.47
35.83
22.80
60.70
43.95
Ours
65.70
56.40
34.50
21.88
58.94
42.93
Ours,
64,94
56.58
35.89
23.07
60.86
44.10
Zero-shot CLIP 55
61.62
54,81
38.71
28.05
64.38
46.49
Linear Probe CLIP 55
59.75
50.05
26,80
19.44
47.19
35.87
CoOp [9
66.60
58.66
39.08
28.89
63.00
47.41
ResNet-101
TaskRes
82
67.70
59.50
41.70
29.87
68.07
49.79
Ours
68.23
59.60
40.83
28.77
67.13
49.08
Ours,
67.87
59,50
41,60
30,00
68.10
49.80
Zero-shot CL.PP 155
62.05
54.79
40.82
29.57
65.99
47.79
Linear Probe CLIP 55
59.58
49.73
28.06
19.67
47.20
36.17
CoOp 89
66.85
58.08
40.44
30.62
64.45
48.40
ViT-B/32
TaskRes 82
68.20
59.20
42.50
31.43
69.33
50.62
Ours
68.80
59.00
41.70
29.57
68.67
49.74
Ours
68.47
59.10
42.70
31.73
69.43
50.74
Zero-shot CLIP 55
66.73
60.83
46.15
47.77
73.96
57.18
Linear Probe CLIP 55
65.85
56.26
34.77
35.68
58.43
46.29
CoOp [9
71.92
64.18
46.71
48.41
74.32
58.41
ViT-B/16
TaskRes 82
73.07
65.30
49.13
50.37
77.70
60.63
Ours
73.68
65.57
48.57
49.23
77.20
60.14
Ours,
73.40
65.60
49.23
50.57
77.73
60.78
TaskRes 82 on benchmark datasets. The experimental results are shown in Fig. where we can
observe that GraphAdapter consistently outperforms previous ETL works for 1-72-14-/-/116shhots on the average performance of 11 benchmark datasets. Particularly, on the 16-shot setting, our
GraphAdapter achieves an average performance of 76. 76.22%. which exceeds the Tip-Adapter-F [86] by
0.57%, and TaskRes [82] by 1.12%. Even for the most challenging dataset FGVCAireraft [48] of
the fine-gra sine-grained classification, our GraphAdapter still performs better than the second-best method Tip-Adapter-F by 1.01 % 16-shot setting, since our Graph/dapter can better exploit the structure knowledge the downstream tasks with the dual knowledge graph.
Generalization. We also investigate the generalization capability of our GraphAdapter on four commonly-used datasets i.e., ImageNet- (58). ImageNNet-Skttch (68), ImageNet-A 23 ImageNet- (22). with different visual backbones, including the pre-trained ResNet- (2) ViT-B/16 [3]
ViT-B/32 [13). The experimental results are shown in Table T where provide two versions "Ours," " and "Ours" of our GraphAdapler for fair comparison. Here, "Ours" denotes the version of GraphAdapter used in few-shot learning. Since the over-fitting on few-shot learning will decrease the generalization capability, we increase the weights a 0.8 of base textual features in the adapted feature 27 as "Oursy" to obviate the wer-fitting on ImageNet Dataset. From Table our GraphAdapter achieves the optimal generalization capability on four cross cross-domain datasets, which outperforms TaskRes [82] by 0.15% on ResNet-50 20). Meanwhile, it obtains the best performance on the source data ImageNet.
4.3 Ablation Studies
The effects of the different variants of GraphAdapter. It is noteworthy that our GraphAdapter can be applied to the textual and visual adapter by introducing fused visual and textual structure knowledge. Therefore, we investigate the effectiveness of another two variants of our GraphAdapter, i.e., GraphAapter-I, and GraphAdapter- T&I. Here, we denote our Graph/dapter in our paper as GraphAdapter-TT since it is used to achieve the feature adapter of the textual branch. GraphAapter-I denotes that we introduce the GraphAdapter into the visual branch, which adjusts the visual feature to adapt the downstream tasks. TraahhAdapter-TTe T&& represents that the GraphAdapter is exploited in the visual and textual branches simultaneously. As shown in Table we can have the following conclusions: 1) the GraphAdapter is more suitable for the textual branch, which outperforms that in

---

Table 2: The ablation study for different variants of our GraphAdapter, where GraphAdapter T, GraphAdapter-I denotes integrating our proposed GraphAdapter into the textual branch and visual branch, respectively. GraphAAapter--T& utilizes the GraphAdapter- and GraphAdapter- simultaneously.
FGVCAirzzaR
StanfordCars
ImageNet
Caltech101
OxfordPets
Flowers 102
Food
SUN397
FaroSAT
UCF101
Average
DTD
Dataset
Method
ClIpe-dapter
18
63.59
92.49
87.84
74.01
93.90
78.25
32.10
69.55
65.96
84.43
76.76
74.44
CoOp 89
62.95
91.83
87.01
73.36
94.51
74.67
31.26
69.26
63.58
83.53
75.71
73.42
TaskRee
82
64.75
88.10
74.93
96.10
78.23
33.73
70.30
67.57
82.57
76.87
75.10
Tip-Adapler
86
65.44
92.63
88.18
75.75
94.23
78.11
35.86
71.00
66.94
84.94
79.03
75.65
Base
58.18
86.29
85.77
55.61
66.14
77.31
17.28
58.52
42.32
37.56
61.46
58.77
GraphAdapters
65.70
93.33
88.57
76.23
96.23
78.63
36.87
71.20
67.57
85.27
78,80
76.22
GraphAdapter-I
63.50
92.10
85.86
71.81
93.24
74.55
30.90
67.39
61.75
78.79
74.40
72.21
GraphAdapter-
65.72
88.59
76.20
96.13
78.40
71.30
85.55
78.67
76.25
Table 3: The ablation studies for two coeffi-
cients a and
0.60
Leumable
CoOp
ImagoNet 64.30
65.1
63.50
63.70
64,95
64.35
CLihAdaster
GrapnAdapior
i8 = 0.7)
0.3
0.6
0.7
08
0.9
Leamable
RN50
RN10L
VIT8/22
viTéas
ImageNet 65.30
65,70
65.37
64.94
63,03
65,42
Figure 4: Comparisons on different backbones.
the visual branch by an average of 4.01% on 11 benchmark datasets. 2) Applying the GraphAdapter in both branches of CLIP only achieves slight gain compared with the GraphAdapter-TT. Therefore, we utilize the GraphAdapter-II in our paper.
The effects of the dual knowledge graph. To investigate the effectiveness of the dual knowledge graph, we conduct experiments on ImageNet by removing the textual knowledge sub-graph (i.e., 1.0), respectively. The experimental
setting 0.0) and visual knowledge sub-gu sub-graph (i.e., setting results are shown in Table We can observe that removing the textual knowledge sub-graph achieves an accuracy of 64.30%, while the performance of removing the visual knowledge sub-graph drops to 64.95%, which reveals that textual structure knowledge is more important for textual feature adapter. Notably, the cooperation of textual and visual structure knowledge can achieve the best performance of 65.70%. This validates the effectiveness of introducing inner-modality and cross- cross-modality structure knowledge for adapter-style tuning.
The effects different coefficients a and B. There are two hyper-parameters a and our GraphAdapter, where the controls the balance of basic textual features from CLIP and the adapted textual feature obtained with GraphAdapter, and 3 is responsible for the weights of textual knowledge graph and visual knowledge graph. As shown in table the performance of GraphAdapter increases first and then drops when the coefficients a and 99 increase. The optimal / and a are 0.7, and 0.6, respectively. We also investigate whether a learnable coefficient can further improve the performance of our GraphAdapter. Experimental results show that the learnable coefficients achieve lower performance with selected fixed coefficients and B, since the coefficients hard to be optimized with the few-shot samples.
The effects of different backbones. As stated in Fig. we also evaluate the effectiveness of our GraphAdapter on different CLIP visual backbones, including ResNet-50, 10sNet-101, ViT-B/32, and ViT-B/16. Our GraphAdapter consistently exceeds previous works for both four visual backbones.
4.4 Complexity Analysis
We compute and compare our methods with existing published efficient transfer learning methods on the ImageNet [12 with the 16-shot setting, from the perspectives of tunable parameters, computational flops, training time, and inference time. All results are measured with the officially released code from GitHub. The experimental results are shown in the Table 4 We can observe our tunable parameters are still less than Tip-Arapte--F [66). For computational flops, our Graphe takes about 5.42 GFlops, almost the same Tip-Adapter-F and TaskRes (82). which is far lower than CLIP-Adapter.

---

Moreover, our training and inference times are less than the adapter- adapter-based work CLIP- and the prompt tuning method CoOp [9]. Therefore, our GraphAdapter satisfies the requirement of efficient transfer learning. Moreover, our GraphAdapter can achieve the best performance.
Table 4: A comparison between GraphAdapter and existing methods on time and model complex-
ity.
Methods
CoOp 89
CLIP-Adapter
18
Tip-Adapter-F
86
TaskRes
82
Ours
Tunable Purameters (M)
0.008
0.524
16.384
1.024
4.145
GFlops
1943.12
1959.44
5.43
5.42
Training time (one epoch) (s)
40.91
45.71
12.36
13.64
23.29
Inference time (s/100)
119.64
275.22
51.03
4.89
4.91
Memory Cost (Training)
18.907
9.257
4.313
6.227
10.75
Memory Cost (Inference)
7.403
7.615
4.161
6.225
4.433
Performance
62,95
63.59
65.44
64.75
65.70
4.5 Applicability
To validate the applicability of GraphAdapter, we select two state-of-the-art adapter-style works, including CaFo [85] and TaskRes* [82]. Here, CaFo [85] incorporates diverse prior knowledge from large pre-trained vision and language models, including DINO's vision-contrastive knowledge, GPT-
3's language-generatinn
knowledge, and DALLE's generative capability. The adapting strategy of CaFo [85] is from the Tip-Adapter [86). The TaskRes* denotes the enhanced version of TaskRes [82
which exploits the enhanced base classifier instead of the original classifier from CLIP 55
For CaFo [85) directly incorporate GraphAdapter into the textual classifier. For TaskRes [22 we replace the task residual with our proposed GraphAdapter and maintain its enhanced textual branch from CLIP. The experimental results ImageNet [12] shown in T Appendix. We can observe that our GraphAdapter can consistently increase the performance of CaFo [85] and TaskRes* [82) few-shot learning with all 112-/4-/8-/16-shor settings. Particularly, the 16-shot setting, ours improves CaFo [85] by 9. and TaskRes* by 1.15%, which validates the powerful applicability of GraphAdapter. Overall, our GraphAdapter is complementary these prior-augmented methods, and can obtain better performance by integrating ours inlo them.
Conclusion
In this paper, we comprehensively review the limitations previous adapter-style tuning methods in the low-data regime as 1) previous works only model the task-specific knowledge with a single modality, and 2) overlooking the exploitation the structure knowledge (i.e., the relation different semantics/classes) in downstream tasks, which is vital for data-efficient tasks. Based on the anal- ysis, we propose a brand-new sttapter-style tuning strategy for visual-language models, termed as GraphAdapter, which introduces the dual knowledge graph to establish the structure knowledge in the textual and visual modalities. By incorporating the fused textual and visual structure knowledge with Graph Convolutional Network (GCN), the text-based classifier effectively with the inner- modalttyy and cross cross- modality structure knowledge. Extension experiments on 11 benchmark datasets revealed the effectiveness of our GraphAdapter few-shot learning and generalization.
be adapted to downstream tasks
Limitations and Broader Impacts. The limitation of our GraphAdapts stems from the textual structure knowledge modeling. In this paper, we utilize the textual features of default prompts like "a photo of [class]" to construct the nodes of the textual graph. However, the prompts are simple and lack enough diversity. We believe more diverse and accurate prompts for downstream tasks can achieve better modeling for textual structure knowledge, which can further improve the performance of our GraphAdapter.
The adapter-style tuning of VLMs aims to efficiently finetune the VLMs for downstream tasks by optimizing few parameters in low-data regime. The possible broader impact our GraphAdapter stems from the tuning of VLMs itself, which has a heavy dependency the pre-trained VLMs. The utilization of our GraphAdapter should follow the privacy and safety of datasets and pre-trained models.
10

---

Acknowledgement
This research supported by the National Research Foundation, Singapore, under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-033) and NSFC under Grant U1908209, 62021001.
References
[1] H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei. Vlmo: Unified vision-language pre-training with mixture-of- -oa-tuue----mmddlll--eeeeeeeee Advances in Neural Information Processing Systems, 35:32897-32912, 2022.
[2] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[3] L. Bossard, M. Guillaumin, and Van Gool. mood-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Purt 13, pages 446-461. Springer, 2014.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[5] M. Caron, H. Touvron, I. Misra, Jégou, J. Mairal, Bojanowski, and A. Joulin. Emerging properties in self- vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[6] G. Chen, W. Yao, X. Song, X. Li. Y. Rao, and K. Zhang. Plot: Prompt learning with optimal transport for vision- laiiunla models. In The Eleventh International Conference on Learning Representations.
[7] R. Chen, T. Chen, X. Hui, H. Wu, G. Li, and L. Lin. Knowledge graph transfer network for few-shot recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 10575-10582, 2020.
[8] T. Chen, M. Xu, X. Hui, H. Wu, and L. Lin. Learning semantic-specific graph representation for multi-label image recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 522-531, 2019.
[9] T. Chen, W. Yu, R. Chen, and L. Lin. Knowledge-embedded routing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6163-6171, 2019.
[10] C.-Y. Chuang, V. Jampani, Y. Li, A. Torralba, and S. Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.
[11] M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2014.
[12] Deng, W. Dong, R. Socher, L.-J. K., Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[13] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2020.

---

[15] B. Du, C. Yuan, R. Barton, T. Neiman, and H. Tong. Hypergraph pre-training with graph neural networks, arXiv preprint arXiv:2105.10862 2021.
[16] G. Fang, X. Ma, M. Song, M. B. Mi, and X. Wang. Depgraph: Towards any structural pruning. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
[17] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training ex- amples: An incremental bayesian approach tested on 101 object categories. 2004 conference on computer vision and pattern recognition workshop, pages 178-172 IEEE, 2004.
[18] Gao, S. Geng, R. Zhang, T. Ma, R. Fang. Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.0454, 2021.
[19] S. Goyal, A. Kumar, S. Garg, Z. Kolter, and A. Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint arXiv:2212.006336 2022.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[21] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266 2019.
[22] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Para- juli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340 8349, 2021.
[23] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[24] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-27999 PMLR, 2019.
[25] T. Huang, J. Chu, and F. Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXivv2204.0364 03649,444 2022.
[26] T. Huang, S. You, F. Wang, C. Qian, and C. Xu. Knowledge distillation from stronger teacher. Advances in Neural Information Processing Systems, 35:33716 (11116-3377,22 2022.
[27] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[28] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan. Maple: Multi-modal prompt learning. arXiv preprini arXiv:2210.033 2022.
[29] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: (etp:::69900000 00009,0290,.55 2016.
[30] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine- Ane-geained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[31] N. Kumari, B. Zhang, S.-Y. Wang, E. Shechtman, R. Zhang, and J.-Y. Zhu. Ablating concepts in text-to-image diffusion models. arXiv preprint arXiv:2303. 13516,, 2023.
[32] N. Kumari, B. Zhang. Zhang, E. Shechtman, and J.-Y. Zhu. Multi-concept customization of text-to-image diffusion. arXiv preprint 2rXiv:2212.04488, 2022.
12

---

[33] D. Lai, M. V. Nguyen, T. H. Nguyen, and F. Dernnncour.. Graph learning regularization and transfer learning for few-shot event detection. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 2172-2176, 2021.
[34] I.. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant bascline for vision and language. arXiv preprint arXiv:1908.0355: 2019.
[35] R. Li, X. Yuan, M. Radfar, P. Marendy, W. Ni. T. J. O'Brien, and P. M. Rasillas-Espinees Graph signal processing, graph neural network and graph learning on biological data: a systematic review, IEEE Reviews in Biomedical Engineering, 2021.
[36] X. Li, X. Jin, J. Fu, X. Yu, B. Tong, and Z. Chen. Few-shot real image restoration via distortion- relation guided transfer learning. arXiv preprint arXiv:2111. 2021.
[37] D. Lian, D. Zhou, J. Feng, and X. Wang. Scaling & shifting your features: A new baseline for efficient model tuning. In Advances Neural Information Processing Systems (NeurIPS), 2022.
[38] Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 55(9):1-35, 2023.
[39] S. Liu, K. Wang, X. Yang. J. Ye, and X. Wang. Dataset distillation via factorization. In Advances in Neural Information Processing Systems, 2022.
[40] S. Liu, J. Ye, R. Yu, and X. Wang. Slimmable dataset condensation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[41] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[42] J. Loedeman, M. C. Stol, T. Han, and Y. M. Asano. Prompt generation networks for efficient adaptation of frozen vision transformers. arXiv preprint arXiv:2210.064666 2022.
[43] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-lane lan-lagg tasks. Advances in neural information processing systems, 32, 2019.
[44] Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian. Prompt distribution learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages $206-5215, 2022.
[45] Z. Lu, S. He, Li, Y.-Z. Song, and T. Xiang. Prediction calibration for generalized few-shot semantic segmentation. IEEE Transactions on Image Processing, 2023.
[46] Z. Lu, S. He, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8741-8750, 2021.
[47] X. Ma. G. Fang, and X. Wang. LLM-Pruner: On the Structural Pruming of Large Language Models. In Advances in Neural Information Processing Systems, 2023.
[48] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft, arXiv preprint 10Xiv:1306.517 315.1.11..11.... 2013.
[49] S. Monka, L. Halilaj, and A. Rettinger. A survey on visual transfer learning using knowledge graphs. Semantic Web, 13(3):477-55 2022.
[50] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074 2021.
13

---

[51] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pages 722-729. IEFE, 2008.
[52] O. Pantazis, G. Brostow, K. Jones, and O. Mac Aodha. Svl-adapter: Self-supervised adapter for vision-language pretrained models. arXiv preprint arXiv:2210.03799. 2022.
[53] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[54] O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2094, 2021.
[55] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
[56] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
[57] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, Zhou, and J. Lu. Denseclip: Language- guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10882110091,00
[58] B. Recht, R. Roelofs, L.. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[59] A. Rezvanian and M. R. Meybodi. Stochastic graph as a model for social networks. Computers in Human Behavior, 64:621 2016.
[60] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[61] Seale Smith, Y.-C. Hsu, I.. Zhang. T. Hua, Z. Kira, Y. Shen, and H. Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv e-prints, pages Xiv- 2304, 2023.
[62] M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test- time prompt tuning for zero-shot generalization in vision-language models. arXiv preprint 0tt:2209900511.. 2022.
[63] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.000, 2012.
[64] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08538 2019.
[65] Y.-.. Sung. J. Cho, and M. Bansal. Vl-adapter: Parameter-efficient transfer learning for vision- and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5237, 2022.
[66] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprin! 10XXvv/900.077
2019.
[67] G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or. Motionclip: Exposing human motion generation to clip space. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 358-374. Springer, 2022.
[68] H. Wang. S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[69] M. Wang, Xing, and Y. Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint 27Xiv:2109.08472, 2021.
14

---

[70] P. Wang, R. Panda, and Z. Wang. Data efficient neural scaling law via model reusing. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors. Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 36193-36204. PMIR, 23-29 Jul 2023.
[71] X. Wang, S. Zhang, J. Cen, C. Gao, Y. Zhang, D. Zhao, and N. Sang. Elip-guided prototype modulating for few-shot action recognition. arXiv preprint arXiv:2303.02982, 2023.
[72] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
[73] S. T. Wasim, M. Naseer, S. Khan, F. S. Khan, and M. Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. arXiv preprint arXiv:2304.03307, 2023.
[74] H. Wu, J. Yin, B. Rajaratnam, and J. Guo. Hierarchical relational learning for few-shot knowledge graph completion. arXiv preprint SSKiv:22000/2005 2022.
[75] Xiao, Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[76] G. Xie, X. Li, X. Pan, and Z. Chen. Diagnosis-oriented medical image compression with efficient transfer learning. arXiv preprint arXiv:2310. 13250, 2023.
[77] W. Xiong. M. Yu, S. Chang, X. Guo, and W. Y. Wang. One-shot relational learning for knowledge graphs. arXiv preprint /-OXXvv::88..... 09040, 2018.
[78] R. Xu, Liu, X. Lu, K. Zhang, and W. Liu. Dmh-fsl: learning. Neural Processing Letters, 54(2):1377-332, 2022.
hypergraph for few-shot [79] X. Yang, J. Ye, and X. Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision, 2022.
[80] X. Yang, D. Zhou, S. Liu, J. Ye, and X. Wang. Deep model reassembly. In Advances in Neural Information Processing Systems, 2022.
[81] Y. Yang. J. Qiu, M. Song, D. Tao, and X. Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[82] T. Yu, Z. Lu, X. Jin, Z. Chen, and X. Wang. Task residual for tuning vision-languar models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[83] Zhang, J. Huang. S. Jin, and S. Lu. Vision-language models for vision tasks: survey. arXiv preprint 2304:2330000088, 2023.
[84] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.
[85] R. Zhang, X. Hu, B. Li, S. Huang, H. Deng, H. Li, Y. Qiao, and P. Gao. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. arXiv preprint N:Xiv:2303.02151 2023.
[86] R. Zhang,
W. Zhang, R. Fang, P. Gao, K. J. Dai, Y. Qiao, and H. Li. Tip-adapter: Training- free adaption of clip for few-shot classification. In Computer EEsi 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 493-510. Springer, 2022.
[87] Y. Zhang, H. Fei, D. Li, Yu, and P. Li. Prompting through prototype: A prototype-based prompt learning on pretrained vasion-language models. arXiv preprint arXiv:2210. 2022.
[88] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision vision-language models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16825,, 2022.
15

---

[89] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[90] X. Zhu, R. Zhang, B. He, A. Zhou, D. Wang, B. Zhao, and P. Gao. Not features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint arX1v:2304.01195, 2023.
16

---

A Applicability
Table 5: The experiments for the applicability of our GraphAdapter. For Cafo 85 we incorporate
our GraphAdapter into the textual classifier. Notably, the Task exploits the enhanced base classifier. Therefore, TaskRes* + Ours denotes that TaskRes* replace the task residual with our proposed GraphAdapter.
Methods
1-shot
2-shot
4-shot
8-shot
16-shot
CaFo [85]
63,80
64.34
65.64
66.86
68.79
+Ours
63.81
64.97
66.17
67.68
69,30
TaskRes* [82
61.43
62.17
62.93
64.03
64.75
+Ours
61.73
62.53
63.47
64.57
65.80
B More Experimental Results
We present the numerical results of "Figure 3 in the main text" as Table 6 We compare our GraphAdapter with the state-of-the-ar works, including the prompt-based method CoOp [9), and adapter-style methods, i.e., CLIP-Adapte [18), Tip-Adapter-F [86). and TaskRes [22) Here, the performance of CoOp 189.. From the table, we can find that the 16-shot few-shot learning, our GraphAdapter outperforms all previous works except for UCF101 663 where its performance comparable. Depart from that, for the average accuracy of 11 benchmark datasets in the 1-/2-/4-/8- 1/2-/4-/8-/16-shoe few-shot learning, our GraphAdapter surpasses previous works with a consistent improvement of 0.57% to 0.76%. We also make the analysis for the Error Bars by providing the standard deviation (Std) of our experimental results in Table 6
is reproduced by [82]. which aims to ensure a fair comparison with
Table 6: A numerical comparison between our GraphAdapter and the state-of-the-an methods.
PGVCAireaa
Caltechl01
Plowzos
Oxforuleas
StamfndCaas
EuOSAT
Fovdl01
ImageNes
SUN397
UCPI01
Methods
Setting
Avg.
Zene-shot CLIP 55
86.29
42.32
37.56
17.28
66.14
77.31
58.18
85.77
55.61
58.52
61.46
$8.77
CoOp (89)
87.53
44.39
50.63
9.64
68.12
74.32
57.15
85.89
55.59
60.29
61.92
$9.59
88
76.82
55.13
62.67
Tip-Adapter.F
81.17
76,
56.78
66.19
Taskkes
88.80
50.17
61.27
21.20
78.77
74.03
61.43
83.50
58.77
61.93
64.57
64.04
Ours (w) Std)
88.90
$1.77
63.30
20.93
79.98
75.43
61,50
84.40
59.70
61.93
64.93
64.80
(d:1
(20.26)
(20.33)
Zem-shet CLIP
86.29
42.33
37.56
CoOp
87.03
61.50
72.44 49
58.28
CLIP-Adapter
1
89.37
51.48
63.90
20.10
81.61
77.22
61.52
86.73
58.74
63.29
67.12
65.55
Tip Vip-Adapter
86
2-shot
89.61
55.32
64.76
21.76
85.40
77.05
61.57
86.06
61.13
63.19
68.99
66.80
Taskiee
90.13
54.53
65.77
23.07
85.63
75.30
62.17
$4.43
62,77
61.33
69.10
67.02
86,30
(=0.22)
(+:1 .6)5
(1:0.65
(=0.25)
12)
(1:0.17)
(500 00)
(60.17)
(=0033)
(10.02)
31)
Zen-shot CLIPP
H6.29
42.32
37.56
17.
66.11
77. 31
58.18
85.77
55.61
$8.52
/4
SK 77
CoOp [8]]
89.55
$3.49
70.18
211 87
86.20
73.33
$9.99
86.70
62.62
63.47
67.03
66.77
Adagter
89.98
36
73.38
22.
87.17
77.92
$7.46
62.45
65.96
69.05
68.61
Tip-Adapter.F
89.53
77.46
Tuskiees
90.63
50
72.97
24.83
89.50
76.23
62.93
86.27
66.50
66.67
69.70
69.61
Ours (wl Stdi
90.97
59.63
75.20
26.97
89.90
76.77
63.12
86.57
66.53
66,70
71.47
70.35
(=0.005
(db0 38)
(=:1.7)
(#:0 21)
(E:1.19)
(360.22 26)
(#:0.19)
(#:1.
(Ei0.29)
(=0.28)
(6:0.16)
(=00.27)
Zero-shot CLIP
86.29
42.32
37.56
17.28
66.14
77.31
85.77
55.61
58.52
61.46
58.77
CoOp
76.73
71.82
CLIP-Aulpter
Tip-Adapier-
8-shot
91.70
91.10
62.93
61.00
77.93
79.33
26 25
30.62
91.00
91,72
77:90
78.04
64.15
62.68
88.25
87.65
69.3 51
67.89
69.23
67.50
73.30
74.76
71.10
72.67
ThckRes h0
92.23
(4..33
78.07
29.35
94,30
76,90
64.113
87.07
70.57
68.
74.77
73.76
92.45
77.73
70.53
75.73
73.40
(=00.88)
(+00.44)
(=1.87)
(#:0.00
(=:0.12)
18)
(+0008)
26)
(6:0.12)
(#00. 12)
(4:0.45)
(=00 200
Zero-shot CLIP
86.29
42.32
37.56
17.28
66.14
77.31
58.18
85.77
55.61
$8.52
61.46
58.77
CoOp (89)
91.83
63.58
83.53
31.26
94,51
14.67
62.95
87.01
73.36
69.26
75.71
73.42
92.19
65.96
81.13
32.10
93.90
78.25
63.59
87.84
74.01
69.55
76.76
71.41
Tip-Adapter-F
92.63
78.
75.75
75.65
TaskRes
07.90
X7.57
33.73
96.10
7%
64.75
74,93
70.30
76.87
75, 10
Ours (wl Sul
93.33
67.57
85.27
36.87
96.23
78.63
65.70
88.57
76.23
71.20
78.80
76.22
(200088)
(..0.00)
(.0.29)
(1.0.00)
(10.15)
(1.0.8)
(10.08)
(1.0.1)
(100177)
(10.008)
(10.25)
(20.111
17

---

C More Dataset and Implementation Details
More Dataset Details. In this paper, we follow previous works, e-g., CoOp CLIP-Adapter [8].
TaskRes 82. and Tip-Adapter [86], and exploit the prompts in Table 7 for the tuning and testing.
More Implementation Details. Our experimental results achieved by running the algorithm three times with different seeds for each setting. The training and inference are implemented with single NVIDIA GeForce RTX 3090. In the implementation of GraphA for the ImageNet [12], we decouple the sub-graph with 1000 nodes for each modality into four graphs with 256 nodes to alleviate the computational cost.
Table 7: The number of classes and used prompt temple for each dataset.
Datasets
# Classes
Prompt Templet
Caltech101 17
100
"a photo of [class]."
DTD [1
"[class] texture."
EuroSAT
21
10
"a centered satellite photo of [class]."
FGVCAireraft
48
100
"a photo of [class], type of aireraft."
Flowers102 51
102
"a photo a [class] type of flower."
Food101 [
101
"a photo [class], type of food."
OxfordPets
533
37
"a photo of [class], a type of pet."
StanfordCars
30
196
"a photo of [class]."
SUN397 75
397
"a photo of [class]."
UCF101
03
101
"a photo of person doing [class]."
Ensemble 7 selected templates, including "itap of [class].", "a bad photo the [class].", "a origami [class]"" "a photo of the large [class]., ","" "a [class] in a video game.", "art the [class]]" and "a photo of the small [class]."
ImageNet 12
1000
D
Visualization of Graph Nodes
To demonstrate how our GraphAdapter works for the adapter-style tuming for VLMs, we visualize the graph nodes for textual features before and after the Adaper. As shown in Figure 5 we randomly sampled 20 classes from ImageNet 122 and utilize the t-SNE visualize the distribution of each node corresponding to the textual fracture for classification. We can observe that with our GraphAdapter, the nodes of different classes move in directions that lead to much larger inter-class distances, thereby improving the performance of adapter-style tuning for VLMs.
18

---

Base Classifier GraphAdapter
2
0-
-2
-4
-6
10
12
Figure 5: Visualization of the variance the graph nodes before and after GraphAdapter. Each node represents the representation of one class. We randomly sampled 20 classes from ImageNet for better visualization. The nodes move toward the direction that leads to much larger inter-class distances after GraphAdapter. The red arrows denote the directions.
19


Reviews:
[{"review_id": "Czjn8wxSP3", "reviewer": "Reviewer_hUWw", "summary": "In this paper, the authors present an adapter-style tuning method, termed as GraphAdapter, that explicitly captures the dual-modality structure knowledge by utilizing a dual knowledge graph, leading to enhanced adapter-style transfer learning. Specifically, the authors identify two key challenges in existing adapter-style approaches for efficient transfer learning, including the deficiency in modeling task-specific knowledge from only a single modality, and the neglect of explicitly exploiting the structural knowledge in downstream tasks. Motivated by these two limitations, the authors propose a novel tuning method for visual-language models that incorporates task-specific knowledge for downstream tasks through the integration of textual and visual structure knowledge, based on graph learning. In particular, the proposed method establishes a dual knowledge graph consisting of a textual knowledge subgraph and a visual knowledge subgraph. Consequently, the feature adapter can effectively leverage the inner-modality and cross-modality structure knowledge for superior tuning performance. Experiments on 11 benchmarks convincingly demonstrate the effectiveness of the proposed GraphAdapter approach.", "questions": "1. Can GCN in GraphAdapter be replaced with other GNNs like GAT and GraphSAGE?\n2. What is the time complexity associated with GraphAdapter?", "limitations": "The limitations have been thoroughly discussed in the main paper.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed GraphAdapter is very well-motivated. The authors have conducted a thorough analysis of the existing ETL approaches, identifying two key limitations, and recognizing the significance of incorporating dual-modality structure knowledge in ETL. Building upon such analysis, the authors develop GraphAdapter, which aims to effectively integrate fused textual and visual structure knowledge using GCN. \n\n2. The experiments are thorough and convincing. The authors perform extensive experiments on 11 few-shot benchmarks, utilizing various backbones such as ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16, as detailed in both the main paper and the supplementary material. In addition, the authors have specifically explored the generalization capability of GraphAdapter on four benchmarks. These experiments convincingly demonstrate the effectiveness of GraphAdapter.\n\n3. The paper is clearly written and easy to follow. The authors extensively elaborate on the details of GraphAdapter, particularly the establishment of the dual knowledge graph.\n\n4. The supplementary material includes extensive implementation details and visualization results. Notably, the authors demonstrate the seamless integration of the proposed GraphAdapter with existing methods such as CaFo and TaskRes*, resulting in consistently improved performance.", "weaknesses": "I’m almost satisfied with this paper, with only a few minor concerns as follows.\n\n1. The authors leverage GCN to integrate dual-modality structural knowledge. However, I am interested in understanding the performance of more advanced GNN mechanisms, such as GAT and GraphSAGE, in this context.\n\n2. The authors are suggested to include an analysis of the time complexity, particularly regarding the construction of the textual and visual knowledge subgraphs.\n\n3. Some related works [1, 2, 3] on KG embedding can be included.\n\n[1] Wu, Han, et al. \"Hierarchical Relational Learning for Few-Shot Knowledge Graph Completion.\" The Eleventh International Conference on Learning Representations. 2022.\n[2] Bordes, Antoine, et al. \"Translating embeddings for modeling multi-relational data.\" Advances in neural information processing systems 26 (2013).\n[3] Xiong, Wenhan, et al. \"One-shot relational learning for knowledge graphs.\" arXiv preprint arXiv:1808.09040 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive comment on our work, along with constructive suggestions for the improvement of our work.  \n\n**Q1:** The authors leverage GCN to integrate dual-modality structural knowledge. However, I am interested in understanding the performance of more advanced GNN mechanisms, such as GAT and GraphSAGE, in this context.\n\n**A1:** Thanks for your valuable comments. In order to delve further into this intriguing question, we replace the GCN in our GraphAdapter with GAT and GraphSAGE. Then we compare them on the Caltech dataset using a 4-shot setting. \nAs shown in the table below, the utilization of a more advanced graph neural network (GNN) mechanism (such as GAT and GraphSAGE) result in a slight improvement in performance. Meanwhile, it also brings more resource const. We will add this analysis in the revision. \n\nMethods| GAT | GraphSAGE | GCN |\n|:-:|:-:|:-:|:-:|\n|Caltech|91.21| 91.19| 90.97|\n\n**Q2:** The authors are suggested to include an analysis of the time complexity, particularly regarding the construction of the textual and visual knowledge subgraphs.\n\n**A2:** Thank you for your insightful suggestions. In response, we have incorporated an analysis of the time complexity for our GraphAdapter in the table below. This includes the training time for each epoch, the inference time for a batch of 100 images, and the associated costs for constructing both the textual and visual graphs. All the time complexity computations were performed with the ImageNet dataset on a 16-shot setting, using a single NVIDIA GeForce 3090. As shown in the table below, the time required for training and inference of our GraphAdapter is obviously less than the typical prompt-based ETL method CoOp and Adapter-based work CLIP-Adapter, which is efficient. Moreover, the textual and visual graphs need only be constructed once at the beginning of the training process. It means the construction of these two graphs is also fast and the cost is minimal. \n|Methods| Training time (one epoch) | Inference time | Textual Graph |Visual Graph| Performance|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|CoOp| 40.91s | 119.64s | -  | - | 62.95 |\n|CLIP-Adapter | 45.71s | 275.22 | - | - | 63.59|\n|Ours| 23.28s| 4.57s | 6.63s | 14.34s| 65.70|\n\n**Q3:** Some related works [1, 2, 3] on KG embedding can be included.\n\n**A3:** Thank you for your valuable suggestions. We will incorporate descriptions of these papers in the \"Graph Learning\" section of our related work.\n\n\n\n\n"}, {"review_id": "7gdjm66w8z", "reviewer": "Reviewer_hFHN", "summary": "The paper proposes to utilize graph learning for efficient transfer learning of large vision-language models. The graph learning consists of scene graphs of two different modalities, first one uses the textual features of prompts and the second one uses the visual features of training samples from the downstream tasks. The feature of each prompt aims to align with the the fused graph features during the optimization process. Experiments on 11 few shot classification benchmarks including a few on fine-grained classification tasks show that this method improves the transfer learning performance. \n", "questions": "This is a well written paper. I have one question though:\nIs there a way to utilize the semantics of visual relationships? The structured knowledge graph in its current form appear to be more of a co-occurrence statistics. e.g. How to distinguish between \"person holding a cup\" vs \"person drinking from a cup\" ?", "limitations": "Broader impacts and limitations are mentioned in the supplementary material, it's the common impact that large vision-language models need to be concerned about - the nature of the pretraining dataset. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea of using structured knowledge is intuitive. Using textual and visual both modalities are interesting. The section of different graph adapter variants, text, visual and T-V is worth adding. The ablation experiments of the different coefficients for different modality graphs have added value. The paper proposes an intuitive idea, explains it well and the paper is well written.   ", "weaknesses": "Not a weakness, but a general question. \nIs there a way to utilize the semantics of visual relationships? The structured knowledge graph in its current form appear to be more of a co-occurrence statistics. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of our work. We have given careful consideration in response to your insightful question. \n\n**Q1:** Is there a way to utilize the semantics of visual relationships? The structured knowledge graph in its current form appears to be more of a co-occurrence statistic. e.g. How to distinguish between \"person holding a cup\" vs \"person drinking from a cup\"?\n\n**A1:** Thanks for your positive comments and insightful questions. Our GraphAdapter, following the setting of CoOp, has been validated on 11 downstream benchmarks for classifications. We believe the semantics of the visual relationships you mentioned can further enhance classification accuracy. Intuitively, the utilization of the visual relationship should be particularly beneficial for fine-grained and multiple-object classification in one image. \n\nBased on your insightful question, we conduct a survey of related works and found that two particular directions are closely correlated with the visual relationship within a single image, such as distinguishing  \"person holding a cup\"\nand \"person drinking from a cup\". These two directions compose of human-object interaction[1, 2, 3] (HOI) and scene graph generation[4, 5] (SGG). Among them, human-object interaction focuses on identifying the relationship between humans and nearby objects.  Scene graph generation is designed to identify the relationship between different objects within a single image. Both of these two tasks typically involve first detecting the objects/humans, followed by classifying their relationship. \n\nThe above directions inspire us to consider two potential approaches to exploit the visual relationship: i) utilizing paired visual images and annotated language description of the relationship to guide both the CLIP textual classifier and the visual encoder, enabling them to be aware of the relationship. ii) detecting the objects within an image and modeling the relationship through graph learning as the SGG works. \n\nTwo potential challenges arise in the utilization of visual relationships.  \n i) Whether the annotated datasets for visual relationships are adequate for classification\n ii) how to design one efficient and effective scheme to exploit the visual relationships for classification. \n\nWe believe it is a very good direction to investigate in future work. \n\n[1] Gkioxari, Georgia, Ross Girshick, Piotr Dollár, and Kaiming He. \"Detecting and recognizing human-object interactions.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8359-8367. 2018.\n\n[2] Liao, Yue, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. \"Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20123-20132. 2022.\n\n[3] Park, Jeeseung, Jin-Woo Park, and Jong-Seok Lee. \"ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17152-17162. 2023.\n\n[4] Lin, Xin, Changxing Ding, Yibing Zhan, Zijian Li, and Dacheng Tao. \"Hl-net: Heterophily learning network for scene graph generation.\" In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19476-19485. 2022.\n\n[5] Tang, Kaihua, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. \"Unbiased scene graph generation from biased training.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3716-3725. 2020.\n"}, {"review_id": "aWofVavJJr", "reviewer": "Reviewer_pwW3", "summary": "This paper proposes a new prompt tuning strategy named GraphAdapter to fuse textual and visual structure knowledge for downstream tasks. It first constructs the dual knowledge graph by taking the textual / visual features of a specific class as nodes and the cosine similarities between these features as edges. After that, the textual feature $z_t$ is warped into the graph space and two GCNs are applied to excavate the knowledge from the textual and visual sub-graphs respectively. Extensive experiments on 11 benchmark datasets show that GraphAdapter consistently outperforms previous works.", "questions": "- In the few-shot training setting, what is the detailed process to obtain the visual features for constructing the visual sub-graph? For example, what is the number of images used here? what kind of data augmentation is applied?\n    \n- what is the overall cost of GraphAdapter compared to previous methods? For example, (a) The training FLOPs; (b) The training wall-time of one epoch; (c) The inference speed; (d) The number of parameters.\n    \n- (Minor) The reference figure in L171 is incorrect.", "limitations": "The limitations are partly addressed. The authors may need to further discuss the limitation of **scalability** as well.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of introducing graph learning into prompt tuning methods is novel.\n    \n- The average performance of GraphAdapter is better than previous methods on few-shot learning, including 1-/2-/4-/8-/16-shots on 11 benchmark datasets.\n    \n- Ablations also show that GraphAdapter can achieve consistent gains when using different backbones.", "weaknesses": "- My major concern is about the **efficiency**. Although modeling all the classes as a knowledge graph is a novel idea, introducing GNNs into prompt tuning are generally time and memory consuming and may violate the original motivation of the *efficient* transfer learning for Vision Language foundation models (VLFMs). In fact, as is mentioned in the appendix (L40), the authors already decouple the original graph of ImageNet (1k nodes) into 4 sub-graphs (256 nodes) to alleviate the computational cost. So the *scalability* of this method maybe limited.\n    \n- Besides that, one of the most important potential of VLFMs is trasferring to *open-world* scenarios, where the classes are unknown and infinite, which may make the graph impossible to build in advance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your great efforts and insightful questions. \n\n**Q1:** About the concern on efficiency and scalability. \n\n**A1:** Thanks for your valuable comments. We will answer the above questions from three perspectives.\n\n(i) As the definition of efficient transfer learning in VLMs, it contains two perspectives: 1) parameter-efficient transfer learning, and 2) data-efficient transfer learning. Our GraphAdapter satisfies these two principles like existing ETL methods since it only utilizes 4.145M learnable parameters and assists few-shot learning.\n\n(ii) To prove that our scheme is efficient, we compute and compare our methods with existing published efficient transfer learning methods on the ImageNet with the 16-shot setting,  from the perspectives of tunable parameters, computational flops, training time, and inference time. All results are measured with the officially released code from GitHub. The experimental results are shown in the table below. We can observe our tunable parameters are still less than Tip-Adapter-F. For computational flops, our GraphAdapter takes about 5.42GFlops, almost the same as Tip-Adapter-F and TaskRes, which is far lower than CLIP-Adapter. Moreover, our training and inference times are less than the adapter-based work CLIP-Adapter and the prompt tuning method CoOp.  Therefore, our GraphAdapter satisfies the requirement of efficient transfer learning.  Moreover, our GraphAdapter can achieve the best performance.\n|    | CoOp| CLIP-Adapter| Tip-Adapter-F| TaskRes| Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Tunable Parameters (M)| 0.008 | 0.524| 16.384 | 1.024| 4.145|\n|GFlops | 1943.12  | 1959.44 | 5.43 | 5.42 | 5.42| \n|Training time (one epoch) (s)| 40.91 | 45.71 | 12.36 | 13.64 | 23.29| \n| Inference time (s/100) | 119.64 | 275.22 | 51.03 | 4.89 | 4.91|  \nMemory Cost (Training) | 18.907| 9.257 | 4.313 | 6.227 | 10.75 | \nMemory Cost (Inference) | 7.403 | 7.615 | 4.161 | 6.225 | 4.433 |\n| Performance | 62.95 | 63.59 | 65.44 | 64.75 | 65.70 |\n\n(iii) For adapter-style efficient transfer learning methods in VLMs, the computational complexity will increase when the number of classes increases in the training process. For example, the complexity of the adapter process in CLIP-Adapter, Tip-Adapter-F, and TaskRes are all $O(kn)$, where $k$ is constant. In our GrapAdapter, **the number of parameters of GCN is invariant for any class number.** The increase in computational cost is primarily caused by edge computing. To reduce the complexity, we decompose the 1000 nodes in ImageNet into four 250 nodes for the trade-off of performance and computational complexity. Here, we give the theoretical derivation. If we have n class, we can divide it into several subgraphs with fixed $m$ nodes, and decrease the edge computing to $O(mn)\\sim O(n)$ when n is large since $m$ is constant like $k$. We also conduct experiments for different $m$ as the table below. We can find with the decrease of $m$, the performance only drops a little. \nTherefore, it owns almost the same scalability as existing adapter-style ETL methods, like TaskRes, CLIP-Adapter, and Tip-Adapter-F. \n| m | 20  | 50 | 100 | 250 | \n|:-:|:-:|:-:|:-:|:-:|\n|1-shot | 61.23 | 61.45 |61.47| 61.50| \n\n**Q2:** One of the most important potentials of VLFMs is transferring to open-world scenarios, where the classes are unknown and infinite, which may make the graph impossible to build in advance.\n\n**A2:** Thanks for your insightful and valuable comments. (i) First, our GraphAdapter follows the CoOp setting, which is followed by existing adapter-style works and is devoted to improving the transferability to the seen task, including seen classes or domain generalization scenarios. Therefore, it is not specifically optimized for open-world scenarios, the same as existing adapter-style ETL works and CoOp. \n(ii) But, we find that our GraphAdapter inherently owns the transferability for open-world scenarios despite no special optimization or design. Notably, **in our GraphAdapter, once the training is finished, the dual knowledge graphs of our GraphAdapter are stored in models, which are not required to be constructed in the inference stage in advance even for open-world scenarios.** As shown in Fig. 2 of our manuscript, given the novel classes, their textual features can warp knowledge from an existing dual knowledge graph constructed with few-shot training data, for the classification of novel classes. CoCoOp is an excellent work for the trade-off between seen classes and unseen classes in an open-source scenario. We follow the base-to-new setting in CoCoOp, where the new classes are unseen in the training process. The experimental results are shown in the Table below:\nMethods | Base  | New | H \n|:-|:-:|:-:|:-:|\nCLIP  | 72.43  | 68.14 | 70.22\nCoOp  | 76.47  | 67.88  | 71.92 \n CoCoOp | 75.98 | 70.43 |  73.10 \n GraphAdapter (Ours) |  77.91 | 70.13 | 74.02\n\nWe can find our GraphAdapter can achieve a better harmonic mean (higher generalization trades-off than CoCoOp). And our GraphAdapter can achieve 70.13\\% accuracy for new classes, which outperforms CoOp by a large margin. \n\n**Q3:** What is the detailed process to obtain the visual features for constructing the visual sub-graph? \n\n**A3:** The visual sub-graph is constructed with the few-shot training samples. Before the training, we exploit the visual encoder of pre-trained CLIP to extract the visual features for these few-shot samples from the same class and average them as the node of this class. The number of images used for each class is decided by the number of shots. For example, for the 2-shot task, two images are used for each node. The data augmentation strategy only contains \"random resized cropping\" and \"random flipping\". We will add this description in our revision. \n\n**Q4:**  What is the overall cost of GraphAdapter **A4:**  Please see the A1. \n\n**Q5:** (Minor) The reference figure in L171 is incorrect. **A5:** Thanks for it. We will revise it in the revision carefully. "}, {"review_id": "a4UWJkfcWM", "reviewer": "Reviewer_NmuM", "summary": "This paper introduces GraphAdapter, a novel adapter-style tuning strategy for vision-language models. GraphAdapter can leverage task-specific structure knowledge by explicitly modeling the dual knowledge graph. The authors validate the proposed method on 11 popular benchmarks on few-shot classification setting.", "questions": "Can the authors also compare the training time/parameters with previous methods to show the efficiency?", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-  GraphAdapter explicitly models the dual-modality structure knowledge with a dual knowledge graph, allowing for the leveraging of task-specific structure knowledge from both textual and visual modalities.\n- The authors conducted comprehensive evaluations on 11 datasets and demonstrated the performance under different shots.", "weaknesses": "- The authors claimed the previous methods overlook the explicit exploitation of the structure knowledge, but the experimental results showed that the combination of text and visual adapters achieved limited gain (compared to text-only adapter). To some extent, it makes the motivation less convincing.\n- The knowledge graph is constructed at the very beginning for once, this can be inconvenient in the setting where there will be new data coming sequentially over time, and then the knowledge graph will need to be updated every time. It's important to consider this because the reason we use adapters is we want a quick adaptation with minimal cost.\n- The performance improvements seem trivial. For example, in Figure 2 and Table 1, most of the improvement over the second-best model is around 1-2%, and sometimes less than 1% or worse. It seems less promising especially considering the complicated design.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The authors claimed the previous methods overlook the explicit exploitation of the structure knowledge, but the experimental results showed that the combination of text and visual adapters achieved limited gain (compared to text-only adapter). To some extent, it makes the motivation less convincing.\n\n**A1:** Thanks for your comments. We will respond in two aspects.\n(i) **An adapter typically exhibits effectiveness on a specific branch.** We claim that previous works, such CLIP-Adapter and TaskRes have shown that different types of adapters are specifically effective on a particular branch while facing performance degradation when extended to both branches.  For example, CLIP-Adapter and TaskRes are working on visual and textual branches, respectively.\nSimilarly, it is reasonable that our GraphAdapter performs superior on the textual branch only.\nMoreover, it is worth noting that our method applying to two branches brings slight gains compared with other methods decreasing the performance. \n(ii)  **Our motivation**. As stated in lines 78-80, we claim that our GraphAdapter mainly captures the structure knowledge via two sub-graphs, i.e., textual sub-graph, and visual sub-graph, ***for adapting text branch only***. To show the effectiveness of exploring structure knowledge, we should compare our GraphAdapter with others that do not utilize such knowledge, instead of contrasting with variants of our own method.\nThe significant performance gains against other methods substantiate our rationale for harnessing structural knowledge. \n\n**Q2:** The knowledge graph is constructed at the very beginning for once, this can be inconvenient in the setting where there will be new data coming sequentially over time, and then the knowledge graph will need to be updated every time. It's important to consider this because the reason we use adapters is we want a quick adaptation with minimal cost.\n\n**A2:** Thanks for your comments. We contend that the necessity for updating the knowledge graph hinges on the nature of incoming data. \nIf the new data sharing the same label space is introduced sequentially, our graph can remain unaffected. \nIn contrast, when new classes or tasks are introduced sequentially, creating a lifelong/incremental learning (IL) set-up, all adapter-style methods need re-optimization including our GraphAdapter. \nFor example, TaskRes necessitates configuring a new residual, while Tip-Adapter needs cache updates and weight relearning.\nOverall, we claim that all these methods are not designed for the IL set-up. \nHowever, we acknowledge that the IL set-up is practical and evaluate our method under a pseudo IL setting where a model is trained on base classes and directly tested on unseen new classes.\nThe results demonstrate the superiority of our method against CoOp and CoCoOp by gains 2.08\\% and 0.90\\%, respectively.\nNote that our method is not specifically designed for this base-to-new set-up.\nFurthermore, we provide the possibility of reformulating our method to real IL, i.e., we can introduce a dynamic graph, which would facilitate the incremental expansion of nodes and limited updating of edges correlated with the new nodes.\n\n**Q3:** The performance improvements seem trivial. For example, in Figure 2 and Table 1, most of the improvement over the second-best model is around 1-2\\%, and sometimes less than 1\\% or worse. It seems less promising, especially considering the complicated design.\n\n**A3:** Thanks for your thoughtful feedback. We acknowledge that the observed improvements may appear modest at first glance. However, we emphasize that while the percentage differences might seem small, they can still be meaningful for the task of tuning VLMs with few-shot samples due to the deficiency of data used for training and limited learnable parameters. For example, regarding the average accuracy over 11 diverse datasets, the number has been improved gradually across methods, e.g., 73.42\\% (CoOp, IJCV) $\\rightarrow$ 74.44\\% (CLIP-Adapter, Arxiv2021) $\\rightarrow$ 75.65\\% (Tip-Adapter, ECCV2022) $\\rightarrow$ 75.10\\% (TaskRes, CVPR2023). This underscores the challenge of enhancing performance in this task and, consequently, highlights that the 1-2\\% gains are not trivial.\n\nMoreover, while the conceptualization of constructing the dual knowledge graph to adapt VLMs might seem intricate, its implementation is remarkably straightforward and the introduced parameters are limited (only 4.145M). \nConsequently, we assert that our method is not overly complex.\n\nIn summary, taking into account the ease of implementation and the modest parameter requirements that lead to 1-2\\% performance gains on such a challenging task, our approach demonstrates its effectiveness and holds the potential to inspire future research. \n\n**Q4:** Can the authors also compare the training time/parameters with previous methods to show the efficiency?\n\n**A4:** Thanks for your great suggestions. In the following table, we compare our method with existing published ETL methods on the ImageNet in 16-shot case from the perspectives of tunable parameters, computational flops, training time, inference time. All results are measured with the officially released code from GitHub. We can observe our tunable parameters are still less than Tip-Adapter-F. For computational flops, our GraphAdapter takes about 5.42GFlops, almost the same as Tip-Adapter-F and TaskRes, which is far lower than CLIP-Adapter. Moreover, our training and inference times are less than the adapter-style work CLIP-Adapter and the prompt tuning method CoOp.  \n\n|    | CoOp| CLIP-Adapter| Tip-Adapter-F| TaskRes| Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Tunable Parameters (M)| 0.008 | 0.524| 16.384 | 1.024| 4.145|\n|GFlops | 1943.12  | 1959.44 | 5.43 | 5.42 | 5.42| \n|Training time (one epoch) (s)| 40.91 | 45.71 | 12.36 | 13.64 | 23.29| \n| Inference time (s/100) | 119.64 | 275.22 | 51.03 | 4.89 | 4.91|  \n| Performance | 62.95 | 63.59 | 65.44 | 64.75 | 65.70 |\n\n"}]

Author rebuttal:
**We thank all reviewers and area chairs for their great efforts and insightful comments!** These suggestions and questions are significantly beneficial to our paper. We believe we have addressed all the concerns of reviewers in the rebuttal. **If you have some new questions/concerns, please let us know.** We will try our best efforts to solve it. Thanks.

Meta-review: Reviewers have reached a positive consensus on this work. The concerns of reviewers are well addressed; the contributions and the effectiveness are supported by reviewers. Overall, it is recommended to be accepted.
Decision: Accept (poster)