SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The meaning of Eq. 4 is seeking the prior knowledge from other nodes in one graph to refine itself based on the correlation (i.e., the adjacent matrix). And thus, we can obtain the adapted/refined textual feature as  $z_{tt} = \mathcal{C}_{tt}^{*}[0,:]$  and  $z_{vt} = \mathcal{C}_{vt}^{*}[0,:]$ . To fuse the inner-modality and cross-modality structure knowledge, we introduce one hyper-parameter  $\beta$  to fuse these two features as  $z_{t}^{\prime} = \beta *z_{tt} + (1 - \beta)*z_{vt}$ . Then the feature adapter is achieved in a residual form, where the hyper-parameter  $\alpha$  is used to adjust the weights of prior knowledge from the dual knowledge graph and original feature:  $z_{t}^{*} = \alpha z_{t} + (1 - \alpha)z_{t}^{\prime}$ .
In the whole training process, only the parameters of GCN  $g_{tt}$  and  $g_{vt}$  are trainable under the constraint of the cross entropy function between the label of samples and the predicted label as Eq. 1.
4.1 Datasets and Implementation Details
Datasets. Following previous adapter-style studies [18, 82, 86], we validate our GraphAdapter on 11 few-shot classification tasks, including ImageNet [12], StandfordCars [30], UCF101 [63], Caltech101 [17], Flowers102 [51], SUN397 [75], DTD [11], EuroSAT [21], FGVCAircraft [48], OxfordPnts [53], and Food101 [3]. Among them, OxfordPnts, Food101, StanfordCars, Flowers102, and FGVCAircraft belong to fine-grained classification tasks, EuroSAT is for remote sensing classification, and DTD is the dataset of texture classification. To investigate the generalization capability of our GraphAdapter, we follow the CoOp [89] and conduct experiments on ImageNetV2 [58], ImageNet-Sketch [68], ImageNet-A [23] and ImageNet-R [22].
Implementation details. If not mentioned, we set the pre-trained backbone ResNet-50 of CLIP [19] as the base layer to produce the visual feature. To estimate the applicability of our method, we also
Figure 3: The performance comparison of our GraphAdapter with the state-of-the-art methods on few-shot learning, including 1-/2-/4-/8-/16-shots on 11 benchmark datasets. We provide all numerical results of this figure in the Supplementary.
apply our GraphAdapter to other CLIP visual encoders, including ResNet-101 [20], ViT-B/32 [14], and ViT-B/16 [14], and validate its effectiveness on ImageNet. We optimize our model for 100 epochs for 1, 2, 4, 8, and 16-shots. In the training process, we utilize the Adam optimizer with an initial learning rate of  $1e^{-3}$ , which drops with the cosine learning rate decay schedule. Notably, to achieve stable training, we follow previous works [82, 89] and utilize the warmup strategy for the training, where the small learning rate  $1e^{-5}$  is applied at the first epoch. The visual sub-graph is constructed with the few-shot training samples. Before the training, we exploit the visual encoder of pre-trained CLIP to extract the visual features for these few-shot samples from the same class and average them as the node of this class. The number of images used for each class is decided by the number of shots. For example, for the 2-shot task, two images are used for each node. The data augmentation strategy only contains "random resized cropping" and "random flipping".
4.2 Comparisons with State-of-the-arts
Few-shot learning. We compare our proposed GraphAdapter with several state-of-the-art works on ETL, including Zero-shot CLIP [55], CoOp [89], clip-adapter [18], Tip-Adapter-F [86], and
Table 1: The performance comparison in terms of generalization capability on four CLIP visual backbones. The ETL methods are optimized with the ImageNet dataset on 16-shot setting and tested on cross-domain datasets, including ImageNet-V2, -Sketch, -A, and -R.
Method	Backbone	Source	Target
ImageNet	-V2	-Sketch	-A	-R	Average
Zero-shot CLIP [55]	58.18	51.34	33.32	21.65	56.00	40.58
Linear Probe CLIP [55]	55.87	45.97	19.07	12.74	28.16	28.16
CoOp [89]	ResNet-50	62.95	55.11	32.74	22.12	54.96	41.23
TaskRes [82]	64.75	56.47	35.83	22.80	60.70	43.95
Ours	65.70	56.40	34.50	21.88	58.94	42.93
\( Ours_g \)	64.94	56.58	35.89	23.07	60.86	44.10
Zero-shot CLIP [55]	61.62	54.81	38.71	28.05	64.38	46.49
Linear Probe CLIP [55]	59.75	50.05	26.80	19.44	47.19	35.87
CoOp [89]	ResNet-101	66.60	58.66	39.08	28.89	63.00	47.41
TaskRes [82]	67.70	59.50	41.70	29.87	68.07	49.79
Ours	68.23	59.60	40.83	28.77	67.13	49.08
\( Ours_g \)	67.87	59.50	41.60	30.00	68.10	49.80
Zero-shot CLIP [55]	62.05	54.79	40.82	29.57	65.99	47.79
Linear Probe CLIP [55]	59.58	49.73	28.06	19.67	47.20	36.17
CoOp [89]	ViT-B/32	66.85	58.08	40.44	30.62	64.45	48.40
TaskRes [82]	68.20	59.20	42.50	31.43	69.33	50.62
Ours	68.80	59.00	41.70	29.57	68.67	49.74
\( Ours_g \)	68.47	59.10	42.70	31.73	69.43	50.74
Zero-shot CLIP [55]	66.73	60.83	46.15	47.77	73.96	57.18
Linear Probe CLIP [55]	65.85	56.26	34.77	35.68	58.43	46.29
CoOp [89]	ViT-B/16	71.92	64.18	46.71	48.41	74.32	58.41
TaskRes [82]	73.07	65.30	49.13	50.37	77.70	60.63
Ours	73.68	65.57	48.57	49.23	77.20	60.14
\( Ours_g \)	73.40	65.60	49.23	50.57	77.73	60.78
TaskRes [82] on 11 benchmark datasets. The experimental results are shown in Fig. 3, where we can observe that our GraphAdapter consistently outperforms previous ETL works for 1-/2-/4-/8-/16-shots on the average performance of 11 benchmark datasets. Particularly, on the 16-shot setting, our GraphAdapter achieves an average performance of  $76.22\%$ , which exceeds the Tip-Adapter-F [86] by  $0.57\%$ , and TaskRes [82] by  $1.12\%$ . Even for the most challenging dataset FGVCAircraft [48] of the fine-grained classification, our GraphAdapter still performs better than the second-best method Tip-Adapter-F by  $1.01\%$  on 16-shot setting, since our GraphAdapter can better exploit the structure knowledge in the downstream tasks with the dual knowledge graph.
Generalization. We also investigate the generalization capability of our GraphAdapter on four commonly-used datasets i.e., ImageNet-V2 [58], ImageNet-Sketch [68], ImageNet-A [23], ImageNet-R [22], with different visual backbones, including the pre-trained ResNet-101 [20], ViT-B/16 [13], ViT-B/32 [13]. The experimental results are shown in Table 1, where we provide two versions "Ours $_g$  and "Ours" of our GraphAdapter for a fair comparison. Here, "Ours" denotes the version of GraphAdapter used in few-shot learning. Since the over-fitting on few-shot learning will decrease the generalization capability, we increase the weights  $\alpha = 0.8$  of base textual features  $z_t$  in the adapted feature  $z_t^*$  as "Ours $_g$  to obviate the over-fitting on ImageNet Dataset. From Table 1, our GraphAdapter achieves the optimal generalization capability on four cross-domain datasets, which outperforms TaskRes [82] by  $0.15\%$  on ResNet-50 [20]. Meanwhile, it obtains the best performance on the source data ImageNet.
The effects of the different variants of GraphAdapter. It is noteworthy that our GraphAdapter can be applied to the textual and visual adapter by introducing fused visual and textual structure knowledge. Therefore, we investigate the effectiveness of another two variants of our GraphAdapter, i.e., GraphAdapter-I, and GraphAdapter-T&I. Here, we denote our GraphAdapter in our paper as GraphAdapter-T, since it is used to achieve the feature adapter of the textual branch. GraphAdapter-I denotes that we introduce the GraphAdapter into the visual branch, which adjusts the visual feature to adapt the downstream tasks. GraphAdapter-T&I represents that the GraphAdapter is exploited in the visual and textual branches simultaneously. As shown in Table 2, we can have the following conclusions: 1) the GraphAdapter is more suitable for the textual branch, which outperforms that in
Table 2: The ablation study for different variants of our GraphAdapter, where GraphAdapter-T, GraphAdapter-I denotes integrating our proposed GraphAdapter into the textual branch and visual branch, respectively. GraphAdapter-T&I utilizes the GraphAdapter-T and GraphAdapter-I simultaneously.
Dataset Method	ImageNet	Caltech101	OxfordPnts	StanfordCars	Flowers102	Food101	FGVC Aircraft	SUN397	DTD	EuroSAT	UCF101	Average
CLIP-Adapter [18]	63.59	92.49	87.84	74.01	93.90	78.25	32.10	69.55	65.96	84.43	76.76	74.44
CoOp [89]	62.95	91.83	87.01	73.36	94.51	74.67	31.26	69.26	63.58	83.53	75.71	73.42
TaskRes [82]	64.75	92.90	88.10	74.93	96.10	78.23	33.73	70.30	67.57	82.57	76.87	75.10
Tip-Adapter [86]	65.44	92.63	88.18	75.75	94.23	78.11	35.86	71.00	66.94	84.94	79.03	75.65
Base	58.18	86.29	85.77	55.61	66.14	77.31	17.28	58.52	42.32	37.56	61.46	58.77
GraphAdapter-T	65.70	93.33	88.57	76.23	96.23	78.63	36.87	71.20	67.57	85.27	78.80	76.22
GraphAdapter-I	63.50	92.10	85.86	71.81	93.24	74.55	30.90	67.39	61.75	78.79	74.40	72.21
GraphAdapter-T&I	65.72	93.37	88.59	76.20	96.13	78.40	36.93	71.30	67.90	85.55	78.67	76.25
Table 3: The ablation studies for two coefficients  $\alpha$  and  $\beta$  .
β (α = 0.6)	0.0	0.3	0.5	0.7	1.0	Learnable
ImageNet	64.30	65.1	65.50	65.70	64.95	64.35
α (β = 0.7)	0.5	0.6	0.7	0.8	0.9	Learnable
ImageNet	65.30	65.70	65.37	64.94	63.03	65.42
Figure 4: Comparisons on different backbones.
the visual branch by an average of  $4.01\%$  on 11 benchmark datasets. 2) Applying the GraphAdapter in both branches of CLIP only achieves a slight gain compared with the GraphAdapter-T. Therefore, we utilize the GraphAdapter-T in our paper.
The effects of the dual knowledge graph. To investigate the effectiveness of the dual knowledge graph, we conduct experiments on ImageNet by removing the textual knowledge sub-graph (i.e., setting  $\beta$  as 0.0) and visual knowledge sub-graph (i.e., setting  $\beta$  as 1.0), respectively. The experimental results are shown in Table 3. We can observe that removing the textual knowledge sub-graph achieves an accuracy of  $64.30\%$ , while the performance of removing the visual knowledge sub-graph drops to  $64.95\%$ , which reveals that textual structure knowledge is more important for textual feature adapter. Notably, the cooperation of textual and visual structure knowledge can achieve the best performance of  $65.70\%$ . This validates the effectiveness of introducing inner-modality and cross-modality structure knowledge for adapter-style tuning.
The effects of different coefficients  $\alpha$  and  $\beta$ . There are two hyper-parameters  $\alpha$  and  $\beta$  in our GraphAdapter, where the  $\alpha$  controls the balance of basic textual features from CLIP and the adapted textual feature obtained with GraphAdapter, and  $\beta$  is responsible for the weights of textual knowledge graph and visual knowledge graph. As shown in table 3, the performance of GraphAdapter increases first and then drops when the coefficients  $\alpha$  and  $\beta$  increase. The optimal  $\beta$  and  $\alpha$  are 0.7, and 0.6, respectively. We also investigate whether a learnable coefficient can further improve the performance of our GraphAdapter. Experimental results show that the learnable coefficients achieve lower performance with our selected fixed coefficients  $\alpha$  and  $\beta$ , since the coefficients are hard to be optimized with the few-shot samples.
The effects of different backbones. As stated in Fig. 4, we also evaluate the effectiveness of our GraphAdapter on different CLIP visual backbones, including ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16. Our GraphAdapter consistently exceeds previous works for both four visual backbones.
4.4 Complexity Analysis
We compute and compare our methods with existing published efficient transfer learning methods on the ImageNet [12] with the 16-shot setting, from the perspectives of tunable parameters, computational flops, training time, and inference time. All results are measured with the officially released code from GitHub. The experimental results are shown in the Table 4. We can observe our tunable parameters are still less than Tip-Adapter-F [86]. For computational flops, our GraphAdapter takes about 5.42 GFlops, almost the same as Tip-Adapter-F and TaskRes [82], which is far lower than CLIP-Adapter.
Moreover, our training and inference times are less than the adapter-based work CLIP-Adapter and the prompt tuning method CoOp [89]. Therefore, our GraphAdapter satisfies the requirement of efficient transfer learning. Moreover, our GraphAdapter can achieve the best performance.
Table 4: A comparison between our GraphAdapter and existing methods on time and model complexity.
Methods	CoOp [89]	CLIP-Adapter [18]	Tip-Adapter-F [86]	TaskRes [82]	Ours
Tunable Parameters (M)	0.008	0.524	16.384	1.024	4.145
GFlops	1943.12	1959.44	5.43	5.42	5.42
Training time (one epoch) (s)	40.91	45.71	12.36	13.64	23.29
Inference time (s/100)	119.64	275.22	51.03	4.89	4.91
Memory Cost (Training)	18.907	9.257	4.313	6.227	10.75
Memory Cost (Inference)	7.403	7.615	4.161	6.225	4.433
Performance	62.95	63.59	65.44	64.75	65.70
4.5 Applicability
To validate the applicability of our GraphAdapter, we select two state-of-the-art adapter-style works, including CaFo [85] and TaskRes* [82]. Here, CaFo [85] incorporates diverse prior knowledge from large pre-trained vision and language models, including DINO's vision-contrastive knowledge, GPT-3's language-generative knowledge, and DALLE's generative capability. The adapting strategy of CaFo [85] is from the Tip-Adapter [86]. The TaskRes* denotes the enhanced version of TaskRes [82], which exploits the enhanced base classifier instead of the original classifier from CLIP [55].
For CaFo [85], we directly incorporate our GraphAdapter into the textual classifier. For TaskRes* [82], we replace the task residual with our proposed GraphAdapter and maintain its enhanced textual branch from CLIP. The experimental results on ImageNet [12] are shown in Table 5 of Appendix. We can observe that our GraphAdapter can consistently increase the performance of CaFo [85] and TaskRes* [82] on few-shot learning with all 1-/2-/4-/8-/16-shots settings. Particularly, on the 16-shot setting, ours improves CaFo [85] by  $0.51\%$ , and TaskRes* by  $1.15\%$ , which validates the powerful applicability of our GraphAdapter. Overall, our GraphAdapter is complementary to these prior-augmented methods, and can obtain better performance by integrating ours into them.
In this paper, we comprehensively review the limitations of previous adapter-style tuning methods in the low-data regime as 1) previous works only model the task-specific knowledge with a single modality, and 2) overlooking the exploitation of the structure knowledge (i.e., the relation of different semantics/classes) in downstream tasks, which is vital for data-efficient tasks. Based on the analysis, we propose a brand-new adapter-style tuning strategy for visual-language models, termed as GraphAdapter, which introduces the dual knowledge graph to establish the structure knowledge in the textual and visual modalities. By incorporating the fused textual and visual structure knowledge with Graph Convolutional Network (GCN), the text-based classifier can be adapted to downstream tasks effectively with the inner-modality and cross-modality structure knowledge. Extension experiments on 11 benchmark datasets revealed the effectiveness of our GraphAdapter on few-shot learning and generalization.
Limitations and Broader Impacts. The limitation of our GraphAdapter stems from the textual structure knowledge modeling. In this paper, we utilize the textual features of default prompts like "a photo of a [class]" to construct the nodes of the textual graph. However, the prompts are simple and lack enough diversity. We believe more diverse and accurate prompts for downstream tasks can achieve better modeling for textual structure knowledge, which can further improve the performance of our GraphAdapter.
The adapter-style tuning of VLMs aims to efficiently finetune the VLMs for downstream tasks by optimizing a few parameters in the low-data regime. The possible broader impact of our GraphAdapter stems from the tuning of VLMs itself, which has a heavy dependency on the pre-trained VLMs. The utilization of our GraphAdapter should follow the privacy and safety of datasets and pre-trained models.
This research is supported by the National Research Foundation, Singapore, under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-023) and NSFC under Grant U1908209, 62021001.
[1] H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems, 35:32897-32912, 2022.
[2] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[3] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
[6] G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang. Plot: Prompt learning with optimal transport for vision-language models. In The Eleventh International Conference on Learning Representations.
[7] R. Chen, T. Chen, X. Hui, H. Wu, G. Li, and L. Lin. Knowledge graph transfer network for few-shot recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10575–10582, 2020.
[8] T. Chen, M. Xu, X. Hui, H. Wu, and L. Lin. Learning semantic-specific graph representation for multi-label image recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 522-531, 2019.
[9] T. Chen, W. Yu, R. Chen, and L. Lin. Knowledge-embedded routing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6163–6171, 2019.
[10] C.-Y. Chuang, V. Jampani, Y. Li, A. Torralba, and S. Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.
[11] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[89] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[90] X. Zhu, R. Zhang, B. He, A. Zhou, D. Wang, B. Zhao, and P. Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint arXiv:2304.01195, 2023.
A Applicability
Table 5: The experiments for the applicability of our GraphAdapter. For Cafo [85], we incorporate our GraphAdapter into the textual classifier. Notably, the TaskRes* exploits the enhanced base classifier. Therefore, TaskRes* + Ours denotes that TaskRes* replace the task residual with our proposed GraphAdapter.
Methods	1-shot	2-shot	4-shot	8-shot	16-shot
CaFo [85]	63.80	64.34	65.64	66.86	68.79
+Ours	63.81	64.97	66.17	67.68	69.30
TaskRes* [82]	61.43	62.17	62.93	64.03	64.75
+Ours	61.73	62.53	63.47	64.57	65.80
B More Experimental Results
We present the numerical results of "Figure 3 in the main text" as Table 6. We compare our GraphAdapter with the state-of-the-art works, including the prompt-based method CoOp [89], and adapter-style methods, i.e., CLIP-Adapter [18], Tip-Adapter-F [86], and TaskRes [82]. Here, the performance of Tip-Adapter-F is reproduced by [82], which aims to ensure a fair comparison with CoOp [89]. From the table, we can find that on the 16-shot few-shot learning, our GraphAdapter outperforms all previous works except for UCF101 [63] where its performance is comparable. Depart from that, for the average accuracy of 11 benchmark datasets in the 1-/2-/4-/8-/16-shot few-shot learning, our GraphAdapter surpasses previous works with a consistent improvement of  $0.57\%$  to  $0.76\%$ . We also make the analysis for the Error Bars by providing the standard deviation (Std) of our experimental results in Table 6.
C More Dataset and Implementation Details
More Dataset Details. In this paper, we follow previous works, e.g., CoOp [89], CLIP-Adapter [18], TaskRes [82], and Tip-Adapter [86], and exploit the prompts in Table 7 for the tuning and testing.
More Implementation Details. Our experimental results are achieved by running the algorithm three times with different seeds for each setting. The training and inference are implemented with a single NVIDIA GeForce RTX 3090. In the implementation of GraphAdapter for the ImageNet [12], we decouple the sub-graph with 1000 nodes for each modality into four graphs with 256 nodes to alleviate the computational cost.
Table 7: The number of classes and the used prompt temple for each dataset.
Datasets	# Classes	Prompt Templet
Caltech101 [17]	100	“a photo of a [class].”
DTD [11]	47	“[class] texture.”
EuroSAT [21]	10	“a centered satellite photo of [class].”
FGVCAircraft [48]	100	“a photo of a [class], a type of aircraft.”
Flowers102 [51]	102	“a photo of a [class], a type of flower.”
Food101 [3]	101	“a photo of a [class], a type of food.”
OxfordPnts [53]	37	“a photo of a [class], a type of pet.”
StanfordCars [30]	196	“a photo of a [class].”
SUN397 [75]	397	“a photo of a [class].”
UCF101 [63]	101	“a photo of a person doing [class].”
ImageNet [12]	1000	Ensemble of 7 selected templates, including “itap of a [class].”, “a bad photo of the [class].”, “a origami [class].”, “a photo of the large [class].”, “a [class] in a video game.”, “art of the [class].” and “a photo of the small [class].”.
D Visualization of Graph Nodes
To demonstrate how our GraphAdapter works for the adapter-style tuning for VLMs, we visualize the graph nodes for textual features before and after the GraphAdapter. As shown in Figure 5, we randomly sampled 20 classes from ImageNet [12] and utilize the t-SNE to visualize the distribution of each node corresponding to the textual fracture for classification. We can observe that with our GraphAdapter, the nodes of different classes move in directions that lead to much larger inter-class distances, thereby improving the performance of adapter-style tuning for VLMs.
Figure 5: Visualization of the variance of the graph nodes before and after GraphAdapter. Each node represents the representation of one class. We randomly sampled 20 classes from ImageNet for better visualization. The nodes move toward the direction that leads to much larger inter-class distances after GraphAdapter. The red arrows denote the directions.