SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin Li $^{1,2}$ ; Dongze Lian $^{2}$ ; Zhihe Lu $^{2}$ ; Jiawang Bai $^{3,2}$ , Zhibo Chen $^{1\dagger}$  and Xinchao Wang $^{2\dagger}$
$^{2}$  National University of Singapore
3 Tsinghua University
lixin666@mail.ustc.edu.cn, {dongze,zhihelu}@nus.edu.sg, bjw19@mails.tsinghua.edu.cn chenzhibo@ustc.edu.cn, xinchao@nus.edu.sg
Adapter-style efficient transfer learning (ETL) has shown excellent performance in the tuning of vision-language models (VLMs) under the low-data regime, where only a few additional parameters are introduced to excavate the task-specific knowledge based on the general and powerful representation of VLMs. However, most adapter-style works face two limitations: (i) modeling task-specific knowledge with a single modality only; and (ii) overlooking the exploitation of the inter-class relationships in downstream tasks, thereby leading to sub-optimal solutions. To mitigate that, we propose an effective adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual adapter by explicitly modeling the dual-modality structure knowledge (i.e., the correlation of different semantics/classes in textual and visual modalities) with a dual knowledge graph. In particular, the dual knowledge graph is established with two sub-graphs, i.e., a textual knowledge sub-graph, and a visual knowledge sub-graph, where the nodes and edges represent the semantics/classes and their correlations in two modalities, respectively. This enables the textual feature of each prompt to leverage the task-specific structure knowledge from both textual and visual modalities, yielding a more effective classifier for downstream tasks. Extensive experimental results on 11 benchmark datasets reveal that our GraphAdapter significantly outperforms previous adapter-based methods. The code will be released at https://github.com/lixinustc/GraphAdapter
Recent large-scale vision-language models (VLMs), such as CLIP [55] and ALIGN [27] have shown their promising representation capability for a series of downstream vision tasks [83], such as classification [44, 85, 42, 18, 86], generation [61, 31, 32, 60], and recognition [71, 73]. Different from previous pre-training models with ImageNet [20, 41, 13], based on large-scale parameters, VLMs can learn more prosperous, fine-grained, and consistent semantics from billions of text-image pairs with contrastive learning. This enables the VLMs with two advantages: 1) performing superior zero-shot generalization capability [89] with simple hand-craft prompts like "a photo of a [class]", and 2) possessing impressive transferability for downstream tasks [67, 69, 57, 65, 86] by finetuning the VLMs in a low-data regime. Notably, traditional fine-tuning methods typically involve tuning all parameters of the model to adapt to downstream tasks. However, this approach may not be well-suited for fine-tuning large VLMs in resource-constrained scenarios, and result in overfitting due to the
Figure 1: The comparison between (a) Zero-shot CLIP [55], (b) CLIP-Adapter [18] (c) TaskRes [82], and (d) our proposed GraphAdapter. We can observe that previous works model task-specific knowledge with a single modality and lacks the exploitation of structure knowledge. In contrast, our GraphAdapter aims to exploit the fused vision and language structure knowledge in data (i.e., the inter-class relationships in dual modalities) for textual feature adapter with graph learning.
limited number of available samples. To mitigate this, efficient transfer learning (ETL) is proposed to transfer the task-relevant knowledge from VLMs to downstream tasks by tuning a few parameters.
There are two popular ETL approaches for VLMs, including prompt tuning [89, 88, 44, 10, 62, 25, 42, 6], and adapter-style tuning [18, 52, 65, 90, 86, 85]. In particular, prompt tuning aims to adjust the textual classifier adaptively toward downstream tasks by adding learnable prompts on the input side, which outperforms zero-shot CLIP by a large margin with few-shot samples, such as CoOp [89], and CoCoOp [88]. Despite that, there is one limitation in the prompt tuning of VLMs [83, 82]: it needs to pass the data through the textual encoder every iteration during training, yielding a higher demand for resources. In contrast, by using the textual encoder once only, adapter-style works tend to refine the textual classifier or visual features with simple but efficient feature modulation for a specific task on the output side. For instance, CLIP-Adapter [18] exploits one simple bottleneck layer to adjust the textual and visual embeddings of VLMs, which exceeds the zero-shot CLIP by  $3.02\%$  on ImageNet with the one-shot setting. TaskRes [82] utilizes learnable task-specific parameters as prior-independent residuals to adjust the textual embeddings. Another popular line [86, 85, 90] seeks to augment the prior knowledge for downstream tasks with the cooperation of CLIP and other pre-trained large vision or language models, such as DINO [5], and GPT [4].
However, there are two limitations in most adapter-style works on ETL: 1) only modeling task-specific knowledge from a single modality perspective, such as CLIP-adapter [18], TaskRes [82] and Tip-Adapter [86], where the adaptation is achieved based on the independent visual or textual feature. 2) overlooking the explicit exploitation of the structure knowledge (i.e., the relation between different semantics/classes) in downstream tasks. A small number of samples in a low-data regime are hard to guide the model to sufficiently excavate the structure knowledge in downstream tasks, leading to the bias for partial attributes in the data, such as color and shape, and causing the sub-optimal transferability and generalization capability [33, 49, 9, 45, 46]. It is vital to model the multi-modality structure knowledge of downstream tasks for the tuning of VLMs in the low-data regime.
To mitigate the above limitations, we propose a brand-new adapter-style tuning strategy, dubbed GraphAdapter, which aims to model task-specific knowledge for downstream tasks with the fused textual and visual structure knowledge. We achieve this by solving two crucial challenges: 1) how to model the structure knowledge (i.e., the inter-class relationship) for downstream tasks, and 2) how to learn the task-specific knowledge by introducing two-modality (i.e., visual and textual) structure knowledge. Recently, graph learning [49, 7, 36] has shown the prevailing performance on modeling data knowledge structure. However, the potential of graphs in improving the efficient transfer learning of VLMs has not yet been fully explored. Inspired by this, for the first challenge, we aim to exploit graph learning to model the structure knowledge for downstream tasks when tuning VLMs. Consequently, we propose the dual knowledge graph, which is composed of a textual sub-graph and a visual sub-graph. Particularly, to establish the relationship between different
semantics/classes in textual space, we regard the textual feature of a specific class as one node of the textual graph and measure inter-class relationships by measuring their corresponding distance of features as edges of the graph. Similarly, the visual knowledge graph aims to model the relationship between different semantics/classes from the visual perspective, where the nodes are constructed with mean features of the training samples from the same class. In this way, our dual knowledge graph contains two-modality structure knowledge from downstream tasks, which enables the feature adapter with sufficient and reliable perception for downstream tasks.
For the second challenge, one intuitive strategy is to introduce the textual/visual structure knowledge for the textual/visual feature adapter, separately. However, this strategy still models the task-specific knowledge with a single modality for each textual/visual adapter (e.g., the textual adapter is only aware of textual structure knowledge). To mitigate this limitation, we introduce the dual knowledge graph for textual/visual adapters, and let each adapter be aware of the structure knowledge in the same modality and cross-modality. (Notably, we only exploit the textual adapter in our paper, since the limited gain with the combination of textual and visual adapters. Please see Table 2.). To exploit the dual knowledge graph effectively, we introduce graph learning to the textual adapter, where each feature of the prompt warps the textual and visual structure knowledge through the graph convolution network (GCN), and then fuse two warped knowledge to modulate the original feature for classification in a residual way. Based on the above strategies, we propose an effective adapter-style tuning strategy, i.e., GraphAdapter, which achieves superior performance in tuning the vision-language models under a low-data regime.
The contributions of this paper are summarized as follows:
- We pinpoint that the dual-modality structure knowledge (i.e., the inter-class relationship in textual and visual space) is vital for the efficient transfer learning (ETL) of VLMs in the low-data regime.
- Based on the above analysis, we propose a brand-new adapter-style tuning strategy, i.e., GraphAdapter, for the ETL of VLMs, which models the dual-modality structure knowledge from the textual and visual space with our proposed dual knowledge graph and enables the feature adapter to leverage the fused visual and language knowledge for better learning of task-specific knowledge from downstream tasks, yielding an effective tuning of VLMs.
- We evaluate our Graphadapter on 11 popular benchmarks on few-shot classification. The experiments demonstrated that our Graphadapter significantly outperforms previous prompt-based or adapter-style works, even on the challenging fine-grained image classification task, such as FGVCAircraft [48].
Vision-Language Pre-training (VLP) aims to learn the universal textual and visual representation simultaneously with amounts of text-image pairs. A series of works have revealed that pre-trained vision-language representations can help lots of downstream tasks to improve their performances, such as few-shot classification [89, 18, 82], cross-modality generation [50, 56, 54], and visual recognition [69]. The methods of VLP can be roughly divided into three types: 1) dual-encoder architecture, which aligns the textual and visual representation with text-image matching or text-image contrastive learning. One typical work is CLIP [55], where the promising visual representation is learned by matching the visual concepts with the corresponding textual representation with contrastive learning. 2) The second type [66, 43, 64, 34, 84, 72] is fusion-encoder architecture, where the textual and visual representation are fused with cross-modal attention. In general, two independent feature extractors are exploited to extract the modal-specific representations, respectively, and the cross-modal transformer is used for learning the text-image shared representation. 3) The third type is a combination of dual-branches and fusion-branch architectures. For instance, VLMo [1] proposes the multiway transformer, which consists of the visual-specific branch, text-specific branch, and text-visual branch. Recently, amounts of studies are devoted to exploring how to tune the large vision-language model to downstream tasks with few parameters and data, i.e., efficient transfer learning. This will be clarified in the next section.
Efficient Transfer Learning (ETL) is proposed to transfer the task-specific knowledge to downstream tasks by tuning the partial parameters [24, 44, 89, 18, 37, 80, 79, 39, 40, 16, 26, 76, 70]. There are two prominent directions for the ETL, i.e., prompt tuning [89, 88, 44] and adapter-style tuning [18, 82, 85, 86]. Prompt engineering stems from natural language process, (NLP), which aims
to formalize various NLP tasks with different prompt templates [38, 47]. Based on this development, prompt engineering is introduced to the visual-language task. For instance, Zero-shot CLIP [55] introduces the simple manual prompt template, like "a photo of a [class]", and outperforms the linear probe by  $1.9\%$  on ImageNet [12]. As the pioneering work, CoOp [89] for the first time introduces the learnable prompt to transfer the task-specific knowledge to few-shot tasks. Following this, amounts of work [88, 87, 85, 44, 28, 57, 62, 10, 25, 42, 73, 6] improve the prompt tuning from multiple perspectives, such as generalization [88], knowledge prototype [87], augmentation [85], and diversity [44]. In contrast, adapter-style works tune the VLMs to downstream tasks by adjusting the textual and visual features. Early work CLIP-Adapter [18] only exploits one bottleneck layer to improve the performance of few-shot classification. TaskRes [82] introduces the task-independent adapter to decouple the prior knowledge of the pre-trained models and task-specific knowledge. There are also some works [86, 85] that seek extra prior knowledge from other pre-trained large models, such as DINO [5], which also yield great performance on adapter-style works. In principle, the above adapter-style methods achieve efficient transfer learning for VLMs from two perspectives: 1) increasing instance-wise or task-wise adaptability based on downstream tasks [18, 82]. 2) taking into account the relation between training and test samples. [86]. However, the above works only model task-specific knowledge with a single modality, and lacks the utilization of structure knowledge in downstream tasks, leading to the sub-optimal solution. In this paper, we aim to excavate the structure knowledge of data with graphs and further improve the performance of adapter-style tuning.
Graph learning [77, 74, 2, 29, 33, 81] performs excellent capability in formalizing the topological structure of data with the nodes and edges, which has been broadly applied in various fields, such as biology [35], and social networks [59]. With the emergence of GCN [29], some works take a step forward to introduce structure graphs into transfer learning [9, 8, 7, 33, 15]. The core challenge for graph learning is how to construct the nodes and edges. Previous works usually construct the nodes from three perspectives, including objects [9], semantic features [8], or model parameters [7]. There are also some works that utilize the hypergraph to achieve better learning of structure knowledge [15, 78]. In contrast, in this paper, we are the first work to introduce graph learning into the efficient transfer learning (ETL) of vision-language tasks.
Contrastive Language-Image Pre-training (CLIP) is designed to learn continuous visual representation based on 0.4 million text-image pairs. Different from the pretraining with ImageNet, where the labels are discrete and cannot delicately describe the attributes of images and the relation between different classes, CLIP learns the visual representation with the supervision of textual embedding by contrastive loss. There are two feature extractors in CLIP, i.e., the textual encoder  $E_{t}$  and visual encoder  $E_{v}$ , which are used to warp the prompts and image into the textual embedding  $z_{t}$  and visual embedding  $z_{v}$ . In the test process,  $K$  prompts like "a photo of a [class]" are inputted to the textual feature extractor to obtain the textual embeddings  $\{z_{t}^{k}\}_{k=1}^{K}$  as the labels of  $K$  classes. The class of each image is achieved with:
$$
P (y = c \mid z _ {v}) = \frac {\exp \left(\operatorname {s i m} \left(z _ {v} , z _ {t} ^ {c}\right) / \tau\right)}{\sum_ {k = 1} ^ {K} \exp \left(\operatorname {s i m} \left(z _ {v} , z _ {t} ^ {k}\right) / \tau\right)}, \tag {1}
$$
where  $c$ ,  $sim$ , and  $\tau$  denote the class, cosine similarity, and learned temperature of CLIP, respectively. Despite CLIP has performed excellently on zero-shot image classification, the efficient transfer learning of CLIP is still required to adapt the pre-trained CLIP to downstream tasks.
3.2 Our Approach: GraphAdapter
To introduce the dual-modality structure knowledge of the downstream tasks to the feature adapter, we introduce dual knowledge graph  $\mathcal{G} = \{\mathcal{G}_t,\mathcal{G}_v\}$ , which is composed of the textual knowledge sub-graph  $\mathcal{G}_t$  and visual knowledge sub-graph  $\mathcal{G}_v$  to obtain and store the textual structure knowledge of prompts and visual structure knowledge of training samples, respectively. Then the feature of the sample/prompt can warp the structure knowledge of downstream tasks in the two modalities to refine themselves, thereby leveraging the dual structure knowledge for better modeling for downstream tasks. The whole framework of our GraphAdapter is depicted in Fig. 3, which is composed of two parts, including the construction of a dual knowledge graph and the adapter-style CLIP-based classification. Particularly, the dual knowledge graph is established only once for the whole training
Figure 2: The pipeline of GraphAdapter, which is composed of the dual knowledge graph and CLIP-based image classification pipeline. Here, the dual knowledge graph contains the delicately designed sub-graphs for textual and visual knowledge, respectively, where the textual knowledge sub-graph is established with the textual feature from the prompts of the downstream task, and the visual knowledge sub-graph is constructed with the visual features of training samples from the downstream task. In the optimization process, the feature of each prompt used for classification will seek the fused visual and textual structure knowledge from the dual knowledge graph to adjust themselves for better classification.  $K$  and  $d$  are the number of classes and the dimension of textual/visual features.
process, and the textual encoder is also only run once to obtain the textual embedding  $z_{t}$  of each prompt. In the optimization process, given one textual feature  $z_{t}$ , the relation between it and the nodes of the dual knowledge graph will be computed as the condition to warp the knowledge from the dual knowledge graph to adjust the textual embedding in the residual form. We will clarify our GraphAdapter as follows:
Textual knowledge sub-graph. To excavate the structure knowledge (i.e., the relation between different semantics) of the downstream tasks in textual space, we construct the textual knowledge sub-graph  $\mathcal{G}_t = \{\mathcal{C}_t,\mathcal{E}_t\}$ , where the nodes  $\mathcal{C}_t$  aims to capture the semantics of different classes and the edges  $\{\mathcal{E}_t\}$  is used to measure the relation of different nodes. Notably, the classifier of CLIP is based on the textual features of the prompts from different classes, where the textual feature of a within-class prompt can represent the semantics of one class. Therefore, given one downstream task with  $K$  classes, the nodes set  $\mathcal{C}_t = \{c_t^i\}_{i = 1}^K\in \mathbb{R}^{K\times d})$  are obtained with the mean feature of the prompts from the different class, where  $c_{t}^{i}$  and  $d$  denotes the textual feature of the  $i^{th}$  class and the dimension of the textual feature. And the edges  $\mathcal{E}_t$  between different nodes are computed with the cosine similarity between different nodes since the classification is achieved by computing the cosine similarity between the features in CLIP.
$$
\mathcal {E} _ {t} = \left\{e _ {t} ^ {i, j} \right\}, e _ {t} ^ {i, j} = \frac {c _ {t} ^ {i} c _ {t} ^ {j} {} ^ {T}}{\left| c _ {t} ^ {i} \right| \cdot \left| c _ {t} ^ {j} \right|}, i, j \in [ 1, K ], \tag {2}
$$
where  $\{e_t^{i,j}\}$  denotes the edge between  $i^{th}$  and  $j^{th}$  nodes.
Visual knowledge sub-graph. Different from the textual knowledge sub-graph, where the structure knowledge is only from the textual concept of a downstream task, the visual knowledge sub-graph can measure more fine-grained relations of different semantics in visual space, since the diverse visual feature of different samples. As shown in Fig. 3, to construct the visual knowledge sub-graph  $\mathcal{G}_v = \{\mathcal{C}_v,\mathcal{E}_v\}$ , we pass the augmented image group from the same class into visual encoder to obtain their visual features, and then compute the mean features of them as the nodes  $\mathcal{C}_v = \{c_v^i\}_{i = 1}^K\in \mathbb{R}^{K\times d}$  of for visual knowledge graph. The edges  $\mathcal{E}_v = \{e_v^{i,j}|i,j\in [1,K]\}$  are computed with the cosine similarity between different nodes in the visual knowledge sub-graph.
Adapting with the dual knowledge graph. After constructing the dual knowledge graph  $\mathcal{G} = \{\mathcal{G}_t,\mathcal{G}_v\}$ , we can achieve the feature adapter by introducing the dual-modality structure knowledge adaptively from the text and image sub-graphs. Notably, previous works on adapter-style tuning only model the task-specific knowledge with a single modality, which lacks the exploitation of cross-modality knowledge. In contrast, our GraphAdapter utilizes both inner-modality and cross-modality structure knowledge for feature adapters. Concretely, given the textual feature  $z_{t}$  from the textual encoder of CLIP, we aim to warp the  $z_{t}$  into the textual and visual knowledge sub-graphs to extract the textual modality and cross-modality structure knowledge, respectively. One simple strategy is to regard the textual/visual features  $z_{t}$  and  $z_{v}$  as the query, and then warp the knowledge from the graph nodes  $\mathcal{C}_t$  and  $\mathcal{C}_v$  based on similarity. However, this ignores that the structure knowledge is also required to be optimized to suit the downstream tasks. To achieve this, in the adapting process, we regard the textual feature  $z_{t}$  as one node and then warp it to the visual and textual sub-graphs, where the textual features obtain dual-modality structure knowledge by interacting with two sub-graphs in the same graph space with the graph convolutional networks (GCN).
Specifically, in the optimization process, we expand the textual knowledge sub-graph and visual knowledge sub-graph by concatenating the nodes  $\mathcal{C}_t / \mathcal{C}_v$  of the textual/visual knowledge sub-graph with the textual features  $z_{t}$ , and then compute their edges (i.e., the correlation between different nodes) as:
$$
\mathcal {C} _ {t t} = \left[ z _ {t}, \mathcal {C} _ {t} \right], \quad \mathcal {C} _ {v t} = \left[ z _ {t}, \mathcal {C} _ {v} \right],
$$
$$
\mathcal {E} _ {t t} = \left[ \begin{array}{c c} 1 & \operatorname {s i m} \left(z _ {t}, \mathcal {C} _ {t}\right) \\ \operatorname {s i m} \left(\mathcal {C} _ {t}, z _ {t}\right) & \mathcal {E} _ {t} \end{array} \right], \quad \mathcal {E} _ {v t} = \left[ \begin{array}{c c} 1 & \operatorname {s i m} \left(z _ {t}, \mathcal {C} _ {v}\right) \\ \operatorname {s i m} \left(\mathcal {C} _ {v}, z _ {t}\right) & \mathcal {E} _ {v} \end{array} \right]. \tag {3}
$$
In this way, we can warp the textual feature to the dual knowledge graph space, which provides the basis for interaction and knowledge transfer between the textual feature and the dual knowledge graph. Then, the Graph Convolution Network (GCN)  $g_{tt}$  and  $g_{vt}$  are utilized to excavate the knowledge for adapting the texture feature from each node in textual and visual knowledge sub-graphs:
$$
\mathcal {C} _ {t t} ^ {*} = g _ {t t} \left(\mathcal {C} _ {t t}, \hat {\mathcal {E}} _ {t t}\right) = \sigma \left(\hat {\mathcal {E}} _ {t t} \mathcal {C} _ {t t} W _ {t t}\right), \quad \mathcal {C} _ {v t} ^ {*} = g _ {v t} \left(\mathcal {C} _ {t t}, \hat {\mathcal {E}} _ {v t}\right) = \sigma \left(\hat {\mathcal {E}} _ {v t} \mathcal {C} _ {v t} W _ {v t}\right), \tag {4}
$$
where  $\hat{\mathcal{E}}_{tt} = D_{tt}\mathcal{E}_{tt}D_{tt}$  and  $\hat{\mathcal{E}}_{vt} = D_{vt}\mathcal{E}_{vt}D_{vt}$  are the adjacent matrix for graph learning.  $\sigma$  is an activation function. And the  $D_{tt}$  and  $D_{vt}$  are the matrices used for Laplace normalization [29] for the edges as:
$$
D _ {t t} = \operatorname {d i a g} \left(\sum_ {p = 1} ^ {K} \left(\mathcal {E} _ {t t} + I\right) _ {p}\right), \quad D _ {v t} = \operatorname {d i a g} \left(\sum_ {p = 1} ^ {K} \left(\mathcal {E} _ {v t} + I\right) _ {p}\right). \tag {5}
$$