SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cheng Cheng $^{1*}$  Lin Song $^{2*}$  Ruoyi Xue $^{1}$  Hang Wang $^{1}$
Hongbin Sun $^{1\dagger}$  Yixiao Ge $^{2}$  Ying Shan $^{2}$
$^{1}$  Xi'an JiaoTong University  $^{2}$ Tencent AI Lab
cheng2016@stu.xjtu.edu.cn, ronnysong@tencent.com
The contrastive vision-language pre-training, known as CLIP, demonstrates remarkable potential in perceiving open-world visual concepts, enabling effective zero-shot image recognition. Nevertheless, few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters on few-shot samples, resulting in longer inference time and the risk of over-fitting in certain domains. To tackle these challenges, we propose the Meta-Adapter, a lightweight residual-style adapter, to refine the CLIP features guided by the few-shot samples in an online manner. With a few training samples, our method can enable effective few-shot learning capabilities and generalize to unseen data or tasks without additional fine-tuning, achieving competitive performance and high efficiency. Without bells and whistles, our approach outperforms the state-of-the-art online few-shot learning method by an average of  $3.6\%$  on eight image classification datasets with higher inference speed. Furthermore, our model is simple and flexible, serving as a plug-and-play module directly applicable to downstream tasks. Without further fine-tuning, Meta-Adapter obtains notable performance improvements in open-vocabulary object detection and segmentation tasks.
The contrastive vision-language pre-training [1-5], known as CLIP [6], has shown the impressive potential in modeling open-world visual concepts [7-17], which benefits multiple vision tasks including image recognition and open-vocabulary perception [7, 10]. It can be mainly attributed to the large-scale datasets [2] and the advanced pre-learning techniques [18]. By constructing prompts based on visual categories, CLIP shows effective zero-shot image classification capabilities and generalization abilities for unseen data [19]. Recently, few-shot learning based on CLIP has garnered increasing research attention. Motivated by the success of feature adapters [20] and prompt tuning [21] for natural language processing, a wide range of few-shot approaches for CLIP [22, 8, 9, 7] are proposed and studied.
Few-shot learning methods for CLIP can be categorized into offline [22, 8, 9] and online approaches [7], according to whether the fine-tuning is required for the few-shot samples of unseen categories. Offline methods extract the knowledge from few-shot samples via parameter optimization. Notable examples include CoOp [8] and CoCoOp [22], which replace the hand-crafted templates in CLIP with learnable continuous tokens by fine-tuning on few-shot samples. Additionally, CLIP-Adapter [9] introduces feature adapters into CLIP by learning task-specific knowledge from few-shot samples. Although the extra components yield promising few-shot learning capabilities, they also incur
(a) Top-1 accuracy with 16 shots.
(b) Top-1 accuracy on ImageNet [29]-Novel.
Figure 1: Comparison of different few-shot learning techniques. The models are trained on the set of base classes and evaluated on the novel classes. The time is measured on a Tesla V100 GPU.
additional training costs and suffer from considerable over-fitting in a certain data distribution [22, 23]. To eliminate training expenses, an online method called Tip-Adapter [7] is proposed. This method proposes a hand-crafted modulation function that adjusts the ratio between category embeddings and few-shot visual embeddings. It obtains the knowledge from few-shot samples without fine-tuning and shows a notable improvement against the zero-shot manner. However, due to its sophisticated hyper-parameter search scheme, we find that Tip-Adapter still tends to over-fit in the distribution of seen data (details referring to Sec. 3.1), resulting in limited generalization capability. Different from previous methods, we attempt to explore a new perceptive: learning an online few-shot learner for CLIP via meta-learning.
To achieve it, we propose the Meta-Adapter that replaces the hand-crafted modulation function and searching scheme in Tip-Adapter with a lightweight residual-style network. The offline few-shot learning methods require additional fine-tuning for few-shot samples from unseen categories. In contrast, our approach employs the meta-testing mechanism [24-27], thereby the categories of training and testing data of our model can be different. By using a limited number of few-shot data, the Meta-Adapter can be trained to enable few-shot learning capability. Without additional fine-tuning, it can further generalize to other unseen data and extract knowledge from the few-shot samples in an online manner. To achieve high efficiency, the Meta-Adapter is constructed by a lightweight network based on the gated multi-head attention mechanism [28], which bridges the gap between few-shot image features and textual features for each category. This procedure can be considered as a learnable filter to refine the category embeddings guided by the few-shot images. Since the Meta-Adapter does not require additional fine-tuning, it only has a slight computational overhead over the zero-shot manner. Compared with the Tip-Adapter, it alleviates the over-fitting problem and demonstrates superior generalization across datasets. Furthermore, the Meta-Adapter is simple and can be applied as a plug-and-play module to various CLIP-based methods, making it a versatile solution for many open-vocabulary downstream tasks.
The extensive experiments demonstrate the effectiveness and efficiency of the Meta-Adapter on image classification, object detection, and segmentation. To verify the generalizability of Meta-Adapter, we conduct a series of ablation studies, including cross-category generalization within a certain dataset, cross-dataset generalization, and cross-task generalization which explores the potential of Meta-Adapter in downstream tasks. As shown in Figure 1(a), by training on the data of base classes, the Meta-Adapter achieves an average of  $3.6\%$  absolute gains over Tip-Adapter across the novel classes of eight image classification datasets under the 16-shot setting. With a larger number of image shots, our method achieves increased performance gain over the Tip-Adapter, which is illustrated in Figure 1(b). Besides, through directly evaluating the ImageNet [29] pre-trained model on other seven classification datasets, our method obtains an average of  $4.9\%$  improvements against Tip-Adapter. In addition, Meta-Adapter shows the potential to improve other tasks, such as open-vocabulary object detection, which leads to consistent improvements in both object detection and instance segmentation.
By integrating the ImageNet-pretrained Meta-Adapter with the open-vocabulary object detection framework, ViLD [11], our method achieves  $1.0\%$  absolute gains on the average precision of rare categories  $\mathrm{AP}_{\mathrm{r}}$  without bells and whistles.
2.1 Vision-Language Pretrained Models
Inspired by the success of pre-trained models in the field of CV and NLP, many works are proposed to pre-train large-scale models to process both vision and language modalities. A typical vision-language model consists of four key components, i.e., vision encoder, language encoder, fusion encoder, and loss function. Recently, following the success of the base models in both CV and NLP [30, 6, 31-36], the community of multi-modal learning can take advantage of these large-scale base models to better elevate the performance. VisualBERT [37], OSCAR [38], Uniter [39] utilize BERT [30] to preprocess the raw text and demonstrate impressive results in multimodal tasks, e.g., visual question answering (VQA). Besides, these methods require a well-designed fusion encoder to integrate the cross-modal interaction. Recently, CLIP [2], DeCLIP [40] and ALIGN [1] demonstrate that vision-language contrastive learning is capable of generating transferable features to downstream tasks and the multimodal interaction can be well interpreted by simply calculating the dot product between vision and language embeddings. Without additional self-attention or cross-attention modules, the multimodal embeddings can be pre-computed and stored, which is more efficient and can be easily adapted to other tasks.
2.2 Vision-Language Model Adaption
Many recent works focus on exploring effective and efficient approaches for adapting vision-language models to downstream tasks [8, 22, 7, 9, 41, 42, 14, 13, 15, 43, 12], which are prompt-tuning methods, e.g., Context-Optimization (CoOp) [8], and feature adapters methods, e.g., Tip-Adapter [7, 14, 13]. Inspired by the success of prompt learning [21, 44], CoOp proposes to replace the hand-crafted templates [2] with continuous tokens that can be optimized in a traditional fine-tuning fashion. Besides, to mitigate the woeful over-fitting issue, CoCoOp further introduces integrate image-specific tokens learned by a shallow MLP. Compared with the manually designed prompt templates, CoOp and CoCoOp achieve impressive performance on few-shot image classification. Different from these prompt tuning methods, CLIP-Adapter and Tip-Adapter conduct residual feature blending to integrate few-shot knowledge with CLIP's zero-shot knowledge. They keep the whole CLIP's parameters frozen and fine-tune an acceptable small number of additional weights and show impressive results in few-shot image classification. Besides, by initializing the linear weights with few-shot knowledge (i.e., cache model in this context), Tip-Adapter can further pose a training-free fashion with preferable performance. Nevertheless, these methods suffer from over-fitting, especially when the domain gap between the source and target datasets is large.
2.3 Meta-Learning
A simple interpretation of meta-learning [24-27] is "learning-to-learn" [45] which corresponds to improving the generalization by searching for the algorithm (inductive bias) that is best suited for a given task family. On the contrary, traditional machine learning algorithms [46, 47] are expected to be improved as more data from a certain single task. Usually, meta-learning is conducted on learning instances sampled from a task family, which is expected to simulate a base learning algorithm that performs well on new tasks sampled from this family. Besides, as mentioned in [24], all training instances can be sampled from a single task in a special case. In the context of adapting the vision-language model to downstream tasks, meta-learning can be viewed as learning the general fine-tuning algorithms which bring consistent gains over different tasks or datasets. Current methods [22, 8, 7, 9] focus mainly on improving the performance of certain tasks or datasets. To the best of our knowledge, this paper is the first one that studies the potential of meta-learning in the field of vision-language model adaption.
Table 1: Comparison of cross-dataset generalization based on ImageNet [29] pre-training. The Tip-Adapter and Meta-Adapter are tuned on ImageNet and frozen for other datasets.  $\Delta$  reflects the generalization ability across datasets.
Method	FGVC	OxfordPets	SUN397	UCF101	Caltech101	DTD	EuroSAT	Avg.	Δ
Zero-shot CLIP	0.42	56.25	28.96	21.05	60.62	10.00	4.17	25.92	-
Tip-Adapter*	13.96	68.75	45.16	40.09	68.33	42.92	56.25	47.92	-
Tip-Adapter	13.96	67.19	43.80	39.47	67.08	40.00	56.25	46.82	-
Meta-Adapter*	19.58	72.66	51.25	52.28	71.46	49.17	64.58	54.43	+6.51
Meta-Adapter	15.21	72.66	48.54	47.54	67.92	48.33	62.50	51.81	+4.99
* indicates searching hyper-parameter or training for each evaluation dataset individually.
In this section, we introduce the proposed Meta-Adapter. In Section 3.1, we first revisit CLIP and Tip-Adapter. In Section 3.2, we elaborate on the implementation of the proposed Meta-Adapter. In Section 3.3, we discuss the difference with other related works.
3.1 Revisiting CLIP and Tip-Adapter
As a vision-language pre-training model, CLIP [2] has shown impressive zero-shot learning potential in modeling open-world visual representation [11, 10] by exploiting contrastive learning with large-scale noisy image-text pairs. To achieve zero-shot image classification, CLIP computes classification scores by measuring the cosine distance between the image features and per-class textual features. Specifically, given an image  $y$ , let  $f \in \mathbb{R}^{D \times 1}$  be the feature of the query image and  $\{w_i\}_{i=1}^N$ ,  $w_i \in \mathbb{R}^{D \times 1}$  be a set of category embeddings generated by the text encoder. The  $D$  indicates the dimension of embedding space and  $N$  denotes the number of total categories. The textual feature  $w_i$  for each class is derived from hand-crafted templates that one of typical form is "a photo of [CLASS]". The class token is then replaced by a specific category name, such as "Alp" or "Lemon", as shown in Figure 2. The predicted logits of the given image  $y$  belonging to the  $i$ -th class can be formulated as:
$$
\operatorname {l o g i t s} \left(y _ {c} = i\right) = \frac {w _ {i} ^ {\top} f}{\| w _ {i} \| \| f \|}, \tag {1}
$$
Tip-Adapter [7] further proposes an online method to learn knowledge from few-shot samples. This method employs a straightforward modulation function with a stochastic hyper-parameter search strategy, achieving impressive few-shot performance for a certain domain. Concretely, two hyperparameters, i.e.,  $\alpha$ , and  $\beta$ , are introduced to adjust the ratio between visual and textual features for different datasets. Given a set of support images  $\mathbf{x} = \{\mathbf{x}_i\}_{i=1}^N$  in  $N$  ways and  $K$  shots, the predicted logits of Tip-Adapter can be formulated as:
$$
\operatorname {l o g i t s} \left(y _ {c} = i | \mathbf {x}, \alpha , \beta\right) = \frac {w _ {i} ^ {\top} f}{\| w _ {i} \| \| f \|} + \alpha \cdot \exp (- \beta (1 - \frac {\mathbf {F} _ {j} ^ {\top} f}{\| \mathbf {F} _ {j} \| \| f \|})) \mathbf {L} _ {j}, \tag {2}
$$
where  $\mathbf{F}_i\in \mathbb{R}^{D\times K}$  is support embeddings of few-shot samples, and  $\mathbf{L}_i\in \mathbb{R}^{N\times K}$  is the corresponding one-hot labels of  $i$ -th class. Tip-Adapter obtains remarkable few-shot performance without additional training for few-shot samples. However, Tip-Adapter relies heavily on the hyper-parameter search strategy on the target dataset, rendering it susceptible to over-fitting within a certain data distribution and limiting its out-of-distribution generalization capabilities. As presented in Table 1, we fix the hyper-parameters searched on ImageNet [29] and directly evaluate the performance of Tip-Adapter on seven other datasets. It demonstrates a poor generalization performance of the Tip-Adapter across different distributions. Compared with the searching scheme for each dataset individually, the Tip-Adapter exhibits a notable decline in performance.
3.2 Meta-Adapter
To address the issue of poor generalization, we propose a learnable Meta-Adapter to replace the handcraft modulation function and searching strategy in Tip-Adapter. Different from the Tip-Adapter,
Figure 2: Diagram of the proposed Meta-Adapter, which employs a learnable network to refine the category embeddings guided by few-shot images.
we model the few-shot learning in a visual-language scheme as a learnable filter on textual features under the guidance of few-shot image samples, to obtain more discriminative category embeddings. Motivated by the non-local filters [48-50] in computer vision, we propose a Meta-Adapter based on the gated multi-head attention mechanism.
As shown in Figure 2, through CLIP encoders, we first extract the support embeddings of input few-shot images and category embedding. The Meta-Adapter then extracts and transfers the few-shot knowledge from visual features into textual features to obtain refined category embeddings. Specifically, we apply the original category embeddings as query and the support embeddings as both key and value into a multi-head attention block. Unlike the standard transformer encoder [28], our approach only introduces Multilayer Perceptron (MLP) layers for query and key. This strategy is crucial since no feature transformations are performed on values, the zero-shot capability is generally not changed after training. The predicted logits with the proposed Meta-Adapter can be formulated as:
$$
\operatorname {l o g i t s} \left(y _ {c} = i | \mathbf {x}\right) = \frac {\hat {w} _ {i} ^ {\top} f}{\| \hat {w} _ {i} \| \| f \|}, \text {w h e r e} \hat {w} = \operatorname {M e t a A d p a t e r} (w, \mathbf {F}). \tag {3}
$$
The  $\hat{w}$  is the refined category embeddings. In the Meta-Adapter, as shown in the right-hand of Figure 2, the proposed method adaptively aggregate the support embeddings according to the affinity between categories and few-shot images. The aforementioned procedure can be implemented by a cross-attention mechanism:
$$
\hat {\mathbf {F}} = \mathbf {F} ^ {\top} \sigma \left(\left(\mathbf {F} W _ {1} ^ {\top}\right) \left(w W _ {2} ^ {\top}\right) ^ {\top} / \sqrt {D}\right) \tag {4}
$$
where  $W_{1}$  and  $W_{2}$  indicate the weights of MLP layers. The  $\sigma$  denotes the Softmax function, and  $\hat{\mathbf{F}}$  represents the aggregated support features. Intuitively, similar to the non-local filters, Meta-Adapter could disregard some outlier samples while paying more attention to the samples that are more related to the category description [42], resulting in robust feature representations.
Besides, the importance of textual and visual features for few-shot learning varies across different data distributions [9]. Therefore, we propose a learnable gating block  $g(\cdot)$ , generating a modulation scalar, to adaptively control the ratio between category embeddings and aggregated support embeddings. Accordingly, the refined category embedding can be obtained by:
$$
\hat {w} = w + g (w) \odot \hat {\mathbf {F}}, \tag {5}
$$
where  $\odot$  denotes Hadamard product. Through training on the few-shot samples, the gating block could adjust the ratio according to the category descriptions. It enables the proposed method to effectively integrated few-shot knowledge with zero-shot knowledge.
Table 2: Quantitative results of in-domain generalization setting on UCF101, Caltech101, DTD, and FGVCAircraft datasets between Meta-Adapter and other methods.
Model	UCF101	Caltech101	DTD	FGVCAircraft
Base	Novel	Base	Novel	Base	Novel	Base	Novel
Zero-shot CLIP	79.42	21.05	93.39	60.62	59.38	10.00	23.84	0.42
Tip-Adapter	85.17	40.09	95.09	68.33	68.36	42.92	30.27	13.96
Meta-Adapter	82.44	52.28	93.39	71.46	64.26	49.17	27.32	19.58
3.3 Comparison with Counterparts
Compared to the offline methods, e.g., CLIP-Adapter [9] and CoOp [8], our Meta-Adapter does not require additional fine-tuning for target samples, significantly reducing the computational costs during inference. In addition, compared with the online methods, e.g., Tip-Adapter [7], the proposed technique replaces the handcrafted hyper-parameter search process with a learnable network on support samples. As shown in Table 1, the Meta-Adapter can better alleviate the over-fitting problem and demonstrates the generalization across datasets without further fine-tuning. Moreover, as the Meta-Adapter refines the textual embedding features directly without altering their dimensions, it can naturally be applied to a variety of downstream tasks based on CLIP.
It should be noted that the distributions of training and testing sets can be identical or dissimilar and it is crucial that Meta-Adapter can well perform in both scenarios. Besides, the potential of Meta-Adapter in downstream tasks is also of great importance. We refer to these three situations as "cross-category generalization", "cross-dataset generalization", and "cross-task generalization", respectively. Specifically, for "cross-category generalization", we split the full categories of each dataset into base and novel sets according to the per-category accuracy predicted by Zero-shot CLIP, that is, the base set contains easy samples and the novel set contains hard samples. This dataset split strategy simulates a rather difficult situation to verify whether Meta-Adapter is able to learn the dataset-irrelevant approach, especially for hard samples. We provide details of dataset splits in the supplementary material. Before diving into experimental analysis, we first give the details of the experimental setup.
Datasets For cross-category generalization experiments, we use 8 representative image classification datasets: ImageNet [29], FGVCAircraft [51], OxfordPets [52], SUN397 [53], UCF101 [54], Caltech101 [55], DTD [56], and EuroSAT [57], which cover a diverse set of classification tasks. As for the cross-dataset generalization experiment, ImageNet is further utilized as the source dataset and its three variants are treated as target datasets, i.e., ImageNet-A [58], ImageNet-R [59], and ImageNet-Sketch [60]. Moreover, to explore the potential of Meta-Adapter on open-vocabulary detection, we conduct experiments on LVIS [61].
Baselines We compare Meta-Adapter with two training-free methods: Zero-shot CLIP [2] and Tip-Adapter [7].
Training Details As for the CLIP backbone, we choose ResNet50 [46] as the visual encoder in most experiments and a transformer [6] as the textual encoder. We adopt the prompt ensemble strategy [2, 7], which inputs 7 templates into the CLIP textual encoder and then averages them as the final prompt embeddings. We optimize the Meta-Adapter on the base set with a batch size of 64 and use AdamW optimizer [62] with a learning rate of 0.0001 and a cosine scheduler for 5 epochs.
4.1 Cross-Category Generalization
Based on empirical studies, it can be observed that Tip-Adapter often requires large hyper-parameters  $(\alpha$  and  $\beta)$  when applied to specific datasets. These hyper-parameters are used to smooth the classification distribution and give significant weight to few-shot knowledge. This phenomenon indicates that Tip-Adapter heavily relies on few-shot knowledge, even for relatively general datasets like ImageNet. As a result, over-fitting issues arise, which hinder its generalization ability. Table 6 demonstrates that
Table 3: Quantitative results on ImageNet of different models utilized various vision backbones.
Model	RN50	RN101	ViT-B/32	ViT-B/16	RN50×16	RN50×64
Zero-shot CLIP	32.82	39.22	40.10	45.77	50.10	54.67
Tip-Adapter	36.51	42.42	43.71	49.84	53.08	57.99
Meta-Adapter	40.19	47.01	46.91	52.60	55.51	60.41
Tip-Adapter achieves slightly higher classification accuracy than Meta-Adapter on the training set for datasets like UCF101 and Caltech101. However, when it comes to novel samples, Tip-Adapter falls behind Meta-Adapter with significant gaps, such as  $40.26\%$  versus  $47.72\%$  on UCF101. This suggests that Tip-Adapter becomes excessively tailored to a specific localized distribution due to its excessive hyper-parameter search strategy.