SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the contrary, thanks to the ingenious and generic ensemble approach, Meta-Adapter enjoys comparable performances on the base set and superior performances on the novel set. As shown in Figure 1(b), Meta-Adapter shows superior performances over other methods on the ImageNet dataset. Compared to Zero-shot CLIP, Meta-Adapter consistently surpasses it on all different few-shot settings. In comparison to Tip-Adapter, the classification accuracy of both methods witnesses relatively steady improvements. When shots are less than 4, Tip-Adapter slightly outperforms Meta-Adapter. The reason is two-fold. First, Tip-Adapter directly calculates classification logits given both few-shot features and their corresponding one-hot labels, which can be viewed as a shortcut solution compared to the generic ensemble approach of Meta-Adapter. Second, Tip-Adapter exploits the potential of few-shot knowledge via a hyper-parameter searching strategy which intends to find out the highest accuracy on certain datasets. Nevertheless, as shots get larger, Meta-Adapter outperforms Tip-Adapter by a clear margin and classification accuracy goes up consistently while Tip-Adapter witnesses performance drops when shots are 32, indicating a possible performance limitation for Tip-Adapter. And we present the quantitative comparison on the other 7 datasets under 16 shots setting in Figure 1(a). It can be observed that Meta-Adapter significantly boosts the classification accuracy over Zero-shot CLIP and surpasses Tip-Adapter with gains of up to  $+7.46\%$ . Due to space limitations, we present the comparison of Meta-Adapter and other methods on the remaining seven datasets under different few-shot settings in the supplementary material.
To further verify the effectiveness of Meta-Adapter, we apply different visual encoders for all methods and conduct experiments on ImageNet. The quantitative comparison is shown in Table 3. Undoubtedly, Meta-Adapter maintains its leading position over Tip-Adapter regardless of the choice of visual encoders. As a more advanced backbone, such as ViT-B/16 [2], is utilized, the classification accuracy of Meta-Adapter further increases. This suggests that the learning potential of Meta-Adapter can be enhanced by employing a more powerful vision-language model. In summary, compared to previous training-free methods, Meta-Adapter not only effectively mitigates the issue of over-fitting but also retains superior generalization ability. As a result, it achieves state-of-the-art classification accuracy on the novel set.
4.2 Cross-Dataset Generalization
It is of great importance that a learned classifier maintains comparable performance when handling different datasets with diverse distributions. It is more challenging because the appearances and shapes can be totally dissimilar across datasets (e.g., from object recognition of ImageNet to textual classification of DTD). Besides, we are interested to know whether Meta-Adapter can learn datasetirrelevant discriminative ensemble approach. To this end, we transfer the optimal hyper-parameters of Tip-Adapter searched on the source dataset and evaluate the performance on target datasets. And Meta-Adapter is first trained on the source dataset, i.e., base set, and evaluated on the target dataset, i.e., novel set, with all learnable parameters frozen. Both experiments are conducted under the 16-shot setting. We report the relative accuracy improvements when transferring from ImageNet (with all categories utilized) to the other 7 datasets, as shown in Figure 3(a). Tip-Adapter base results are set as the baseline in Figure 3(a), thus the corresponding relative accuracy improvements are always 1.0. Considering that ImageNet contains a variety of classes, e.g., different animal breeds and different kinds of vehicles, it is not surprising that both models retain comparable accuracy compared with their counterparts. Nevertheless, Tip-Adapter poses rather clear performance drops compared with Meta-Adapter. Moreover, transferring from ImageNet to SUN397, UCF101, and EuroSAT
(a) ImageNet to Others
(b) Others to ImageNet
Figure 3: Relative accuracy improvements of Tip-Adapter and Meta-Adapter in cross-dataset generalization experiments.
Table 4: Quantitative results of domain generalization experiments between Tip-Adapter and Meta-Adapter. The data in parentheses records the changes brought by comparing with Zero-shot CLIP.
CLIP Backbone	Model	Target Datasets
ImageNet-A	ImageNet-R	ImageNet-Sketch
RN50	Zero-shot CLIP	23.88	60.54	35.45
Tip-Adapter	23.25(-0.63)	58.73(-1.81)	34.77(-0.68)
Meta-Adapter	23.71(-0.17)	59.96(-0.58)	35.54(+0.09)
ViT-B/16	Zero-shot CLIP	50.65	77.82	48.42
Tip-Adapter	49.89(-0.76)	76.94(-0.88)	48.13(-0.29)
Meta-Adapter	51.12(+0.47)	77.54(-0.28)	48.76(+0.34)
results in surpassing their baselines, i.e., directly trained on the target datasets, which suggests that the learning potential of Meta-Adapter can benefit from a generalized dataset. Besides, we report the relative accuracy improvements of Tip-Adapter and Meta-Adapter by transferring from 7 small classification datasets (with all available categories utilized) to ImageNet, as shown in Figure 3(b). Surprisingly, Meta-Adapter maintains comparable results while Tip-Adapter poses clear performance drops, particularly on OxfordPets and Caltech101. Thus, it can be concluded that Meta-Adapter retains better transferability against diverse distributions and domain shifts.
Furthermore, we conduct domain generalization experiments as in [22]. Humans have a natural ability to generalize to out-of-distribution data, which raises the question of whether Meta-Adapter possesses the same advantage. To this end, we transfer Tip-Adapter's optimal hyper-parameters  $(\alpha$  and  $\beta)$  and Meta-Adapter's weights searched and optimized on ImageNet to its three variants, i.e., ImageNet-A [58], ImageNet-R [59], and ImageNet-Sketch [60]. Besides, we report the performance of Zero-shot CLIP on these three datasets as the baseline. The quantitative results are presented in Table 4. It is clear that directly transferring Tip-Adapter's optimal hyper-parameters searched on ImageNet to its variants leads to performance drops, even worse than Zero-shot CLIP. This phenomenon accords with the previous statement that Tip-Adapter is sensitive to hyper-parameter settings, or to say Tip-Adapter suffers from severe over-fitting issues. On the contrary, Meta-Adapter can better adapt to domain shifts, demonstrating comparable performances with Zero-shot CLIP.
4.3 Cross-Task Generalization
It is a major concern whether few-shot learning methods can benefit downstream tasks. To this end, we integrate Meta-Adapter and Tip-Adapter with open-vocabulary object detection method ViLD [11]. ViLD utilizes CLIP to explore the open-vocabulary potential, it replaces the typical classifier in
Table 5: Comparison of different few-shot methods on LVIS [61] val set.
Methods	Object Detection	Instance Segmentation
APr	APc	APf	APr	APc	APf
ViLD	18.1	27.3	32.0	17.4	25.5	28.5
ViLD + Tip-Adapter	11.3	23.3	26.7	10.8	21.8	24.2
ViLD + Meta-Adapter	19.1	27.3	31.7	18.3	25.5	28.3
object detection framework [63, 47, 64, 65, 17, 66-69] with textual features generated by CLIP's text encoder and aligns the textual features and ROI features by knowledge distillation.
We first generate pre-processed region features of LVIS [61] given its annotations as the few-shot samples. Following ViLD, object categories in the LVIS dataset are split into "frequent", "common", and "rare" according to their frequency in the training set. Among them, 866 frequent and common categories of LVIS are taken as the base set, and 337 rare categories are held out as the novel set. For Tip-Adapter, the prediction logits of ViLD is modified by adding an additional few-shot term.
$$
\operatorname {l o g i t s} (\hat {r}) = f (\hat {r}) w ^ {T} + \alpha \cdot \varphi (f (\hat {r}) \mathbf {F} ^ {\top}) \mathbf {L}, \text {w h e r e} \varphi (x) = \exp (- \beta (1 - x)) \tag {6}
$$
where  $f(\hat{r})$  represents the region features of proposal  $\hat{r}$ ;  $\varphi(\cdot)$  is the modulation function introduced in Tip-Adapter;  $\mathbf{F}$  and  $\mathbf{L}$  denotes few-shot region features and their corresponding one-hot labels. It should be noted that Equation 6 is applied to both ViLD-text and ViLD-image as introduced in ViLD. As for the hyper-parameter settings  $(\alpha$  and  $\beta)$  of Tip-Adapter, we observe that setting them to be the ones that are searched on ImageNet will cause ViLD collapses. Thus, we set  $\alpha$  to 0.05 and  $\beta$  to 1, which encourages ViLD to rely more on CLIP knowledge to avoid collapse. Besides, we utilize the ImageNet pre-trained Meta-Adapter to integrate the pre-processed LVIS few-shot knowledge into the original textual features. And for ViLD, we utilize the re-implement version as suggested in DetPro [10] which replaces the pre-trained ResNet-50 [46] with self-supervised pre-trained SoCo [70]. We report the average precision in Table 5. From Table 5, it is clear that Meta-Adapter can boost the detection ability on rare categories by a clear margin while Tip-Adapter corrupts the detection performances due to its poor transferability and its necessity to modify the original prediction score of ViLD. Besides, considering that part of the LVIS annotations contains rather small bounding boxes (usually around  $10 \times 10$ ), the quality of the LVIS few-shot dataset may not be as good as the image classification counterparts. As mentioned before, Tip-Adapter naturally and heavily relies on few-shot knowledge, which may be one of the main causes of poor performance in open-vocabulary object detection. On the contrary, Meta-Adapter learns the generic ensemble approach and can be easily integrated into open-vocabulary object detection methods without changing their formulation of prediction scores.
Table 6: Comparison of Zero-Shot CLIP, Tip-Adapter, and Meta-Adapter on UCF101 and Caltech101 datasets in in-domain generalization setting. H: Harmonic mean (to highlight the generalization trade-off [22]).
Dataset	ImageNet	UCF101	Caltech101
Model	Base	Novel	H	Base	Novel	H	Base	Novel	H
Zero-shot CLIP	71.9	32.8	45.0	79.4	21.1	33.4	95.4	60.6	74.1
CLIP-Adapter	76.3	15.1	25.3	89.4	5.4	10.2	97.3	39.3	54.0
CoOp	75.3	2.7	5.2	89.3	1.0	2.0	97.2	31.3	47.4
CoCoOp	75.5	33.9	46.8	86.5	9.1	16.5	96.8	60.9	74.8
Meta-Adapter	76.3	40.8	53.2	82.4	47.7	0.4	94.9	76.1	84.4
4.4 Comparison with Offline Methods
As shown in Table 6, we provide the ablation studies between our meta-adapter and other offline methods [22, 8, 9]. For a fair comparison, similar to CoCoOp, the experiments adopt a base-to-novel generalization setting. The results demonstrate that our method has significant advantages in generalization on novel classes, e.g. improving over CoCoOp by  $6.9\%$  on ImageNet. Additionally,
as in previous works, we introduce Harmonic Mean to measure overall performance on base and novel classes. It can be observed that our approach also shows clear superiority in terms of overall performance. More importantly, finetuning is not needed for our method when applying to new datasets or tasks.
4.5 More ablation studies of the Meta-Adapter
A2: Thanks for this insightful suggestion. Accordingly, we further conduct more ablation studies to demonstrate the advantages of our design. As shown in Table 7, the results demonstrate that multi-head attention contributes most significantly to improving accuracy. The proposed learnable gating block can further enhance the performance while introducing value projection leads to decreased generalization capability. Besides, as shown in Table 8, we increase the model scale of the meta-adapter by widening the projection layers (Wider) or cascading multiple modules (Deeper). The results show that increasing the number of modules can improve the parameter size and accuracy slightly, but brings a significant efficiency decrease.
Table 7: Ablation study on different components. The 'VP', 'MHA', and 'LGB' indicate the value projection layer, multi-head attention block, and learnable gating block, respectively.
Method	ImageNet	SUN397	UCF101	DTD
Meta-Adapter w/ VP	25.6	18.2	5.9	5.0
Meta-Adapter w/o MHA	32.9	28.9	21.4	10.0
Meta-Adapter w/o LGB	39.6	51.3	49.8	53.8
Meta-Adapter	40.2	52.7	51.4	54.6
Table 8: Comparison of different model scales on several datasets.
Method	ImageNet	SUN397	UCF101	DTD	#Param	Latency
Meta-Adapter	40.8	52.6	51.0	55.8	2.1M	3ms
+ Wider (×2)	40.1	51.4	50.0	55.0	4.2M	5ms
+ Wider (×4)	40.5	51.3	50.4	55.0	8.4M	9ms
+ Deeper (×2)	40.4	52.9	52.2	56.7	4.2M	6ms
+ Deeper (×4)	38.9	52.7	51.4	56.3	8.4M	11ms
Limitations Based on the findings presented in Table 6, it can be observed that the learning potential of Meta-Adapter may encounter limitations when the classification accuracy of Zero-shot CLIP is high, as seen in the cases of UCF101 and Caltech101. The underlying reason behind this could be the combination of image-image similarity scores with text-image similarity scores, which might impede the potential of few-shot learning. In other words, in such scenarios, Meta-Adapter appears to favor zero-shot knowledge over few-shot knowledge, resulting in an imbalance between the two. Furthermore, empirical studies indicate that Meta-Adapter struggles to be applied to open-vocabulary semantic segmentation tasks where obtaining high-quality few-shot datasets is challenging. It is possible that incorporating external data could help alleviate this issue. These aforementioned challenges are left for future research and investigation.
Conclusion This paper demonstrates the significant potential of Meta-Adapter, a new few-shot learning method for CLIP, which is designed to overcome the limitations of previous methods in terms of poor generalization ability and low efficiency. The Meta-Adapter, employing a meta-testing mechanism and a lightweight residual-style network, extracts knowledge from few-shot samples without the need for additional fine-tuning, thus alleviating the over-fitting issue while maintaining high efficiency. Our findings highlight the impressive performance of the Meta-Adapter in various tasks, including image classification, object detection, and segmentation, indicating its superior generalization capabilities across datasets and tasks. Future work could focus on further refining the Meta-Adapter and exploring its potential applications in other vision tasks, advancing the capabilities of few-shot learning techniques in visual concept modeling.
This work was supported by the National Natural Science Foundation of China (No. 62204200).