SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multi-modal prompt tuning: Moving beyond single-modal prompt tuning, MPT is a recently introduced task that learns textual prompts and visual prompts at the same time. This jointly tuning strategy not only distills the multi-modal knowledge but enables the dynamic alignments between prompts across modalities, showing better generalization. Zang et al. [27] propose a unified prompt tuning framework (UPT) [27] that shares an initial prompt across different modalities and designs a tiny network to generate the modality-specific prompts together. Almost parallel to UPT, Khattak et al. [28] proposed multi-modal prompt tuning (MaPLe) and adopted a projection matrix to condition vision prompts on their language counterparts explicitly allowing mutual propagation of gradients to promote synergy. In comparison, this work aims to learn multi-modal multi-mode prompts to better meet the requirement of diverse comprehensive representations. Besides, unlike measuring the similarity between images and labels by the global prompt-level features, we model each prompt as an empirical distribution over the token-level embedding space, and the similarity score is calculated by combining the prompt and token-level features under a hierarchical OT framework, which provides a novel and elegant tool to adapt PVLs into downstream tasks.
Datasets To make a comprehensive evaluation, we performed extensive experiments on 4 task settings, such as few-shot image recognition, base-to-new generalization, cross-dataset transfer learning, and domain generalization. Those experiments are conducted on 15 widely used image datasets, varying in scale and domains, including ImageNet [44], Caltech101 [45], OxfordPets [46], StanfordCars [47], Flowers102 [48], Food101 [49], FGVCAircraft [50], EuroSAT [51], UCF101 [52], DTD [53], SUN397 [54], ImageNetV2 [55], ImageNet-Sketch [56], ImageNet-A [57], and ImageNet-R [58]. The details of each dataset are provided in the Appendix Table. B.1.
Baselines We compare ALIGN with the state-of-the-art methods, including: CLIP [1], which provides the base results without prompt tuning; the single-modal prompt tuning methods,e.g.,
TPTs: CoOP [3], CoCoOp [4] and PLOT [17], and VPTs: VPT [6], and multi-modal prompt tuning methods: UPT [27] and MaPLe [28]. Note that we modified the official code of PLOT and changed the backbone to ViT-B/16 for a fair comparison.
Implementation Details Following previous MaPLe [28], we load the pre-trained Vit-B/16 CLIP model as our backbone, where  $d_{l} = 512$ ,  $d_{v} = 768$  and  $d = 512$ . We set the number of textual and visual prompts  $M = N = 4$ , the length of prompt tokens  $b = 2$ , the hyperparameter  $\lambda = 0.1$ , and  $\beta = 1$ . The maximum iteration number in the Sinkhorn algorithm is set as 100. For all tasks, we train our model with a batch-size of 4, a learning rate of 0.0035, and an optimizer as SGD. For each task, we optimize the number of epochs. Following MaPLe we run 2 epochs to train ImageNet as a source model with a learning rate of 0.0026. The reported results are the averaged value over 3 seeds. Please refer to the Appendix Sec. B for more details. For all baselines, we set the length of prompts as 4 and collect their results according to the original papers or previous works. Thus, some experimental results may be missing.
Figure 3: The few-shot learning results on 7 datasets (more detailed results of other datasets can be found in the Appendix Table. D. 1.). The red solid line denotes our ALIGN method, and the dotted lines represent various baselines. All results are reported as the mean value over three seeds.
5.2 Evaluation with the Standard Setting
Few-Shot Learning. We first evaluate our model on few-shot classification, where models are trained on 1,2,4,8, and 16 shots and then applied to the test sets. We report the accuracy scores of all models across 11 datasets at Fig. 3. Overall, our proposed ALIGN outperforms others in most cases and achieves consistent improvement over CoOp on all datasets. Among the multimodal prompt tuning methods, our method shows superior performance compared to UPV and MaPLe in general, except for the EuroSAT datasets. This demonstrates that ALIGN has the ability to distill the across-modalities knowledge and efficiently adapt PVLs into downstream tasks. Although there is a small margin between our model and those two models on some datasets, the competing models usually cannot achieve high performance over all datasets, e.g., ALIGN exhibits  $9.06 / 3.56 / 2.56 / 2.61 / 0.78(\%)$  accuracy improvements compared with UPT on DTD datasets and achieves  $6.56 / 8.75 / 7.63 / 6.64 / 4.47(\%)$  improvements compared with MaPLe. In addition, we also find that our ALIGN performs better on  $1 / 2 / 4$  shots settings, showing the efficiency of our fine-grained alignments, which provide the token-level comparison during prediction. This capability contributes to more accurate classification, even with limited training samples.
Base-to-New Generalization. To assess the generalizability of our model, we follow CoCoOp [4] to equally split the classes into base and new sets, and models are only trained on the base classes while tested on the new sets. Table. 1 reports the results, and we have the following observations: First, our proposed ALIGN surpasses previous baselines by achieving the highest average scores,
Table 1: Base-to-New on 11 datasets. The prompts are learned from the 16-shots base set. We report the classification accuracy on base set (Base), new set (New), and their harmonic mean (H), where  $\mathrm{H} = (2 \times \text{Base} \times \text{New}) / (\text{Base} + \text{New})$ . The best results are highlighted.
Average	ImageNet	Caltech 101	Oxford Pets
Base	New	H	Base	New	H	Base	New	H	Base	New	H
CLIP	69.34	74.22	71.69	72.34	68.14	70.21	96.84	94.00	95.39	91.17	97.26	94.11
CoOp	82.66	63.22	71.65	76.14	67.88	71.77	98.00	89.81	93.72	93.67	95.29	94.47
CoCoOp	80.47	71.69	75.83	75.98	70.43	73.10	97.96	93.81	95.84	95.20	97.69	96.43
PLOT	77.20	60.38	67.76	75.97	69.23	72.44	96.53	82.86	89.17	93.45	79.76	86.06
MaPLe	82.28	75.14	78.55	76.66	70.54	73.47	97.74	94.36	96.02	95.43	97.76	96.58
ALIGN	83.38	75.51	79.25	76.89	72.15	74.45	98.37	94.70	96.50	95.67	97.93	96.79
Stanford Cars	Flowers 102	Food 101	FGVC Aircraft
Base	New	H	Base	New	H	Base	New	H	Base	New	H
CLIP	63.37	74.89	68.65	72.08	77.80	74.83	90.10	91.22	90.65	27.19	36.29	31.08
CoOp	78.12	60.40	68.12	97.60	59.67	74.06	88.33	82.26	85.18	40.44	22.30	28.74
CoCoOp	70.49	73.59	72.10	94.87	71.75	81.71	90.70	91.29	90.99	33.41	23.71	27.74
PLOT	61.41	42.69	50.37	95.26	56.03	70.56	88.45	85.28	86.84	29.63	16.17	20.92
MaPLe	72.94	74.00	73.47	95.92	72.46	82.56	90.71	92.05	91.38	37.44	35.61	36.50
ALIGN	77.24	76.38	76.80	97.70	73.3	83.75	90.77	92.07	91.42	37.56	36.97	37.26
SUN 397	DTD	EuroSAT	UCF 101
Base	New	H	Base	New	H	Base	New	H	Base	New	H
CLIP	69.36	75.35	72.23	53.24	59.90	56.37	56.48	64.05	60.02	70.53	77.50	73.85
CoOp	80.60	65.89	72.50	79.44	41.18	54.24	92.19	54.74	68.69	84.69	56.05	67.45
CoCoOp	79.74	76.86	78.27	77.01	56.00	64.85	87.49	60.04	71.21	82.33	73.45	77.64
PLOT	78.56	72.34	75.32	69.87	53.63	60.68	87.39	67.63	74.30	72.71	41.51	52.84
MaPLe	80.82	78.70	79.75	80.36	59.18	68.16	94.07	73.23	82.35	83.00	78.66	80.77
ALIGN	82.47	79.68	81.05	82.13	54.17	65.28	94.03	74.9	83.38	84.43	78.33	81.27
Table 2: Cross-dataset transfer learning accuracy results. Here we use the key letters to denote the datasets. The best results are highlighted.
Method	Source	Target
ImageNet	Cal	Pets	Cars	Flo	Food	FGCV	SUN397	DTD	EuroSAT	UCF101	Average
CoOp	71.51	93.70	89.14	65.41	68.71	85.30	18.47	64.15	41.92	46.39	66.55	63.88
CoCoOp	71.02	94.43	90.14	65.32	71.88	86.06	22.94	67.36	45.73	45.37	68.21	65.74
MaPLe	70.72	93.53	90.49	65.57	72.23	86.20	24.74	67.01	46.49	48.06	68.69	66.30
ALIGN	72.03	93.91	90.55	65.84	73.75	86.40	24.95	67.59	46.75	47.25	69.60	67.15
thereby illustrating the superiority of the proposed framework. Second, ALIGN outperforms others in terms of H score across all datasets, except for the DTD dataset which indicates our method offers a more favorable trade-off between the base and new sets. We attribute this success to the token-level multi-mode prompt tuning strategy, where the multi-mode prompts enhance the ability to identify diverse visual concepts, which plays an essential role in unseen category prediction. Furthermore, for datasets that have small intra-class variances, such as Stanford Cars and FGVCAircraft, ALIGN achieves a noticeable improvement over MaPLe. The token-level alignment in ALIGN might account for this improvement, as it makes it more effective for fine-grained image classification.
Transfer Learning and Domain Generalization. To investigate the generalizability across-datasets or across-domains, we first train our model on ImageNet, utilizing all 1,000 classes, and subsequently apply it to 1) other 10 datasets and 2) other 4 domain shift datasets. We report those results at Table. 2 and 3, respectively. Based on those results, we find that our approach outperforms the baseline methods on 8/10 datasets with the best average accuracy score on dataset transfer learning task and 3/4 datasets on domain shift setting. These overall improvements highlight that ALIGN is less susceptible to the distribution shift between the source and target domains, thus revealing the robust generalizability of our model. Despite the marginal performance gain of ALIGN in contrast to MaPLe and UPT, our method outperforms them in most cases in terms of all four tasks and provides a novel multi-mode token-level alignment alternative for prompt tuning.
Figure 4: Visualization of the learned prompts and tokens. p_m denotes the  $m$ -th prompt and p_m_t_l denotes the  $l$ -th token of  $m$ -th prompt.
Table 3: Cross-domain generalization accuracy results. The best results are highlighted.
Method	Learnable	Source	Target
ImageNet	ImageNetV2	ImageNet-Sketch	ImageNet-A	ImageNet-R
CLIP	X	66.73	60.83	46.15	47.77	73.96
CoOp	✓	71.51	64.20	47.99	49.71	75.21
CoCoOp	✓	71.02	64.07	48.75	50.63	76.18
VPT	✓	70.57	63.67	47.66	43.85	74.42
UPT	✓	72.63	64.35	48.66	50.66	76.24
MaPLe	✓	70.72	64.07	49.15	50.90	76.98
ALIGN	✓	72.03	64.64	49.96	50.94	76.16
Qualitative Analysis Besides the extensive quantitative results, we are also interested in the learned visual concepts of the proposed token-level alignments. Fortunately, the transport plan learned in our token-level OT provides us with access to a convenient tool to visualize the most related visual patches. We report the qualitative analysis at Fig. 2(b) and Fig. 4. From Fig. 2(b), we find that our model prefers to learn separable representations in both base and new classes. Recalling that  $h^n$  denotes the global feature of  $n$ -th prompts, to visualize the learned prompt and obtain the attention map over the patch space, we calculate the cosine similarity between  $h^n$  and all patch embeddings  $\hat{e} \in R^{d \times O}$ . We then view the normalized cosine similarity as the attention map and visualize the learned  $N = 4$  prompts at the top row of Fig. 4. We observe that different prompts tend to align different patch regions, each of which contributes to the final prediction. This finding also meets with the motivation of the multi-mode prompt tuning, where each prompt aims to learn specific visual semantics.
Moving beyond the prompt-level visualization, we also visualize the token-level concepts. Specifically, for  $l$ -th column of the learned transport plan  $\hat{\mathbf{T}}_l \in R^J$  in token-level OT, it measures how likely the  $l$ -th token is transported to  $J = b + O$  patches. Here we focus on the  $O$  image patches and visualize the transport plan of an image sampled from the base set and new set at the second and third row in Fig. 4, respectively. We find that 1) different tokens within a prompt can capture various patches with similar visual concepts. For example, both p_1_t_2 and p_1_t_3 attend to the head of the cat; 2) Learning from the base set, the prompt tokens prefer to align the similar patches in the new set, which reveals that our token-level alignment module has the ability to transfer from the base set to the new set, rather than over-fitting to the base categories.
Complexity Analysis As discussed above, one of the key ideas of the proposed ALIGN is to learn multiple prompts for vision and language inputs and explore the token-level alignments under the hierarchical OT framework. To demonstrate the computation cost, we report the complexity analysis at Table. 4, where we focus on the number of trainable parameters (#Paras) and inference speed (fps). We find that 1) Overall, the multimodel prompts tuning methods (last three) require more trainable parameters and inference time than single-modal methods. 2) The proposed ALIGN requires
Table 4: Complexity analysis over various baselines. we report the number of trainable parameters (#Paras) and frames per second(fps). We can not report the fps result of UPT because of its unreleased code.
Methods	CoOp	CoCoOp	VPT	PLOT	UPT	MAPLE	ALIGN
#Paras	2,048	35,360	13,824	8,192	3,555,072	3,555,072	3,582,720
fps	645	37	152	583	-	282	62
slightly more training parameters than UPT and MAPLE because of the multiple prompts. And it also requires more inference time than MAPLE, due to the hierarchical OT operations. 3) Thanks to the independent OT operations, which can be calculated parallelly with the GPU, ALIGN has a faster testing time than CoCoOp, and achieves 62 fps at the test stage.
This paper introduces a novel multi-mode token-level alignment framework for multi-modal prompt tuning under optimal transport. We first employ the prompt-level OT to model the multi-mode prompts across modalities, and then introduce the token-level OT by viewing each prompt itself as a set over token embedding space. By coupling those two-level OT via the cost matrix, the final prediction is obtained by combining the prompt-level features and the token-level embeddings, enabling fine-grained alignments. Extensive experiments have been conducted, showing that our proposed model achieves competing performance on four settings. In terms of the limitations, the users may still need large GPU memory to load the pre-trained weights of PVLs to apply the proposed model to the test process. One potential solution is to combine prompt tuning with knowledge distillation. We leave it as a future study. Thanks to the open-world visual concept understanding of PVLs, our model shows promising zero-shot/few-shot ability for image recognition, which has the potential to encourage researchers to derive new and better methods for prompt tuning. Our work may indirectly lead to a negative impacts if there is a sufficiently malicious or ill-informed choice of a few-shot classification task.
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22160; in part by the 111 Project under Grant B18039, in part by the Fundamental Research Funds for the Central Universities; in part by the Innovation Fund of Xidian University under Grant YJSJ23016;
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 8748-8763. PMLR, 2021.
[2] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. PMLR, 2022.
[3] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[4] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[50] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
[51] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[52] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
[53] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[54] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[55] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize toImagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[56] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[57] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[58] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
We in this paper propose ALIGN, a unified framework for multi-modal prompt tuning, where multi-mode modality-specific prompts are learned via the token-level alignment strategy. Moving beyond the single-model methods, which focus on textual prompt tuning or visual prompt tuning, ALIGN allows one to learn textual and visual prompts simultaneously, resulting in better representations in the shared vision-text embedding space. Compared to recent multi-modal methods, such as UPT [27] and MaPLe [28], ALIGN prefers to learn multi-mode prompts to capture diverse class attributes and develop the token-level alignment for fine-grained comparisons. This provides ALIGN with an efficient tool to calculate the similarity between prompts. We find that many previous works can be merged into our ALIGN framework with special hyperparameter settings. We summarize this relationship at Table. 5. The N/A in Table. 5 means that PLOT calculates the similarity between the prompt-level label embeddings and the visual patch embeddings, which is not the case in ALIGN, where we calculate the similarity of prompt-level OT between textual label embeddings and visual image embeddings. and calculate the similarity of token-level OT between token embeddings and patch embeddings.
Table 5: Most previous works can be merged into our ALIGN framework.  $M$  : Number of visual prompts.  $N$  : Number of textual prompts.  $\beta$  : Weight of token-level OT in Eq.6 in the manuscript.
Methods	Type	M	N	β
CoOp [3]	Textal Prompt Tuning	0	1	0
VPT [6]	Visual Prompt Tuning	1	0	0
PLOT [17]	Textual Prompt Tuning	0	≥ 1	N/A
UPT [27]	Multi-modal Prompt Tuning	1	1	0
MaPLe [28]	Multi-modal Prompt Tuning	1	1	0
ALIGN(Ours)	Multi-modal Prompt Tuning	≥ 0	≥ 0	≥ 0
B Data statistics and Hyperparameter setting
We thoroughly evaluate our proposed ALIGN framework across four distinct tasks: few-shot recognition, base-to-new generalization, cross-dataset transfer, and cross-domain generalization. These extensive experiments are conducted on a diverse set of fifty commonly used vision datasets, covering various contexts. These datasets include ImageNet [44] and Caltech101 [45] for generic image classification, OxfordPets [46], StanfordCars [47], Flowers102 [48], Food101 [49], and FGVCAircraft [50] for fine-grained image recognition, SUN397 [54] for scene recognition, UCF101 [52] for action recognition, DTD [53] for texture classification, and EuroSAT [51] for satellite imagery recognition. In the case of the cross-domain generalization task, our model is trained on ImageNet and subsequently tested on ImageNetV2 [55], ImageNet-Sketch [56], ImageNet-A [57], and ImageNet-R [58]. We summarize data statistics at Table. B. 1
The evaluation pipeline for each task follows the approach employed by previous works [3, 28]. The specific details of this pipeline are summarized below:
Few-shot Recognition. To evaluate the efficiency of the proposed ALIGN on the few-shot case, we follow CoOp [3], and first partition the dataset into base and novel sets. Those two sets share the same categories. Models are trained on the base set using a variety of shot settings, including 1, 2, 4, 8, and 16 shots per class, and then tested on the full novel set. The accuracy scores are reported to compare the performance. The training epoch is set as 10 for 1, 2, and 4 shots and 40 for 8 and 16 shots.
Base-to-New Generalization. To show the Generalizability of unseen categories, we first divide the dataset into two separate subsets: the base subset and the new subset. Importantly, these subsets do not share the same categories. The base subset contains a specific set of categories used for model training, while the new subset consists of previously unseen categories that the model has not
been exposed to during training. Besides reporting the accuracy score on base and novel sets, we also calculate the harmonic mean  $\mathrm{H} = (2\times \mathrm{Base}\times \mathrm{New}) / (\mathrm{Base} + \mathrm{New})$ , which acts as a trade-off between Base and New, providing a comprehensive measure of overall model performance. The training epoch is set as 8.
Cross-Dataset Transfer. To determine the transferability of our model across different datasets, we first train our model on the source dataset (ImageNet) and then evaluate it on 10 different target datasets. The training epoch is set as 2 and the learning rate is set as 0.0026.
Cross-Domain Generalization. To determine the robustness of our model on the distribution-shift setting, we trained our model on the source dataset (ImageNet) and then assess it on 4 domain-shifted datasets, including ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R. The training epoch is set as 2 and the learning rate is set as 0.0026.
The other training hyperparameters in the previous experiments are set according to MaPLe [28], which are detailed listed at Table B.2.
Table B. 1: Statistics of the used 15 datasets. N/A denotes that we do not use the corresponding training or validation sets.
Dataset	Domains	#Classes	#Train	#Val	#Test
ImageNet	generic object	1000	1.28M	N/A	50,000
Caltech101	generic object	100	4,128	1,649	2,465
OxfordPnts	fine-grained object	37	2,944	736	3,669
StanfordCars	fine-grained object	196	6,509	1,635	8,041
Flowers102	fine-grained object	102	4,093	1,633	2,463
Food101	fine-grained object	101	50,500	20,200	30,300
FDVCAircraft	fine-grained object	100	3,334	3,333	3,333
SUN397	scene recognition	397	15,880	3,970	19,850
UCF101	action recognition	101	7,639	1,808	3,783
DTD	texture recognition	47	2,820	1,128	1,692
EuroSAT	satellite object	10	13,500	5,400	8,100
ImageNetV2	generic object	1000	N/A	N/A	10,000
ImageNet-Sketch	sketch object	1000	N/A	N/A	50,889
ImageNet-A	generic object	200	N/A	N/A	7,500
ImageNet-R	generic object	200	N/A	N/A	30,000
C Training Algorithm