SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Image-Text Retrieval. Table 1 shows performances for image-text retrieval tasks on MSCOCO [47] and FLICKR30K [54]. It can be observed that our Aurora  $(\mathrm{R} = 64)$  achieves comparable results with the state-of-the-art frozen backbone method while using only  $0.5\%$  of its parameters. When we
Table 2: Results on two benchmark video-text retrieval datasets, MSR-VTT and DiDemo. Recall@  $K$  and MdR are utilized as the evaluation metric, where MdR measures the median rank of target items in the retrieved ranking list. Input means the sampling number and frame shape of each video.
Method	Input	# Pretrain	# Tunable	MSR-VTT	DiDemo
R@1	R@5	R@10	MdR	R@1	R@5	R@10	MdR
Methods with full fine-tuning
ClipBERT	16×448	5M	135M	22.0	46.8	59.9	6.0	20.4	48.0	60.8	6.0
Frozen in Time	32×224	5M	180M	31.0	59.5	70.5	3.0	34.6	65.0	74.7	3.0
ALPRO	8×224	5M	245M	33.9	60.7	73.2	3.0	35.9	67.5	78.8	3.0
VIOLET	5×224	138M	306M	34.5	63.0	73.4	-	32.6	62.8	74.7	-
All-in-one	9×224	138M	110M	37.9	68.1	77.1	-	32.7	61.4	73.5	3.0
CLIP4Clip	12×224	400M	124M	43.1	70.4	80.8	2.0	42.8	68.5	79.2	2.0
CLIP-Hhiker	120×224	400M	124M	47.7	74.1	82.9	-	-	-	-	-
Methods with frozen backbone
CLIP-Prompt	16×224	400M	64M	36.7	64.6	-	-	-	-	-	-
LoRA (r=32)	8×224	129M	10.6M	49.9	72.0	81.3	2.0	50.9	75.3	82.4	2.0
UniAdapter (r=128)	8×224	129M	4.6M	49.7	71.9	81.5	2.0	49.0	75.5	83.3	2.0
UniAdapter (r=512)	8×224	129M	18.8M	50.6	73.4	81.6	1.0	52.1	77.3	85.2	1.0
Aurora (ours, r=64)	8×224	129M	0.1M	52.4	73.9	82.0	1.0	53.1	77.4	85.3	1.0
ΔPETL	-	-	0.5%	+1.8	+0.5	+0.4	+0.0	+1.0	+0.1	+0.1	+0.0
Table 3: Results on two visual question answering datasets, VQAv2 and MSRVTT-QA. # Tunable represents the number of learnable parameters. For VQAv2, we report the test-dev and test-standard results, for MSRVTT-QA, accuracy is used as the evaluation metric.
Method	# Tunable	VQAv2	Method	# Tunable	MSRVTT-QA test acc
test-dev	test-standard
Methods with full fine-tuning
VL-T5/BART	165M	-	71.30	CLIPBERT	135M	37.4
SOHO	155M	73.25	73.47	ALPRO	245M	42.1
OSCAR	330M	73.61	73.82	Just-Ask	200M	41.5
UNITER	330M	73.82	74.03	VIOLET	306M	43.9
ALBEF	266M	75.84	76.04	MERLOT	233M	43.1
BLIP	337M	77.44	77.48	All-in-one	110M	44.3
Methods with frozen backbone
UniAdapter (r=128)	4.6M	73.72	73.71	UniAdapter (r=128)	4.6M	44.2
UniAdapter (r=512)	18.8M	75.44	75.56	UniAdapter (r=512)	18.8M	44.7
Aurora (ours, r=64)	0.1M	77.69	77.87	Aurora (ours, r=64)	0.1M	44.8
ΔPETL	-	+2.25	+2.31	ΔPETL	-	+0.1
increase the rank to 128, Aurora can further boost the performance, surpassing all the frozen backbone methods, and even outperforming some full fine-tuning methods with fewer trainable parameters.
Video-Text Retrieval. To further verify the performance of our Aurora in the field of video-text retrieval, we conduct experiments on two video datasets, MSRVTT [69] and DiDemo [5], and the results are presented in Table 2. With only about 0.1M trainable parameters, our Aurora directly achieves better performances than all the frozen backbone methods, and we outperform most full fine-tuning methods. This indicates that our Aurora has an excellent understanding ability under video-text scenes, even with relatively few trainable parameters.
Visual Question Answering. In an effort to further explore the potential of our Aurora, we evaluate it on VQA/VideoQA tasks with VQAv2 [24]/MSRVTT-QA [68] datasets and demonstrate the evaluation results in Table 3. Unlike retrieval tasks, VQA task needs to verify the model's multimodal generative ability. We share the trainable parameters of the multimodal encoder and multimodal decoder to further reduce the parameter amount. From the results, we find that our Aurora outperforms UniAdapter and all the full fine-tuning methods, indicating that Aurora can have powerful transfer ability for downstream generative tasks.
4.3 Performance Comparisons on Zero-shot Setting
To evaluate the generalization ability of Aurora, we conduct experiments on cross-modal tasks with zero-shot setting, and make comparisons with the pretrained version, full fine-tuning method, LoRA, and UniAdapter.
Table 4: Zero-shot performance analysis.
Method	# Parameter	MSRVTT (T→V)	DiDemo (T→V)
R@1	R@5	R@10	R@1	R@5	R@10
BLIP	223M	41.5	62.0	70.7	42.1	59.6	67.3
BLIP + FFT	223M	42.5	62.8	71.6	43.0	60.5	68.3
BLIP + LoRA	10.6M	42.7	62.8	71.4	43.4	60.3	68.2
BLIP + UniAdapter	18.8M	42.2	62.6	71.1	43.1	60.2	67.9
BLIP + Aurora	0.1M	43.1	63.5	72.0	44.6	61.4	68.6
(b)
Figure 3: The answer to how rank  $R$  affects Aurora. (a), (b), and (c) show the performance increase accompanied with larger  $R$  on three different cross-modal tasks. Notably, our results are divided on two  $y$ -axes for clear demonstration, where Recall@1 is shown on the left axis and Recall@5/10 are on the right one. (d) compares the parameter scalability with other PETL methods.
Specifically, experiments are based on the pre-trained large-scale multimodal foundation models, i.e., BLIP, which is further tuned in each corresponding method on MSCOCO. Then, we utilize two video datasets as the zero-shot data for validation. Table 4 provides an overview of the performance of Aurora on various zero-shot multimodal tasks. It is obvious that Aurora achieves the highest zero-shot performance while requiring the least number of trainable parameters during vision-language pre-training, which represents more powerful general-purpose understanding ability.
4.4 Analysis of Different Designs
We thoroughly evaluate how the core operations in Aurora affect results, i.e., the rank  $R$ , parameter sharing, context enhancement, and the gated query transformation. We conduct experiments to analyze the impacts of different designs, which are implemented on several cross-modal tasks.
How Rank of CP Decomposition Affects Aurora?
Results on Flickr30K and DiDemo are shown in the left three columns in Figure 3. A fundamental conclusion is that as  $R$  increases, the dimensionality of the model representation also increases, leading to better performance on downstream tasks. However, when  $R$  reaches a certain range, the rate of increase slows down, which may be related to the redundancy of high-dimensional features.
In addition, we show in (d) of Figure 3 that as  $R$  increases, the growth rate of parameter size in Aurora is much slower than that of LoRA and UniAdapter. As another tensor decomposition method, our scalability and ease of deployment are much stronger than the baselines.
How Does Aurora Benefit from Informative Context Enhancement?
For an in-depth investigation, we introduce three ablated variants, i.e., Aurora w/o C removes the context enhancement, Aurora w. R replaces the context with random vectors, and Aurora w. M simply takes the average features of all queries. Ablation results are shown in Figure 4, where we separate the results on two  $y$ -axes respectively, which can better demonstrate the differences across different indicators more clearly. And the results of our Aurora are marked by the black upper lines. Surprisingly, using random vectors to supplement fused features still achieves better
Figure 4: Analysis of the impact of the informative context enhancement module.
results than removing this module. Moreover, we observe that adaptive weight  $\alpha$  causes better performance than the uniform weight. These phenomena fully demonstrate that with few parameters,
Figure 5: Visualization of cross-attention map comparisons on Flickr30K, which shows the capability to locate the most semantic-related visual parts for specific words in the text.
Figure 7: The left three columns show the parameter distribution of the pre-trained large-scale multimodal foundation model (BLIP) vs. our Aurora, which is tuned on MSCOCO. And the right part is full fine-tuned model vs. Aurora. Notably,  $\mathcal{W}_q$  is the stack of the query projection matrices in different modality branches.
supplementing more context information to fused features is crucial to promote modality alignment for downstream tasks, and our context enhancement is the optimal choice.
How Does Aurora Benefit from Parameter Sharing?
See Figure 5 for the effectiveness of parameter sharing. We observe that the parameter sharing strategy can achieve a better capability to locate the most semantic-related visual parts for specific words in the text. It should be noted that we are able to reduce 0.4M parameters through parameter sharing. Moreover, we also conduct quantitative experiments on FLICKR30K, in which parameter sharing results in better performance with an average improvement of 0.1. Therefore, we can draw the conclusion that parameter sharing during training not only decreases the parameter dependence but also boost the cross-modal interaction for better performances.
How Does Aurora Benefit from Gated Query Transformation?
To further validate the effectiveness of the Gated Query Transformation module, we conduct a comprehensive ablation study on MSCOCO and DiDemo datasets with two variants. The experimental results are shown in Figure 6, where Aurora w/o Q (blue bars) represents the removal of the Gated Query Transformation and Aurora w. O (red bars) initializes the transformation matrix and bias with ones. The gap between the blue bars and ours shows that this module helps promote the final performance to some extent, which also indicates that adaptively incorpora
Figure 6: Analysis of the impact of the gated query transformation module.
ing text information into the modality fusion branch has a positive effect on modality alignment under the deep network architecture. In addition, the difference between the red bars and ours shows that initializing from zeros is more conducive to adaptively controlling the proportion of text feature fusion, and can gradually achieve better modality alignment for better results.
4.5 Visualization Analysis
Parameter Distribution. We visualize the parameter distributions of the pre-trained model, fully fine-tuned model, and our Aurora in Figure 7. Aurora involves gradually embedding the knowledge learned from downstream tasks into the parameters of a pre-trained model, while amplifying the original parameters in a certain direction. From the left part, we can see that our Aurora only adjusts the pre-trained model parameters in a small local range, but it can have a better effect on downstream
tasks. Meanwhile, we can see from the right part that Aurora's parameter distribution is very close to that of the fully fine-tuned model, with only small changes in a small range. These small changes enable our method to achieve superior performance on many downstream tasks.
This paper proposes Aurora, a graceful prompt framework for cross-modal transfer. We first leverage mode approximation to implement multimodal prompt tuning, which explores the low intrinsic dimension with only  $0.04\%$  parameters of the pretrained model. Then, to better reduce the modality gap, we propose Informative Context Enhancement and Gated Query Transformation modules under extremely low parameter scenarios. Extensive evaluation of Aurora on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even surpasses full fine-tuning approach.
Limitations. The representation ability of our Aurora to some extent depends on the setting of the rank  $R$ . However, it also unavoidably increases the parameter size and thus the computational cost. Therefore, it is crucial for us to select a proper rank in the trade-off between parameter size and performance. Unfortunately, it is challenging for us to determine a suitable rank beforehand with the variations of downstream tasks and data sizes, which poses a challenge to the application of our model. We will further optimize this issue in future work.
Broader Impact. Multimodal pre-trained large models have wide applications in the real world, including image-text understanding, retrieval, question answering, and generation. As models scale up, they will achieve more amazing performances. Our Aurora enables lightweight transfer of large-scale models, which allows us to better utilize cloud-based large models and guide them to produce high-performance edge applications under constrained computational resources.
[1] RZdunek ACichocki et al. Nonnegative matrix and tensor factorizations: Applications to exploratory multi-way data analysis and blind source separation, 2009.
[2] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[4] Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of machine learning research, 15:2773-2832, 2014.
[5] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017.
[6] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[7] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. A clip-hitchhiker's guide to long video retrieval. arXiv preprint arXiv:2205.08508, 2022.
[8] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[74] Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al. Visual tuning. arXiv preprint arXiv:2305.06061, 2023.
[75] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-23651, 2021.
[76] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
[77] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
To provide a comprehensive demonstration of our approach, we will supplement additional details. The arrangement of these sections is as follows: First, we demonstrate the core concepts of our Aurora for clarity. Second, we comprehensively make overall comparisons with existing methods on various multimodal tasks. Then, we provide details regarding the datasets and baselines in Section C, while the concrete training details are outlined in Section D. We then conduct a comprehensive analysis of the computational costs, including time and memory consumption, along with algorithmic complexity in Section E. Furthermore, we provide theoretical support for our approach in Section F. Finally, we present visualizations of our proposed Aurora for several cross-modal tasks to facilitate qualitative comparisons in Section G. To represent our method clearly and concisely, we use lowercase letters for scalars, bold lowercase letters for vectors, italicized uppercase letters for MATRICES, and bold italicized uppercase letters for TENSORS in the equations, respectively.
A Concept Definition
According to [1, 51], we will offer a precise definition of the fundamental notions underpinning our key Mode Approximation component.
First, the definition of tensors can be demonstrated as follows:
Definition 1 (Tensor). Let  $\mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_N \in \mathbb{N}$  denote index upper bounds, a tensor  $\mathcal{W} \in \mathbb{R}^{\mathcal{D}_1 \times \dots \times \mathcal{D}_N}$  of order  $N$  is an  $N$ -way array where elements  $\mathcal{W}_{d_1, d_2, \dots, d_n}$  are indexed by  $d_n \in \{1, 2, \dots, \mathcal{D}_n\}$ , for  $1 \leq n \leq N$ .
Then, the concept of the mode is formulated as follows:
Definition 2 (Mode). Let  $\mathcal{W} \in \mathbb{R}^{n_1 \times n_2 \times \dots \times n_d}$  be a  $d$ -dimensional tensor. The mode- $k$  matricization of  $\mathcal{W}$ , denoted as  $\mathcal{W}^{(k)} \in \mathbb{R}^{n_k \times (n_1 \dots n_{k-1} n_{k+1} \dots n_d)}$ , is obtained by unfolding the tensor along its  $k$ -th mode and arranging the entries as rows in a matrix.
Given the mathematical definition of the mode, we can implement decomposition in the context of CP decomposition. We stack all the weight matrices in the attention layer (i.e.,  $W_{q}, W_{k}, W_{v}$ ) of all the branches into a tensor, which needs to be updated as  $\Delta \mathcal{W}$ . Since our method assumes that the stack of weight matrices is a three-order tensor,  $k$  is three in our settings, and thus the CP decomposition can be illustrated as follows:
$$
\boldsymbol {\Delta} \boldsymbol {\mathcal {W}} = \sum_ {r = 1} ^ {R} \lambda_ {r} \mathbf {u} _ {r} \circ \mathbf {v} _ {r} \circ \mathbf {p} _ {r}, \tag {6}
$$
where  $R$  is the rank,  $\lambda_r$  are non-negative scalar weights, and  $\mathbf{u}_r \in \mathbb{R}^{n_1}$ ,  $\mathbf{v}_r \in \mathbb{R}^{n_2}$ ,  $\mathbf{p}_r \in \mathbb{R}^{n_3}$  are non-zero factor vectors. And the mode- $k$  unfolding of the tensor  $\Delta \mathcal{W}$  is  $U, V, P$  respectively.
Our proposed Aurora aims to approximate the latent mode matrix with randomly initialized learnable parameters, which can learn knowledge on downstream tasks in a lightweight manner.
B Overall Comparison
We compare all the baselines for three downstream tasks and presented a comprehensive illustration in Figure 8. The arrow in the figure points towards better performance on dual metrics, as it moves towards the upper right corner. We rank the size of the model parameters and use it as a basis for determining the size of the bubbles, which are also displayed in Figure 8. It is evident that our method performs remarkably well even with smaller parameter sizes, and in several instances, outperforms the fully fine-tuned approach, demonstrating the strength of our architecture.
C More Details for the Baselines & Datasets.
Baselines. For Frozen Backbone methods, UniAdapter [49] is currently the state-of-the-art method for parameter-efficient transfer learning in the field of multimodality and can be considered as a
Figure 8: Performance-Parameter comparison of different methods on each multimodal downstream task. Note that the bubble size denotes the size of total tunable parameters.
representative of the Adapter class of methods in the multimodal domain. LoRA [25] is another important branch of parameter-efficient transfer learning methods. Its core idea is to use low-rank decomposed matrices to calculate the incremental change of model parameters during adaptation on downstream tasks. To enable comparison with a wider range of baselines, we replicate many of the settings from prior works and reuse their experiment results whenever possible. It should be noted that this means some baselines only appear in specific experiments.
As for Full Fine Tuning methods, we apply UNITER [13], VILLA [23], OSCAR [43], ALIGN [29], ALBEF [42] and BLIP [41] for image-text retrieval tasks, then we use ClipBERT [37], Frozen in Time [6], ALPRO [40], VIOLET [22], All-in-one [65], CLIP4Clip [50] and CLIP-Hhiker [7] for text-video retrieval tasks, finally we adopt ClipBERT[37], ALPRO [40], Just-Ask [70], VIOLET [22], MERLOT [75], All-in-one [65] for VideoQA task while adopt VL-T5/BART [14], SOHO [27], OSCAR [43], UNITER [13], ALBEF [42] and BLIP [41] for VQA tasks.
Datasets. We provide a comprehensive introduction to the datasets of various downstream tasks in the multimodal scenario, as outlined below:
- MSCOCO [47] is a large scale image-text dataset and each image is annotated with five captions. Following [49, 34], we use Karpathy split of MSCOCO: 5,000 images for testing, 5,000 images for validation, and the rest for training.
- Flickr30K [54] contains 31,000 images collected from Flickr. Each image is usually referenced with five human annotations. Following previous works [49, 21], we use 1,000 images for testing, another 1000 for validation, and the rest for training.
- MSR-VTT [69] contains 10,000 video clips and each video clip is annotated with 20 English sentences. Following recent works [49, 50], we adopt 1k-test split for training and testing.
- DiDemo [5] is one of the most commonly used datasets for the temporal localization of events in videos. It contains about 10,000 videos and 40,000 annotations. we follow [49, 6] to concatenate all descriptions corresponding to the same video into a single sentence to conduct actually paragraph-to-video retrieval task.
- VQAv2 [24] is one of the most famous visual question answering datasets which contains  $83\mathrm{k} / 41\mathrm{k} / 81\mathrm{k}$  images for training/validation/testing. Following [49, 42, 41], we use both training and validation splits of VQAv2 and additional training samples from Visual Genome [36] for training. The results should be evaluated by the official server and we report the results on the test-dev and test-standard splits.
- MSRVTT-QA [68] is one of the most popular video question answering datasets. It's constructed based on MSRVTT and has 243k open-ended questions associated with 10k videos. We follow [49, 37] to employ the standard split for training and testing.
D More Training Details
D.1 Frozen Backbone
Another important design in BLIP is CapFlit, It contains a Captioner to generate captions given web-searched images and a Filter to remove noisy image-text pairs, both Captioner and Filter are finetuned
Figure 9: Model architecture for the Visual Question Answering tasks on both images and videos.
individually on the COCO dataset while using different objective loss. In addition, BLIP uses momentum technology to further improve the correctness of the image-text matching relationship.
D.2 Architecture for VQA Tasks
Figure 9 shows the architecture of Aurora for Visual Question Answering tasks. Compared to Retrieval tasks, the VQA architecture has an additional answer decoder. During fine-tuning, the images/videos are first encoded by a single-modal visual encoder, and then the image/video-text pairs are fused using a multimodal encoder and given to the decoder for prediction. Answers are used as ground truth and Language-Model Loss is utilized for parameter updating throughout the entire training process. As the ITM Loss is no longer needed, we remove the Informative Context Enhancement module from the VQA architecture. Meanwhile, we retain the Gated Query Transformation module to preserve the complete semantic information of the question representations as much as possible. Finally, in order to further reduce the number of parameters, we share the learnable parameters of the multimodal encoder and the multimodal decoder.
In this section, we give more training details about our Aurora.