SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haixin Wang $^{1,\ast}$ , Xinlong Yang $^{1,\ast}$ , Jianlong Chang $^{2,\dagger}$ , Dian Jin $^{3}$ , Jinan Sun $^{1,\dagger}$ , Shikun Zhang $^{1}$ , Xiao Luo $^{1}$ , Qi Tian $^{2}$
$^{1}$ Peking University,  $^{2}$ Huawei Cloud & AI,  $^{3}$ University of Wisconsin-Madison {wang.hx, xinlong.yang} @stu.pku.edu.cn, {jianlong.chang, tian.qi1} @huawei.com, djin38@wisc.edu, {sjn, zhangsk, xiaoluo} @pku.edu.cn
Driven by the progress of large-scale pre-training, parameter-efficient transfer learning has gained immense popularity across different subfields of Artificial Intelligence. The core is to adapt the model to downstream tasks with only a small set of parameters. Recently, researchers have leveraged such proven techniques in multimodal tasks and achieve promising results. However, two critical issues remain unresolved: how to further reduce the complexity with lightweight design and how to boost alignment between modalities under extremely low parameters. In this paper, we propose A graceful prompt framework for cross-modal transfer (Aurora) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate 0.1M trainable parameters to implement the multimodal parameter-efficient tuning, which explores the low intrinsic dimension with only  $0.04\%$  parameters of the pre-trained model. Then, for better modality alignment, we propose the Informative Context Enhancement and Gated Query Transformation module under extremely few parameters scenes. A thorough evaluation on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even outperforms the full fine-tuning approach. Our code is available at: https://github.com/WillDreamer/Aurora.
The era of large models in deep learning has arrived, with increasingly more large-scale pre-trained models exhibiting remarkable generation and inference capabilities in the fields of text [38, 9, 63], vision [57, 53], and multi-modality [56, 41, 3, 76]. Nowadays, the mainstream strategy in the community involves pre-training models on large-scale data, and then fine-tuning them for each downstream task. This approach has been demonstrated to be effective in various cross-modal tasks, as evidenced by [40, 50, 67].
However, this high-performing strategy is not always advantageous, as several practical factors limit its further development. The primary concern is excessive parameter dependency. For example, GPT-3 [9] (175B params) obtains striking performances while coming at the cost of a massive parameter count. The enormous parameter size has two obvious drawbacks. Firstly, it incurs significant computations and physical storage costs, making pre-training and transfer very expensive. Besides, fine-tuning limits pre-trained knowledge's effectiveness in small-scale downstream tasks. Both of them hinder the further extension of large models from specific datasets to more general scenarios.
To relieve the high cost of large pre-trained models, a series of parameter-efficient transfer learning (PETL) methods [17, 74] have been proposed. The common paradigm is to freeze the backbone
network of the large model and introduce a small number of additional parameters. Recently, some works have begun to focus on PETL in the multimodal field, such as UniAdapter [49], VL-Adapter [62], and MAPLE [33]. However, their common idea is to combine existing architectures used in NLP for multimodal models, which simply inserts learnable parameters in the backbone networks of unimodal and multimodal branches to achieve good performances. Their simple designs cannot integrate the essence of efficient parameter transfer into multimodal models. There are still two main challenges that need to face: one is how to transfer knowledge in an extremely lightweight manner, and the other is how to boost alignment between modalities under extremely low parameters.
To overcome the challenges mentioned above, we propose A graceful prompt framework for cross-modal transfer, named Aurora, which surpasses the state of the art with a remarkably small parameter budget of merely  $\sim 100\mathrm{K}$ , thereby aptly living up to its name due to lightness. First, inspired by the idea of CANDECOMP/PARAFAC (CP) decomposition [64], we utilize mode approximation to generate lightweight learnable parameters for the attention-based architectures. As attention is proven to inevitably fit the background noise [45], most other features are redundant. For example, the categories in CLIP [56] are essentially infinite, which results in only a few features responding to a single class. This can be easily proven by the fact that the logits output by CLIP are mostly between 0.2 to 0.3 with very little fluctuation, indicating that most features are consistent. In other words, although pre-trained models have large-scale high-dimensional parameters, the intrinsic dimension corresponding to each downstream task is not large [55]. Thus, our mode approximation can fine-tune a very small number of parameters to achieve good performance on downstream tasks theoretically.
Second, in contrast to existing methods that directly insert a small network for cross-modal fusion, we mine deeper into the unimodal information and fuse them in a controllable manner with multimodal representations, thereby achieving desirable results with lower parameter dependence. In our proposed Informative Context Enhancement module, each feature is enriched by contextual information to boost the modality fusion. Furthermore, we adaptively control the ratio of textual information supplied for modality fusion in our proposed Gated Query Transformation module.
Finally, we evaluate Aurora on six cross-modal tasks and two zero-shot tasks, which obtains state-of-the-art performance compared to other PETL methods. Compared to the full fine-tuning, Aurora even obtains  $1.8\%$  and  $0.5\%$  performance improvement on MSRVTT and VQAv2 benchmarks averagely but only with about  $0.05\%$  trainable parameters of the pre-trained model.
2.1 Vision-Language Models
In vision-language pre-training, large-scale image-text pairs are used to pre-train the Vision-Language Models (VLMs) for downstream tasks [13, 56, 41, 72, 3, 76]. The architecture of VLMs typically consists of a visual encoder, a textual encoder, and a cross-modal fusion module. VLMs have been applied to various tasks, including image generation [71], image captioning [39, 26], visual question answering [12, 11, 20], and cross-modal retrieval [58, 28]. This article focuses on how to efficiently fine-tune VLMs on downstream tasks and utilizes BLIP-base [41] for vision-language tasks.
2.2 Parameter-efficient Transfer Learning
As the size of recent models increases rapidly, updating the models in parameter-efficient ways becomes crucial. The huge computational burden brought by these pre-trained models will undoubtedly impose unnecessary burdens on transfer learning. PETL [17, 74] methods diverge from the conventional approach of fine-tuning the entire pre-trained model, instead only learning a few additional parameters for knowledge transfer. There are three common categories, prompt tuning [77, 30, 66, 46], adapter tuning [32, 10, 62] and parameter tuning [25, 31, 61]. We aim to relieve the computational burden by developing a graceful prompt framework specifically for cross-modal transfer, adapting frozen pre-trained multimodal models to downstream tasks across a broad distribution.
2.3 Tensor Decomposition
Tensor decomposition [35, 4, 59, 51] is an important research area studied for decades, which aims to approximate a tensor through a set of low-rank factors with diverse applications. In general, the
Figure 1: Comparison of existing PETL methods for downstream cross-modal tasks. (a) Adapter, which involves inserting a learnable small network into a pre-trained model; (b) LoRA, which employs a down and up tensor as updated parameters for low-rank approximation  $(\mathbb{R}\ll \mathrm{d})$ , added to the pre-trained model; and (c) our proposed Aurora, which utilizes mode approximation to further reduce the number of trainable parameters added to the pre-trained model. Notably, the red blocks represent trainable parameters, while the blue ones indicate the frozen backbone.
most widely used decompositions are Tucker decomposition [15] and CANDECOMP/PARAFAC (CP) decomposition [64], both of which can be seen as generalizations of the matrix singular value decomposition (SVD) [60]. CP decomposition can also be seen as a special Tucker decomposition whose core tensor is diagonal, but is more lightweight and explainable, meaning that a high-order tensor can be uniquely represented as the sum of a set of rank-one tensors theoretically. Here, we leverage the idea of CP decomposition to implement mode approximation for lightweight prompts.
Frozen Backbone. BLIP [41] is a unified VLP framework which has multimodal mixture of encoder-decoder(MED) architecture with both understanding and generation capabilities. We utilize BLIP-base as the frozen backbone and the pre-trained weights can be downloaded from Salesforce. Its visual encoder is ViT-B [19] and the text encoder is the same as BERT [16] while the text decoder replaces the self-attention layers with causal self-attention layers. It uses cross-attention layers to gather information from encoded visual representations using the textual representations as query. It's flexible to choose different components in the BLIP architecture to perform different multimodality downstream tasks.
We start with the core attention-based Transformer architecture mostly utilized in existing large-scale multimodal models [56, 44, 73, 41] for representation. The input image/text is divided into several non-overlapping patches, and then these patches appended a [CLS] token are fed into an embedding layer followed by the Transformer blocks with multi-head attention as the core operation. It copies the input embedding into query, key, and value by three projection matrices  $W_{q}^{l}, W_{k}^{l}$ , and  $W_{v}^{l} \in \mathbb{R}^{d \times d}$  respectively, where  $l$  represents the  $l$ -th layer. The pre-trained model contains many dense layers which perform matrix multiplication as follows:
$$
\operatorname {A t t e n t i o n} \left(X _ {q, m} ^ {l}, X _ {k, m} ^ {l}, X _ {v, m} ^ {l}\right) = \operatorname {S o f t m a x} \left(\frac {X _ {q , m} ^ {l} W _ {q , m} ^ {l} \left(X _ {k , m} ^ {l} W _ {k , m} ^ {l}\right) ^ {T}}{\sqrt {d}}\right) X _ {v, m} ^ {l} W _ {v, m} ^ {l}
$$
where  $m \in \{I, T, C\}$  denotes the vision, text, and cross-modal modality. For the unimodal vision/text modality branch,  $m$  are all from the vision/text modality. For the cross-modal modality branch,  $X_{q,m}^{l}$  is from the text modality, while the other two are from the visual modality. Assume there are  $L$  layers in total, we can stack all of the attention-based weight matrices in the multimodal pre-trained model, and derive the tensor  $\mathcal{W}_0 = \{W_q^l, W_k^l, W_v^l\}_{l=1}^L \in \mathbb{R}^{d \times d \times N}$ , where  $d$  is the dimension of the embedding token, and  $N$  denotes the total number of the weight matrices.
Parameter Update. For downstream tasks, directly updating the weight tensor with full fine-tuning will consume a huge amount of computation and brings about a heavy storage burden. We aim to update the knowledge with a few additional trainable parameters following the idea of PETL methods. Due to the redundancies of  $\mathcal{W}_0$ , we hope to implement the mode approximation of the tensor  $\mathcal{W}_0$  to get the new learnable weight tensor  $\Delta \mathcal{W}$  for downstream knowledge transfer. The
Figure 2: Demonstration of the overall framework. The frozen backbone network is shown in grey. The trainable parameters in color represent: blue for vision tasks, pink for text tasks, and the gradient color for fused modalities. Notably, globally shared parameters are represented in purple.
differences between our Aurora and existing PETL methods are demonstrated in Figure 1. Therefore, the backward propagation on the downstream training data  $\mathcal{D}$  can be expressed as:
$$
\nabla_ {\boldsymbol {W}} = \nabla_ {\boldsymbol {\Delta} \boldsymbol {W}} = \frac {\partial \mathcal {L} (\mathcal {D} ; \boldsymbol {W} _ {0} + \boldsymbol {\Delta} \boldsymbol {W})}{\partial \boldsymbol {\Delta} \boldsymbol {W}} \tag {1}
$$
3.2 Lightweight Design for PETL
To ensure a successful parameter-efficient tuning framework, it is important to prioritize a lightweight design that enables easy deployment and scalability. Recent research [45] has shown that when self-attention-based architecture is pre-trained on large-scale datasets, there is a significant amount of feature redundancy due to the limited responses of individual classes in a nearly infinite feature space. In fact, there exists a low-dimensional reparameterization that is as effective for fine-tuning as the full parameter space [2]. This inherent dimensionality describes the minimum dimension required to solve the optimization problem it defines to a certain level of precision. Considering the pre-trained parameters as a tensor, we recognize that approximation can effectively preserve low-rank yet discriminative non-redundant features, and scale down the weight tensor of pre-trained large-scale models along certain directions, thus making them more suitable for downstream tasks. Therefore, we propose a novel mode approximation method named Aurora, which utilizes mode approximation to update the frozen  $\mathcal{W}_0$ . Specifically, we borrow the idea of CANDECOMP/PARAFAC (CP) decomposition to decompose the learnable parameters  $\Delta \mathcal{W}$  into a series rank-one tensors to explore the inherent dimensionality embedded in the features. The detailed architecture is shown in Figure 2.
CP Decomposition. In the typical tensor decomposition method, a 3D tensor has three modes (further introduced in Appendix), each of which can be viewed as the reduced projection of the tensor in a specific dimension. Given the 3D updated weight tensor  $\Delta \mathcal{W} \in \mathbb{R}^{d \times d \times N}$ , CP decomposition factorizes this tensor into a sum of  $R$  rank-one components in total, and each component can be formalized as the outer product of three decomposed vectors in the formulation of:
$$
\boldsymbol {\Delta} \boldsymbol {\mathcal {W}} = \sum_ {r = 1} ^ {R} \lambda_ {r} \left(\boldsymbol {u} _ {r} \circ \boldsymbol {v} _ {r} \circ \boldsymbol {p} _ {r}\right) \tag {2}
$$
where  $\pmb{u}_r \in \mathbb{R}^d$ ,  $\pmb{v}_r \in \mathbb{R}^d$ , and  $\pmb{p}_r \in \mathbb{R}^N$  are decomposed vectors for the  $r$ -th component, while each vector belongs to the corresponding mode matrix, i.e.,  $\pmb{u}_r$  is the column vector in  $U =$
$[u_1, \dots, u_r, \dots, u_R]$ . In addition,  $\circ$  represents the outer product,  $\lambda_r$  is the coefficient scalar of each component, and  $R$  is the rank of CP decomposition. For better understanding, each component contributes to the value of the tensor in the sum of scalar products:
$$
\Delta \boldsymbol {W} _ {i j k} = \sum_ {r = 1} ^ {R} \lambda_ {r} \left(u _ {r, i} v _ {r, j} p _ {r, k}\right), \quad i, j \in \{1, 2, \dots , d \}, k \in \{1, 2, \dots , N \} \tag {3}
$$
where  $i,j,k$  denote the indices of the three modes.
Mode Approximation. In multimodal tasks, learning modality-specific representations and modality fusion representations are both important. Therefore, we aim to implement mode approximation to update the frozen attention-based weight tensor  $\mathcal{W}_0$ , including the self-attention module in the vision/text encoders, and the cross-attention module in the multimodal encoders, which are based on the pre-trained multimodal foundation model like BLIP [41]. We first approximate the attention-based weight in these modules by initializing three mode factors, i.e.,  $U = [u_{1},\dots ,u_{R}]$ ,  $V = [v_{1},\dots ,v_{R}]$ , and  $P = [p_1,\dots ,p_R]$ .  $U,P$  are randomly initialized with Gaussian distribution, and  $V$  is with zeros, so that  $\Delta \mathcal{W} = 0$  before training. It should be noted that  $U,V$  are shared as global factors utilized for mode approximation, which means that our Aurora considers the cross-modal interaction and share the knowledge between these weight matrices in each modality. Besides, to further capture the discriminative features of each modality, we randomly initialize the learnable coefficient vector  $\lambda^m\in \mathbb{R}^R$  for each weight matrix on the modality  $m$  respectively. With these three mode factors, we can implement the mode approximation in the forward propagation by the inverse progress of CP decomposition with input tensor  $\mathcal{X}^m$  as follows:
$$
\boldsymbol {\mathcal {H}} ^ {m} = \boldsymbol {\mathcal {W}} _ {0} \boldsymbol {\mathcal {X}} ^ {m} + \left(\sum_ {r = 1} ^ {R} \lambda_ {r} ^ {m} \left(\boldsymbol {u} _ {r} \circ \boldsymbol {v} _ {r} \circ \boldsymbol {p} _ {r}\right)\right) \boldsymbol {\mathcal {X}} ^ {m} \tag {4}
$$
Analyzed from the perspective of prompt learning, our idea of approximating the pre-trained weight parameters  $\mathcal{W}_0$  with additional trainable parameters  $\Delta \mathcal{W}$  can be essentially understood as the soft prompts, which learns on downstream data based on CP decomposition. Such prompts not only provide better guidance for downstream tasks, but also are very lightweight in design, which greatly facilitates the application of pre-trained models to many cross-modal tasks in the unified mechanism.
3.3 Modality Alignment Design
Unlike existing methods that directly insert learnable networks to explicitly achieve cross-modal alignment, we further propose two effective modules to align different modalities with few trainable parameters. Thus, with the addition of mode approximation above, we can achieve a graceful prompt framework for cross-modal transfer, which is both lightweight and high-performance.
Informative Context Enhancement. For better modality alignment, we aim to provide prompts that can activate the fusion features after the cross-attention module. Inspired by the development in In-Context Learning [52, 18], we realize that the demonstration template is important for the prompts. The most intuitive approach is to align image-text pairs to obtain more cross-modal contextual information. However, even with relevant image regions, there may still be more than one way to describe these regions. Some texts may accurately summarize the content of an image, whereas others may not. Without a priori for matched textual information, we determine to introduce the context enhancement module to provide coverage of possible textual information.
We adopt the image-grounded text branch from BLIP and design a specific demonstration template for cross-modal prompt tuning. Given the fusion features of the image-grounded text branch  $F = \{f_1, \dots, f_{|\mathcal{B}|}\} \in \mathbb{R}^{|\mathcal{B}| \times E}$  and the textual query features of the self-attention module  $T = \{t_1, \dots, t_{|\mathcal{B}|}\} \in \mathbb{R}^{|\mathcal{B}| \times E}$ , we utilize all of the query features with dimension  $E$  in a batch  $\mathcal{B}$  as the context for enhancement. Specifically, we calculate the attention score  $\alpha \in \mathbb{R}^{|\mathcal{B}| \times |\mathcal{B}|}$  between feature  $f_i$  and each textual query feature from  $t_1$  to  $t_{|\mathcal{B}|}$  as:
$$
\alpha_ {i j} = \frac {\exp \left(\boldsymbol {f} _ {i} \cdot \boldsymbol {t} _ {j}\right)}{\sum_ {b = 1} ^ {| \mathcal {B} |} \exp \left(\boldsymbol {f} _ {i} \cdot \boldsymbol {t} _ {b}\right)} \tag {5}
$$
In order to generate a more effective advanced fusion feature  $F'$ , all the adaptively weighted query features  $T$  within a batch are collected together with a specific fusion feature  $f_i$  to form the demon-
Table 1: Results on image-text retrieval datasets MSCOCO and FLICKR30K. Using the text query, we simplify retrieving images as  $\mathrm{T}\rightarrow \mathrm{I}$  and vice versa. Recall@  $K$  represents the recall of top-  $K$  returned samples. # Tunable is the size of the learnable parameters in the backbone network.  $\Delta_{PETL}$  represents the performance gap between our Aurora and the best PETL method.
Method	# Tunable	MSCOCO (I→T)	MSCOCO (T→I)	FLICKR30K (I→T)	FLICKR30K (T→I)
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Methods with full fine-tuning
UNITER	330M	65.7	88.6	93.8	52.9	79.9	88.0	87.3	98.0	99.2	75.6	94.1	96.8
VILLA	330M	-	-	-	-	-	-	87.9	97.5	98.8	76.3	94.2	96.8
OSCAR	330M	73.5	92.2	96.0	57.5	82.8	89.8	-	-	-	-	-	-
ALIGN	820M	77.0	93.5	96.9	59.9	83.3	89.8	95.3	99.8	100.0	84.9	97.4	98.6
ALBEF	210M	77.6	94.3	97.2	60.7	84.3	90.5	95.9	99.8	100.0	85.6	97.5	98.9
BLIP	223M	81.9	95.4	97.8	64.3	85.7	91.5	97.3	99.9	100.0	87.3	97.6	98.9
Methods with frozen backbone
LoRA (r = 32)	10.6M	80.0	94.1	97.2	62.1	84.4	90.6	96.2	99.7	99.8	85.8	97.1	98.4
UniAdapter (r=128)	4.6M	79.8	94.2	97.5	62.3	84.5	90.8	97.1	100.0	100.0	86.5	97.4	98.8
UniAdapter (r=512)	18.8M	80.1	94.6	97.4	62.6	84.6	90.9	97.1	99.9	100.0	86.4	97.4	98.9
Aurora (ours, r=64)	0.1M	80.2	95.1	97.7	62.4	84.5	91.0	96.8	100.0	100.0	86.7	97.8	98.7
ΔPETL	0.5%	+0.1	+0.5	+0.2	-0.2	-0.1	+0.1	-0.3	+0.0	+0.0	+0.2	+0.4	-0.2
Aurora (ours, r=128)	0.2M	80.7	95.3	97.8	62.8	84.8	91.0	97.2	100.0	100.0	86.8	97.6	98.9
ΔPETL	1%	+0.6	+0.7	+0.3	+0.2	+0.2	+0.1	+0.1	+0.0	+0.0	+0.3	+0.2	+0.0
stration template. This form can adaptively absorb context query information to derive a better enhanced fusion feature for the image-text matching loss  $\mathcal{L}_{\mathrm{ITM}}$  as  $\pmb{f}_i' = \pmb{f}_i + \sum_{j=1}^{|B|} \alpha_{ij} \pmb{t}_j$ .
Gated Query Transformation. Another reason why modality alignment is difficult is that the multimodality fusion branch network is deep, which can cause textual information to be lost during training. To address this issue, we propose a novel approach inspired by gating mechanisms to explicitly model the relative contributions of textual information during modality alignment. Specifically, instead of directly concatenating the fusion representation  $\mathbf{f}$  (output of the Cross-Attention block) with the query representation  $\mathbf{t}$  (output of the Self-Attention block) as residuals in existing methods [49], we learn a gated query function to balance the contributions of both modalities. Our gated query transformation involves two steps. First, we implement the transformation as follows:  $\mathbf{t}' = \sigma(\gamma \odot \mathbf{t}) + \beta$ , where  $\gamma$  and  $\beta$  are zero-initialized learnable transformation matrix and bias with an activation function  $\sigma$ . Second, we calculate the query gate  $\mathbf{g}$  by computing the product of  $\mathbf{t}'$  and  $\mathbf{f}$  with Softmax. It should be noted that  $\gamma$  and  $\beta$  are zero initialized, so that  $\mathbf{t}'$  is zero at the beginning of training. Hence, the query gate explicitly measures the contribution of the query representation in the formulation of  $\mathbf{g} \odot \mathbf{f} + (\mathbf{1} - \mathbf{g}) \odot \mathbf{t}'$  to update the fusion representation  $\mathbf{f}$ .
Datasets & Baselines. We evaluate Aurora on six benchmarks spanning four cross-modal tasks: image-text retrieval, question answering (QA), and video-text retrieval and QA. We compare Aurora with two types of approaches: full fine-tuning methods including SOTA for each task, and frozen backbone methods including LoRA [25] and UniAdapter [49]. See more details in Appendix.
Implementation Details. Our implementation is based on Salesforce's open-source codebase [41]. Following [49], we also apply BLIP-base [41] as our vision-language backbone for all the multimodal downstream tasks. We use PyTorch to implement all experiments on  $8 \times$  NVIDIA V100 GPU (32G). We use AdamW [48] optimizer with a weight decay of 0.05 and a learning rate of 1e-4 obtained from grid search for all experiments. Note that during the fine-tuning process, the parameters of the backbone model are kept frozen. More training details can be seen in Appendix.
4.2 Performance Comparisons on Cross-modal Tasks