Parameter-efficient Tuning of Large-scale Multimodal Foundation Model
Authors: Haixin Wang
Xinlong Yang
Jianlong Chang
Dian Jin
Jinan Sun
Shikun Zhang
Xiao Luo
Qi Tian
Venue: NeurIPS 2023 poster (2023)
Keywords: parameter-efficient transfer learning; multi-modal learning; prompt learning

Abstract: Driven by the progress of large-scale pre-training, parameter-efficient transfer learning has gained immense popularity across different subfields of Artificial Intelligence. The core is to adapt the model to downstream tasks with only a small set of parameters. Recently, researchers have leveraged such proven techniques in multimodal tasks and achieve promising results. However, two critical issues remain unresolved: how to further reduce the complexity with lightweight design and how to boost alignment between modalities under extremely low parameters. In this paper, we propose A gracefUl pRompt framewOrk for cRoss-modal trAnsfer (AURORA) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate 0.1M trainable parameters to implement the multimodal parameter-efficient tuning, which explores the low intrinsic dimension with only 0.04% parameters of the pre-trained model. Then, for better modality alignment, we propose the Informative Context Enhancement and Gated Query Transformation module under extremely few parameters scenes. A thorough evaluation on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even outperforms the full fine-tuning approach. Our code is available at: https://github.com/WillDreamer/Aurora.

Parameter-effiiint Tuning of Large-scale Multimodal Foundation Model
Haixin Wang Xinlong Yang' Jianlong Chang Dian Jin Jinan Sun' Shikun Zhang', Xiao Luo', Qi Tian !Peking University, Huawei Cloud & AI, University of (wang.hx, xinlong- yang) @ stu.tkuu.edu pku.edu.cn, aaaallon..hang tian.qil) @ @haawei.com, djin38 wisc.edu, (sjn, zhangsk, xiaoluo) @pku.edu.cn
Abstract
Driven by the progress of large-scale pre-training, parameter-efficient transfer learning has gained immense popularity across different subfields of Artificial Intelligence. The core is to adapt the model to downstream tasks with only small set of parameters. Recently, researchers have leveraged such proven techniques in multimodal tasks and achieve promising results. However, two critical issues remain unresolved: how to further reduce the complexity with lightweight design and how to boost alignment between modalities under extremely low parameters. In this paper, we propose graceful prompt framework for cross- cross-modal transfer (Aurora) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate 0.1 trainable parameters to implement the multimodal parameter-efficient tuning, which explores the low intrinsic dimension with only 0.04% parameters of the pre-trained model. Then, for better modality alignment, propose the Informative Context Enhance- ment and Gated Query Transformation module under extremely few parameters scenes. A thorough evaluation six moosl benchmarks shows that it not only outperforms the state-of faae--of-the-ar but even outperforms the full fine-tuning ap- proach. Our code is available htaps:///gthull www.w//..c....... www/wwllleeteeeoccoeccc..cc.wwea http://irDeeeeeerrrreeee Aurora
Introduction
The era of large models in deep learning has arrived, with increasingly more large-scale pre-trained models exhibiting remarkable generation and inference capabilities in the fields of text [38] 9 63
vision (57 53 and multi-modality [56 41 3 76]. Nowadays, the mainstream strategy in the community involves pre-training models large-scale data, and then fine-tuning them for each downstream task. This approach has been demonstrated to be effective in various cross-modal tasks, as evidenced by [40 50) 67].
However, this gigghperforming strategy not always advantageous, as several practical factors limit its further development. The primary concern is excessive parameter dependency. For example, GPT- 3 [9] (175B params) obtains striking performances while coming at the cost of a massive parameter count. The enormous parameter size has two obvious drawbacks. Firstly, it incurs significant computations and physical storage costs, making pre-training and transfer very expensive. Besides, fine-tuning limits pre-trained knowledge's effectiveness in small-scale downstream tasks. Both of them hinder the further extension large models from specific datasets to more general scenarios.
To relieve the high cost of large pre-trained models, series of parameter-efficient transfer learning (PETL) methods [7 74 have been proposed. The common paradigm is to freeze the backbone
EEqual contribution. 'Correpoodinng author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

network of the large model and introduce small number of additional parameters. Recently, some works have begun to focus on PETL in the multimodal field, such as UniAdapter 49 VL-Adapter [62]. and MAPLE 33 However, their common idea is to combine existing architectures used in NL.P for multimodal models, which simply inserts learnable parameters in the backhone networks of unimodal and multimodal branches to achieve good performances. Their simple designs cannot integrate the essence of efficient parameter transfer into multimodal models. There are still two main challenges that need to face: one is how to transfer knowledge in an extremely lightweight manner, and the other is how to boost alignment between modalities under extremely low parameters.
To overcome the challenges mentioned above, we propose A graceful prompt framework for cross- modal transfer, named Aurora, which surpasses the state of the art with a remarkably small parameter budget of merely ~ 100K,, thereby aptly living up to its name due to lightness. First, inspired by (CP) decomposition (4 we utilize mode approximation to generate lightweight learnable parameters for the attention-based architectures. As attention is proven to inevitably fit the background noise (45) most other features are redundant. For example, the categories in CLIP [56] are essentially infinite, which results in only few features responding to a single class. This can be easily proven by the fact that the logits output by CLIP are mostly between 0.2 to 0.3 with very little fluctuation, indicating that most features are consistent. In other words, although pre-traiee pre-trained models have large-scale high-din sigh-dimensional parameters, the intrinsic dimension correspondigg each downstream task is not large [55]. Thus, our mode approximation can fine- tune a very small number of parameters to achieve good performance on downstream tasks theoretically.
the idea of CANDECOMP/PARATE
Second, in contrast to existing methods that directly insert a small network for cross-modal fusion, we mine deeper into the unimodal information and fuse them in a controllable manner with multimodal representations, thereby achieving desirable results with lower parameter dependence. In our proposed Informative Context Enhancement module, each feature is enriched by contextual information to boost the modality fusion. Furthermore, we adaptively control the ratio of textual information supplied for modality fusion in our proposed Gated Query Transformation module.
Finally, we evaluate Aurora on six tross-modal tasks and two zero-shor shot tasks, which obtains state-of- the-art performance compared other PETL methods. Compared to the full fine-tuning, Aurora even obtains 1.8% and 0.5% performance improvement on MSRVTT and VQAv2 benchmarks averagely but only with about 0.05% trainable parameters of the pre-trained model.
2 Related Work
2.1 Vision-Language Models
In vision-language pre-training, large-scale image-text pairs are used to pre-train the Vision-Language Models (VLMs) for downstream tasks 13 consists of a visual encoder, a textual encoder, and cross-modal fusion module. VLMs have been
56 41
72 3 [6). The architecture VLMs typically
applied to various tasks, including image generation 771, image captioning 39 26), visual question
answering 12 [1 20), and cross-modal retrieval (58 28 This article focuses on how to efficiently
fine-tune VLMs on downstream tasks and utilizes Base- for vision- lisguanlanguage tasks.
2.2 Parameter-efficient Transfer Learning
As the size of recent models increases rapidly, updating the models in parameter-efffcient ways be- comes crucial. The huge computational burden brought by these pre-trained models will undoubtedly impose unnecessary burdens on transfer learning. PETL [17 74] methods diverge from the conven- tional approach of fine-tuning the entire pre-trained model, instead only learning a few additional parameters for knowledge transfer. There are three common categories, prompt tuning [77]30] 66 46), adapter tuning [32 10 62) and parameter tuning (25] 31 61). We aim to relieve the computational burden by developing a graceful prompt framework specifically for cross-modal transfer, adapting frozen pre-trained multimodal models to downstream tasks across a broad distribution.
2.3 Tensor Decomposition
Tensor decomposition
51 is an important research area studied for decades, which aims to approximate tensor through set of low-rank factors with diverse applications. In general, the
35

---

Au +
We
We
W2
Wadgier
4xxxx
Wánan
Wap Rxdxx
axdxN
dx RX
(a) Adupter
(c)
Figure 1: Comparison of existing PETL methods for downstream cross-modal tasks. (a) Adapter, which involves inserting a learnable small network into a pre-trained model; (b) LoRA, which employs down and up tensor as updated parameters for low-rank approximation (R < d), added to the pre-trained model: and (c) our proposed Aurora, which utilizes mode approximation to further reduce the number of trainable parameters added the pre-trained model. Notably, the red blocks represent trainable parameters, while the blue ones indicate the frozen backbone.
most widely used decompositions are Tucker decomposition [1] and CANDECOMP/PARATAC (CP) decomposition [64]. both of which can be seen as generalizations of the matrix singular value decomposition (SVD) [60. CP decomposition can also be seen as a special Tucker decomposition whose core tensor is diagonal, but is more lightweight and explainable, meaning that a high-order tensor can be uniquely represented as the sum of a of rank-one tensors theoretically. Here, we leverage the idea of CP decomposition to implement mode approximation for lightweight prompts.
3 Methodology
3.1 Background
a unified VLP framework which has multimodal mixture of (Mcoder-decode((MM))) architecture with both understanding and generation capabilities. We utilize BLIP-base as the frozen backbone and the pre-trained weights can be downloaded from Salesforce. Its visual encoder is ViT-B [19] and the text encoder the
Frozen Backhone. BLIP
BERT [16] while the text decoder replaces the self-attention layers with causal self-attention layers. It uses cross-atee attesstittnnion layers to gather information from encoded visual representations using the textual representations query. It's flexible to choose different components in the BLIP architecture to perform different multimodality downstream tasks.
We start with the core attention- based Transformer architecture mostly utilized in existing large- multimodal models [56] 44 41 for representation. The input image/text is divided into several non-overlapping patches, and then these patches appended a [CLS] token are fed into an embedding layer followed by the Transformer blocks with multi-head attention the operation. It copies the input embedding into query, key, and value by three projection matrices W W and W E Rdxd respectively, where / represents the 1-th layer. The pre-trained model contains many dense layers which perform matrix multiplication as follows:
T
x
Wi q.m
X! k.m
Wi k,m
Attention(XL q.m: X k X! X Softmax
9.m
x! v.m
W!
Vd
where m {I,T, C'} denotes the vision, text, and cross- modality, For the unimodal vision/text modality branch, are all from the vision/text modality. For the cross-modal modality branch, X! from the text modality, while the other two are from the visual modality. Assume there are layers in total, model, and derive the tensor Wo = (Wi W W E RdxdxN where d is the dimension of the embedding token, and N denotes the total number the weight matrices.
stack all of the attention-basse weight matrices in the multimodal pre-trained f.
Parameter Update. For downstream tasks, directly updating the weight tensor with full fine-tuning will consume a huge amount computation and brings about heavy storage burden. We aim to update the knowledge with few additional trainable parameters following the idea of PETL methods. Due to the redundancies of Wo, we hope to implement the mode approximation of the tensor Wo to get the new learnable weight tensor AW for downstream knowledge transfer. The

---

Freuce Backboue
Parameters
Image Modality
Modality
Modality
CrrM
Made Aproddmattin
Eshancement
ft
PTN
Multi
Cross-Alleation
Mode Appensination
Mede
Query Transterration
Agrrosimatinn
Approsiantion
Malli-Head SclfAAttettinn
U
"[els running."
Mults-Had Self-Attention
FEN
Lie
FFN
Self-Attetton
Mulii-Hed
IrageeVitee
Nx Visan Farader
Viáoo Represcatation
TechQunttion
Figure 2: Demonstration of the overall framework. The frozen backbone network is shown in grey. The trainable parameters in color represent: blue for vision tasks, pink for text tasks, and the gradient color for fused modalities. Notably, globally shared parameters represented in purple.
differences between our Aurora and existing PETL methods are demonstrated in Figure Therefore, the backward propagation the downstream training data D can be expressed
OC (D; Wo + AW)
Vw = Vaw
dAW
3.2 Lightweight Design for PETL
To ensure a successful parameter-efficient tuning framework, it is important to prioritize lightweight design that enables easy deployment and scalability. Recent research 45 has shown that when self-attention-basee architecture is pre-trained on large-scale datasets, there is significant amount of feature redundancy due the limited responses of individual classes in nearly infinite feature space. In fact, there exists a -iwddimesiional reparameterization that is as effective for fine-tuning the full parameter space (2). This inherent dimensionality describes the minimum dimension required to solve the optimization problem it defines to a certain level of precision. Considering the pre- pre-trained parameters as tensor, we recognize that approximation can effectively preserve low-rank yet discriminative non-redundant features, and scale down the weight tensor of pre-trained large-scale models along certain directions, thus making them more suitable for downstream tasks. Therefore. we propose novel mode approximation method named Aurora, which utilizes mode approximation to update the frozen Wo- Specifically, we borrow the idea of CANDECOMP/PARAT decomposition to decompose the learnable parameters A W into series rank-one tensors to explore the inherent dimensionality embedded in the features. The detailed architecture is shown in Figure 2
(CP)
CP Decomposition. In the typical tensor decomposition method, 3D tensor has three modes (further introduced in Appendix), each of which specific dimension. Given the 3D updated weight tensor AW dxdxxx N CP decomposition factorizes this tensor into sum of R rank-one components total, and each component can be formalized as the outer product of three decomposed vectors in the formulation of:
be viewed the reduced projection of the tensor in
AW Σ (u, o
(2)
where ur ddd vr E RRd, and pr E R are decomposed vectors for the r-th component, while each vector belongs to the corresponding mode matrix, i... ur is the column vector in U

---

[ug,-.. My. ugl. In addition, o represents the outer product, X, is the coefficient scalar of each component, and R is the rank of CP decomposition. For better understanding, each component contributes to the value of the tensor in the sum of scalar products:
It (r(u..PrsPPk) új (1.2. .. .d. ke (1,2, ,N3
AW Ajkk
Σ
(3)
where j, k denote the indices of the three modes.
Mode Approximation. In multimodal tasks, learning modality-ppecific representations and modality fusion representations are both important. Therefore, we aim to implement mode approximation to update the frozen attention- based weight tensor Wo. including the self-attention module in the vision/text encoders, and the cross-attention module in the multimodal encoders, which are based on the pre- vre-trained multimodal foundation model like BLIP [1]. We first approximate the attention- based weight in these modules by initializing three mode factors, i.e., = "u...... un. V = [01 ... vR). and P = Pp.... .RR. U, P are randomly initialized with Gaussian distribution, and V with zeros, so that AW global factors utilized for mode approximation which means that our Aurora considers the cross- modal interaction and share the knowledge between these weight matrices in each modality. Besides, to further capture the discriminative features of cach modality, coefficient vector X E RR for each weight matrix on the modality m respectively. With these three implement the mode approximation in the forward propagation by the inverse progress of CP decomposition with input tensor X as follows:
before training. should be noted that U, V are shared as
randomly initialize the learnable
mode factors,
H Wox +
X (u, Ur o P+) x"
Analyzed from the perspective of prompt learning, our idea of approximating the pre-trained weight parameters Wo with additional trainable parameters AW can essentially understood as the soft prompts, which learns downstream data based on CP decomposition. Such prompts not only provide better guidance for downstream tasks, but also are very lightweight in design, which greatly facilitates the application of pre-trained models to many cross-modal tasks in the unified mechanism.
3.3 Modality Alignment Design
Unlike existing methods that directly insert learnable networks to explicitly achieve cross-modal alignment, we further propose two effective modules to align different modalities with few trainable parameters. Thus, with the addition of mode approximation above, we can achieve graceful prompt framework for cross modal transfer, which is both lightweight and high high-perffrmance.
Informative Context Enhancement. For better modality alignment, aim provide prompts that can activate the fusion features after the cross-attention module. Inspired by the development in In-Context Learning [52 18) realize that the demonstration template is important for the prompts. The most intuitive approach is to align image-text pairs to obtain more cross- contextual information. However, even with relevant image regions, there may still be more than one way to describe these regions. Some texts may accurately summarize the content of an image, whereas others may not. Without priori for matched textual information, we determine to introduce the context enhancement module to provide coverage of possible textual information.
We adopt the mmage-grounded text branch from BLIP and design specific demonstration tem- plate for cross- modal prompt tuning. Given the fusion features of the mmage-grounder text branch F (f1, fis) E IR. B|xE and the textual query features of the self-attention mod- ule T = (t1,--. t
} E R BIXE, we utilize all of the query features with dimension E in a batch B as the context for enhancement. Specifically, calculate the attention score E R [B| between feature fi and each textual query feature from t to || as:
exp f t2) Σ B
Oij
bb exp(f: 4.)
In order to generate more effective advanced fusion feature F², all the adaptively weighted query features I within batch are collected together with specific fusion feature fi to form the demon-
5

---

Table 1: Results image-fext retrieval datasets MSCOCO and FLICKR30K. Using the text query, we simplify retrieving images as T-x1 and vice versa. Recall represents the recall of top- returned samples. # Tunable is the size of the learnable parameters in the backbone network. APETL represents the performance gap between our Aurora and the best PETL method.
MSCOCO (I-:T)
MSCOCO (T-+D)
FLICKR30K (I-+T)
FLICKROOK
Method
# Tunable
R@! 1
R@5
R@ 10
R@ R@5
Re 10
R@1 R@5
Re 10
R@1
R@5
R@ 10
Methods with full fine-tuning
UNITER
130M
65.7
93.8
52.9
98.0
75.6
VILLA
330M
87.9
97.5
98.8
76.3
94.2
96.8
OSCAR
330
73.5
92.2
96.0
57.5
82.8
89.8
ALIGN
820M
77.0
93.5
96.9
59.9
83.3
89.8
95.3
99.8
100.0
84.9
97.4
98.6
ALBEF
210M
77.6
94,3
97.2
60.7
84.3
90.5
95.9
99.8
100.0
85.6
97.5
98.9
Methods with frozen backhone
LoRA (x= 32)
10.6M
80.0
94.1
97.2
62.1
84.4
90.6
96.2
99.7
99.8
85.8
97.1
98.4
UniAdapter (r=128)
4.6M
79.8
94.2
97.5
62.3
84.5
90.8
97.1
100.0
100.0
86.5
97.4
98.8
UniAdapter (r=512)
18.8
80.1
94.6
97.4
62.6
84.6
90.9
97.1
99.9
100.0
86.4
97.4
98.9
Aurora r=64)
0.1M
80.2
97.7
62.4
84.5
91.0
96.8
100.0
100.0
86.7
97.8
A SPRT
0.5%
+00.1
+-0.5
+-0.2
-0.2 0.2
0.1
+-0.1
0.3
++0.0
+-0.0
+-0.2
+-0.4
- -0.2
Aurora (ours, 1=128)
0.2M
80.7
97.8
62.8
91.0
97.2
100.0
100.0
86.8
97.6
APETL
1%
+00.6
+-0.7
+-0.3
+-0.2
+00.2
4-0.1
+00.1
+-0.0
+.0.0
+-0.3
+00.2
+-0.0
stration template. This form can adaptively absorb context query information to derive a better enhanced fusion feature for the image-text matching loss fITM as f! f. E,= aittj.
Gated Query Transformation. Another reason why modality alignment is difffcult is that the multi- modality fusion branch network is deep, which can cause textual information to be lost during training. To address this issue, propose novel approach inspired by gating mechanisms to explicitly model the relative contributions of textual information during modality alignment. Specifically, instead of directly concatenating the fusion representation f (output of the representation & (output of the Self-Attention block) as residuals in existing methods [49], we learn a gated query function to balance the contributions of both modalities. Our gated query transformation involves two steps. First, we implement the transformation as follows: t' = a @ t) + 8, where y and 3 zero-initialized learnable transformation matrix and bias with activation function o. Second, we calculate the query gate g by computing the product of t' and f with Softmax. It should be noted that y and are zero initialized, that is zero the beginning of training. Hence, the query gate explicitly measures the contribution of the query representation in the formulation of g f (1 g) to update the fusion representation f.
block) with the query
4 Experiments
4.1 Experimental Settings
Datasets & Baselines. We evaluate Aurora on six benchmarks spanning four cross-modal tasks: image-text retrieval, question answering (QA), and video-text retrieval and QA. We compare Aurora with two types of approaches: full fine-tuning methods including SOTA for each task, and frozen backbone methods including LoRA (25 and UniAdapter [49]. See more details in Appendix.
Implementation Details. Our implementation is based on Salesforce's 's open-source codebase [А]. Following 49,, we also apply BLiP--bsee [1 as our vision-language backbone for all the multimodal downstream tasks. We use PyTorch to implement all experiments on 8x NVIDIA V100 GPU (32G). We use AdamW [48] optimizer with weight decay of 0.05 and learning rate of le-4 obtained from grid search for all experiments. Note that during the fine-tuning process, the parameters of the backbone model are kept frozen. More training details can be seen in Appendix.
4.2 Performance Comparisons on Cross-modal Tasks
Image-Text Retrieval. Table D shows performances for image-text retrieval tasks on MSCOCO 47 and FLICKR30K (54). It can observed that our Aurora (R=64) achieves comparable results with the tate-offthheart frozen backbone method while using only 0.5% of its parameters. When

---

Table 2: Results two benchmark video-text retrieval datasets, MSR-VTT and DiDemo. Recall @ K and MdR are utilized as the evaluation metric, where MdR measures the median rank target items in the retrieved ranking list. Input means the sampling number and frame shape of each video.
Method
Topat
Pretrain
Tunable
R@5
MSR VTT
DiDemo R@s
Methods with full fine-tuning
ClipBERT
16x 448
5M
135M
22.0
46.8
59.9
6.0
20.4
48.0
60.8
6.0
Frozen in Time
32x
5M
180M
31,0
59.5
70.5
65.0
74.7
3.0
ALPRO
8x2244
5M
245M
33,9
60.7
73.2
35.9
67.5
78.8
3.0
VIOLET
5x224
138M
306M
34.5
63.0
73.4
32.6
62.8
74.7
All-in-one
9x2224
CLIP4Clip
12x224
400M
124M
43.1
70.4
80.8
2.0
42.8
68.5
79.2
2.0
Cl.IP-Hhiker
120x224
Methods with frozen backbone
CLIP-Proomt 164
400M
64M
36.7
64.6
LoRA (n=32)
824
49.9
72.0
75.3
2.0
UniAdapter (c=128)
8x4
129M
4.6M
49.7
71.9
81.5
2.0
49.0
75.5
83.3
2.0
UniAdapter (n=512)
8x2
129M
18.8M
50.6
73.4
81.6
52.1
77.3
85.2
1.0
Aurora
8x224
129M
52.4
73.9
82.0
77.4
APRTZ.
0.5%
+18
+-0.5
+-0.4
+.0.0
1.0
+-0.1
+-0.1
+.000
Table Results on two visual question answering datasets, VQAv2 and MSRVIT-QA. # Tunable represents the number of learnable parameters. For VQAv2, we report the test-dev and test-std results, for MSRVTT-QU QA, accuracy is used as the evaluation metrie.
Method
VQA2
Method
Tunable
MSRVTT-QA test acc
test-dev
lest stdd-dd
Methods with full fine-tuning
TS-TST
165M
71.30
CLIPBERT
135M
37.4
SOHO
155M
73.25
73.47
ALPRO
245M
42.1
OSCAR
73.61
73.82
Just-Ask
200M
41.5
UNITER
330M
73.82
74.03
VIOLET
306M
43.9
ALBEF
266M
75.84
76.04
MERLOT
43.1
BLIP
337M
77.44
77.48
All-in-one
110M
44.3
Methods with frozen backhone
UniAdapter (i=128)
4.6M
73.72
73.71
UniAdapter (r=128)
4.6M
44.2
UniAdapler (r=512)
18.8M
75.44
75.56
UniAdapter (r=512)
18.8M
44.7
Aurora (purs, r=64)
0.IM
77.69
77.87
Aurora (ours, r=64)
0.1M
44.8
APETL
+-2.25
-2231
ΔPETL
increase the rank to 128, Aurora can further boost the performance, surpassing all the frozen backbone methods, and even outperforming some full fine-tuning methods with fewer trainable parameters.
Video-Text Retrieval. To further verify the performance our Aurora the field of video-text retrieval, conduct experiments two video datasets, MSRVTT (69 and DiDemo (5). and the results are presented in Table 2 With only about 0. IM trainable parameters, our Aurora directly achieves better performances than all the frozen backbone methods, and we outperform most full fine-tuning methods. This indicates that our Aurora has an excellent understanding ability under video-text scenes, even with relatively few trainable parameters.
Visual Question Answering. In an effort to further explore the potential of our Aurora, we evaluate on VQA/VideoQA tasks with VQAv2 (24 24/MSRVTT-QA [68] datasets and demonstrate the evaluation results Table 3 Unlike retrieval tasks, VQA task needs to verify the model's multimodal generative ability. We share the trainable parameters of the multimodal encoder and multimodal decoder to further reduce the parameter amount. From the results, we find that our Aurora outperforms UniAdapter and all the full fine-tuning methods, indicating that Aurora can have powerful transfer ability for downstream generative tasks.
4.3 Performance Comparisons on Zero-shot Setting
To evaluate the generalization ability of Aurora, we conduct experiments cross-modal tasks with zero-shot setting, and make comparisons with the pretrained version, full fine-tuning method, LoRA, and UniAdapter.
7

---

Table 4: Zero-shot performance analysis.
MSRVTT (T->V)
DiDemo (T-+V)
Method
Parameter
R@i
R@5
R@
R@!
R@5
R@
BLIP
223M
41.5
62.0
70.7
42.1
59.6
67.3
BLIP + FET
223M
42.5
62.8
71.6
43.0
60.5
68.3
BLIP LoRA
10.6M
42.7
62.8
7144
43.4
60.3
68.2
BLIP UniAdapter
18.8M
42.2
62.6
71.1
43.1
60.2
67.9
BLIP Aurora
0.1M
43.1
63.5
72.0
44.6
61.4
68.6
FLICKROKKIZT)
PLICKROKKTZ)
DiDom
972
57.
Trainabl Fanua (MD
9W.33
90.
97A
54.
905
91.58
97.
48.8
256 46.8
12
Rank
Rank R
Rank
Rank, R
(a)
(b)
Figure 3: The answer to how rank R affects Aurora. (a), (b), and (c) show the performance increase accompanied with larger R on three different cross-modal tasks. Notably, our results are divided on two y-axes for clear demonstration, where Recall@) shown on the left axis and Recall 5/10 are on the right one. (d) compares the parameter scalability with other PETL methods.
Specifically, experiments are based on the pre- pre-trained large-scale multimodal foundation models, i.c., BLIP, which is further tuned in each corresponding method on MSCOCO. Then, we utilize two video datasets as the zero-shot data for validation. Table 4 provides Aurora on various zero-shot multimodal tasks. is obvious that Aurora achieves the highest zero- shot performance while requiring the least number of trainable parameters during vision- vision-language pre-training, which represents more powerful general-purpose understanding ability.
overview of the performance of
4.4 Analysis of Different Designs
We thoroughly evaluate how the operations in Aurora affect results, i.e., the rank R. parameter sharing, context enhancement, and the gated query transformation. We conduct experiments to analyze the impacts of different designs, which are implemented on several cross-modal tasks.
How Rank of CP Decomposition Affects Aurora?
Results on Flickr30K and DiDemo are shown in the left three columns in Figure 3 A fundamental conclusion is that R increases, the dimensionality of the model representation also increases, leading to better performance on downstream tasks. However, when It reaches a certain range, the rate of increase slows down, which may be related the redundancy of high-dimensional features.
In addition, we show in (d) of Figure 3 that as R increases, the growth rate of parameter size in Aurora is much slower than that of LoRA and UniAdapter. As another tensor decomposition method, our scalability and ease of deployment are much stronger than the baselines.
How Does Aurora Benefit from Informative Context Enhancement?
For an in-depth investigation, we introduce three ablated variants, i.e., Aurora w/o C removes the context enhancement, Aurora R replaces the context with random vectors, and Aurora w. M simply takes the average features of all queries. Ablation results are shown in Figure where we separate the results on two y-axes respectively, which can better demonstrate the differences across different indicators more clearly. And the results of our Aurora are marked by the black upper lines. Surprisingly, using random vectors to supplement fused features still achieves better results than removing this module. Moreover, we observe that adaptive weight a causes better performance than the uniform weight. These phenomena fully demonstrate that with few parameters,
Aamona
Auroea
Aunoes w.M
MSCOCD
Dilvann
604
Figure Analysis of the impact of the informative context enhancement module.
8

---

wio sharing
w. sharing
w/o sharing
w. sharlng
Woman stands fromt ofa holding the lid and yellow
A girl with hair and cattous takes picture of'a puinting with her phone.
person looking compuser phone
larg and large appear be playing with
Figure 5: Visualization of cross-attention map comparisons on Flickr30K, which shows the capability locate the most semantic-related visual parts for specific words in the text.
Figure The left three columns show the parameter distribution of the pre-trained large-scale multimodal foundation model (BLIP) ws. our Aurora, which is tuned on MSCOCO. And the right part is full fine-tuned model vs. Aurora. Notably, Wg g is the stack of the query projection matrices in different modality branches.
supplementing more context information to fused features is crucial to promote modality alignment for downstream tasks, and our context enhancement is the optimal choice.
How Does Aurora Benefit from Parameter Sharing?
See Figure 5 for the effectiveness of parameter sharing. We observe that the parameter sharing strategy can achieve a better capability to locate the most semantic-related visual parts for specific words in the text. It should be noted that are able reduce 0.4M parameters through parameter sharing. Moreover, we also conduct quantitative experiments on FLICKR30K, which parameter sharing results in better performance with an average improvement of 0.1. Therefore, we can draw the conclusion that parameter sharing during training not only decreases the parameter dependence but also boost the tross-moda interaction for better performances.
How Does Aurora Benefit from Gated Query Transformation?
To further validate the effectiveness of the Gated Query Transformation module, we conduct comprehensive ablation study on MSCOCO and DiDemo datasets with two variants. The experi- mental results are shown in Figure 6 where Au- rora w/o (blue bars) represents the removal of the Gated Query Transformation and Aurora w. (red bars) initializes the transformation matrix and bias with ones. The gap between the blue bars and ours shows that this module helps promote the final performance to some extent, which also indicates that adaptively incorporat- ing text information into the modality fusion branch has a positive effect on modality alignment under the deep network architecture. In addition, difference between the red bars and ours shows that initializing from zeros is more conducive adaptively controlling the proportion of text feature fusion, and can gradually achieve better modality alignment for better results.
NA CT ATTEREATTT
Figure 6: Analysis of the impact of the gated query transformation module.
4.5 Visualization Analysis
Parameter Distribution. We visualize the parameter distributions of the pre-trained model, fully fine-tuned model, and our Aurora Figure Aurora involves gradually embedding the knowledge learned from downstream tasks into the parameters of pre-trained model, while amplifying the original parameters in a certain direction. From the left part, we can see that our Aurora only adjusts the pre-trained model parameters in small local range, but can have better effect downstream

---

tasks. Meanwhile, we can see from the right part that Aurora's parameter distribution is very close to that of the fully fine-tuned model, with only small changes small range. These small changes enable our method to achieve superior performance on many downstream tasks.
Conclusion
This paper proposes Aurora, a graceful prompt framework for cross-modal transfer. We first leverage mode approximation to implement multimodal prompt tuning, which explores the low intrinsic dimension with only 0.04% parameters of the pretrained model. Then, to better reduce the modality gap, we propose Informative Context Enhancement and Gated Query Transformation modules under extremely low parameter scenarios. Extensive evaluation of Aurora six pross-modal benchmarks shows that it not only outperforms the state-of-the-ar but even surpasses full fine-tuning approach.
Limitations. The representation ability of our Aurora some extent depends on the setting of the rank R. However, also unavoidably increases the parameter size and thus the computational cost. Therefore, it is crucial for to select proper rank in the trade-off between parameter size and performance. Unfortunately, it is challenging for us to determine a suitable rank beforehand with the variations of downstream tasks and data sizes, which poses challenge to the application of our model. We will further optimize this issue in future work.
Broader Impact. Multimodal pre-trained large models have wide applications in the real world, including image-text understanding, retrieval, question answering, and generation. As models seale up, they will achieve more amazing performances. Our Aurora enables lightweight transfer of large-scale models, which allows us to better utilize cloud-based large models and guide them to produce high-pe hegh-performance edge applications under constrained computational resources.
References
[1] RZdunek ACichocki et al. Nonnegative matrix and tensor factorizations: Applications to exploratory multi-way data analysis and blind source separation, 2009.
[2] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXi::2012.13255, 2020.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736. 2022.
[4] Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of machine learning research, 15:2773-2832, 2014.
[5] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference computer vision, pages 5803-5812, 2017.
[6] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[7] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. A clip-hitchhiier's guide to long video retrieval. arXiv preprini arXiv:2205.088 2022.
[8] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:187771901,, 2020.
10

---

[10] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205. 13535, 2022.
[11] Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: Vision-aadioo-language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304008345, 2023.
[12] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint 12XXiv22099066794 2022.
[13] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[14] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-andd-aagguae tasks via text generation. In International Conference Machine Learning, pages 1931-1942. PMLR, 2021,
[15] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 21(4):1253-1275, 2000.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480. 2018.
[17] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, pages 1-16, 2023.
[18] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in in-context learning. arXiv preprint arXiv:2301. 002344, 2022.
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
[20] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson. Quan Vuong, Tianhe Yu, et al. Palm-c: An embodied multimodal language model. arXiv preprint arXiv:2303..m 03337, 2023.
[21] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26, 2013.
[22] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end ladeo-la transformers with masked visual-token modeling. arXiv preprint 3rXiv:2/11.12662 2021.
[23] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for lision-and-language representation learning. Advances in Neural Information Processing Systems, 33:661 16 6626628, 2020.
[24] Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Flevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6913,, 2017.
[25] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00685, 2021.
11

---

[26] Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. Expansionnet v2: Block static expansion fast end to end training for image captioning. arXiv preprint arXiv:2208.065514 2022.
[27] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12976-12985, 2021.
[28] Yuqi Huo. Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. Wenlan: Bridging vision and language by large-scale multi-modal pre-training. arXiv preprint arXiv:2 06561, 2021.
[29] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Lc, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[30] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 709-727, Springer, 2022.
[31] Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. arXiv preprint urXiv:2212.03145, 2022.
[32] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035, 2021.
[33] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint a7X1v:2210.03117, 2022.
[34] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: usiio-anddlanguuag transformer without convolution region supervision. In International Conference Machine Learning, pages 5583-5594, PMLR, 2021.
[35] Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3):455-500 2009.
[36] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-72 73, 2017.
[37] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and languang learning via sparse sampling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7331-7341, 2021.
[38] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.146611 2019.
[39] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient lasguae learning by cross-modal skip-connettions. arXiv preprint arXiv:220s. 12005, 2022.
[40] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: iideo-and-languag pre-training with entity prompts. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 4953-4963, 2022.
12

---

[41] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- laigual understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[42] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[43] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[44] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208. 2021.
[45] Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653, 2023.
[46] Junfan Lin, Jianlong Chang, Lingho Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang- wen Chen. Being comes from not-being: Open-vocabulary text-to-motion generation with wordless training. arXiv preprint arXiv:22 15929, 2023.
[47] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCVV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v11711.05101, 2017.
[49] Haoyu Lu, Mingyu Ding, Yuqi Huo, Guoxing Yang. Zhiwu Lu, Masayoshi Tomizuka, and Wei Zhan. Uniadapter: Unified parameter-efficien transfer learning for cross-modal modeling. arXiv preprint arXiv:2302.0m 06000,66555. 2023.
[50] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293-304, 2022.
[51] Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. tensorized transformer for language modeling. Advances in neural information processing systems, 32, 2019.
[52] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
[53] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077193 2023.
[54] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phranl correspondences for richer image-to-sentenee models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[55] Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, al. Exploring low-dimensional intrinsic task subspace via prompt tuning. arXiv preprint arXiv:21 2021.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
13

---

[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[58] Fangxun Shu, Biaolong Chen, Yue Liao, Shuwen Xiao, Wenyu Sun, Xiaobo Li, Yousong Zhu, Jinqiao Wang, and Si Liu. Masked contrastive pre-training for efficient video-text retrieval. arXiv preprini wwr.x:22222.0088000 2022.
[59] Nicholas D Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E Papalex- akis, and Christos Faloutsos. Tensor decomposition for signal processing and machine learning. IEEE Transactions on Signal Processing, (5(13):3551-358) 2017.
[60] Gilbert W Stewart. On the early history of the singular value decomposition. SIAM review, 35(4):351-566, 1993.
[61] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder sidc-tuning for parameter and memory efficient transfer learning. arXiv preprint arXiv:2206.. 22t.::22000006622,, 2022.
[62] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for vision-and-lanp language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5237, 2022.
[63] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[64] Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279-31 221,,11,,11111 1966.
[65] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. arXiv preprint 0tti::20330030333 2022.
[66] Haixin Wang, Jianlong Chang, Xiao Luo, Jinan Sun, Zhouchen Lin, and Qi Tian. Lion: Implicit vision prompt tuning. arXiv preprint arXiv:2303.0999) 2023.
[67] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal. Subhojit Som, al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[68] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[69] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2288-5296, 2016.
[70] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Leaming to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686-1697, 2021.
[71] Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. Diffusioon-based scene graph image generation with masked contrastive pre-training. arXiv preprint vrXiv:2211.11138 2022.
[72] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong. Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 2022.
[73] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training, arXiv preprint arXiv:2111.07783, 2021.
14

---

[74] Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, al. Visual tuning. arXiv preprint 0rXiv:2305.06061 2023.
[75] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-2335 2021.
[76] Haotian Zhang, Pengehuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
[77] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for langon- models. International Journal of Computer Vision, 130(9))2337-2346 2022.
15

---

APPENDIX
To provide a comprehensive demonstration of our approach, will supplement additional details. The arrangement of these sections is as follows: First, we demonstrate the core concepts our Aurora for clarity. Second, comprehensively make overall comparisons with existing methods various multimodal tasks. Then, we provide details regarding the datasets and baselines in Section C while the concrete training details outlined in Section [ We then conduct comprehensive analysis of the computational costs, including time and memory consumption, along with algorithmic complexity in Section E Furthermore, we provide theoretical support for our approach in Section [F] Finally, present visualizations of our proposed Aurora for several pross-modal tasks to facilitate qualitative method clearly and concisely,
comparisons in Section G To represent for scalars, bold lowercase letters for vectors, italicized uppercase letters for MATRICES, and bold italicized uppercase letters for TENSORS in the equations, respectively.
lowercase letters
A Concept Definition
According to ST, we will offer precise definition of the fundamental notions underpinning our key Mode Approximation component.
First, the definition of tensors be demonstrated as follows:
Definition (Tensor). Let D1, D2....... ,N N denote index upper bounds, a tensor W RP xPPxx DDN of order N is an N-way array where elements Wai,da, da are indexed by dn E (1,2, D.). (..). for
S. N.
Then, the concept of the mode is formulated as follows:
Definition 2 (Mode). Let W E Xpm1Xaxx-XXa be a d-dimensional tensor. The mode-k matricization is obtained by unfolding the tensor along its
of W, denoted W((k E gguxxmm----mepp k-th mode and arranging the entries as rows in a matrix.
Given the mathematical definition of the mode, we can implement decomposition in the context of CP decomposition. We stack all the weight matrices in the attention layer (i.e., Wg, W,, W.) of all the branches into tensor, which needs to be updated as AW. Since our method assumes that the stack of weight matrices a three-order tensor, & is three in our settings, and thus the CP decomposition can be illustrated as follows:
R AW Σ
u, o V, o pr.
(6)
where the rank, 'y are non-neguen non-negative scalar weights, and u, RI non-zero factor vectors. And the mode-k unfolding of the tensor AW is V.V. respectively.
V, R Pr E R are
Our proposed Aurora aims approximate the latent mode matrix with randomly initialized learnable parameters, which can learn knowledge on downstream tasks in a lightweight manner.
B Overall Comparison
We compare all the baselines for three downstream tasks and presented a comprehensive illustration Figure g The arrow in the figure points towards better performance on dual metrics, as moves towards the upper right corner. We rank the size of the model parameters and use it as basis for determining the size of the bubbles, which are also displayed in Figure It is evident that our method performs remarkably well even with smaller parameter sizes, and in several instances, outperforms the fully fine-tuned approach, demonstrating the strength of our architecture.
C More Details for the Baselines & Datasets.
Baselines. For Frozen Backbone methods, UniAdapter [49] currently the state-of-the-art method for earfmeter-efffcient transfer learning in the field of multimodality and can be considered
16

---

MSCOCO
VQAV2
Video Text Retrieval
56.9
Image Text
19.0
Anrren
Agrene
Aarora
Laka
Unisalgeer
62.5
LaRA
Ualkdagerr
50800
Unikdapter
ALSET
Alki-Oox
77.8
$1.5
87.0
92.0
74.8
71,1
55.
GAB
65.
7B.8
Text -> Image
Test-std
MSRVTT(T V)
Figure 8: Performance- Performanee-Parameeer comparison of different methods each multimodal downstream task. Note that the bubble size denotes the size of total tunable parameters.
is another representative of the Adapter class of methods in the multimodal domain. LoRA important branch of earameter-efficient transfer learning methods. Its core idea is to use low-rank decomposed matrices to calculate the incremental change of model parameters during adaptation on downstream tasks. To enable comparison with a wider range of baselines, we replicate many of the settings from prior works and reuse their experiment results whenever possible. It should be noted that this means some baselines only appear in specific experiments.
25
As for Full Fine Tuning methods, we apply UNITER [3). VILLA [23]. OSCAR 43 ALIGN 29 ALBEF 42) and BLIP [4] for image-text retrieval tasks, then we use ClipBERT (37, Frozen in
Time 6 ALPRO
44) VIOLET [22]. All-in-one 665 CLIP4Clip (50] and CLIP-Hhiker 7 for
text-video retrieval tasks, finally we adopt ClipBERT 37 ALPRO 40, Just-Ask 70, VIOLET 22
MERLOT (75. All-in-one [65] for VideoQA task while adopt VL-T5/BART
14 SOHO 27
OSCAR 43 UNITER [13, ALBEF 422 and BLIP
41 for VQA tasks.
Datasets. We provide comprehensive introduction to the datasets of various downstream tasks in the multimodal scenario, outlined below:
MSCOCO 47 is a large scale image-text dataset and each image is annotated with five captions. use Karpathy split of MSCOCO: 5,000 images for testing. 5,000 images for validation, and the rest for training.
Following 44 34
Flickr30K [54] contains 31,000 images collected from Flickr. Each image usually referenced
with five human annotations. Following previous works [9 21, we use 1,000 images for testing,
another 1000 for validation, and the rest for training.
MSR-VTT 69 contains 10,000 video clips and each video clip is annotated with 20 English we adopt k-test split for training and testing.
sentences. Following recent works 44 50]
DiDemo (5] is one of the most commonly used datasets for the temporal localization of events in videos. It contains about 10,000 videos and 40,000 annotations. we follow [49 6 to concatenate all
descriptions corresponding to the same video into a single sentence to conduct actually paragraph-
to-video retrieval task.
VQAv2 [24] is one of the most famous visual question answering datasets which contains 83k/41k/81k images for training/allidationnessing Following [49 42 41,
both train-
ing and validation splits of VQAv2 and additional training samples from Visual Genome (36] for training. The results should be evaluated by the official server and we report the results on the test-dev and test-std splits.
MSRVTT-QA (68) is one of the most popular vidco question answering datasets, It's constructed based on MSRVTT and has 243k open-ended questions associated with 10k videos. We follow to employ the standard split for training and testing.
49
D More Training Details
D.1 Frozen Backbone
Another important design in BLIP is CapFlit, It contains a Captioner to generate captions given web- searched images and a Filter to remove noisy image-text pairs, both Captioner and Filter are finetuned
17

---

Answer
Query Transformation
Image Encoder
Question Encoder
Answer Decoder
Approximation
Mode A Approximation
t "[Encode] Q"
Vmdge/Video
"[Decode]"
Visual Encoder
Multimodal Encoder
Multimodal Decoder
Figure 9: Model architecture for the Visual Question Answering tasks on both images and videos.
individually on the COCO dataset while using different objective loss. In addition, BLIP uses momentum technology to further improve the correctness of the image-text matching relationship.
D.2 Architecture for VQA Tasks
the architecture of Aurora for Visual Question Answering tasks. Compared to Retrieval tasks, the VQA architecture has an additional answer decoder. During fine-tuning, the images/videos are first encoded by single-modal visual encoder, and then the image/video-text pairs are fused using a multimodal encoder and given to the decoder for prediction. Answers are used as ground truth and Language-Model Loss is utilized for parameter updating throughout the entire training process. As the ITM Loss no longer needed, we remove the Informative Context Enhancement module from the VQA architecture. Mcanwhile, to preserve the complete semantic information of the question representations as much as possible. Finally, in order further reduce the number of parameters, share the learnable parameters of the multimodal encoder and the multimodal decoder.
retain the Gated Query Transformation module
D.3 Implementation Details
In this section, we give more training details about our Aurora.
For image-text downstream tasks, we set the image size into 384x 384. We use cosine decay to update the learning rate during training. We set the batch size to 16 for each GPU and train a total of epochs.
For video-text retrieval tasks, we randomly sample T 8(16) frames for each video during training(testing) while setting the frame size to be 224x224. We use cosine decay to update the learning rate, we set the batch size per GPU during training and train for total of 5 epochs.
For VideoQA task, we also sample 8 frames per video for training while the frame size is changed 384. While training, we set batch size to 4 per GPU and train 10 epochs in total. During the evaluation, we randomly sample 16 frames for each video, and we use greedy search to generate the next token when producing answer for its corresponding question.
For VQA task, we set the image size to 480x480 for training/inferenee and adopt batch size of 16 for each GPU. We also adopt cosine decay to change the value learning rate at different epochs and train 10 epochs.
We also perform simple cleaning of the text data and truncate all words beyond the maximum length of the sentence. All data processing and partitioning are consistent with UniAdapter and LoRA to ensure fair comparison. When implementing CP decomposition, textual encoder, visual encoder, and multimodal encoder share the same global mode factor U and factor do parameter sharing and initialize the weights of factor Vto be zero.
18

---

Table 5: Training time and GPU memory comparison.
Method
#Tunable
MSCOCO
FLICKR30K
MSRVTT-QA
VQAv2
DiDemo Time
MSRVTT
Memory
Time
Memory
Time
Memory
Time
Memory
Memory
Time
Memory
UniAdapter (t=512)
18.8M
1.00
1.00
1.00
1.00
1.00
1,00
1.00
1.00
1.00
1.00
1,00
1,00
UniAdapter (r=128)
4.6M
0.86
0.95
0.93
0.95
0.89
0.91
0.79
0.92
0.88
0.94
0.94
0.94
(r=128)
0.2M
0.94
0.88
0.89
0.79
0.91
0.90
0.92
0.95
0.93
Aurora (mm64)
0.IM
0.84
0.93
0.92
0.94
0.86
0.89
0.76
0.89
0.88
0.88
0.94
0.92
E Cost Analysis
In T following [49], we report the training time and GPU memory cost for both retrieval and VQA tasks. We regard the training time and memory cost of UniAdapter(r=512) as one unit. Since we adopt the same backbone models, the forward propagation process of the two methods, Aurora and UniAdapter, is almost consistent, and the time cost similar. However, our Aurora has fewer trainable parameters, resulting in slightly smaller GPU memory footprint.
Then, We will give theoretical analysis of the parametric complexity of the three PETL methods. Assume that the frozen backbone's 's visual, textual, and multimodal encoder all have transformer layers while multimodal encoders contain cross-attention modules and visual, textual encoders contain celf-attention modules. We only approximate the Query/Key/Vaalee attention-basee modules. Let donates the dimension of the hidden feature and r for rank, so the parametric complexity of the LoRA is L X (3 + 6) x 2dr ~ OLdrr), the complexity of the UniAdapter x 4dr O(Ldr), and the complexity our Aurora is L x (3 + 6) 2r 2dr + 2Ld ((L+d)r), normally and L are much smaller than d, so from the above analysis we can draw the conclusion that when increases, our Aurora can achieve the lowest parameter cost.
weight matrix in these
F Theoretical Analysis
Write AW =A >r vr pr), and the ki, elemmnt of AW is:
R A U.
AW
ijk
(7)
Recall that the Frobenius norm of tensor X' RddxxxN is given by:
1/2
d
d
N
|
Σ
M
Σ
(8)
Hence, it suffices to analyze the convergence rate of our parameter tensors in the Euclidean space Rddd where ddN denotes the product of d x N order to distinguishing from the space of multi-dimensional arrays of size d d x N.
We identify RddxxNN with RddáN and let the loss function L be defined on RdddN while we still write L(W) where W ddxdxN a tensor. We will also use Frobenius norm and @² norm interchangeably, which means that:
Assumption F.1
| =112112-
(9)
Assumption F.2 We assume that the loss function L TR.dN TR. has the following property:
1. L is injective.
2. L is strongly convex: there exist and M such that:
mI C() MI.
(10)
That is. V+((X) ml is positive semidefinite and 2 (((( M1 is negative semidefinite.
19

---

(0) (0) Let (A,") , or P decomposition, where x E RR u² IR of (0)
u,
-...., .R] be the randomly initialized vectors used for tensor
E Rd Pr
RRN Denote
AW Wo Σ u
(11)
Let ( be the parameter tensor returned by the nihh training epoch, that is,
AW
(n) of p!!))
(12)
Theorem F.1 Under the above assumptions, and suppose that we train for n epochs with 1j S 1/M using gradient descent. Let W be the optimal parameter tensor, then,
L < (WWm) + (WV"
(n - (m))
(13)
Moreover, W unique.
Proof F.1 The follows from 18). For notational convenience let X = Wo + AW("), and denote the optimal value L(W") by X. We will begin by analyzing the convergence using arbitrary N.Y E RddN and then plug in our parameter tensors. By Taylor's theorem we can write:
((Y) L(X) + (C -x) 2 -x X),
(14)
where Z lies in the line segment joining X and V. By the strong convexity assumption, we have,
1 5 x) x) 2 / (Y) x)=mm))mm) m(y X)
1
m || x112
(15)
2
Hence
(()) 2 ((x) + 1 x12
- m
(16)
Now we use |L(x)) 1o bound L(X) - X The right-hand side of 16 is a convex quadratic function of y, hence ) X 1/m VL()) is the minimizer, thus,
(()) 2 ((X) + V2(2) - x) ? m 18 21112
(17)
[(X)+ + VL(X)* (- (--
m 1 -V((
(18)
2
= -(() - 2m |L(X')
(19)
Since y arbitrary, plugging X' = (n),, we have
X 2 ((x(1))
(20)
2m
By the assumption V ((X) = MI we have
M
((Y) < ((X) + VC(X)"(V- x)
11 111-
(21)
2
Plugging in y (n) v/ C(((()) yields
Z(t) < -( 1 Mn² |
(22)
Now we minimize over on both sides of (2) and denote the optimal value by Σ(n"). The right-hand side of (2) is simple quadratic, hence is minimized by r 1/M, and
min(RHS) L(X) 2M [V
(23)
20

---

My- Last
16 Payer?
M)- Layer 12
8.1
Pretrained
8.0
80
Aurora
Aurora
Auror
6.1
6.0
6.0
231
2.0
2.
011
11.
0.0
-0.1
-033
-0.2
-0.1
0.0
1
Layer
15.0
Pretrained
Pretrained
Pretralned
154
Aurora
10.8
7.
7.5
54
2,5
68
1L.
0.0
-1.10
-L005
0,00
0.85
0,10
Figure 10: We represent the parameter distribution different layers of the pre-trained large-scale multimodal foundation model (BLIP) vs. our Aurora, which is tuned on MSCOCO for image-text retrieval. Notably, WA and W. the stack of the key and value projection matrices in different modality branches.
Now;
C(X(()) + ()X(1)) -((t) S ((X(())
1
[V(()
(24)
2M
Subtracting X* on both sides, have,
C (xx())) ( -)x("))) -x s ((x(1)) X
V [C(2))
(25)
2M
by 20 we have,
2 (x((att))= = (x + +)d((T)) - -x s (1-
m
(c(x(()) - xx))
(26)
M
By mathematical induction, obtain,
((x(()) - x³
" M) (C(x(()) -x-)0000>~000000000000>00 (n - 20),
-
(27)
therefore ((x())) + X as + xc.
Suppose V is another optimal parameter tensor, then by the same argument we have L(X(())
L (V") as n + co. Since RddN is a Hausdorff space, C ((V") (W"). By our assumption that is injective, W =
G Visualization Analysis
G.1 Parameter Distribution
Figure 10land and Figure show more details on the parameter distribution comparisons between our Aurora and the pre-trained model and the full fine-tuned model, in which similar results can be observed on WW and W.. We can see that the mode approximation parameters adjust the original weights, and change the distribution of weights and biases to the downstream task. It can be concluded that Aurora has several advantages over traditional fine-tuning approaches. First, it avoids over-fitting to specific downstream tasks by only adjusting the pre-trained model parameters small local range. Second, it reduces the amount of training required on new data, making it more efficient and cost-effective. Last but not least, Aurora can further improve the model's performance on downstream tasks.
21

---

10.0
Layer
10,0
-aaer?
10.
W. Layer
Fall Fine-tumed
Full Fine-tuned
Full Fine-tuned
Aurora
Aurora
Aurora
4.0
4.0
4.0
2.0
2.0
20
-1.2
-0.1
6:9
1L1
-1.2
-0.1
0.0
0,1
0.2
-0.2
+1.1
[LB
0.1
6.2
1t, Layer
W;, Layer
2001
Wu Layer 12
Pall Fine-tuned
Pull Fine-tuned
Foll Fine-tuned
12.5
Aurora
12.5
Aurora
15.
Aurora
10.0
10.11
58
5.0
25
2.5
0.10
0.15
0.10
0.10
0.05
0.80
0.85
0.10
-0.05
Figure 11: We represent the parameter distribution different layers of the full fine-tuned model vs. our Aurora, which is tuned on MSCOCO for image-text retrieval. Notably, We and W, are the stack of the key and value projection matrices in different modality branches.
Caption7462: He drew a beautiful picture
Caption9623: man is on a cell phone while people are fighting
Caption9451: Kids feeding and playing with the horse
Caption7756: A group of women are singing
Caption8662:
man woman cooking cooking show
Figure 12: Video-Text retrieval cases MSRVTT test set.
G.2 Case Study
Visual-Text Retrieval. Figure 12 demonstrates some actual examples of Aurora performing text- to-video task on MSRVTT test set. In conclusion, the results presented highlight the exceptional performance of Aurora in searching relevant videos from textual descriptions. The accuracy and realism of the returned videos demonstrate the effectiveness of our proposed method in understanding the relationship between text and visual content.
Visual Question Answering, Figure 13 gives some question-answering examples of Aurora and Udaptrr on the MSRVTT-QA dataset. Specifically, our method able to reason about the meaning of the text and video information to answer the questions more accurately than UniAdapter This is an important result because the ability to reason about the meaning of both text and visual information is essential for understanding multimodal data.
22

---

Query191201: What is guy doing?
Fround-Trutian ask
Aurora: ask
UniAdapter: talk
Query191306: What is a man standing talking?
Fround-Truthh kitchen
Aurora: kitchen
UniAdapter: kitchen
Query191382: What does man fold?
Ground-Truth: paper
Aurora: paper
UniAdapter: paper
Query191949: How many men play in tennis match?
Ground- Truth: two
Aurora: two
UniAdapter: two
Query1924000 What does hillary clinton speak to?
Ground-Truth: crowd
Aurora: crowd
UniAdapter: people
Figure 13: Video Question Answering cases on MSRVTT-QA test set.
Overall, the qualitative results shown in Figure and Figure 13 demonstrate the effectiveness of proposed method in both multimodal retrieval and question-answerte anwwering tasks. We believe that our approach has the potential to be used in many multimodal applications, where understanding and analyzing multimedia data is essential.
23


Reviews:
[{"review_id": "y4qZnHWgc4", "reviewer": "Reviewer_NuU2", "summary": "The paper proposes AURORA, a method which uses mode approximation to boost the knowledge transfer in Vision Language models and enhances alignment between the modalities in a lightweight parameter efficient manner. In addition to this, the paper further proposes Context Enhancement module and  Gated Query Transformation module to boost the modality fusion in an adaptively controllable way. The proposed method is evaluated on six cross-modal tasks and two zero-shot tasks and is compared to existing PETL methods.", "questions": "- Can the method be applied to a broader range of vision-language tasks, and how well does it adapt to different modalities or data distributions?", "limitations": "- Please have a look at Questions and Weaknesses", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper seems to be the first which has used mode approximation in Vision-Language models to efficiently achieve modality fusion.\n- The proposed method is novel with respect to existing prompt tuning methods.\n- The proposed methods works by tuning a very small number of parameters which can save time and computational resources.", "weaknesses": "- The motivation for using Mode approximation is not very clear in the paper. For instance there could be other methods which can solve the redundancies in the attention weights. There is no detailed explanation of the theoretical basis or formal analysis of how mode approximation aids in prompt learning. Adding a brief theoretical background could provide more insights into the method's underlying principles.\n- The authors haven't provided any solid proof behind the argument regarding loss of textual information in multi-modality fusion branches, which essentially forms the basis of introducing Gated Query Transformation.\n- The authors should have compared their method with previous prompt learning methods [1, 2] for zero-shot out-of-distribution classification tasks.\n\n\n[1] M. U. khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, “Maple: Multi-modal prompt learning”. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023\n\n[2] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models”. International Journal of Computer Vision (IJCV), 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your professional comments. We will continue to improve based on your feedback, and we believe that our Aurora has a very positive impact on promoting the efficient transfer of multimodal large models in the community.\n\n> **Motivation not clear.** The motivation for our mode approximation module lies in two folds. **First**, it can approximate the weights in large-scale model with a little learnable parameters. **Second**, it is based on the algorithm of CP decomposition which has good theoretical convergence property. We agree with you that theoretical background could provide more insights into the method. Therefore, we have provided a detailed concept definition in Appendix Part A . **Third**, we provide detailed theoretical analysis and derivation in Appendix Part F to validate that the proposed method can achieve good convergence.\\\nWe acknowledge that there are other methods to address redundancy in attention weights. However, Aurora employs mode approximation to generate lightweight prompts, which is an effective approach that achieves good performance with very few parameters. We believe the advantage of this approach is that it enables efficient parameter transfer with multimodal prompt tuning under extremely few parameters while reducing computational and storage costs while maintaining performance. The advantages of Aurora are further demonstrated in the experimental results.\n\n> **No solid proof.** We provide the motivation for designing the Gated Query Transformation module in the following text, which is that the information content of textual information decreases as the layer depth of cross-attention increases.\\\nSpecifically, in the cross-attention mechanism, the encoding vectors are used to calculate the attention distribution for generating the output vectors. And the query vectors are used to calculate the weights of the attention distribution, where the query vectors come from the textual tokens under the multimodal setting. \\\nSuppose we have $L$ layers in the cross-attention mechanism, and the **encoding vector (from visual modality) for layer $l$ is denoted as $e_l$**, and the **query vector (from textual modality) for layer $l$ is denoted as $q_l$**. We aim to prove that with the increase of the layer number $L$, the textual information content of the query vector $q_L$, which is the encoding vector, becomes lower and lower.\\\nWe can use the concept of entropy to measure the information content of the query vector. \nThen we can view the query vector $q_L$ as a random variable that has all possible encoding vectors $e_l$ as its possible values. To compute the entropy of the query vector $q_L$, we need to calculate the probability distribution $p(q_L)$. We can assume that $p(q_L|e_1, ..., e_L)$ is the probability distribution of the query vector $q_L$ given all the $L$ encoding vectors. Then, we can use Bayes' theorem to transform $p(q_L|e_1, ..., e_L)$ into the product form of $p(e_1, ..., e_L|q_L)$ and $p(q_L)$ as follows:\\\n$$ p(q_L|e_1, ..., e_L) = \\frac{p(e_1, ..., e_L|q_L) p(q_L)}{p(e_1, ..., e_L)} $$\\\nSince $p(e_1, ..., e_L)$ does not depend on $q_L$, we can treat it as a constant and get:\\\n$$ p(q_L) = \\frac{p(e_1, ..., e_L|q_L) p(q_L)}{p(e_1, ..., e_L)} $$\\\nThen, we can express the entropy of the query vector $q_L$ as:\\\n$$H(q_L) = -\\sum_{e_1, ..., e_L} p(e_1, ..., e_L) \\sum_{q_L} p(q_L|e_1, ..., e_L) \\log p(q_L|e_1, ..., e_L) \\\n= -\\sum_{e_1, ..., e_L} p(e_1, ..., e_L) \\sum_{q_L} p(q_L) \\log p(q_L) \\\n\\sum_{e_1, ..., e_L} p(e_1, ..., e_L) \\sum_{q_L} p(e_1, ..., e_L|q_L) \\log p(e_1, ..., e_L|q_L)$$\nThe first summation term represents the entropy of the marginal distribution of $q_L$, which is independent of $e_1, ..., e_L$. Therefore, we can treat it as a constant and rewrite the second summation term as: $ -\\sum_{e_1, ..., e_L} p(e_1, ..., e_L) H(e_1, ..., e_L|q_L) $.\\\nHere, $H(e_1, ..., e_L|q_L)$ is the conditional entropy of $e_1, ..., e_L$ given $q_L$. It represents the uncertainty of all possible encoding vectors $e_1, ..., e_L$ given the query vector $q_L$. Since the conditional entropy increases with the uncertainty of the conditioning variable, we can infer that the information content of $q_L$ decreases with the increase of $L$, as the uncertainty of the encoding vectors given $q_L$ increases.\\\nIn summary, as the layer number $L$ increases in the Transformer, the information content of the query vector $q_L$ decreases, indicating that the query vector becomes less informative about the encoding vectors in the subsequent layer.\n\n>**Lack of comparison.** Our paper mainly copes with the typical multi-modal tasks based on the BLIP architecture including cross-modal retrieval, VQA on both image and video modalities. We will further add the comparisons with the work you mentioned in the final version. During the rebuttal period, we would like to first add parts of the results which are shown below. \n| Datasets | ImageNet |       | Caltech101 |       | StanfordCars |       |\n|----------|:--------:|:-----:|:----------:|:-----:|:------------:|:-----:|\n| Method   |   base   | novel |  base  | novel |   base  | novel |\n| MaPLe    |   76.66  | 70.54 |  97.74 | 94.36 |  72.94 | 74.00 |\n| Aurora   |   76.59  | 70.75 |    98.13   | 94.52 |  73.75  | 74.28 |\n\n> **Generalization results.** Following your suggestion to further validate the generalization ability, we extend our Aurora to a more recent sota vision-language model, InstructBLIP[1]. We apply Aurora to the Q-former architecture in InstructBLIP, and the results are shown below. \n| Method              | OKVQA | A-OKVQA | COCO Caption |\n|----------|:--------:|:-----:|:----------:|\n| InstructBLIP+FFT（188M） | 54.9  | 55.9   | 68.0  |\n| InstructBLIP+LoRA（11M | 53.3  | 52.8    | 67.4  |\n| InstructBLIP+UniAdapter（18M） | 53.0  | 53.5    | 67.2 |\n| InstructBLIP+Aurora（0.5M) | 53.7  | 54.1    | 67.6  |\n\n[1] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. arXiv preprint arXiv:2304.08485, 2023."}, {"review_id": "IwR5366RSi", "reviewer": "Reviewer_37bN", "summary": "The paper addresses the problems of (i) transfer learning and (ii) reducing the multimodality gap in multimodal models.\n\nTo address (i) it presents a technique which can be viewed as a generalization of LoRA; instead of independently factoring representing matrices as a low rank representation, all the matrices of the transformer get stacked together and are represented as a low rank tensor. \n\nTo address (ii) two techniques are presented: one which aims to improve the representations by allowing information exchange between different examples in the batch (Informative Context Enhancement) and another which aims to prevent the loss of text information for deep models (Gated Query Transformation). \n\nAblation studies are performed to justify the different design choices.\n", "questions": "1. Why are some methods present in ‘Methods with frozen backbone’ in Table 2 omitted from the same section in Table 3? (e.g. LoRA)\n2. Is |B| in the ‘Informative Context Enhancement’ section referring to the number of tokens in the entire batch?\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Mode approximation (Generalizing LoRA by stacking the matrices of all the transformer layers and using CP decomposition to represent that) is a nice idea and experimentally shows to be a parameter efficient way (beats LoRA) of adapting frozen models to new domains\n\n2. Thorough ablation studies are performed to demonstrate the impact and justify existence of each presented component in the final model.\n", "weaknesses": "1. Informative Context Enhancement seems to allow information exchange between different examples in the batch. That makes it dependent on the batch size, but the impact of changing it is not evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your highly accurate summary of our work and recognition of our comprehensive experiments. We will continue to improve based on your feedback, and we believe that our Aurora has a very positive impact on promoting the efficient transfer of multimodal large models in the community.\n\n>**Impact of the batch size.** Thanks for the comment on our work and we greatly appreciate your valuable feedback. We implement different numbers of batch size to investigate the effectiveness of Informative Context Enhancement. Some results are shown below:\n>\n>| DataSet                          | Batch Size = 4 | Batch Size = 8 | Batch Size = 16 | Batch Size = 32 |\n>| -------------------------------- | :------------: | :------------: | :-------------: | :-------------: |\n>| MSCOCO (I$\\rightarrow$T, R@1)    |      80.4      |      80.6      |      80.7       |      80.8       |\n>| Flickr30K (I$\\rightarrow$T, R@1) |      96.9      |      97.1      |      97.2       |      97.2       |\n>| DiDemo (R@1)                     |      53.1      |      53.1      |      53.2       |      53.4       |\n>| MSRVTT-QA                        |      44.4      |      44.7      |      44.8       |      45.0       |\n\n>**Lack of results.** Thanks for the comment on our work and we greatly appreciate your valuable feedback. We apologize for the omission of the comparison experiment results on LoRA in Table 3. We have added them, and the complete results for Table 3 are shown below:\n>\n>| Method                       | #Tunable | test-dev  test-std | Method         | #Tunable | test acc |\n>| -------------------------------- | :----------: | :--------------------: | ------------------ | :----------: | :----------: |\n>| Methods with frozen backbone |              |                        |                    |              |              |\n>| LoRA (r=32)                      |    10.6M     |      74.11   74.24      | LoRA (r=32) |    10.6M     |     44.3     |\n|UniAdapter (r=512) | 18.8M | 75.44 75.56 | UniAdapter (r=512)| 18.8M | 44.7|\n|Aurora (r=64) | 0.1M | 77.69 77.87 | Aurora (r=64) | 0.1M | 44.8|\n\n>**Symbol not clear.** Thanks for the comment on our work and we greatly appreciate your valuable feedback. $| \\mathcal{B}|$ is the number of image-text pairs in the entire batch, and \"feature\" means the [cls] token of each image and text. In other words, each [cls] token represents the global information of the image and text."}, {"review_id": "5pLaciEH4P", "reviewer": "Reviewer_ME39", "summary": "This paper proposes a parameter efficient adaptation technique Aurora for multi-modal models. Particularly, the proposed method motivates their design by suggesting that the original pre-trained weight matrices have redundancies due to their high dimensional nature and the downstream task often requires low-dimensional reparameterization only. Aurora supplements the original weight matrices with series-one rank tensors which are only learned during the fine-tuning process. \n\nIn addition, to enhance the modality alignment between the vision and text representations, Aurora utilizes informative context enhancement module and gated query transformation which fuses and explicitly relates the textual representations with the multi-modal fusion representations in the cross-attention block of BLIP. \n\nExtensive experiments over various benchmarks shows the effectiveness of Aurora in comparison with fine-tuning and parameter efficient adaptation approaches.", "questions": "Please refer to the weaknesses section. ", "limitations": "The authors have discussed the limitations and societal impacts are highlighted.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n\n(1) The idea of decomposing learnable pre-trained matrices into small rank tensors is encouraging, as it explicitly allows to adapt only necessary amount of parameters for efficient and effective adaptation.\n\n(2) The paper has performed extensive evaluations with proper ablation studies which justifies their design choices.\n\n(3) The proposed methods performs favorably well with very less number of learnable parameters.", "weaknesses": "Weaknesses:\n\n(1) The overall paper presentation style is very confusing, specially the main methodology section. There is no preliminaries on the baseline architecture on which the proposed solution has been built. It is very difficult for the readers to grasp the contents without knowing the main model architecture. For example, in line 161-162, the authors have mentioned the cross-attention module, but unfortunately no prior information about that block is provided anywhere in the manuscript. Also the writing is not clear and I found it difficult to understand the manuscript. \n\n(2) How the proposed solution is considered as a prompt learning variant? If I understood correctly, the additional learnable parameters are utilized as part of the model and the input tensor has to be multiplied with it. It is not the case where the learnable parameters are part of the inputs, which is the core definition of prompt learning.\n\n(3) The proposed multi-modal alignment module seems to be heavily designed for the BLIP multi-modal model. It is not clear if these components could be utilized in other multi-modal models. It will be good to see the generalization of proposed approach to other recent VL models. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our design idea and comprehensive experiments. We will continue to improve based on your feedback, and we believe that our Aurora has a very positive impact on promoting the efficient transfer of multimodal large models in the community.\n\n> **Confusing presentation.** Thanks for the comment on our work and we greatly appreciate your valuable feedback. We apologize for any confusion caused by the lack of preliminaries on the baseline architecture used to build our proposed solution. Due to the page limit, we put the details of the whole architecture of the pretrained model BLIP in Part D of the Appendix. Following your sincere suggestion to introduce more preliminary knowledge of our base model, we will revise the paper to include a more detailed explanation of the network architecture and its role in our proposed solution in a more clear way. We will work to improve the clarity of our writing to make it more accessible to readers in the final version. \n\n> **Wrong use of prompt learning.** Thanks for the comment on our work and we greatly appreciate your valuable feedback. Actually, prompt tuning is one typical way of parameter-efficient transfer learning (PETL). Our work is based on the existing roadmap of the PETL by decomposing the pre-trained networks with learnable parameters. These learnable parameters are multiplied with the input as the \"soft prompts\" for the pre-trained parameters to implement PETL. Therefore, our Aurora is quite different from the typical prompts as the inputs. In the final version, we will replace the \"prompt\" with \"multimodal parameter-efficient transfer learning based on mode approximations\" to better reflect the essence of our method.\n\n> **Generalization results.** Following your suggestion to further validate the generalization ability, we extend our Aurora to a more recent sota vision-language model, InstructBLIP [1]. We apply Aurora to the Q-former architecture in InstructBLIP, and the results are shown below. \n| Method              | OKVQA | A-OKVQA | COCO Caption |\n|----------|:--------:|:-----:|:----------:|\n| InstructBLIP+FTE（188M） | 54.9  | 55.9    | 68.0         |\n| InstructBLIP+LoRA（11M）        | 53.3  | 52.8    | 67.4         |\n| InstructBLIP+UniAdapter（18M）        | 53.2  | 53.5    | 67.2         |\n| InstructBLIP+Aurora（0.5M）      | 53.7  | 54.1    | 67.6         |\n\n\n[1] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. arXiv preprint arXiv:2304.08485, 2023."}, {"review_id": "C5erle39l6", "reviewer": "Reviewer_kJXf", "summary": "This paper aims to design a lightweight prompt tuning method (i.e. Aurora) for cross-modal transfer. The main idea follows the observation by LoRA [15] that most of the features are redundant and a low-rank ∆W can be learned to adapt the features. Different from LoRA, they adopt CP decomposition [47] to decompose the learnable parameters into a series rank-one tensors. They also propose Informative Context Enhancement and Gated Query Transformation for better modality alignment. However, the connection between the two parts is not clear. Experiments show that Aurora performs better than LoRA and is at least comparable to full finetuning methods.", "questions": "The author is suggested to address the concerns in the weakness section.", "limitations": "The limitation is mentioned at the end of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength \n1.\tFigure 1 highlights the difference between the proposed method and the baselines.\n2.\tThe proposed method is lightweight and effective.\n3.\tAurora is applicable to both image-text and video-text retrieval.\n", "weaknesses": "Major\n1.\tThe main idea of the paper is lightweight adaptation achieved by adopting CP decomposition [47] for the adapter weights. While it is shown to be effective and parameter-efficient, there is limited technical novelty introduced by this work. In terms of story-level novelty, it mainly follows the observation and approach in LoRA [15], and it’s not considered novel.\n\n2.\tThe improvement of Aurora over UniAdapter [32] is marginal in Table 1.\n\n3.\tWhat are BLIP, BLIP+LoRA zero-shot performance in Table 4?\n\n4.\tThe new thing of this paper is Informative Context Enhancement and Gated Query Transformation. However, there is loose connection between the main idea and these two modules. These two modules are orthogonal to the low-rank approximation part. What will be the performance if LoRA is integrated with these two modules?\n\n5.\tWhich part of the method does Parameter Sharing in L248-L255 correspond to?\n\nMinor\n6.\tWhat are x-axis and y-axis in Figure 7?\n\n7.\tThe main flow of the architecture is not clear. In section 3.3, Gated Query Transformation is presented after Context Enhancement. However, Gated Query Transformation manipulates f, but Context Enhancement depends on f. Is the former performed prior to the latter?\n\n8.\tThe insight for Gated Query Transformation is unclear. How about replacing t’ in L191 with t?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the lightweight design, effectiveness, and comprehensive experiments of our method. We will continue to improve based on your feedback, and we believe that our Aurora has a very positive impact on promoting the efficient transfer of multimodal large models in the community.\n\n>**Limited technical novelty.** We believe that our paper has certain novelty and contributes to the entire community for three reasons. **First**, our main advantage is that the mode approximation based on CP decomposition has a lighter parameter decomposition architecture compared to LoRA. **Second**, our low-rank decomposition method has better mathematical interpretability as theoretical support (please refer to Appendix F). Most importantly, **third**, Aurora is not significantly dependent on rank, demonstrating true high parameter efficiency.\nSpecifically, in Figure 3 of the main text, part (d), we can observe that as the rank increases, our method Aurora does not exponentially increase the burden of learnable parameters. Experimental results have further validate our idea.\n\n> **Marginal improvement in Table 1.** We would like to point out that our advantages beyond UniAdapter are actually clear not only on parameter efficiency but also on performance.\n**First**, in the cross-modal retrieval task in Table 1, we achieve better results than UniAdapter even when the fine-tuned accuracy is already near 100\\%. \n**Second**, in the tasks of Table 2 and 3, we achieve even greater advantages over UniAdapter, with around a 3\\% improvement in performance in both video-text retrieval and VQA tasks.\n**Third**, our Aurora already achieves a leading advantage with rank=64. In fact, when we compare fairly with UniAdapter using rank=512, our leading advantage will be further expanded to around 5\\%.\n\n> **Zero-shot performance in Table 4.** We add additional experiments on BLIP and BLIP+LoRA, where BLIP is the base pretrained version and BLIP+LoRA is fintuned with LoRA. The experiment results in Table are shown below: \n| Method | \\# Parameter ||   MSRVTT (T2V)  |   | | DiDemo (T2V)   |   |\n|-- | :------: | :--: | :--: | :--: | :--: | :--: | :--: |\n|    |    | R@1 | R@5  | R@10 | R@1 | R@5  | R@10 |\n| BLIP |  223M |  41.5 | 62.0 | 70.7 |42.1   | 59.6 | 67.3 |\n| BLIP + LoRA |  10.6M| 42.7  | 62.8 | 71.4 |  43.3   | 60.3 | 68.2 |\n\n>**Loose connection between main modules.** Mode approximation is designed for high-efficiency transfer, however, there exists no pure module that can be adapted and suffers from no modality alignment pain. Therefore, how to utilize the feature outputs of the mode approximation module to boost the performances on multi-modal tasks is the core motivation of Informative Context Enhancement and Gated Query Transformation. We also add some experiments on **LoRA integrated with Informative Context Enhancement and Gated Query Transformation**. The results are shown below, we can draw the following conclusions: **First**, Informative Context Enhancement and Gated Query Transformation indeed boost the performances of multi-modal tasks even with LoRA, which validates the effectiveness of our proposed module. **Second**, the increase on LoRA is obviously lower than that on Aurora, which can be attributed to that better representations learned on downstream tasks cause better modality alignment results.\n| \\#Tunable |  | MSCOCO I2T |  |  | MSCOCO T2I |  |  | FLICKR30K I2T |  |  | FLICKR30K T2I |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| - | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |\n|  10.6M  | 80.1 | 94.4 | 97.5 | 62.3 | 84.5 | 90.9 | 96.5 | 99.9 | 100.0 | 86.2 | 97.4 | 98.7 |\n| **\\#Tunable** |  | **MSRVTT** |  |  |  |  |  | **DiDemo** |  |  |  |  |\n| -  | R@1 | R@5 | R@10 | MdR |  |  | R@1 | R@5 | R@10 | MdR |  |  |\n| 10.6M  | 50.6 | 72.7 | 81.6 | 2.0 |  |  | 51.7 | 76.0 | 83.4 | 2.0 |  |  |\n\n> **Parameter Sharing explanation.** The parameter sharing described in L248-L255 means that the trainable parameters of the textual branch, visual branch, and multimodal fusing branch in BLIP share the same U and V decomposition factors when performing mode approximation. Detailed descriptions can be found in L144-L146.\n\n>**Figure 7 explanation.** Figure 7 shows a comparison of the distribution statistics of the parameters for the pre-trained model and our Aurora after efficient fine-tuning. The x-axis represents the parameter values, and the y-axis represents the frequency of occurrence.\n\n>**Demonstration order.** Gated Query Transformation is implemented prior to Context Enhancement following Figure 2. We will change the order in which these two modules are presented in the final version to help readers better understand.\n\n>**Insight for Gated Query Transformation.** Loss of textual information in deep multi-modal fusion branches essentially forms the basis of introducing Gated Query Transformation. The solid proof is also given in A2 for reviewer NuU2. Since $t'$ is learned by autograd, we output the mean value of the zero-initialized learnable transformation matrix $\\gamma$ and $\\beta$, which is 1.17, 0.23 (1.09, 0.36) on Didemo (Flickr30K) separately. It demonstrates that scaling textual information is beneficial for training."}, {"review_id": "kEGiTjoXqJ", "reviewer": "Reviewer_Srms", "summary": "This paper proposes a novel approach to address the challenge of high learning costs when migrating large models to specific downstream tasks. The proposed method aims to reduce task complexity and improve the consistency across the different modal outputs of multimodal models.\n\nThe authors employ a LoRA-like technique that involves fine-tuning the transformer by appending adjustment matrices to the qkv matrices. This process is further optimized through CP Decomposition, an approach used to reduce the scale of the fine-tuning parameters. Then use Informative Context Enhancement mechanism to compute weights and the mixed graph-text features are adjusted according to these weights, facilitating the generation of superior fused features. To prevent the loss of textual information during the alignment of deep multimodal networks, the authors propose using a Gated Query Transformation. This approach calculates the blending ratio of text features for enhancement.\n\nThe paper claims that these techniques allow the model to outperform the current best methods on several downstream tasks, using fewer fine-tuning parameters, and even surpasses fully fine-tuned methods.", "questions": "(1) Justification for Terminology: Could you clarify why you choose to refer to your method as a 'multimodal prompt'? Given the technique's resemblance to low-rank fine-tuning, the terminology might be misleading. Understanding the rationale behind this terminology would greatly aid in interpreting your work.\n(2) Clarification on Gated Query Transformation: The paper could benefit from further details on the application of the Gated Query Transformation. Specifically, is the text information fused with the image features before entering the cross-attention, according to the gate values? This part of the methodology was not entirely clear in the paper, and further explanation would enhance the clarity of your methods and potentially support the broader understanding and application of your techniques.", "limitations": "(1) Hyperparameter Selection: The need for manual selection of the rank hyperparameter is a key limitation of the proposed method. While it is acknowledged by the authors, there is no substantial discussion on how this limitation can be addressed. \n(2) Training Time: Another potential limitation that should be acknowledged is the impact on training time. Although the proposed method reduces the parameter count for downstream tasks, it doesn't seem to significantly decrease training time.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "（1） Innovative and Effective Approach: Building upon the foundations of similar works like LoRA, this paper makes significant strides by achieving better fine-tuning performance with lower parameter usage. \n（2） Proposes Novel Techniques: The paper introduces innovative methods such as Informative Context Enhancement and Gated Query Transformation to enhance the fusion of modalities. These methods appear to be highly effective in improving model performance, and could potentially be applied in a range of different contexts.\n（3） High-Quality Visualizations: The paper includes aesthetically pleasing and informative figures and tables, which contribute to the clarity and overall quality of the work. The visualizations effectively aid in the understanding of complex concepts and methodologies.", "weaknesses": "(1)  Not Clearly Written: There appears to be some confusion in the terminology used, especially around 'soft prompts'. While the paper asserts that its fine-tuning approach can be seen as a 'soft prompt', the proposed method is more akin to parameter-efficient transfer learning. Therefore, the use of the term 'prompt' may not be accurate in this context. A clearer and more precise usage of technical terminology would be beneficial for readers and would also strengthen the overall quality of the paper. (Now author states that in the final version, they will replace the inappropriate term \"prompt\" with \"multimodal parameter-efficient transfer learning based on mode approximations\" to better reflect the essence of our method.)\n(2) Lack of Details on Gated Query Transformation: The paper does not provide a clear and thorough explanation of the implementation details for the Gated Query Transformation. For future improvements, it would be beneficial to include more technical details of this novel technique, which would improve the clarity of the paper and make it easier for others in the field to replicate and build upon this work. (Now author states the final version will be more detailed)\n(3) Manual Parameter Tuning: The paper suggests that the rank hyperparameter needs to be manually adjusted, which could pose an obstacle to scalability and efficiency. More research is required on how to optimally choose this hyperparameter's value. (Author said they use a detailed grid search experiments to get good rank)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the novelty and effectiveness of our approach as well as the visualization demonstration. We will continue to improve based on your feedback, and we believe that our Aurora has a very positive impact on promoting the efficient transfer of multimodal large models in the community.\n\n>**Not Clearly Written.** Thanks for the comment on the terminology and we greatly appreciate your valuable feedback. We agree with your statement, and our method is essentially a parameter-efficient transfer learning method. Our initial understanding is that the features obtained by passing the input through the learnable parameters constructed by mode approximation are prompts, which are added to features obtained by the pre-trained network. Therefore, in the final version, we will replace the inappropriate term \"prompt\" with \"multimodal parameter-efficient transfer learning based on mode approximations\" to better reflect the essence of our method.\n\n>**Lack of Details on Gated Query Transformation.** \nThanks for your sincere comment on the detail supplement and we greatly appreciate your valuable feedback. We will add solid proof behind the argument regarding the loss of textual information and implementation details in our final version for better understanding.\n\n>**Manual Parameter Tuning.** \nThanks for the comment on the hyper-parameter and we greatly appreciate your valuable feedback. Actually, we borrow the idea of [1] to automatically estimate the intrinsic dimension and then tune the model according to the estimated rank value (around 60). And then to further investigate the effectiveness of our low-rank decomposition, we implement a detailed grid search experiments on different rank values, which is shown in the Figure 3 in the paper. And an obvious phenomenon is that the performance improvement starts to level off after the rank increases to 64. Furthermore, we will attempt to intergrate the intrinsic dimension estimator with our method more tightly.\n\n>**Justification for Terminology.**\nOur initial understanding is that the features obtained by passing the input through the learnable parameters constructed by mode approximation are added to the features obtained by the pre-trained network as prompts. As a result, our mode approximation modules in the multi-modal network are called `multimodal prompt'. In the final version, we will replace the inappropriate term \"prompt\" with \"multimodal parameter-efficient transfer learning based on mode approximations\" to more accurately reflect the nature of our method.\n\n>**Clarification on Gated Query Transformation.**\nThanks for your sincere comment on the detail supplement and we greatly appreciate your valuable feedback. Loss of textual information in deep multi-modal fusion branches essentially forms the basis of introducing Gated Query Transformation. Please refer to the answer for reviewer NuU2 for details of solid proof. Gated Query Transformation utilizes the gated function to fuse textual information with the feature in the fusion branch as the input for the cross-attention in the next layer to avoid textual information loss. We will add the solid proof behind the argument regarding the loss of textual information and implementation details in our final version for better understanding.\n\n\n[1] Chen B, Huang K, Raghupathi S, et al. Automated discovery of fundamental variables hidden in experimental data[J]. Nature Computational Science, 2022, 2(7): 433-442."}]

Author rebuttal:

Meta-review: This is a borderline submission. The rebuttal and the following discussion clarified the remaining concerns and we believe that the propose approach achieves solid empirical results and provides sufficient insight to the relevant community. We urge the authors to incorporate the results from the rebuttal and discussion.
Decision: Accept (poster)