Title: Parameter-efficient Tuning of Large-scale Multimodal Foundation Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: parameter-efficient transfer learning; multi-modal learning; prompt learning
Abstract: Driven by the progress of large-scale pre-training, parameter-efficient transfer learning has gained immense popularity across different subfields of Artificial Intelligence. The core is to adapt the model to downstream tasks with only a small set of parameters. Recently, researchers have leveraged such proven techniques in multimodal tasks and achieve promising results. However, two critical issues remain unresolved: how to further reduce the complexity with lightweight design and how to boost alignment between modalities under extremely low parameters. In this paper, we propose A gracefUl pRompt framewOrk for cRoss-modal trAnsfer (AURORA) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate 0.1M trainable parameters to implement the multimodal parameter-efficient tuning, which explores the low intrinsic dimension with only 0.04% parameters of the pre-trained model. Then, for better modality alignment, we propose the Informative Context Enhancement and Gated Query Transformation module under extremely few parameters scenes. A thorough evaluation on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even outperforms the full fine-tuning approach. Our code is available at: https://github.com/WillDreamer/Aurora.

Parameter-efficient Tuning of Large-scale Multimodal Foundation Model

Abstract

Driven by the progress of large-scale pre-training, parameter-efficient transfer learning has gained immense popularity across different subfields of Artificial Intelligence. The core is to adapt the model to downstream tasks with only a small set of parameters. Recently, researchers have leveraged such proven techniques in multimodal tasks and achieve promising results. However, two critical issues remain unresolved: how to further reduce the complexity with lightweight design and how to boost alignment between modalities under extremely low parameters. In this paper, we propose A gracef u l p r ompt framew o rk for c r oss-modal tr a nsfer ( Aurora ) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate 0.1M trainable parameters to implement the multimodal parameter-efficient tuning, which explores the low intrinsic dimension with only 0.04% parameters of the pre-trained model. Then, for better modality alignment, we propose the Informative Context Enhancement and Gated Query Transformation module under extremely few parameters scenes. A thorough evaluation on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even outperforms the full fine-tuning approach. Our code is available at: https://github.com/WillDreamer/Aurora .

1 Introduction

The era of large models in deep learning has arrived, with increasingly more large-scale pre-trained models exhibiting remarkable generation and inference capabilities in the fields of text [38, 9, 63], vision [57, 53], and multi-modality [56, 41, 3, 76]. Nowadays, the mainstream strategy in the community involves pre-training models on large-scale data, and then fine-tuning them for each downstream task. This approach has been demonstrated to be effective in various cross-modal tasks, as evidenced by [40, 50, 67].

However, this high-performing strategy is not always advantageous, as several practical factors limit its further development. The primary concern is excessive parameter dependency. For example, GPT3 [9] (175B params) obtains striking performances while coming at the cost of a massive parameter count. The enormous parameter size has two obvious drawbacks. Firstly, it incurs significant computations and physical storage costs, making pre-training and transfer very expensive. Besides, fine-tuning limits pre-trained knowledge's effectiveness in small-scale downstream tasks. Both of them hinder the further extension of large models from specific datasets to more general scenarios.

To relieve the high cost of large pre-trained models, a series of parameter-efficient transfer learning (PETL) methods [17, 74] have been proposed. The common paradigm is to freeze the backbone

network of the large model and introduce a small number of additional parameters. Recently, some works have begun to focus on PETL in the multimodal field, such as UniAdapter [49], VL-Adapter [62], and MAPLE [33]. However, their common idea is to combine existing architectures used in NLP for multimodal models, which simply inserts learnable parameters in the backbone networks of unimodal and multimodal branches to achieve good performances. Their simple designs cannot integrate the essence of efficient parameter transfer into multimodal models. There are still two main challenges that need to face: one is how to transfer knowledge in an extremely lightweight manner , and the other is how to boost alignment between modalities under extremely low parameters.

To overcome the challenges mentioned above, we propose A gracef u l p r ompt framew o rk for c r ossmodal tr a nsfer, named Aurora , which surpasses the state of the art with a remarkably small parameter budget of merely ∼ 100K, thereby aptly living up to its name due to lightness. First , inspired by the idea of CANDECOMP/PARAFAC (CP) decomposition [64], we utilize mode approximation to generate lightweight learnable parameters for the attention-based architectures. As attention is proven to inevitably fit the background noise [45], most other features are redundant. For example, the categories in CLIP [56] are essentially infinite, which results in only a few features responding to a single class. This can be easily proven by the fact that the logits output by CLIP are mostly between 0.2 to 0.3 with very little fluctuation, indicating that most features are consistent. In other words, although pre-trained models have large-scale high-dimensional parameters, the intrinsic dimension corresponding to each downstream task is not large [55]. Thus, our mode approximation can fine-tune a very small number of parameters to achieve good performance on downstream tasks theoretically.

Second , in contrast to existing methods that directly insert a small network for cross-modal fusion, we mine deeper into the unimodal information and fuse them in a controllable manner with multimodal representations, thereby achieving desirable results with lower parameter dependence. In our proposed Informative Context Enhancement module, each feature is enriched by contextual information to boost the modality fusion. Furthermore, we adaptively control the ratio of textual information supplied for modality fusion in our proposed Gated Query Transformation module.

Finally, we evaluate Aurora on six cross-modal tasks and two zero-shot tasks, which obtains state-ofthe-art performance compared to other PETL methods. Compared to the full fine-tuning, Aurora even obtains 1.8% and 0.5% performance improvement on MSRVTT and VQAv2 benchmarks averagely but only with about 0.05% trainable parameters of the pre-trained model.

2 Related Work

2.1 Vision-Language Models

In vision-language pre-training, large-scale image-text pairs are used to pre-train the Vision-Language Models (VLMs) for downstream tasks [13, 56, 41, 72, 3, 76]. The architecture of VLMs typically consists of a visual encoder, a textual encoder, and a cross-modal fusion module. VLMs have been applied to various tasks, including image generation [71], image captioning [39, 26], visual question answering [12, 11, 20], and cross-modal retrieval [58, 28]. This article focuses on how to efficiently fine-tune VLMs on downstream tasks and utilizes BLIP-base [41] for vision-language tasks.

2.2 Parameter-efficient Transfer Learning

As the size of recent models increases rapidly, updating the models in parameter-efficient ways becomes crucial. The huge computational burden brought by these pre-trained models will undoubtedly impose unnecessary burdens on transfer learning. PETL [17, 74] methods diverge from the conventional approach of fine-tuning the entire pre-trained model, instead only learning a few additional parameters for knowledge transfer. There are three common categories, prompt tuning [77, 30, 66, 46], adapter tuning [32, 10, 62] and parameter tuning [25, 31, 61]. We aim to relieve the computational burden by developing a graceful prompt framework specifically for cross-modal transfer, adapting frozen pre-trained multimodal models to downstream tasks across a broad distribution.

2.3 Tensor Decomposition

Tensor decomposition [35, 4, 59, 51] is an important research area studied for decades, which aims to approximate a tensor through a set of low-rank factors with diverse applications. In general, the

+

+

(a) Adapter

(b) LoRA

most widely used decompositions are Tucker decomposition [15] and CANDECOMP/PARAFAC (CP) decomposition [64], both of which can be seen as generalizations of the matrix singular value decomposition (SVD) [60]. CP decomposition can also be seen as a special Tucker decomposition whose core tensor is diagonal, but is more lightweight and explainable , meaning that a high-order tensor can be uniquely represented as the sum of a set of rank-one tensors theoretically. Here, we leverage the idea of CP decomposition to implement mode approximation for lightweight prompts.

3 Methodology

3.1 Background

Frozen Backbone. BLIP [41] is a unified VLP framework which has multimodal mixture of encoder-decoder(MED) architecture with both understanding and generation capabilities. We utilize BLIP-base as the frozen backbone and the pre-trained weights can be downloaded from Salesforce. Its visual encoder is ViT-B [19] and the text encoder is the same as BERT [16] while the text decoder replaces the self-attention layers with causal self-attention layers. It uses cross-attention layers to gather information from encoded visual representations using the textual representations as query. It's flexible to choose different components in the BLIP architecture to perform different multimodality downstream tasks.

We start with the core attention-based Transformer architecture mostly utilized in existing large-scale multimodal models [56, 44, 73, 41] for representation. The input image/text is divided into several non-overlapping patches, and then these patches appended a [CLS] token are fed into an embedding layer followed by the Transformer blocks with multi-head attention as the core operation. It copies the input embedding into query, key, and value by three projection matrices W l q , W l k , and W l v ∈ R d × d respectively, where l represents the l -th layer. The pre-trained model contains many dense layers which perform matrix multiplication as follows:

where m ∈ { I, T, C } denotes the vision, text, and cross-modal modality. For the unimodal vision/text modality branch, m are all from the vision/text modality. For the cross-modal modality branch, X l q,m is from the text modality, while the other two are from the visual modality. Assume there are L layers in total, we can stack all of the attention-based weight matrices in the multimodal pre-trained model, and derive the tensor W 0 = { W l q , W l k , W l v } L l =1 ∈ R d × d × N , where d is the dimension of the embedding token, and N denotes the total number of the weight matrices.

Parameter Update. For downstream tasks, directly updating the weight tensor with full fine-tuning will consume a huge amount of computation and brings about a heavy storage burden. We aim to update the knowledge with a few additional trainable parameters following the idea of PETL methods. Due to the redundancies of W 0 , we hope to implement the mode approximation of the tensor W 0 to get the new learnable weight tensor ∆ W for downstream knowledge transfer. The

Mode

Approximation

P

V

U

N

Vision Encoder

"[cls] A dog is running."

Multi-Head

Self-Attention

FFN

Multi-Head

Self-Attention

N

Text Encoder

Text Representation

Image/Video

Text/Question

FFN

"[cls] A dog is running."

FFN

Multi-head

Cross-Attention

Mode

Approximation

P

V

U

Mode

Approximation

Multi-Head

Self-Attention

U

V

P

Parameter Fusion

Mode

Approximation

P

U

V

Context Enhancement

+

+

Shared Parameters

Image Modality

Text Modality

Cross Modality

Query Transformation

Vision Representation

differences between our Aurora and existing PETL methods are demonstrated in Figure 1. Therefore, the backward propagation on the downstream training data D can be expressed as:

3.2 Lightweight Design for PETL

To ensure a successful parameter-efficient tuning framework, it is important to prioritize a lightweight design that enables easy deployment and scalability. Recent research [45] has shown that when self-attention-based architecture is pre-trained on large-scale datasets, there is a significant amount of feature redundancy due to the limited responses of individual classes in a nearly infinite feature space. In fact, there exists a low-dimensional reparameterization that is as effective for fine-tuning as the full parameter space [2]. This inherent dimensionality describes the minimum dimension required to solve the optimization problem it defines to a certain level of precision. Considering the pre-trained parameters as a tensor, we recognize that approximation can effectively preserve low-rank yet discriminative non-redundant features, and scale down the weight tensor of pre-trained large-scale models along certain directions, thus making them more suitable for downstream tasks. Therefore, we propose a novel mode approximation method named Aurora, which utilizes mode approximation to update the frozen W 0 . Specifically, we borrow the idea of CANDECOMP/PARAFAC (CP) decomposition to decompose the learnable parameters ∆ W into a series rank-one tensors to explore the inherent dimensionality embedded in the features. The detailed architecture is shown in Figure 2.

CP Decomposition. In the typical tensor decomposition method, a 3D tensor has three modes (further introduced in Appendix), each of which can be viewed as the reduced projection of the tensor in a specific dimension. Given the 3D updated weight tensor ∆ W ∈ R d × d × N , CP decomposition factorizes this tensor into a sum of R rank-one components in total, and each component can be formalized as the outer product of three decomposed vectors in the formulation of:

where u r ∈ R d , v r ∈ R d , and p r ∈ R N are decomposed vectors for the r -th component, while each vector belongs to the corresponding mode matrix, i.e., u r is the column vector in U =

[ u 1 , · · · , u r , · · · , u R ] . In addition, ◦ represents the outer product, λ r is the coefficient scalar of each component, and R is the rank of CP decomposition. For better understanding, each component contributes to the value of the tensor in the sum of scalar products:

where i, j, k denote the indices of the three modes.

Mode Approximation. In multimodal tasks, learning modality-specific representations and modality fusion representations are both important. Therefore, we aim to implement mode approximation to update the frozen attention-based weight tensor W 0 , including the self-attention module in the vision/text encoders, and the cross-attention module in the multimodal encoders, which are based on the pre-trained multimodal foundation model like BLIP [41]. We first approximate the attention-based weight in these modules by initializing three mode factors, i.e., U = [ u 1 , · · · , u R ] , V = [ v 1 , · · · , v R ] , and P = [ p 1 , · · · , p R ] . U, P are randomly initialized with Gaussian distribution, and V is with zeros, so that ∆ W = 0 before training. It should be noted that U, V are shared as global factors utilized for mode approximation, which means that our Aurora considers the crossmodal interaction and share the knowledge between these weight matrices in each modality. Besides, to further capture the discriminative features of each modality, we randomly initialize the learnable coefficient vector λ m ∈ R R for each weight matrix on the modality m respectively. With these three mode factors, we can implement the mode approximation in the forward propagation by the inverse progress of CP decomposition with input tensor X m as follows:

Analyzed from the perspective of prompt learning, our idea of approximating the pre-trained weight parameters W 0 with additional trainable parameters ∆ W can be essentially understood as the soft prompts, which learns on downstream data based on CP decomposition. Such prompts not only provide better guidance for downstream tasks, but also are very lightweight in design, which greatly facilitates the application of pre-trained models to many cross-modal tasks in the unified mechanism.

3.3 Modality Alignment Design

Unlike existing methods that directly insert learnable networks to explicitly achieve cross-modal alignment, we further propose two effective modules to align different modalities with few trainable parameters. Thus, with the addition of mode approximation above, we can achieve a graceful prompt framework for cross-modal transfer, which is both lightweight and high-performance.

Informative Context Enhancement. For better modality alignment, we aim to provide prompts that can activate the fusion features after the cross-attention module. Inspired by the development in In-Context Learning [52, 18], we realize that the demonstration template is important for the prompts. The most intuitive approach is to align image-text pairs to obtain more cross-modal contextual information. However, even with relevant image regions, there may still be more than one way to describe these regions. Some texts may accurately summarize the content of an image, whereas others may not. Without a priori for matched textual information, we determine to introduce the context enhancement module to provide coverage of possible textual information.

We adopt the image-grounded text branch from BLIP and design a specific demonstration template for cross-modal prompt tuning. Given the fusion features of the image-grounded text branch F = { f 1 , · · · , f |B| } ∈ R |B|× E and the textual query features of the self-attention module T = { t 1 , · · · , t |B| } ∈ R |B|× E , we utilize all of the query features with dimension E in a batch B as the context for enhancement. Specifically, we calculate the attention score α ∈ R |B|×|B| between feature f i and each textual query feature from t 1 to t |B| as:

In order to generate a more effective advanced fusion feature F ′ , all the adaptively weighted query features T within a batch are collected together with a specific fusion feature f i to form the demon-

stration template. This form can adaptively absorb context query information to derive a better enhanced fusion feature for the image-text matching loss L ITM as f ′ i = f i + ∑ |B| j =1 α ij t j .

Gated Query Transformation. Another reason why modality alignment is difficult is that the multimodality fusion branch network is deep, which can cause textual information to be lost during training. To address this issue, we propose a novel approach inspired by gating mechanisms to explicitly model the relative contributions of textual information during modality alignment. Specifically, instead of directly concatenating the fusion representation f (output of the Cross-Attention block) with the query representation t (output of the Self-Attention block) as residuals in existing methods [49], we learn a gated query function to balance the contributions of both modalities. Our gated query transformation involves two steps. First, we implement the transformation as follows: t ′ = σ ( γ ⊙ t ) + β , where γ and β are zero-initialized learnable transformation matrix and bias with an activation function σ . Second, we calculate the query gate g by computing the product of t ′ and f with Softmax. It should be noted that γ and β are zero initialized, so that t ′ is zero at the beginning of training. Hence, the query gate explicitly measures the contribution of the query representation in the formulation of g ⊙ f +( 1 -g ) ⊙ t ′ to update the fusion representation f .

4 Experiments

4.1 Experimental Settings

Datasets & Baselines. We evaluate Aurora on six benchmarks spanning four cross-modal tasks: image-text retrieval, question answering (QA), and video-text retrieval and QA. We compare Aurora with two types of approaches: full fine-tuning methods including SOTA for each task, and frozen backbone methods including LoRA [25] and UniAdapter [49]. See more details in Appendix.

Implementation Details. Our implementation is based on Salesforce's open-source codebase [41]. Following [49], we also apply BLIP-base [41] as our vision-language backbone for all the multimodal downstream tasks. We use PyTorch to implement all experiments on 8 × NVIDIA V100 GPU (32G). We use AdamW [48] optimizer with a weight decay of 0.05 and a learning rate of 1e-4 obtained from grid search for all experiments. Note that during the fine-tuning process, the parameters of the backbone model are kept frozen. More training details can be seen in Appendix.

4.2 Performance Comparisons on Cross-modal Tasks

Image-Text Retrieval. Table 1 shows performances for image-text retrieval tasks on MSCOCO [47] and FLICKR30K [54]. It can be observed that our Aurora (R=64) achieves comparable results with the state-of-the-art frozen backbone method while using only 0.5% of its parameters. When we

increase the rank to 128, Aurora can further boost the performance, surpassing all the frozen backbone methods, and even outperforming some full fine-tuning methods with fewer trainable parameters.

Video-Text Retrieval. To further verify the performance of our Aurora in the field of video-text retrieval, we conduct experiments on two video datasets, MSRVTT [69] and DiDemo [5], and the results are presented in Table 2. With only about 0.1M trainable parameters, our Aurora directly achieves better performances than all the frozen backbone methods, and we outperform most full fine-tuning methods. This indicates that our Aurora has an excellent understanding ability under video-text scenes, even with relatively few trainable parameters.

Visual Question Answering. In an effort to further explore the potential of our Aurora, we evaluate it on VQA/VideoQA tasks with VQAv2 [24]/MSRVTT-QA [68] datasets and demonstrate the evaluation results in Table 3. Unlike retrieval tasks, VQA task needs to verify the model's multimodal generative ability. We share the trainable parameters of the multimodal encoder and multimodal decoder to further reduce the parameter amount. From the results, we find that our Aurora outperforms UniAdapter and all the full fine-tuning methods, indicating that Aurora can have powerful transfer ability for downstream generative tasks.

4.3 Performance Comparisons on Zero-shot Setting

To evaluate the generalization ability of Aurora, we conduct experiments on cross-modal tasks with zero-shot setting, and make comparisons with the pretrained version, full fine-tuning method, LoRA, and UniAdapter.

(a)

(b)

(c)

(d)

FLICKR30K(I2T)

FLICKR30K(T21)

DiDemo

3.0

98.0

97.5

+

+

Recall@1

Recall@10

Recall@5

101.00

100.75

88.0

87.0

Recall@5

Recall10

Reeall@1

99.5

100.0

60.0

58.0

+

Recall@1

Recall@5

Recall@10

06

88

(M)

2.5

LoRA

#

Recal

97.0

96.5

100.25

100.50

100.00

Recal

eca

86.0

85.0

99.0

98.5

98.0

5/10

ecal

1

Recal

56.0

54.0

52.0

84

82

86

Recall@5/10

nable

Param

2.0

1.5

96.0

99.75

84.0

97.5

R

50.0

80

1.0

95.5

99.50

99.25

83.0

96.5

97.0

48.0

76

78

0.5

95.0

16

32

64

128

256

99.00

82.0

16

32

64

128

256

96.0

46.0

16

32

64

128

256

74

0.0

1

2

4

8

16

32

64

Rank R

Rank R

RankR

Rank R

Specifically, experiments are based on the pre-trained large-scale multimodal foundation models, i.e., BLIP, which is further tuned in each corresponding method on MSCOCO. Then, we utilize two video datasets as the zero-shot data for validation. Table 4 provides an overview of the performance of Aurora on various zero-shot multimodal tasks. It is obvious that Aurora achieves the highest zeroshot performance while requiring the least number of trainable parameters during vision-language pre-training, which represents more powerful general-purpose understanding ability.

4.4 Analysis of Different Designs

We thoroughly evaluate how the core operations in Aurora affect results, i.e., the rank R , parameter sharing, context enhancement, and the gated query transformation. We conduct experiments to analyze the impacts of different designs, which are implemented on several cross-modal tasks.

How Rank of CP Decomposition Affects Aurora?

Results on Flickr30K and DiDemo are shown in the left three columns in Figure 3. A fundamental conclusion is that as R increases, the dimensionality of the model representation also increases, leading to better performance on downstream tasks. However, when R reaches a certain range, the rate of increase slows down, which may be related to the redundancy of high-dimensional features.

In addition, we show in (d) of Figure 3 that as R increases, the growth rate of parameter size in Aurora is much slower than that of LoRA and UniAdapter. As another tensor decomposition method, our scalability and ease of deployment are much stronger than the baselines.

How Does Aurora Benefit from Informative Context Enhancement?

For an in-depth investigation, we introduce three ablated variants, i.e., Aurora w/o C removes the context enhancement, Aurora w. R replaces the context with random vectors, and Aurora w. M simply takes the average features of all queries. Ablation results are shown in Figure 4, where we separate the results on two y -axes respectively, which can better demonstrate the differences across different indicators more clearly. And the results of our Aurora are marked by the black upper lines. Surprisingly, using random vectors to supplement fused features still achieves better

Recall (I12T)

99.0

98.0

97.0

96.0

MSCOCO

92.0

90.0

88.0

Recall (T21)

?

Recall

54.0

53.5

53.0

52.5

52.0

DiDemo

87.0

86.0

85.0

84.0

81.0

83.0

82.0

01/S

Recall@5

94.0

93.0

95.0

86.0

84.0

51.5

51.0

50.5

80.0

76.0

79.0

78.0

77.0

92.0

R@5(12T)R@10(12T)R@5(T2I)R@10(T21)

82.0

50.0

R@1

RaS

Rg10

75.0

results than removing this module. Moreover, we observe that adaptive weight α causes better performance than the uniform weight. These phenomena fully demonstrate that with few parameters,

Woman stands in front of a large pot holding the

lid

and wears a yellow apron.

w/o sharing

A girl with orange hair and tattoos takes a picture of a painting with her

phone

.

w/o sharing

forVx

A person is looking at the computer on a desk with a phone

and a box.

A large black dog and a large yellow

dog appear to be playing with a soccer ball.

W - Layer 7

8.0

8.0

6.0

Pretrained

6.0

Pretrained

8.0

6.0

Pretrained

8.0

6.0

Full Fine-tuned

8.0

6.0

Full Fine-tuned

8.0

6.0

Full Fine-tuned

4.0

4.0

4.0

4,0

4,0

4.0

2.0

2.0

2.0

2.0

2.0

2.0

0.0

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.0

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.0

f'0-

0.2

0.1

0.0

0.1

0.2

0.3

0U0

0.3

0.2

1°0

0U9

0.1

0.2

0.3

00

0,3

0.2

0.1

0U9

0.1

0.2

0.3

0.0

0.2

0.1

0U0

0.1

2'0

W - Layer 12

supplementing more context information to fused features is crucial to promote modality alignment for downstream tasks, and our context enhancement is the optimal choice.

How Does Aurora Benefit from Parameter Sharing?

See Figure 5 for the effectiveness of parameter sharing. We observe that the parameter sharing strategy can achieve a better capability to locate the most semantic-related visual parts for specific words in the text. It should be noted that we are able to reduce 0.4M parameters through parameter sharing. Moreover, we also conduct quantitative experiments on FLICKR30K, in which parameter sharing results in better performance with an average improvement of 0.1. Therefore, we can draw the conclusion that parameter sharing during training not only decreases the parameter dependence but also boost the cross-modal interaction for better performances.

How Does Aurora Benefit from Gated Query Transformation?

To further validate the effectiveness of the Gated Query Transformation module, we conduct a comprehensive ablation study on MSCOCO and DiDemo datasets with two variants. The experimental results are shown in Figure 6, where Aurora w/o Q (blue bars) represents the removal of the Gated Query Transformation and Aurora w. O (red bars) initializes the transformation matrix and bias with ones. The gap between the blue bars and ours shows that this module helps promote the final performance to some extent, which also indicates that adaptively incorporat-

99.0

MSCOCO

92.0

54.0

DiDemo

87.0

Recall

98.0

97.0

96.0

90.0

88.0

Recall(T21)

Recall

53.5

53.0

52.5

52.0

81.0

0°98

85.0

84.0

83.0

82.0

95.0

86.0

R

51.5

80.0

93.0

94.0

84.0

51.0

50.5

78.0

77.0

76.0

106L

92.0

R@5(12T)R@10(12T)R@5(T21)Rg10(T2)

82.0

0°0S

R@1

Ra5

R@10

75.0

ing text information into the modality fusion branch has a positive effect on modality alignment under the deep network architecture. In addition, the difference between the red bars and ours shows that initializing from zeros is more conducive to adaptively controlling the proportion of text feature fusion, and can gradually achieve better modality alignment for better results.

4.5 Visualization Analysis

Parameter Distribution. We visualize the parameter distributions of the pre-trained model, fully fine-tuned model, and our Aurora in Figure 7. Aurora involves gradually embedding the knowledge learned from downstream tasks into the parameters of a pre-trained model, while amplifying the original parameters in a certain direction. From the left part, we can see that our Aurora only adjusts the pre-trained model parameters in a small local range, but it can have a better effect on downstream

tasks. Meanwhile, we can see from the right part that Aurora's parameter distribution is very close to that of the fully fine-tuned model, with only small changes in a small range. These small changes enable our method to achieve superior performance on many downstream tasks.

5 Conclusion

This paper proposes Aurora, a graceful prompt framework for cross-modal transfer. We first leverage mode approximation to implement multimodal prompt tuning, which explores the low intrinsic dimension with only 0.04% parameters of the pretrained model. Then, to better reduce the modality gap, we propose Informative Context Enhancement and Gated Query Transformation modules under extremely low parameter scenarios. Extensive evaluation of Aurora on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even surpasses full fine-tuning approach.

Limitations. The representation ability of our Aurora to some extent depends on the setting of the rank R . However, it also unavoidably increases the parameter size and thus the computational cost. Therefore, it is crucial for us to select a proper rank in the trade-off between parameter size and performance. Unfortunately, it is challenging for us to determine a suitable rank beforehand with the variations of downstream tasks and data sizes, which poses a challenge to the application of our model. We will further optimize this issue in future work.

Broader Impact. Multimodal pre-trained large models have wide applications in the real world, including image-text understanding, retrieval, question answering, and generation. As models scale up, they will achieve more amazing performances. Our Aurora enables lightweight transfer of large-scale models, which allows us to better utilize cloud-based large models and guide them to produce high-performance edge applications under constrained computational resources.

References

RZdunek ACichocki et al. Nonnegative matrix and tensor factorizations: Applications to exploratory multi-way data analysis and blind source separation, 2009.

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 , 2020.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of machine learning research , 15:2773-2832, 2014.

Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision , pages 5803-5812, 2017.

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1728-1738, 2021.

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. A clip-hitchhiker's guide to long video retrieval. arXiv preprint arXiv:2205.08508 , 2022.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13535 , 2022.

Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: Vision-audio-language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345 , 2023.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 , 2022.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX , pages 104-120. Springer, 2020.

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning , pages 1931-1942. PMLR, 2021.

Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications , 21(4):1253-1278, 2000.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence , pages 1-16, 2023.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234 , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems , 26, 2013.

Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681 , 2021.

Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems , 33:6616-6628, 2020.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. Expansionnet v2: Block static expansion in fast end to end training for image captioning. arXiv preprint arXiv:2208.06551 , 2022.

Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12976-12985, 2021.

Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. Wenlan: Bridging vision and language by large-scale multi-modal pre-training. arXiv preprint arXiv:2103.06561 , 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII , pages 709-727. Springer, 2022.

Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. arXiv preprint arXiv:2212.03145 , 2022.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems , 34:1022-1035, 2021.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint arXiv:2210.03117 , 2022.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , pages 5583-5594. PMLR, 2021.

Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review , 51(3):455-500, 2009.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123:32-73, 2017.

Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7331-7341, 2021.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint arXiv:2205.12005 , 2022.

Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video-and-language pre-training with entity prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4953-4963, 2022.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16 , pages 121-137. Springer, 2020.

Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 , 2021.

Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653 , 2023.

Junfan Lin, Jianlong Chang, Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang-wen Chen. Being comes from not-being: Open-vocabulary text-to-motion generation with wordless training. arXiv preprint arXiv:2210.15929 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Haoyu Lu, Mingyu Ding, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Masayoshi Tomizuka, and Wei Zhan. Uniadapter: Unified parameter-efficient transfer learning for cross-modal modeling. arXiv preprint arXiv:2302.06605 , 2023.

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing , 508:293-304, 2022.

Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. Advances in neural information processing systems , 32, 2019.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pages 2641-2649, 2015.

Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, et al. Exploring low-dimensional intrinsic task subspace via prompt tuning. arXiv preprint arXiv:2110.07867 , 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Fangxun Shu, Biaolong Chen, Yue Liao, Shuwen Xiao, Wenyu Sun, Xiaobo Li, Yousong Zhu, Jinqiao Wang, and Si Liu. Masked contrastive pre-training for efficient video-text retrieval. arXiv preprint arXiv:2212.00986 , 2022.

Nicholas D Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E Papalexakis, and Christos Faloutsos. Tensor decomposition for signal processing and machine learning. IEEE Transactions on Signal Processing , 65(13):3551-3582, 2017.

Gilbert W Stewart. On the early history of the singular value decomposition. SIAM review , 35(4):551-566, 1993.

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. arXiv preprint arXiv:2206.06522 , 2022.

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5227-5237, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika , 31(3):279-311, 1966.

Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. arXiv preprint arXiv:2203.07303 , 2022.

Haixin Wang, Jianlong Chang, Xiao Luo, Jinan Sun, Zhouchen Lin, and Qi Tian. Lion: Implicit vision prompt tuning. arXiv preprint arXiv:2303.09992 , 2023.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia , pages 1645-1653, 2017.

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5288-5296, 2016.

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1686-1697, 2021.

Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. Diffusion-based scene graph to image generation with masked contrastive pre-training. arXiv preprint arXiv:2211.11138 , 2022.

Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys , 2022.

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 , 2021.

Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al. Visual tuning. arXiv preprint arXiv:2305.06061 , 2023.

Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems , 34:23634-23651, 2021.

Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems , 35:36067-36080, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

APPENDIX

To provide a comprehensive demonstration of our approach, we will supplement additional details. The arrangement of these sections is as follows: First, we demonstrate the core concepts of our Aurora for clarity. Second, we comprehensively make overall comparisons with existing methods on various multimodal tasks. Then, we provide details regarding the datasets and baselines in Section C, while the concrete training details are outlined in Section D. We then conduct a comprehensive analysis of the computational costs, including time and memory consumption, along with algorithmic complexity in Section E. Furthermore, we provide theoretical support for our approach in Section F. Finally, we present visualizations of our proposed Aurora for several cross-modal tasks to facilitate qualitative comparisons in Section G. To represent our method clearly and concisely, we use lowercase letters for scalars, bold lowercase letters for vectors , italicized uppercase letters for MATRICES , and bold italicized uppercase letters for TENSORS in the equations, respectively.

A Concept Definition

According to [1, 51], we will offer a precise definition of the fundamental notions underpinning our key Mode Approximation component.

First, the definition of tensors can be demonstrated as follows:

Definition 1 (Tensor). Let D 1 , D 2 , · · · , D N ∈ N denote index upper bounds, a tensor W ∈ R D 1 ×···×D N of order N is an N -way array where elements W d 1 ,d 2 , ··· ,d n are indexed by d n ∈ { 1 , 2 , · · · , D n } , for 1 ≤ n ≤ N.

Then, the concept of the mode is formulated as follows:

Definition 2 (Mode). Let W ∈ R n 1 × n 2 ×···× n d be a d -dimensional tensor. The modek matricization of W , denoted as W ( k ) ∈ R n k × ( n 1 ··· n k -1 n k +1 ··· n d ) , is obtained by unfolding the tensor along its k -th mode and arranging the entries as rows in a matrix.

Given the mathematical definition of the mode, we can implement decomposition in the context of CP decomposition. We stack all the weight matrices in the attention layer (i.e., W q , W k , W v ) of all the branches into a tensor, which needs to be updated as ∆ W . Since our method assumes that the stack of weight matrices is a three-order tensor, k is three in our settings, and thus the CP decomposition can be illustrated as follows:

where R is the rank, λ r are non-negative scalar weights, and u r ∈ R n 1 , v r ∈ R n 2 , p r ∈ R n 3 are non-zero factor vectors. And the modek unfolding of the tensor ∆ W is U, V, P respectively.

Our proposed Aurora aims to approximate the latent mode matrix with randomly initialized learnable parameters, which can learn knowledge on downstream tasks in a lightweight manner.

B Overall Comparison

We compare all the baselines for three downstream tasks and presented a comprehensive illustration in Figure 8. The arrow in the figure points towards better performance on dual metrics, as it moves towards the upper right corner. We rank the size of the model parameters and use it as a basis for determining the size of the bubbles, which are also displayed in Figure 8. It is evident that our method performs remarkably well even with smaller parameter sizes, and in several instances, outperforms the fully fine-tuned approach, demonstrating the strength of our architecture.

C More Details for the Baselines & Datasets.

Baselines. For Frozen Backbone methods, UniAdapter [49] is currently the state-of-the-art method for parameter-efficient transfer learning in the field of multimodality and can be considered as a

MSCOCO

VQAv2

Video-TextRetrieval

81.0

79.0

75.0

x

T

↑

80.0

79.0

10.6M

18.8M

D.2M

223M

est-dev

78.0

77.0

76.0

330M

337M

0.2M

E

>

↑

67.5

72.5

70.0

65.0

10.6M

18.82

77.0

78.0

330M

820M

210M

LoRA

BLIP

ALIGN

UniAdapter

ALBEF

OSCAR

73.0

75.0

74.0

155M

330M

266M

18.8M

BLIP

UniAdapter

SOHO

UNITER

ALBEF

OSCAR

Demo

62.5

57.5

55.0

60.0

245M

306M

110M

124M

LoRA

UniAdapter

All-in-One

CLIP4CLIP

ALPRO

VIOLET

76.0

72.0

86.0

87.0

88.0

89.0

90.0

91.0

92.0

93.0

73.0

74.0

75.0

76.0

77.0

78.0

55.0

60.0

65.0

70.0

Text→Image

Test-std

MSRVTT(T→V)

representative of the Adapter class of methods in the multimodal domain. LoRA [25] is another important branch of parameter-efficient transfer learning methods. Its core idea is to use low-rank decomposed matrices to calculate the incremental change of model parameters during adaptation on downstream tasks. To enable comparison with a wider range of baselines, we replicate many of the settings from prior works and reuse their experiment results whenever possible. It should be noted that this means some baselines only appear in specific experiments.

As for Full Fine Tuning methods, we apply UNITER [13], VILLA [23], OSCAR [43], ALIGN [29], ALBEF [42] and BLIP [41] for image-text retrieval tasks, then we use ClipBERT [37], Frozen in Time [6], ALPRO [40], VIOLET [22], All-in-one [65], CLIP4Clip [50] and CLIP-Hhiker [7] for text-video retrieval tasks, finally we adopt ClipBERT[37], ALPRO [40], Just-Ask [70], VIOLET [22], MERLOT [75], All-in-one [65] for VideoQA task while adopt VL-T5/BART [14], SOHO [27], OSCAR [43], UNITER [13], ALBEF [42] and BLIP [41] for VQA tasks.

Datasets. We provide a comprehensive introduction to the datasets of various downstream tasks in the multimodal scenario, as outlined below:

MSCOCO [47] is a large scale image-text dataset and each image is annotated with five captions. Following [49, 34], we use Karpathy split of MSCOCO: 5,000 images for testing, 5,000 images for validation, and the rest for training.

Flickr30K [54] contains 31,000 images collected from Flickr. Each image is usually referenced with five human annotations. Following previous works [49, 21], we use 1,000 images for testing, another 1000 for validation, and the rest for training.

MSR-VTT [69] contains 10,000 video clips and each video clip is annotated with 20 English sentences. Following recent works [49, 50], we adopt 1k-test split for training and testing.

DiDemo [5] is one of the most commonly used datasets for the temporal localization of events in videos. It contains about 10,000 videos and 40,000 annotations. we follow [49, 6] to concatenate all descriptions corresponding to the same video into a single sentence to conduct actually paragraphto-video retrieval task.

VQAv2 [24] is one of the most famous visual question answering datasets which contains 83k/41k/81k images for training/validation/testing. Following [49, 42, 41], we use both training and validation splits of VQAv2 and additional training samples from Visual Genome [36] for training. The results should be evaluated by the official server and we report the results on the test-dev and test-std splits.

MSRVTT-QA [68] is one of the most popular video question answering datasets. It's constructed based on MSRVTT and has 243k open-ended questions associated with 10k videos. We follow [49, 37] to employ the standard split for training and testing.

D More Training Details

D.1 Frozen Backbone

Another important design in BLIP is CapFlit, It contains a Captioner to generate captions given websearched images and a Filter to remove noisy image-text pairs, both Captioner and Filter are finetuned

Image

Encoder

Image/Video

Visual Encoder

Mode

Approximation

"[Decode]"

Multimodal Decoder

Answer

Decoder

" Answer "

Query

Transformation

Question

Encoder

"[Encode] + Q"

Multimodal Encoder

+ Mode

Approximation

individually on the COCO dataset while using different objective loss. In addition, BLIP uses momentum technology to further improve the correctness of the image-text matching relationship.

D.2 Architecture for VQA Tasks

Figure 9 shows the architecture of Aurora for Visual Question Answering tasks. Compared to Retrieval tasks, the VQA architecture has an additional answer decoder. During fine-tuning, the images/videos are first encoded by a single-modal visual encoder, and then the image/video-text pairs are fused using a multimodal encoder and given to the decoder for prediction. Answers are used as ground truth and Language-Model Loss is utilized for parameter updating throughout the entire training process. As the ITM Loss is no longer needed, we remove the Informative Context Enhancement module from the VQA architecture. Meanwhile, we retain the Gated Query Transformation module to preserve the complete semantic information of the question representations as much as possible. Finally, in order to further reduce the number of parameters, we share the learnable parameters of the multimodal encoder and the multimodal decoder.

D.3 Implementation Details

In this section, we give more training details about our Aurora.

For image-text downstream tasks, we set the image size into 384 × 384. We use cosine decay to update the learning rate during training. We set the batch size to 16 for each GPU and train a total of 6 epochs.

For video-text retrieval tasks, we randomly sample T = 8(16) frames for each video during training(testing) while setting the frame size to be 224 × 224. We use cosine decay to update the learning rate, we set the batch size to 8 per GPU during training and train for a total of 5 epochs.

For VideoQA task, we also sample 8 frames per video for training while the frame size is changed to 384 × 384. While training, we set the batch size to 4 per GPU and train 10 epochs in total. During the evaluation, we randomly sample 16 frames for each video, and we use greedy search to generate the next token when producing answer for its corresponding question.

For VQA task, we set the image size to 480 × 480 for training/inference and adopt a batch size of 16 for each GPU. We also adopt cosine decay to change the value of learning rate at different epochs and we train 10 epochs.

We also perform a simple cleaning of the text data and truncate all words beyond the maximum length of the sentence. All data processing and partitioning are consistent with UniAdapter and LoRA to ensure fair comparison. When implementing CP decomposition, textual encoder, visual encoder, and multimodal encoder share the same global mode factor U and factor V to do parameter sharing and we initialize the weights of factor V to be zero.

E Cost Analysis

In Table 5, following [49], we report the training time and GPU memory cost for both retrieval and VQA tasks. We regard the training time and memory cost of UniAdapter(r=512) as one unit. Since we adopt the same backbone models, the forward propagation process of the two methods, Aurora and UniAdapter, is almost consistent, and the time cost is similar. However, our Aurora has fewer trainable parameters, resulting in a slightly smaller GPU memory footprint.

Then, We will give a theoretical analysis of the parametric complexity of the three PETL methods. Assume that the frozen backbone's visual, textual, and multimodal encoder all have L transformer layers while multimodal encoders contain cross-attention modules and visual, textual encoders contain self-attention modules. We only approximate the Query/Key/Value weight matrix in these attention-based modules. Let d donates the dimension of the hidden feature and r for rank, so the parametric complexity of the LoRA is L × (3 + 6) × 2dr ∼O (Ldr), the complexity of the UniAdapter is L × 4dr ∼O (Ldr), and the complexity of our Aurora is L × (3 + 6) × 2r + 2dr + 2Ld ∼O ((L+d)r), normally r and L are much smaller than d, so from the above analysis we can draw the conclusion that when r increases, our Aurora can achieve the lowest parameter cost.

F Theoretical Analysis

Write ∆ W = ∑ R r =1 λ r ( u r ◦ v r ◦ p r ) , and the ( i, j, k ) -element of ∆ W is:

Recall that the Frobenius norm of a tensor X ∈ R d × d × N is given by:

Hence, it suffices to analyze the convergence rate of our parameter tensors in the Euclidean space R ddN , where ddN denotes the product of d × d × N in order to distinguishing from the space of multi-dimensional arrays of size d × d × N .

Assumption F.1 We identify R d × d × N with R ddN , and let the loss function L be defined on R ddN , while we still write L ( W ) where W ∈ R d × d × N is a tensor. We will also use Frobenius norm and ℓ 2 norm interchangeably, which means that:

Assumption F.2 We assume that the loss function L : R ddN → R has the following property:

L is injective.

L is strongly convex: there exist m and M such that:

That is, ∇ 2 L ( X ) -mI is positive semidefinite and ∇ 2 L ( X ) -MI is negative semidefinite.

Let { λ (0) r , u (0) r , v (0) r , p (0) r : r = 1 , · · · , R } be the randomly initialized vectors used for tensor decomposition, where λ (0) r ∈ R , u (0) r ∈ R d , v (0) r ∈ R d , p (0) r ∈ R N . Denote

Let ∆ W ( n ) be the parameter tensor returned by the n th training epoch, that is,

Theorem F.1 Under the above assumptions, and suppose that we train for n epochs with η ≤ 1 /M using gradient descent. Let W ∗ be the optimal parameter tensor, then,

Moreover, W ∗ is unique.

Proof F.1 The proof follows from [8]. For notational convenience let X ( n ) = W 0 +∆ W ( n ) , and denote the optimal value L ( W ∗ ) by λ ∗ . We will begin by analyzing the convergence using arbitrary X , Y ∈ R ddN , and then plug in our parameter tensors. By Taylor's theorem we can write:

where Z lies in the line segment joining X and Y . By the strong convexity assumption, we have,

Hence

Now we use ∥∇L ( X ) ∥ 2 to bound L ( X ) -λ ∗ . The right-hand side of (16) is a convex quadratic function of Y , hence Y ∗ = X 1 /m ∇L ( X ) is the minimizer, thus,

Since Y is arbitrary, plugging X = X ( n ) , we have

By the assumption ∇ 2 L ( X ) ⪯ MI we have

Plugging in Y = X ( n ) -η ∇L ( X ( n ) ) yields

Now we minimize over η on both sides of (22), and denote the optimal value by ˜ L ( η ∗ ) . The right-hand side of (22) is simple quadratic, hence it is minimized by η = 1 /M , and

W-Layer 1

W-Layer7

Wk-Layer 12

8.0

Pretrained

8.0

Pretrained

8.0

Pretrained

6.0

6.0

6.0

4.0

4.0

4.0

2.0

2.0

2.0

0.0

0.0

0.0

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.3

0.2

0.1

0.0

0.1

0.2

0.3

W-Layer1

15.0

W-Layer7

W-Layer 12

15.0

Pretrained

Pretrained

Pretrained

12.5

15.0

12.5

10.0

10.0

10.0

7.5

7.5

5.0

5.0

5.0

2.5

2.5

0.0

0.0

0.0

0.2

0.1

0.0

0.1

0.2

0.2

0.1

0.0

0.1

0.2

0.10

S0'0-

0.00

0.05

0.10

Now,

Subtracting λ ∗ on both sides, we have,

by (20) we have,

By mathematical induction, we obtain,

therefore L ( X ( n ) ) → λ ∗ as n →∞ .

Suppose V ∗ is another optimal parameter tensor, then by the same argument we have L ( X ( n ) ) → L ( V ∗ ) as n →∞ . Since R ddN is a Hausdorff space, L ( V ∗ ) = L ( W ∗ ) . By our assumption that L is injective, W ∗ = V ∗ .

G Visualization Analysis

G.1 Parameter Distribution

Figure 10 and Figure 11 show more details on the parameter distribution comparisons between our Aurora and the pre-trained model and the full fine-tuned model, in which similar results can be observed on W k and W v . We can see that the mode approximation parameters adjust the original weights, and change the distribution of weights and biases to fit the downstream task. It can be concluded that Aurora has several advantages over traditional fine-tuning approaches. First, it avoids over-fitting to specific downstream tasks by only adjusting the pre-trained model parameters in a small local range. Second, it reduces the amount of training required on new data, making it more efficient and cost-effective. Last but not least, Aurora can further improve the model's performance on downstream tasks.

W-Layer1

W-Layer7

W-Layer 12

10.0

10.0

10.0

Full Fine-tuned

Full Fine-tuned

Full Fine-tuned

8.0

8.0

8.0

6.0

6.0

6.0

4.0

4.0

4.0

2.0

2.0

2.0

0.0

0.0

0.0

0.2

0.1

0.0

0.1

0.2

0.2

0.1

0.0

0.1

0.2

0.2

0.1

0.0

0.1

0.2

W-Layer 1

W-Layer7

20.0

W-Layer 12

15.0

Full Fine-tuned

15.0

Full Fine-tuned

Full Fine-tuned

12.5

12.5

15.0

10.0

10.0

7.5

7.5

10.0

5.0

5.0

5.0

2.5

2.5

0.0

0.0

0.0

0.10

S0°0-

0.00

0.05

0.10

0.10

S0'0-

0.00

0.05

0.10

0.10

0.05

0.00

0.05

0.10

Caption7462

：

He drew a beautiful picture

Caption9623

：

A man is on a cell phone while people are fighting

Caption9451

：

Kids feeding and playing with the horse

Caption7756

：

A group of women are singing

Caption8662

：

A man a woman cooking on a cooking show

G.2 Case Study

Visual-Text Retrieval. Figure 12 demonstrates some actual examples of Aurora performing textto-video task on MSRVTT test set. In conclusion, the results presented highlight the exceptional performance of Aurora in searching relevant videos from textual descriptions. The accuracy and realism of the returned videos demonstrate the effectiveness of our proposed method in understanding the relationship between text and visual content.

Visual Question Answering. Figure 13 gives some question-answering examples of Aurora and UniAdapter on the MSRVTT-QA dataset. Specifically, our method is able to reason about the meaning of the text and video information to answer the questions more accurately than UniAdapter. This is an important result because the ability to reason about the meaning of both text and visual information is essential for understanding multimodal data.

Query191201

：

What is a guy doing?

Query191306

：

What is a man standing in talking?

Query191382

：

What does a man fold?

Query191949

：

How many men play in a tennis match?

Ground-Truth: ask       Aurora: ask       UniAdapter: talk

Ground-Truth: kitchen       Aurora: kitchen       UniAdapter: kitchen

Ground-Truth: paper       Aurora: paper       UniAdapter: paper

Query192400

：

Ground-Truth: two      Aurora: two      UniAdapter: two

Ground-Truth: crowd      Aurora: crowd      UniAdapter: people

HANSY

JOPDANHAMSY

Celebrity.Ch

1.18

ROLEX

fame

nfame

qfame

Overall, the qualitative results shown in Figure 12 and Figure 13 demonstrate the effectiveness of our proposed method in both multimodal retrieval and question-answering tasks. We believe that our approach has the potential to be used in many multimodal applications, where understanding and analyzing multimedia data is essential.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to address the challenge of high learning costs when migrating large models to specific downstream tasks. The proposed method aims to reduce task complexity and improve the consistency across the different modal outputs of multimodal models.\n\nThe authors employ a LoRA-like technique that involves fine-tuning the transformer by appending adjustment matrices to the qkv matrices. This process is further optimized through CP Decomposition, an approach used to reduce the scale of the fine-tuning parameters. Then use Informative Context Enhancement mechanism to compute weights and the mixed graph-text features are adjusted according to these weights, facilitating the generation of superior fused features. To prevent the loss of textual information during the alignment of deep multimodal networks, the authors propose using a Gated Query Transformation. This approach calculates the blending ratio of text features for enhancement.\n\nThe paper claims that these techniques allow the model to outperform the current best methods on several downstream tasks, using fewer fine-tuning parameters, and even surpasses fully fine-tuned methods.\n\nSTRENGTHS:\n（1） Innovative and Effective Approach: Building upon the foundations of similar works like LoRA, this paper makes significant strides by achieving better fine-tuning performance with lower parameter usage. \n（2） Proposes Novel Techniques: The paper introduces innovative methods such as Informative Context Enhancement and Gated Query Transformation to enhance the fusion of modalities. These methods appear to be highly effective in improving model performance, and could potentially be applied in a range of different contexts.\n（3） High-Quality Visualizations: The paper includes aesthetically pleasing and informative figures and tables, which contribute to the clarity and overall quality of the work. The visualizations effectively aid in the understanding of complex concepts and methodologies.\n\nWEAKNESSES:\n(1)  Not Clearly Written: There appears to be some confusion in the terminology used, especially around 'soft prompts'. While the paper asserts that its fine-tuning approach can be seen as a 'soft prompt', the proposed method is more akin to parameter-efficient transfer learning. Therefore, the use of the term 'prompt' may not be accurate in this context. A clearer and more precise usage of technical terminology would be beneficial for readers and would also strengthen the overall quality of the paper. (Now author states that in the final version, they will replace the inappropriate term \"prompt\" with \"multimodal parameter-efficient transfer learning based on mode approximations\" to better reflect the essence of our method.)\n(2) Lack of Details on Gated Query Transformation: The paper does not provide a clear and thorough explanation of the implementation details for the Gated Query Transformation. For future improvements, it would be beneficial to include more technical details of this novel technique, which would improve the clarity of the paper and make it easier for others in the field to replicate and build upon this work. (Now author states the final version will be more detailed)\n(3) Manual Parameter Tuning: The paper suggests that the rank hyperparameter needs to be manually adjusted, which could pose an obstacle to scalability and efficiency. More research is required on how to optimally choose this hyperparameter's value. (Author said they use a detailed grid search experiments to get good rank)\n\nQUESTIONS:\n(1) Justification for Terminology: Could you clarify why you choose to refer to your method as a 'multimodal prompt'? Given the technique's resemblance to low-rank fine-tuning, the terminology might be misleading. Understanding the rationale behind this terminology would greatly aid in interpreting your work.\n(2) Clarification on Gated Query Transformation: The paper could benefit from further details on the application of the Gated Query Transformation. Specifically, is the text information fused with the image features before entering the cross-attention, according to the gate values? This part of the methodology was not entirely clear in the paper, and further explanation would enhance the clarity of your methods and potentially support the broader understanding and application of your techniques.\n\nLIMITATIONS:\n(1) Hyperparameter Selection: The need for manual selection of the rank hyperparameter is a key limitation of the proposed method. While it is acknowledged by the authors, there is no substantial discussion on how this limitation can be addressed. \n(2) Training Time: Another potential limitation that should be acknowledged is the impact on training time. Although the proposed method reduces the parameter count for downstream tasks, it doesn't seem to significantly decrease training time.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to design a lightweight prompt tuning method (i.e. Aurora) for cross-modal transfer. The main idea follows the observation by LoRA [15] that most of the features are redundant and a low-rank ∆W can be learned to adapt the features. Different from LoRA, they adopt CP decomposition [47] to decompose the learnable parameters into a series rank-one tensors. They also propose Informative Context Enhancement and Gated Query Transformation for better modality alignment. However, the connection between the two parts is not clear. Experiments show that Aurora performs better than LoRA and is at least comparable to full finetuning methods.\n\nSTRENGTHS:\nStrength \n1.\tFigure 1 highlights the difference between the proposed method and the baselines.\n2.\tThe proposed method is lightweight and effective.\n3.\tAurora is applicable to both image-text and video-text retrieval.\n\nWEAKNESSES:\nMajor\n1.\tThe main idea of the paper is lightweight adaptation achieved by adopting CP decomposition [47] for the adapter weights. While it is shown to be effective and parameter-efficient, there is limited technical novelty introduced by this work. In terms of story-level novelty, it mainly follows the observation and approach in LoRA [15], and it’s not considered novel.\n\n2.\tThe improvement of Aurora over UniAdapter [32] is marginal in Table 1.\n\n3.\tWhat are BLIP, BLIP+LoRA zero-shot performance in Table 4?\n\n4.\tThe new thing of this paper is Informative Context Enhancement and Gated Query Transformation. However, there is loose connection between the main idea and these two modules. These two modules are orthogonal to the low-rank approximation part. What will be the performance if LoRA is integrated with these two modules?\n\n5.\tWhich part of the method does Parameter Sharing in L248-L255 correspond to?\n\nMinor\n6.\tWhat are x-axis and y-axis in Figure 7?\n\n7.\tThe main flow of the architecture is not clear. In section 3.3, Gated Query Transformation is presented after Context Enhancement. However, Gated Query Transformation manipulates f, but Context Enhancement depends on f. Is the former performed prior to the latter?\n\n8.\tThe insight for Gated Query Transformation is unclear. How about replacing t’ in L191 with t?\n\nQUESTIONS:\nThe author is suggested to address the concerns in the weakness section.\n\nLIMITATIONS:\nThe limitation is mentioned at the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parameter efficient adaptation technique Aurora for multi-modal models. Particularly, the proposed method motivates their design by suggesting that the original pre-trained weight matrices have redundancies due to their high dimensional nature and the downstream task often requires low-dimensional reparameterization only. Aurora supplements the original weight matrices with series-one rank tensors which are only learned during the fine-tuning process. \n\nIn addition, to enhance the modality alignment between the vision and text representations, Aurora utilizes informative context enhancement module and gated query transformation which fuses and explicitly relates the textual representations with the multi-modal fusion representations in the cross-attention block of BLIP. \n\nExtensive experiments over various benchmarks shows the effectiveness of Aurora in comparison with fine-tuning and parameter efficient adaptation approaches.\n\nSTRENGTHS:\nStrengths:\n\n(1) The idea of decomposing learnable pre-trained matrices into small rank tensors is encouraging, as it explicitly allows to adapt only necessary amount of parameters for efficient and effective adaptation.\n\n(2) The paper has performed extensive evaluations with proper ablation studies which justifies their design choices.\n\n(3) The proposed methods performs favorably well with very less number of learnable parameters.\n\nWEAKNESSES:\nWeaknesses:\n\n(1) The overall paper presentation style is very confusing, specially the main methodology section. There is no preliminaries on the baseline architecture on which the proposed solution has been built. It is very difficult for the readers to grasp the contents without knowing the main model architecture. For example, in line 161-162, the authors have mentioned the cross-attention module, but unfortunately no prior information about that block is provided anywhere in the manuscript. Also the writing is not clear and I found it difficult to understand the manuscript. \n\n(2) How the proposed solution is considered as a prompt learning variant? If I understood correctly, the additional learnable parameters are utilized as part of the model and the input tensor has to be multiplied with it. It is not the case where the learnable parameters are part of the inputs, which is the core definition of prompt learning.\n\n(3) The proposed multi-modal alignment module seems to be heavily designed for the BLIP multi-modal model. It is not clear if these components could be utilized in other multi-modal models. It will be good to see the generalization of proposed approach to other recent VL models.\n\nQUESTIONS:\nPlease refer to the weaknesses section.\n\nLIMITATIONS:\nThe authors have discussed the limitations and societal impacts are highlighted.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper addresses the problems of (i) transfer learning and (ii) reducing the multimodality gap in multimodal models.

To address (i) it presents a technique which can be viewed as a generalization of LoRA; instead of independently factoring representing matrices as a low rank representation, all the matrices of the transformer get stacked together and are represented as a low rank tensor. 

To address (ii) two techniques are presented: one which aims to improve the representations by allowing information exchange between different examples in the batch (Informative Context Enhancement) and another which aims to prevent the loss of text information for deep models (Gated Query Transformation). 

Ablation studies are performed to justify the different design choices.

STRENGTHS:
1. Mode approximation (Generalizing LoRA by stacking the matrices of all the transformer layers and using CP decomposition to represent that) is a nice idea and experimentally shows to be a parameter efficient way (beats LoRA) of adapting frozen models to new domains

2. Thorough ablation studies are performed to demonstrate the impact and justify existence of each presented component in the final model.

WEAKNESSES:
1. Informative Context Enhancement seems to allow information exchange between different examples in the batch. That makes it dependent on the batch size, but the impact of changing it is not evaluated.

QUESTIONS:
1. Why are some methods present in ‘Methods with frozen backbone’ in Table 2 omitted from the same section in Table 3? (e.g. LoRA)
2. Is |B| in the ‘Informative Context Enhancement’ section referring to the number of tokens in the entire batch?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes AURORA, a method which uses mode approximation to boost the knowledge transfer in Vision Language models and enhances alignment between the modalities in a lightweight parameter efficient manner. In addition to this, the paper further proposes Context Enhancement module and  Gated Query Transformation module to boost the modality fusion in an adaptively controllable way. The proposed method is evaluated on six cross-modal tasks and two zero-shot tasks and is compared to existing PETL methods.\n\nSTRENGTHS:\n- The paper seems to be the first which has used mode approximation in Vision-Language models to efficiently achieve modality fusion.\n- The proposed method is novel with respect to existing prompt tuning methods.\n- The proposed methods works by tuning a very small number of parameters which can save time and computational resources.\n\nWEAKNESSES:\n- The motivation for using Mode approximation is not very clear in the paper. For instance there could be other methods which can solve the redundancies in the attention weights. There is no detailed explanation of the theoretical basis or formal analysis of how mode approximation aids in prompt learning. Adding a brief theoretical background could provide more insights into the method's underlying principles.\n- The authors haven't provided any solid proof behind the argument regarding loss of textual information in multi-modality fusion branches, which essentially forms the basis of introducing Gated Query Transformation.\n- The authors should have compared their method with previous prompt learning methods [1, 2] for zero-shot out-of-distribution classification tasks.\n\n\n[1] M. U. khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, “Maple: Multi-modal prompt learning”. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023\n\n[2] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models”. International Journal of Computer Vision (IJCV), 2022\n\nQUESTIONS:\n- Can the method be applied to a broader range of vision-language tasks, and how well does it adapt to different modalities or data distributions?\n\nLIMITATIONS:\n- Please have a look at Questions and Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}