SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- For image-text downstream tasks, we set the image size into  $384 \times 384$ . We use cosine decay to update the learning rate during training. We set the batch size to 16 for each GPU and train a total of 6 epochs.
- For video-text retrieval tasks, we randomly sample  $\mathrm{T} = 8(16)$  frames for each video during training(testing) while setting the frame size to be  $224 \times 224$ . We use cosine decay to update the learning rate, we set the batch size to 8 per GPU during training and train for a total of 5 epochs.
- For VideoQA task, we also sample 8 frames per video for training while the frame size is changed to  $384 \times 384$ . While training, we set the batch size to 4 per GPU and train 10 epochs in total. During the evaluation, we randomly sample 16 frames for each video, and we use greedy search to generate the next token when producing answer for its corresponding question.
- For VQA task, we set the image size to  $480 \times 480$  for training/inference and adopt a batch size of 16 for each GPU. We also adopt cosine decay to change the value of learning rate at different epochs and we train 10 epochs.
We also perform a simple cleaning of the text data and truncate all words beyond the maximum length of the sentence. All data processing and partitioning are consistent with UniAdapter and LoRA to ensure fair comparison. When implementing CP decomposition, textual encoder, visual encoder, and multimodal encoder share the same global mode factor  $U$  and factor  $V$  to do parameter sharing and we initialize the weights of factor  $V$  to be zero.
Table 5: Training time and GPU memory comparison.
Method	#Tunable	MSCOCO	FLICKR30K	MSRVTT-QA	VQAv2	DiDemo	MSRVTT
Time	Memory	Time	Memory	Time	Memory	Time	Memory	Time	Memory	Time	Memory
UniAdapter (r=512)	18.8M	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
UniAdapter (r=128)	4.6M	0.86	0.95	0.93	0.95	0.89	0.91	0.79	0.92	0.88	0.94	0.94	0.94
Aurora (r=128)	0.2M	0.90	0.93	0.93	0.94	0.88	0.89	0.79	0.91	0.90	0.92	0.95	0.93
Aurora (r=64)	0.1M	0.84	0.93	0.92	0.94	0.86	0.89	0.76	0.89	0.88	0.88	0.94	0.92
E Cost Analysis
In Table 5, following [49], we report the training time and GPU memory cost for both retrieval and VQA tasks. We regard the training time and memory cost of UniAdapter(r=512) as one unit. Since we adopt the same backbone models, the forward propagation process of the two methods, Aurora and UniAdapter, is almost consistent, and the time cost is similar. However, our Aurora has fewer trainable parameters, resulting in a slightly smaller GPU memory footprint.
Then, We will give a theoretical analysis of the parametric complexity of the three PETL methods. Assume that the frozen backbone's visual, textual, and multimodal encoder all have L transformer layers while multimodal encoders contain cross-attention modules and visual, textual encoders contain self-attention modules. We only approximate the Query/Key/Value weight matrix in these attention-based modules. Let d donates the dimension of the hidden feature and r for rank, so the parametric complexity of the LoRA is  $\mathrm{L} \times (3 + 6) \times 2\mathrm{dr} \sim \mathcal{O}(\mathrm{L}\mathrm{dr})$ , the complexity of the UniAdapter is  $\mathrm{L} \times 4\mathrm{dr} \sim \mathcal{O}(\mathrm{L}\mathrm{dr})$ , and the complexity of our Aurora is  $\mathrm{L} \times (3 + 6) \times 2\mathrm{r} + 2\mathrm{dr} + 2\mathrm{L}\mathrm{d} \sim \mathcal{O}((\mathrm{L} + \mathrm{d})\mathrm{r})$ , normally r and L are much smaller than d, so from the above analysis we can draw the conclusion that when r increases, our Aurora can achieve the lowest parameter cost.
F Theoretical Analysis
Write  $\Delta \mathcal{W} = \sum_{r=1}^{R} \lambda_r (\pmb{u}_r \circ \pmb{v}_r \circ \pmb{p}_r)$ , and the  $(i,j,k)$ -element of  $\Delta \mathcal{W}$  is:
$$
\Delta \boldsymbol {W} _ {i j k} = \sum_ {r = 1} ^ {R} \lambda_ {r} u _ {r i} v _ {r j} p _ {r k}. \tag {7}
$$
Recall that the Frobenius norm of a tensor  $\mathcal{X} \in \mathbb{R}^{d \times d \times N}$  is given by:
$$
\left\| \mathcal {X} \right\| _ {F} = \left(\sum_ {i _ {1} = 1} ^ {d} \sum_ {i _ {2} = 1} ^ {d} \sum_ {i _ {3} = 1} ^ {N} \left| x _ {i _ {1}, i _ {2}, i _ {3}} \right| ^ {2}\right) ^ {1 / 2}, \tag {8}
$$
Hence, it suffices to analyze the convergence rate of our parameter tensors in the Euclidean space  $\mathbb{R}^{ddN}$ , where  $ddN$  denotes the product of  $d\times d\times N$  in order to distinguishing from the space of multi-dimensional arrays of size  $d\times d\times N$ .
Assumption F.1 We identify  $\mathbb{R}^{d\times d\times N}$  with  $\mathbb{R}^{ddN}$ , and let the loss function  $\mathcal{L}$  be defined on  $\mathbb{R}^{ddN}$  while we still write  $\mathcal{L}(\mathcal{W})$  where  $\mathcal{W}\in \mathbb{R}^{d\times d\times N}$  is a tensor. We will also use Frobenius norm and  $\ell^2$  norm interchangeably, which means that:
$$
\left\| \mathcal {X} \right\| _ {F} = \left\| \mathcal {X} \right\| _ {2}. \tag {9}
$$
Assumption F.2 We assume that the loss function  $\mathcal{L}:\mathbb{R}^{ddN}\to \mathbb{R}$  has the following property:
1.  $\mathcal{L}$  is injective.
2.  $\mathcal{L}$  is strongly convex: there exist  $m$  and  $M$  such that:
$$
m I \preceq \nabla^ {2} \mathcal {L} (\mathcal {X}) \preceq M I. \tag {10}
$$
That is,  $\nabla^2\mathcal{L}(\mathcal{X}) - mI$  is positive semidefinite and  $\nabla^2\mathcal{L}(\mathcal{X}) - MI$  is negative semidefinite.
Let  $\{\lambda_r^{(0)},\pmb {u}_r^{(0)},\pmb {v}_r^{(0)},\pmb {p}_r^{(0)}:r = 1,\dots ,R\}$  be the randomly initialized vectors used for tensor decomposition, where  $\lambda_r^{(0)}\in \mathbb{R},\pmb {u}_r^{(0)}\in \mathbb{R}^d,\pmb {v}_r^{(0)}\in \mathbb{R}^d,\pmb {p}_r^{(0)}\in \mathbb{R}^N$  . Denote
$$
\Delta \boldsymbol {\mathcal {W}} ^ {(0)} = \boldsymbol {\mathcal {W}} _ {0} = \sum_ {r = 1} ^ {R} \lambda_ {r} ^ {(0)} \boldsymbol {u} _ {r} ^ {(0)} \circ \boldsymbol {v} _ {r} ^ {(0)} \circ \boldsymbol {p} _ {r} ^ {(0)}. \tag {11}
$$
Let  $\Delta \mathcal{W}^{(n)}$  be the parameter tensor returned by the  $n$ th training epoch, that is,
$$
\Delta \boldsymbol {\mathcal {W}} ^ {(n)} = \sum_ {r = 1} ^ {R} \lambda_ {r} ^ {(n)} \boldsymbol {u} _ {r} ^ {(n)} \circ \boldsymbol {v} _ {r} ^ {(n)} \circ \boldsymbol {p} _ {r} ^ {(n)}. \tag {12}
$$
Theorem F.1 Under the above assumptions, and suppose that we train for  $n$  epochs with  $\eta \leq 1 / M$  using gradient descent. Let  $\mathcal{W}^*$  be the optimal parameter tensor, then,
$$
\mathcal {L} \left(\boldsymbol {W} _ {0} + \Delta \boldsymbol {W} ^ {(n)}\right)\rightarrow \mathcal {L} \left(\boldsymbol {W} ^ {*}\right) \quad (n \rightarrow \infty). \tag {13}
$$
Moreover,  $\mathcal{W}^*$  is unique.
Proof F.1 The proof follows from [8]. For notational convenience let  $\mathcal{X}^{(n)} = \mathcal{W}_0 + \Delta \mathcal{W}^{(n)}$ , and denote the optimal value  $\mathcal{L}(\mathcal{W}^*)$  by  $\lambda^*$ . We will begin by analyzing the convergence using arbitrary  $\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{ddN}$ , and then plug in our parameter tensors. By Taylor's theorem we can write:
$$
\mathcal {L} (\mathcal {Y}) = \mathcal {L} (\mathcal {X}) + \nabla \mathcal {L} (\mathcal {X}) ^ {T} (\mathcal {Y} - \mathcal {X}) + \frac {1}{2} (\mathcal {Y} - \mathcal {X}) ^ {T} \nabla^ {2} \mathcal {L} (\mathcal {Z}) (\mathcal {Y} - \mathcal {X}), \tag {14}
$$
where  $\mathcal{Z}$  lies in the line segment joining  $\mathcal{X}$  and  $\mathcal{Y}$ . By the strong convexity assumption, we have,
$$
\frac {1}{2} (\mathcal {Y} - \mathcal {X}) ^ {T} \nabla^ {2} \mathcal {L} (\mathcal {Z}) (\mathcal {Y} - \mathcal {X}) \geq \frac {1}{2} (\mathcal {Y} - \mathcal {X}) ^ {T} m (\mathcal {Y} - \mathcal {X}) = \frac {m}{2} \| \mathcal {Y} - \mathcal {X} \| _ {2} ^ {2}. \tag {15}
$$
$$
\mathcal {L} (\mathcal {Y}) \geq \mathcal {L} (\mathcal {X}) + \nabla \mathcal {L} (\mathcal {X}) ^ {T} (\mathcal {Y} - \mathcal {X}) + \frac {m}{2} \| \mathcal {Y} - \mathcal {X} \| _ {2} ^ {2}. \tag {16}
$$
Now we use  $\| \nabla \mathcal{L}(\mathcal{X})\|_2$  to bound  $\mathcal{L}(\mathcal{X}) - \lambda^*$ . The right-hand side of (16) is a convex quadratic function of  $\mathcal{V}$ , hence  $\mathcal{Y}^* = \mathcal{X} - 1 / m\nabla \mathcal{L}(\mathcal{X})$  is the minimizer, thus,
$$
\begin{array}{l} \mathcal {L} (\mathcal {Y}) \geq \mathcal {L} (\mathcal {X}) + \nabla \mathcal {L} (\mathcal {X}) ^ {T} \left(\mathcal {Y} ^ {*} - \mathcal {X}\right) + \frac {m}{2} \| Y ^ {*} - \mathcal {X} \| _ {2} ^ {2} (17) \\ = \mathcal {L} (\mathcal {X}) + \nabla \mathcal {L} (\mathcal {X}) ^ {T} \left(- \frac {1}{m} \nabla \mathcal {L} (\mathcal {X})\right) + \frac {m}{2} \left\| \frac {1}{m} \nabla \mathcal {L} (X) \right\| _ {2} ^ {2} (18) \\ = \mathcal {L} (\mathcal {X}) - \frac {1}{2 m} \| \nabla \mathcal {L} (\mathcal {X}) \| _ {2} ^ {2}. (19) \\ \end{array}
$$
Since  $\mathcal{V}$  is arbitrary, plugging  $\mathcal{X} = \mathcal{X}^{(n)}$ , we have
$$
\lambda^ {*} \geq \mathcal {L} \left(\mathcal {X} ^ {(n)}\right) - \frac {1}{2 m} \left\| \nabla \mathcal {L} \left(\mathcal {X} ^ {(n)}\right) \right\| _ {2} ^ {2} \tag {20}
$$
By the assumption  $\nabla^2\mathcal{L}(\mathcal{X})\preceq MI$  we have
$$
\mathcal {L} (\mathcal {Y}) \leq \mathcal {L} (\mathcal {X}) + \nabla \mathcal {L} (\mathcal {X}) ^ {T} (\mathcal {Y} - \mathcal {X}) + \frac {M}{2} \| \mathcal {Y} - \mathcal {X} \| _ {2} ^ {2}. \tag {21}
$$
Plugging in  $\mathcal{Y} = \mathcal{X}^{(n)} - \eta \nabla \mathcal{L}(\mathcal{X}^{(n)})$  yields
$$
\widetilde {\mathcal {L}} (t) \leq \mathcal {L} \left(\mathcal {X} ^ {(n)}\right) - \eta \| \nabla \mathcal {L} (\mathcal {X}) \| _ {2} ^ {2} + \frac {M \eta^ {2}}{2} \| \nabla \mathcal {L} (\mathcal {X}) \| _ {2} ^ {2}. \tag {22}
$$
Now we minimize over  $\eta$  on both sides of (22), and denote the optimal value by  $\tilde{\mathcal{L}} (\eta^{*})$ . The right-hand side of (22) is simple quadratic, hence it is minimized by  $\eta = 1 / M$ , and
$$
\min  (\operatorname {R H S}) = \mathcal {L} (\mathcal {X}) - \frac {1}{2 M} \| \nabla \mathcal {L} (\mathcal {X}) \| _ {2} ^ {2}. \tag {23}
$$
Figure 10: We represent the parameter distribution on different layers of the pre-trained large-scale multimodal foundation model (BLIP) vs. our Aurora, which is tuned on MSCOCO for image-text retrieval. Notably,  $\mathcal{W}_k$  and  $\mathcal{W}_v$  are the stack of the key and value projection matrices in different modality branches.
$$
\mathcal {L} \left(\mathcal {X} ^ {(n)} + \eta \Delta \mathcal {X} ^ {(n)}\right) = \widetilde {\mathcal {L}} \left(t ^ {*}\right) \leq \mathcal {L} \left(\mathcal {X} ^ {(n)}\right) - \frac {1}{2 M} \| \nabla \mathcal {L} (\mathcal {X}) \| _ {2} ^ {2}. \tag {24}
$$
Subtracting  $\lambda^{*}$  on both sides, we have,
$$
\mathcal {L} \left(\mathcal {X} ^ {(n)} + \eta \Delta \mathcal {X} ^ {(n)}\right) - \lambda^ {*} \leq \mathcal {L} \left(\mathcal {X} ^ {(n)}\right) - \lambda^ {*} - \frac {1}{2 M} \| \nabla \mathcal {L} (\mathcal {X}) \| _ {2} ^ {2}, \tag {25}
$$
by (20) we have,
$$
\mathcal {L} \left(\mathcal {X} ^ {(n + 1)}\right) = \mathcal {L} \left(\mathcal {X} ^ {(n)} + \eta \Delta \mathcal {X} ^ {(n)}\right) - \lambda^ {*} \leq \left(1 - \frac {m}{M}\right) \left(\mathcal {L} \left(\mathcal {X} ^ {(n)}\right) - \lambda^ {*}\right). \tag {26}
$$
By mathematical induction, we obtain,
$$
\mathcal {L} \left(\mathcal {X} ^ {(n)}\right) - \lambda^ {*} \leq \left(1 - \frac {m}{M}\right) ^ {n} \left(\mathcal {L} \left(\mathcal {X} ^ {(0)} - \lambda^ {*}\right)\rightarrow 0 (n \rightarrow \infty), \right. \tag {27}
$$
therefore  $\mathcal{L}(\mathcal{X}^{(n)})\to \lambda^*$  as  $n\to \infty$
Suppose  $\mathcal{V}^*$  is another optimal parameter tensor, then by the same argument we have  $\mathcal{L}(\mathcal{X}^{(n)})\to$ $\mathcal{L}(\mathcal{V}^*)$  as  $n\rightarrow \infty$  . Since  $\mathbb{R}^{ddN}$  is a Hausdorff space,  $\mathcal{L}(\mathcal{V}^*) = \mathcal{L}(\mathcal{W}^*)$  . By our assumption that  $\mathcal{L}$  is injective,  $\mathcal{W}^{*} = \mathcal{V}^{*}$
G Visualization Analysis
G.1 Parameter Distribution
Figure 10 and Figure 11 show more details on the parameter distribution comparisons between our Aurora and the pre-trained model and the full fine-tuned model, in which similar results can be observed on  $\mathcal{W}_k$  and  $\mathcal{W}_v$ . We can see that the mode approximation parameters adjust the original weights, and change the distribution of weights and biases to fit the downstream task. It can be concluded that Aurora has several advantages over traditional fine-tuning approaches. First, it avoids over-fitting to specific downstream tasks by only adjusting the pre-trained model parameters in a small local range. Second, it reduces the amount of training required on new data, making it more efficient and cost-effective. Last but not least, Aurora can further improve the model's performance on downstream tasks.
Figure 11: We represent the parameter distribution on different layers of the full fine-tuned model vs. our Aurora, which is tuned on MSCOCO for image-text retrieval. Notably,  $\mathcal{W}_k$  and  $\mathcal{W}_v$  are the stack of the key and value projection matrices in different modality branches.
Caption7462: He drew a beautiful picture
Caption9623: A man is on a cell phone while people are fighting
Caption9451: Kids feeding and playing with the horse
Caption7756: A group of women are singing
Caption8662: A man a woman cooking on a cooking show
Figure 12: Video-Text retrieval cases on MSRVTT test set.
G.2 Case Study
Visual-Text Retrieval. Figure 12 demonstrates some actual examples of Aurora performing text-to-video task on MSRVTT test set. In conclusion, the results presented highlight the exceptional performance of Aurora in searching relevant videos from textual descriptions. The accuracy and realism of the returned videos demonstrate the effectiveness of our proposed method in understanding the relationship between text and visual content.
Visual Question Answering. Figure 13 gives some question-answering examples of Aurora and UniAdapter on the MSRVTT-QA dataset. Specifically, our method is able to reason about the meaning of the text and video information to answer the questions more accurately than UniAdapter. This is an important result because the ability to reason about the meaning of both text and visual information is essential for understanding multimodal data.
Query191201: What is a guy doing?
Ground-Truth: ask
Aurora: ask
UniAdapter: talk
Query191306: What is a man standing in talking?
Ground-Truth: kitchen
Aurora: kitchen
UniAdapter: kitchen
Query191382: What does a man fold?
Ground-Truth: paper
Aurora: paper
UniAdapter: paper
Query191949: How many men play in a tennis match?
Ground-Truth: two
Aurora: two
UniAdapter: two
Query192400: What does hillary clinton speak to?
Figure 13: Video Question Answering cases on MSRVTT-QA test set.
Ground-Truth: crowd
Aurora: crowd
UniAdapter: people
Overall, the qualitative results shown in Figure 12 and Figure 13 demonstrate the effectiveness of our proposed method in both multimodal retrieval and question-answering tasks. We believe that our approach has the potential to be used in many multimodal applications, where understanding and analyzing multimedia data is essential.