SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ao Zhang $^{1}$  Hao Fei $^{1*}$  Yuan Yao $^{2*}$  Wei Ji $^{1}$  Li Li $^{1}$  Zhiyuan Liu $^{2}$  Tat-Seng Chua $^{1}$ $^{1}$  NExT++ Lab, School of Computing, National University of Singapore  
 $^{2}$  Department of Computer Science and Technology, Tsinghua University  
zhanga6@outlook.com haofei37@nus.edu.sg yaoyuanthu@163.com
Project: https://vpgtrans.github.io
Since developing a new multimodal LLM (MLLM) by pre-training on a tremendous amount of image-text pairs from scratch is exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG component of the MLLM still incurs significant computational costs, such as thousands of GPU hours and millions of training data points. An alternative solution is to transfer an existing VPG from one MLLM to the target MLLM. In this work, we investigate VPG transferability across LLMs for the first time, aiming to reduce the cost of VPG transfer. Specifically, we explore VPG transfer across different LLM sizes (e.g., small-to-large) and types. We identify key factors to maximize the transfer efficiency, based on which we develop a simple yet highly effective two-stage transfer framework, called VPGTrans. Notably, it enables VPG transfer from BLIP-2  $\mathrm{OPT}_{2.7\mathrm{B}}$  to BLIP-2  $\mathrm{OPT}_{6.7\mathrm{B}}$  with less than  $10\%$  of GPU hours using only  $10.7\%$  of the training data compared to training a VPG for  $\mathrm{OPT}_{6.7\mathrm{B}}$  from scratch. Furthermore, we provide a series of intriguing findings and discuss potential explanations behind them. Finally, we showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with the recently released LLaMA and Vicuna LLMs.
Figure 1: (a) The general architecture of MLLMs, e.g., BLIP-2 [30] and PaLM-E [16], including a visual prompt generator (VPG), a linear projector and a backbone LLM. Typically, to tune the MLLM, only the VPG and the projector are updated, while the LLM is kept frozen. (b) This work investigates the VPG transferability across LLMs, including different LLM sizes and LLM types.
Background. Recent years have witnessed a great rise in large-scale language models (LLMs) in ushering the human-like artificial intelligence. Text-based LLMs [42, 7, 44] are further enhanced
*Corresponding Author: Hao Fei, Yuan Yao
by associating with other modalities such as vision, leading to the multimodal LLMs (MLLMs), such as BLIP-2 [30], Flamingo [2], GPT-4 [8] for multimodal dialog system, and PaLM-E [16] for embodied AI system. To construct a MLLM, a visual prompt generator (VPG) module (cf. Fig. 1(a)) that produces soft prompts for the input images/videos is added<sup>2</sup> for bridging the gap between vision and language modalities. Currently, such architecture has been frequently adopted by many popular MLLMs [30, 27]. For example, BLIP-2 pre-trains a CLIP-ViT [43] combined with a Q-Former as VPG. To obtain the final MLLM, the VPG needs to be tuned. Ideally, the vast LLM backbone can remain untouched, leaving only the relatively lightweight VPG module to be fully or partially updated.<sup>3</sup>
Motivation. However, building a MLLM is inevitably computation-expensive, due to the huge overhead brought by the LLM. For example, training a BLIP-2  $\mathrm{FlanT5}_{\mathrm{XXL}}$  needs over 600 A100-GPU hours on over 100 million image-text pairs. Hopefully, transferring a pre-trained VPG (which is the main body of trainable parts) from an existing MLLM to a novel LLM instead of training from scratch, $^4$  offers a promising solution. Intuitively, all the MLLMs literally can share the same VPG infrastructure and utility, $^5$  which makes the VPG transfer theoretically feasible. In this work, we thus investigate the potential of transferring VPG across LLMs.
**Proposal.** Specifically, this paper examines the transferability of VPG across LLMs: 1) with different sizes (the same type), i.e., transfer across LLM sizes, and 2) across different LLM types, i.e., transfer across LLM type, as illustrated in Fig. 1(b).
- [Transfer across LLM Sizes (TaS)]. It has been a typical practice for LLM-related research [8] to validate the training strategy and the hyperparameter on smaller models (e.g.,  $\mathrm{OPT}_{2.7\mathrm{B}}$ ) and then scale up to larger ones (e.g.,  $\mathrm{OPT}_{6.7\mathrm{B}}$ ). It is thus worth exploring whether a VPG trained on a smaller LLM can be transferred to a larger LLM, resulting in reduced computational costs & data, and maintaining comparable performance.
- [Transfer across LLM Types (TaT)]. With a well-tuned VPG for a type of LLM, it is interesting to see if VPG can be transferred to other types of LLMs even with different architectures (e.g., decoder v.s. encoder-decoder). If the transfer can be achieved, how to make it more efficient?
We conduct a series of exploratory analyses (cf. §3.1) to identify the key factors for transfer efficiency. Based on our empirical study, we design a two-stage transfer learning framework (cf. §3.2), namely VPGTrans, that includes a projector warm-up (stage-1) and vanilla fine-tuning (stage-2). For stage-1, we find that warming up the projector before VPG tuning can effectively reduce the training step for adapting a pre-trained VPG to a new LLM, and avoid the potential performance drop in the adaptation. To achieve an efficient warm-up, the projector will be well-initialized and then trained with an extremely large learning rate  $(5 \times lr)$ . For stage-2, there is a vanilla fine-tuning of both the VPG and projector. Despite its simplicity, VPGTrans is able to significantly speed up the VPG-transfer process without harming the performance.
Results and Findings. Via extensive experiments on the transfer across LLM sizes and types (cf. §4 & §5), we gain the following key observations:
- VPGTrans helps to avoid the performance drop caused by directly inheriting the VPG and achieves at most 10 times acceleration for the small-to-large transfer across LLMs in the same type.
- VPGTrans can also achieve comparable or better performance than training from scratch and achieve at most 5 times acceleration for the transfers between different model types.
- Notably, our VPGTrans helps to achieve a BLIP-2 ViT-G OPT $_{2.7\mathrm{B} \rightarrow 6.7\mathrm{B}}$  transfer with less than  $10\%$  of GPU hours and  $10.7\%$  of training data required for the original model training.
- Furthermore, our framework can even outperform the original BLIP-2  $\mathrm{OPT}_{6.7\mathrm{B}}$  on most of the evaluated datasets, with a  $+2.9$  improvement on VQAv2 and a  $+3.4$  improvement on OKVQA.
Our investigation further reveals some intriguing findings, for which we provide possible explanations:
Also including a linear projector for dimension matching.
Note that Flamingo also inserts some tunable parameters into the LLM part, but recent works [30, 16] found that freezing LLM can be more efficient.
<sup>4</sup>It is not a rigorous expression, because the VPG is typically a pre-trained model, like CLIP [43]. We use it for simplicity in this paper.
while the projector can not be shared due to the dimension mismatch.
Figure 2: Comparing the cost between training VPG from scratch vs. transferring VPG via our VPGTrans strategy. Note the LLM via VPGTrans is  $\mathrm{FlanT5_{XL\to XXL}}$  and  $\mathrm{OPT}_{2.7B\to 6.7B}$ , respectively.
- When conducting TaS from  $\mathrm{LLM}_{\mathrm{src}}$  to  $\mathrm{LLM}_{\mathrm{tgt}}$ , the size of  $\mathrm{LLM}_{\mathrm{src}}$  is not the larger the better. The transfer sometimes even follows a counterintuitive principle of "the smaller the  $LLM_{src}$  size, the more speed-up and better performance" (cf. §4.2).
- When conducting TaT, efficient VPG transfer can not be achieved between two small LLMs with our VPGTrans, due to the large gap between small LLMs' embedding space (cf. §5.2).
Contributions. In this study, we show for the first time that effective VPG transfer across LLMs can be achieved under most conditions, suggesting that it is possible to build a new MLLM with considerably lower computational cost, as seen in Fig. 2. To summarize, we make the following key contributions:
- Effective approach. We investigate the key factors for VPG-transfer efficiency and propose a two-stage transfer framework VPGTrans. The approach helps to achieve a highly-efficient VPG transfer across LLMs with less training data and even task improvements.
- Intriguing findings. By exploring the VPG transfer across LLMs, we reveal several intriguing findings and provide potential explanations that will shed light on further research.
- Open source. We showcase how to customize a novel GPT-4-like MLLM with our VPGTrans (cf. §6), and release two multimodal-version MLLMs: VL-LLaMA and VL-Vicuna. All codes and models is released at https://github.com/VPGTrans/VPGTrans.
This section will outlines the existing prevailing MLLMs, and elaborates on the settings of the exploratory analyses of these MLLMs.
2.1 MLLM
Architecture. As illustrated in Fig. 1(a), current MLLMs mostly adopt a common architecture, including a visual prompt generator (VPG), a projector, and a backbone LLM. Typically, VPG takes images/videos as inputs, and encodes the visual input into a fixed length of soft prompts. Then, a linear projector is employed to align the soft prompt's dimension to LLM's word embedding dimension. Finally, the LLM will generate sentences based on the information from the soft prompt. We list some of the recent representative MLLMs in Table 1.
Table 1: MLLMs architectures and pre-training paradigm. †: it is a GPT-2-like LLM with relative position embeddings.
MLLMs	VPG	VPG Trainable	LLM	LLM Trainable
KOSMOS-1 [19]	CLIP [43]	All	Rand. Init. LM	All
Frozen [54]	NF-ResNet-50 [6]	NF-ResNet-50	GPT-2-like† [42]	No
Flamingo [2]	NFNet-F6 [6]+Resampler [21]	Resampler	Chinchilla [18]	Xattn-Dense
PaLM-E [16]	ViT [15]/OSRT [49]	All	PaLM [11]	No
BLIP-2 [30]	EVA-CLIP [52] + Q-Former [30]	Q-Former	OPT [60] / Flan-T5 [12]	No
Training Paradigm. Given a MLLM, typically the VPG and linear projector will be trained, fully or partially. For example, PaLM-E updates all of the parameters of VPG in the pre-training stage, while BLIP-2 and Flamingo freeze the ViTs and tune their Q-Former and Resampler, respectively. As the main part of the whole architecture, the LLM is usually frozen during the training or tuned only a small portion (e.g., 10B for Flamingo-80B). KOSMOS-1 is an exception, which does not use a pre-trained LLM but trains the LLM from scratch. Such a training paradigm typically results in much longer training time and data (both multimodal and pure text corpus). Recent works [30, 16] show that
adopting an existing LLM and freezing all of its parameters can also achieve excellent performance with significantly reduced computational cost, which leads to the trend of adapting frozen pre-trained LLM. For example, BLIP-2 FlanT5 $_{\mathrm{XXL}}$  (12.1B) can achieve better zero-shot VQAv2 performance (65.0% in Acc.) compared with KOSMOS-1 (51.0% in Acc.) and Flamingo-80B (56.3% in Acc.). Thus, in this paper, we mainly focus on VPG transfer across frozen LLMs.
2.2 Experiment Settings
Architecture. We adopt BLIP-2's architecture and training paradigm. In our exploration experiments, we consider using the VPG that consists of a CLIP ViT-L/14 [43], and a Q-Former that has already undergone a BLIP-like pre-training (the 1st stage pre-training in BLIP-2's paper [30]).
Training Data. For all of the exploration experiments, we adopt human-annotated COCO caption dataset [34] and web image-text pairs SBU dataset [40], which results in 1.4 million image-text pairs.
Transfer Direction. For the small-to-large model transfer among the same type of LLMs, we investigate: 1) OPT [60] (decoder-only) series including 125M, 350M, 1.3B, and 2.7B, and 2) FlanT5 [12] (encoder-decoder) ranging base, large, and XL. For the transfer across different types of LLMs, we consider the ones of OPT and FlanT5 with similar sizes.
Evaluation. To evaluate the performance of MLLMs, we choose two caption datasets: (1) COCO caption [34] (2) NoCaps [1], and three VQA datasets: (3) VQAv2 [4] (4) GQA [20] (5) OKVQA [37]. We make evaluations after the pre-training without task-specific fine-tuning and report the CIDEr [55] for all caption tasks and accuracy for all VQA tasks.
Implementation Details. We follow the same implementation details of BLIP-2, via the open code.6 Concretely, we use FP16 and BFloat16 for OPT and FlanT5 respectively in the model training. For the learning rate, we first conduct a linear warm-up from 1e-6 to 1e-4, and then use a cosine learning rate schedule with the minimal  $lr = 1$  e-5 for 10 epochs. Due to the limited data amount, we slightly decrease the batch size, which we find beneficial for the final performance. Specifically, we set the batch size of 1,728 and 1,152 for OPT and FlanT5-based models, respectively.
3 Maximizing the Transfer Efficiency with a Two-stage Transfer Strategy
In this section, we first identify the key factors for maximizing transfer efficiency, based on which we then motivate our solution for better transfer.
3.1 Exploratory Analysis: Identifying Key Factors for VPG Transfer
Via selected experiments of small-to-large transfer among OPT models, we can obtain the following key observations. More systematical comparisons are conducted in the later section (cf. §4).
- Inheriting the trained VPG can accelerate training. To demonstrate this, we compare the convergence rates of VPG training on  $\mathrm{OPT}_{350\mathrm{M}}$  from scratch, and inheriting VPG trained on  $\mathrm{OPT}_{125\mathrm{M}}$ . The patterns are shown in Fig. 3. Overall, we find that inheriting VPG trained on  $\mathrm{OPT}_{125\mathrm{M}}$  accelerates convergence, particularly for two caption tasks. However, for datasets that require fine-grained visual perception such as VQAv2 and GQA, directly conduct continue training with an inherited VPG will harm the performance. We hypothesize that tuning VPG with a randomly initialized projector will compromise the existing fine-grained visual perception ability of VPG. The possible reason can be that, the VPG is typically a pre-trained model with powerful visual perception ability, and thus updating based on the gradient passed through a random projector will mislead the VPG at the initial steps [2, 33, 23].
- Warming up the linear projector can prevent performance drop and expedite VPG training. To verify this, we first conduct a warm-up training of the linear projector for 3 epochs, during which both VPG and LLM are frozen. Subsequently, we jointly train VPG and the projector and plot the performance curve in Fig. 4 (the warm-up process is not included in this figure). The results show that the performance drop observed in Fig.3 can be avoided in Fig.4. Additionally, we observe that the warm-up training leads to fewer training steps required for VPG and projector joint training. However, we must emphasize that warming up is a costly step. In the case of a large LLM, such as 6.7B, the trainable parameters of BLIP-2's VPG will account for less than  $10\%$  of the total parameters, where freezing VPG can only lead to a reduction of  $5.4\%$  of A100 hours (36.9 out of 684.0 A100
<sup>6</sup>https://github.com/salesforce/lavis
Figure 3: Comparisons between i) inheriting VPG from  $\mathrm{OPT}_{125\mathrm{M}}$  and training it with randomly initialized projector for  $\mathrm{OPT}_{350\mathrm{M}}$  and ii) training VPG and randomly initialized projector for  $\mathrm{OPT}_{350\mathrm{M}}$  from scratch.
hours). We will elaborate on how to accelerate the linear projector warm-up in our later discussion (cf. 3.1).
- Initializing  $\mathrm{LLM}_{\mathrm{tgt}}$  's projector with the help of the word converter can accelerate the linear projector warm-up. In fact, the VPG and projector trained on  $\mathrm{LLM}_{\mathrm{src}}$  have already learned how to map the visual content to  $\mathrm{LLM}_{\mathrm{src}}$  's understandable soft prompt [39]. If we can convert the  $\mathrm{LLM}_{\mathrm{src}}$  's soft prompt to  $\mathrm{LLM}_{\mathrm{tgt}}$  's soft prompt, we can directly get a VPG suitable for  $\mathrm{LLM}_{\mathrm{tgt}}$ . One natural idea is to leverage the word embeddings of both models as a proxy for the soft prompt [25]. The intuition behind the scene is that, the soft prompt works in the same format as normal words.
Figure 4: First warming-up then transferring can avoid performance drop on VQAv2 and accelerate convergence for COCO caption.
To validate our hypothesis, we conduct an experiment on the transfer from  $\mathrm{OPT}_{125\mathrm{M}}$  to  $\mathrm{OPT}_{1.3\mathrm{B}}$ . After training a linear word embedding converter (cf. §3.2(b)), we initialize the projector for  $\mathrm{OPT}_{1.3\mathrm{B}}$  with the merged linear operation of the projector for  $\mathrm{OPT}_{125\mathrm{M}}$  and converter. As shown in Table 2, we observe that the initialization can reduce the 3 epochs' warm-up to 2 epochs.
- Linear projector warm-up enables faster convergence with an extremely large learning rate. To determine the most efficient transfer practice, we experiment with training the projector using different learning rates. Surprisingly, we find that the linear projector enables fast and stable convergence with an extremely large learning rate. Specifically, by setting the learning rate to 5 times of the original value, the COCO caption's CIDEr score can reach 133.1 with 1 epoch training, which is higher than the 3 epochs results of w/o init. as shown in Table 2.
Table 2: Comparison between linear projector warm-up with/without word embedding initialization. The metric is COCO caption's CIDEr.
Epoch	w/ init.	w/o init.
1	130.2	126.1
2	132.7	131.6
3	133.4	132.8
3.2 A Two-stage VPG Transfer Framework
By connecting all the dots as discussed above in §3.1, we now design our two-stage VPGTrans framework for more efficient VPG transfer. As shown in Fig. 5, the stage-1 of VPGTrans performs projector warm-up and the stage-2 carries out a vanilla fine-tuning. Our results demonstrate that the VPGTrans is simple yet effective that can significantly speed up the transfer without compromising performance. Detailed results are given in the later sections (cf. §4 & 5).
$\triangleright$  Stage-1: Projector Warm-up.
(a) Inherit VPG. We first initialize the VPG for  $\mathrm{LLM}_{\mathrm{tgt}}$  with the VPG trained on  $\mathrm{LLM}_{\mathrm{src}}$ .
(b) **Projector Initialization**. Then, we initialize the projector for  $\mathrm{LLM}_{\mathrm{tgt}}$  merged from the projector of  $\mathrm{LLM}_{\mathrm{src}}$  and a linear word converter. Formally, we define the linear projector of  $\mathrm{LLM}_{\mathrm{src}}$  as  $f_s(x) = W_s x + b_s$ , the linear projector for  $\mathrm{LLM}_{\mathrm{tgt}}$  as  $f_t(x) = W_t x + b_t$ , and the word converter as  $g_c(x) = W_c x + b_c$ .
The word converter is a linear layer trained with text-only caption data to convert the  $\mathrm{LLM}_{\mathrm{src}}$  's word embeddings to  $\mathrm{LLM}_{\mathrm{tgt}}$  's word embeddings. We experiment with optimizing losses based on cosine similarity or Euclidean distance, and observe no significant difference between the two losses. Thus we simply use cosine similarity in our experiments. In cases where  $\mathrm{LLM}_{\mathrm{src}}$  and  $\mathrm{LLM}_{\mathrm{tgt}}$  use different
Figure 5: Our two-stage VPGTrans framework. **Stage-1** is to first (a) inherit the VPG of  $\mathrm{LLM}_{\mathrm{src}}$  and (b) initialize the projector by merging the projector of  $\mathrm{LLM}_{\mathrm{src}}$  and word converter. (c) Then the projector will be warmed up for 1 epoch with a large learning rate. **Stage-2** is to (d) conduct a vanilla fine-tuning for the VPG and projector for  $n$  epochs.
tokenization methods, we optimize based on the overlapped tokens. Formally, for every given token  $k$ , we denote its word embeddings of  $\mathrm{LLM}_{\mathrm{src}}$  and  $\mathrm{LLM}_{\mathrm{tgt}}$  as  $x_{s}$  and  $x_{t}$ . Then, we minimize the loss:
$$
\mathcal {L} = 1 - \operatorname {s i m} \left(g _ {c} \left(x _ {s}\right), x _ {t}\right). \tag {1}
$$
Once we obtain the word converter  $g_{c}(\cdot)$ , we can easily merge it with the projector of  $\mathrm{LLM}_{\mathrm{src}}$  as:
$$
f _ {t} (x) = f _ {s} \left(g _ {c} (x)\right) = W _ {s} \left(W _ {c} x + b _ {c}\right) + b _ {s}, \tag {2}
$$
resulting in  $f_{t}$ 's weight and bias as  $W_{t} = W_{s}W_{c}$  and  $b_{t} = W_{s}b_{c} + b_{s}$ .
(c) Warm-up Training. Then, we only train the projector in this stage with a frozen VPG and LLM. Specifically, we train the projector for 1 epoch with 5 times of the normal learning rate.
$\triangleright$  Stage-2: Vanilla Fine-tuning.
(d) Vanilla Fine-tuning. In the final step, we conduct a joint training of VPG and projector for  $n$  epochs with a normal learning rate.
4 Exp-I: Transfer across Different Model Sizes
In this section, we conduct experiments to systematically illustrate the effectiveness of our VPGTrans and analyze the relationship between transfer efficiency and model size. For simplicity, we use TaS to represent the transfer across different model sizes.
In this part, we introduce baselines and transfer variants. For details about training data and implementation details, please refer to the experiment settings in the Preliminary (cf. 2.2).
Baselines. We mainly compare our VPGTrans with training from scratch (TFS) and VPG inheritance (VPG Inheritance), where we report their performance on the aforementioned 5 tasks without further task-specific fine-tuning. For our VPGTrans, the word converter training only requires updating a linear layer on tokenized text data and typically takes less than 10 minutes on 1 A100 GPU with less than 15G GPU memory. Meanwhile, freezing the VPG can lead to at least 14 A100 minutes speed-up per epoch. Therefore, we consider the whole stage-1 training as the 1st epoch for simplicity.
Transfer Variants. We conducted experiments on transfer learning using 1) the OPT model across four different sizes: 125M, 350M, 1.3B, and 2.7B, and 2) the FlanT5 model across three sizes: base, large, and XL. However, we encountered significant instability during training with  $\mathrm{FlanT5}_{\mathrm{large}}$ . As a result, we mainly present the transfer results between  $\mathrm{FlanT5}_{\mathrm{base}}$  and  $\mathrm{FlanT5}_{\mathrm{XL}}$ .
4.2 VPGTrans Enabling Faster Convergence without Performance Drop under TaS
First of all, as shown in Fig. 6, our VPGTrans can consistently accelerate the model convergence. For COCO caption and NoCaps that require more training steps to converge, our VPGTrans (green line) can be higher than the other two lines (blue and orange lines). To give a quantitative evaluation of the
Figure 6: Comparison between different methods across 3 TaS variants on 5 tasks. Note that the model is directly evaluated after pre-training without further fine-tuning. Please refer to Appendix\S C for other transfer variants.
Table 3: The speed-up rate of our VPGTrans compared with training from scratch (TFS). The symbol "-" means VPGTrans can not achieve better performance than TFS.
Transfer	COCO Caption	NoCaps	VQAv2	GQA	OKVQA
OPT125M→350M	1.7	3.0	1.0	5.0	5.0
OPT125M→1.3B	9.0	10.0	9.0	-	2.0
OPT350M→1.3B	4.5	5.0	9.0	2.0	2.0
OPT125M→2.7B	10.0	10.0	2.0	2.0	3.0
OPT350M→2.7B	10.0	10.0	2.0	-	3.0
OPT1.3B→2.7B	3.3	3.3	2.0	-	1.5
FlanT5base→XL	1.0	1.1	3.0	4.0	2.0
FlanT5XL→OPT2.7B	5.0	5.0	2.0	2.0	3.0
OPT2.7B→FlanT5XL	1.7	2.0	-	2.0	-