SlowGuess/NeurIPS_2023_OCR huggingface dataset:
speed-up rate, we show the speed-up rate in Table 3. The speed-up rate is calculated by considering the number of epochs reduced to achieve the best TFS performance on a particular dataset. Formally, given a dataset  $D$ , TFS obtains the best performance  $p$  on  $D$  at epoch  $e_{\mathrm{tfs}}$ , whereas VPGTrans first achieves a better performance than  $p$  at epoch  $e_{\mathrm{vt}}$ . The speed-up rate on  $D$  is given by  $\frac{e_{\mathrm{tfs}}}{e_{\mathrm{vt}}}$ . According to Table 3, our VPGTrans can achieve at least 4 times speed-up on  $40\%$  of Transfer-Task variants. Furthermore, for the two caption datasets, which take a long time to converge, our VPGTrans  $\mathrm{OPT}_{125\mathrm{M} \rightarrow 2.7\mathrm{B}}$  delivers a 10 times speed-up.
Moreover, when compared with the VPG inherit in Fig. 6, our VPGTrans can achieve a higher speed-up rate on all of the variants on caption tasks, and achieve better performance on most variants except for  $\mathrm{OPT}_{1.3\mathrm{B}\rightarrow 2.7\mathrm{B}}$ . We refer the readers to Appendix §C.3 for more comparisons.
We provide interesting findings with respect to the efficiency transfer by VPGTrans in the following.
- The smaller size of  $\mathrm{LLM}_{\mathrm{src}}$ , the easier the transfer. In our OPT based experiments, we notice an interesting phenomenon: when transferring to a given  $\mathrm{LLM}_{\mathrm{tgt}}$ , both the convergence rate and optimal performance are roughly inversely proportional to the size of  $\mathrm{LLM}_{\mathrm{src}}$ . For example, as shown in Table 3, the  $\mathrm{OPT}_{125\mathrm{M}\rightarrow 2.7\mathrm{B}}$  and  $\mathrm{OPT}_{350\mathrm{M}\rightarrow 2.7\mathrm{B}}$  have much higher speed-up rate than  $\mathrm{OPT}_{1.3\mathrm{B}\rightarrow 2.7\mathrm{B}}$  on all of the datasets. Meanwhile, as demonstrated in Fig. 6, the optimal performance of  $\mathrm{OPT}_{125\mathrm{M}\rightarrow 2.7\mathrm{B}}$  is better than  $\mathrm{OPT}_{1.3\mathrm{B}\rightarrow 2.7\mathrm{B}}$  on 3 VQA tasks.
Table 4: Comparison between models built with our VPGTrans and the original BLIP-2 ViT-G OPT $_{6.7B}$  and BLIP-2 ViT-G FlanT5 $_{XXL}$ .
Models	VQAv2 val	GQA test-dev	OKVQA test	GPU hours	training data
BLIP-2 ViT-G OPT6.7B	54.3	36.4	36.4	631.5	129M
BLIP-2 ViT-G OPT2.7B→6.7B (ours)	57.2	36.2	39.8	59.0	13.8M
VL-LLaMA7B (ours)	58.1	37.5	37.4	67.1	13.8M
BLIP-2 ViT-G FlanT5XXL	65.2	44.7	45.9	684.0	121.6M
BLIP-2 ViT-G FlanT5XL→XXL (ours)	65.2	45.0	45.0	32.4	5.3M
Figure 8: Comparison between different methods across 2 TaT variants on 5 tasks. Note that the model is directly evaluated after pre-training without further fine-tuning. Please refer to Appendix\SD for other transfer variants.
We hypothesize that training VPG on larger OPT will have a worse influence on VPG's existing fine-grained perception ability, which might be caused by the enlarging embedding dimensions. To validate our hypothesis, we fix the VPG weight and only tune linear projectors to test VPGs trained on different  $\mathrm{LLM}_{src}$  through cross-size transfer. The SPICE [3] metric on COCO caption is used to evaluate the VPG's visual perception ability, where SPICE is specifically designed for visual concept perception in captions. As shown in Fig. 7, for each row, given the  $\mathrm{LLM}_{tar}$ , the performance of VPG trained on smaller  $\mathrm{LLM}_{src}$  can outperform the larger ones in most conditions, which indicates a better visual perception ability of VPG trained on smaller  $\mathrm{LLM}_{src}$ . Therefore, adapting a VPG from a smaller OPT model which is less affected, is helpful to take fewer steps to reach the TFS's best performance and achieve even better performance.
4.3 Scale-up Experiments
To validate the effectiveness of our VPGTrans on the real-world application level, we experiment on transferring from BLIP-2 ViT-G  $\mathrm{OPT}_{2.7\mathrm{B}}$  to  $\mathrm{OPT}_{6.7\mathrm{B}}$  and from BLIP-2 ViT-G  $\mathrm{FlanT5_{XL}}$  to  $\mathrm{FlanT5_{XXL}}$ . Please refer to Appendix\S C.4 for implementation details.
Speed-up with non-degenerated performances. As shown in Table 4, we can see that (1)  $\mathrm{OPT}_{2.7\mathrm{B} \rightarrow 6.7\mathrm{B}}$ : our VPGTrans achieves a 10.7 times speed-up with only  $10.7\%$  training data, while the performance on VQAv2 and OKVQA have over 2 points improvement. (2)  $\mathrm{FlanT5}_{\mathrm{XL} \rightarrow \mathrm{XXL}}$ : VPGTrans can achieve 21.1 times speed-up with less than  $5\%$  training data while achieving the same performance on VQAv2, higher performance on GQA and slightly lower performance on OKVQA. Note that continuing training the  $\mathrm{FlanT5}_{\mathrm{XL} \rightarrow \mathrm{XXL}}$  only shows improvement on VQAv2 and GQA. Thus, we do not show a checkpoint with more training steps.
Figure 7: The confusion matrix. Only linear layers are trained for VPG evaluation. Models are tested on COCO caption with SPICE metric to compare the VPGs trained on different  $\mathrm{LLM}_{\mathrm{src}}$ .
Table 5: Comparison between our VL-Vicuna and SOTA MLLMs for multimodal conversation. The evaluation is done by Multimodality Chatbot Arena platform via user voting.
Rank	Model	Elo Rating
1	LLaMA-Adapter v2 [17]	1023.0
2	LLaVA [35]	1019.9
3	VL-Vicuna (ours)	1012.1
4	MiniGPT-4 [62]	1011.9
5	InstructBLIP [13]	999.5
6	mPLUG-Owl [59]	996.3
7	Otter [26]	981.5
8	BLIP-2 [30]	955.8
5 Exp-II: Transfer across Different Model Types
In this section, we further investigate the transfer across different model types. For simplicity, we mark this type of transfer as TaT.
Baselines. We mainly compare our VPGTrans with training from scratch (TFS), and report the performance on the aforementioned 5 tasks. Other details (cf. 4.1) are totally the same with TaS experiments.
Transfer Variants. We conducted experiments on transfer learning between 1)  $\mathrm{OPT}_{350\mathrm{M}}$  and  $\mathrm{FlanT5_{base}}$ , and 2)  $\mathrm{OPT}_{2.7\mathrm{B}}$  and  $\mathrm{FlanT5_{XL}}$ .
5.2 VPGTrans Enabling Faster Convergence only on Large LLMs under TaT
- There is no speed-up of TaT between two small LLMs. A finding is that on TaT our VPGTrans does not show speed-up for small models, and even shows a degeneration of training speed in the initial several epochs. As shown in Fig. 8, when transferring from  $\mathrm{OPT}_{350\mathrm{M}}$  to  $\mathrm{FlanT5_{base}}$ , the convergence speed of VPGTrans is even slower than TFS in the initial several epochs.
- Speed-up of VPGTrans happens in large LLMs. However, when moving to the large LLMs like  $\mathrm{OPT}_{2.7\mathrm{B}}$  and  $\mathrm{FlanT5_{XL}}$ , there is an obvious speed-up. As shown in Table 3, we can see at least 2 times speed-up when transferring from  $\mathrm{FlanT5_{XL}}$  to  $\mathrm{OPT}_{2.7\mathrm{B}}$ . We empirically find that the soft prompts for larger LLM are more linear transferrable among different LLM types. As shown in Fig. 8, when transferring between  $\mathrm{FlanT5_{base}}$  and  $\mathrm{OPT}_{350\mathrm{M}}$ , the VGPTrans' 1st epoch results on two caption datasets are limited, where only a linear operation can be trained. The result of  $\mathrm{OPT}_{350\mathrm{M}} \rightarrow \mathrm{FlanT5_{base}}$  on the COCO caption is even near to zero. By contrast, the result of  $\mathrm{FlanT5_{XL}} \rightarrow \mathrm{OPT}_{2.7\mathrm{B}}$  with our VPGTrans are obviously higher than TFS. We hypothesize that larger LLM typically learned more generalizable text embeddings and share more similarity among relative word distances, which enables an easier VPG transfer.
6 Customizing New MLLMs with Any LLMs
Above, we thoroughly certify the efficacy of our proposed VPGTrans approach for higher efficient transfer of VPG. In this section, we illustrate how to apply the VPGTrans framework for VPG transfer to customize new MLLMs with any LLMs.
VL-LLaMA. By applying our VPGTrans, we can equip the recently released LLaMA [53] model with a VPG trained on BLIP-2  $\mathrm{OPT}_{6.7B}$  to perceive the visual information. As shown in Table 4, we can see that our VL-LLaMA can outperform the original BLIP-2  $\mathrm{OPT}_{6.7B}$  on all datasets.
VL-Vicuna. An exciting application of our VPGTrans is to build a GPT-4 [8] style multimodal conversation chatbot. To achieve our goal, we employ Vicuna [10] as our base LLM. Similarly, we transfer the VPG from BLIP-2  $\mathrm{OPT}_{6.7\mathrm{B}}$  and add an extra instruction tuning using MiniGPT-4's self-instruct data [62]. We compare our model with MiniGPT-4 in Fig. 9. When compared to MiniGPT-4, our VL-Vicuna shows better visual perception ability. Please refer to Appendix\SE for more cases. In
Figure 9: Comparison between MiniGPT-4 and our VL-Vicuna.
addition, we also report the ranking and ELO ratings for our VL-Vicuna compared with the other 7 SOTA multimodal chatbots in Table 5 (The evaluation is done by Multimodality Chatbot Arena platform). The results show the effectiveness of our VL-Vicuna compared with existing SOTA models.
In this work, we conduct a comprehensive investigation to the problem of VPG transferability across LLMs. We first explore the key factors for maximizing the transfer efficiency under the VPG transfer across different LLM sizes and types. Based on the key findings, we propose a novel two-stage transfer framework, namely VPGTrans, which can help to achieve comparable or better performance while significantly reducing the training costs. Moreover, a list of important findings and possible reasons behind them are shown and discussed. Finally, we demonstrate the practical value of our VPGTrans, by customizing new MLLMs via VPG transfer from existing MLLMs.
This research is supported by NExT++ Lab, Singapore Ministry of Education Academic Research Fund Tier 2 under MOE's official grant number T2EP20221-0023, and CCF-Baidu Open Fund.
[1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the
<sup>7</sup>http://vlarena.opengvlab.com
[56] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022.
[57] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
[58] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[59] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
[60] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[61] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[62] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
A.1 Vision and Language Models
Vision and language models (VLMs) [50, 29, 32, 31] aim at understanding visual and textual information with a single model. Previously, the VLM mainly employs a pre-trained object detector as its feature extractor and conducts unsupervised training on a huge amount of image-text pairs. For example, VL-Bert [50], ViL-Bert [36] and Uniter [9] adopt Faster-RCNN [47] to extract image information into object features and take advantage of the masked language model as their pre-training task. Later, due to the prevalence of vision transformer (ViT), the VLM paradigm is turned into end-to-end training with a ViT as the visual encoder. The representative works include ALBEF [28], BLIP [29], and BEiT-v3 [56], which show the state-of-the-arts supervised training performance on a wide range of downstream tasks.
Recently, LLMs have shown their remarkable capability as zero/few-shot learners [7] and a series of emergent abilities [57] like in-context learning [7], and chain-of-thoughts reasoning [58]. A new paradigm, i.e., MLLMs is created by associating the VLM or pure vision encoders with LLMs. As we illustrated before, the VLM or visual encoders are typically able to convert the input vision signals into LLM-understandable soft prompts, and thus we call them VPG. The MLLMs advance in inheriting the great potentials of the backbone LLMs, and thus are capable of achieving excellent zero/few-shot performances [2, 30] on downstream tasks or be equipped with visual planning ability [16]. However, connecting the VPG to the existing LLMs with further tuning is costly. Even the BLIP-2 [30], targeted at efficient training, will take over 600 A100 GPU hours on over 100M image-text pairs for its largest model. With this regard, our proposed VPGTrans can effectively reduce the cost of building new MLLMs with the help of existing ones.
A.2 Prompt Transfer
In this paper, we investigate the VPG transfer, where the soft prompt is to represent the content of specific inputs like images and videos. In addition to the content prompt, the more explored soft prompt is the task prompt [24, 61, 22], where a sequence of soft prompts are tuned to assist the pre-trained models to achieve better performance on specific tasks. There have already been some works exploring the transferability of task prompts. For example, Su et al. [51] conducts a series of experiments to illustrate the transferability across tasks and models. Specifically, Su et al. [51] find that the transfer between similar tasks is beneficial for training speed-up and better performance. Lester et al. [25] proposes to recycle soft prompts across models with vocab-to-vocab transformations, or linear-combination transformations. Note that our word converter initialization is similar to the idea of vocab-to-vocab transformations. However, we do not observe a zero-shot transfer in our experiments like them, which indicates a potential difference between the content prompts and task prompts. Another way of soft prompt transfer [14] is to conduct prompt tuning on discrete prompts, and thus the discrete prompts can be directly shared across models. Different from these task prompts transfer works, our VPG transfer scenario actually suffers from fewer limitations. For example, the task soft prompts transfer suffers from the dimension change problem, where the main body of the trainable parameters should be processed. However, our VPG (the main trainable parameters) can naturally be shared among LLMs with different embedding dimensions and leave the dimension change problem to a simple projector with ignorable parameters.
B Extended Findings in Exploratory Analysis
In this section, we show extended findings of exploratory analysis (cf. §3.1).
- 1. Merely tuning the projector can not achieve the best performance. We want to clarify that merely tuning the projector is insufficient for achieving the best performance. Notably, as shown in Fig. 10, significant performance gaps are observed between the "only linear" (green curve) and "train from scratch" (orange curve) approaches for COCO caption and NoCaps. Therefore, if the goal is to build a multimodal conversation robot using carefully collected dialog data, training only the linear projector is insufficient to align with the provided data.
Figure 10: Comparisons between i) inheriting VPG from  $\mathrm{OPT}_{125\mathrm{M}}$  and training it with randomly initialized projector for  $\mathrm{OPT}_{350\mathrm{M}}$  and ii) training VPG and randomly initialized projector for  $\mathrm{OPT}_{350\mathrm{M}}$  from scratch. iii) training only the projector.
Figure 11: Interpreting generated soft prompts for  $\mathrm{OPT}_{125\mathrm{M}}$  with nearest words.
- 2. Word embedding converter can not replace a trained linear projector. As demonstrated in Fig. 11, we observe a common pattern: the last token of soft prompts is closest to EOS, while the middle tokens represent the image content. Such a phenomenon indicates a similarity between soft prompts and word embeddings. However, they are not identical. For instance, the norm of soft prompts is typically around 10 times the average norm of word embeddings. It is important to note that the linear projector initialization cannot replace the warm-up training. Using only the linear projector initialization even yields a random performance. We believe that a better understanding of how prompt works will further benefit the VPG's transfer learning.
- 3. The projector warm-up is robust to a larger learning rate, while VPG can not. The first thing we want to clarify is that the 5 times normal learning rate will result in a training crash for VPG. Additionally, we find that although increasing the learning rate to 10 times in the projector warm-up does not yield any additional acceleration, the projector can converge without crashing during training.
C Extended TaS Experiments
In this section, we first illustrate extending findings of TaS experiments (cf. §4). Then, we introduce the implementation details of scale-up experiments. We also plot a more complete version of Fig. 6 in Fig. 13.
C.1 VPGTrans Enabling Stable Training under TaS
As we illustrated before,  $\mathrm{FlanT5}_{\mathrm{large}}$  training is extremely unstable. Even when the learning rate is adjusted to one-tenth of its original value, the model does not converge or shows a very slow convergence rate after 4 epochs. However, we find that by lowering the learning rate for stage-2 training, our VPGTrans can achieve stable training on  $\mathrm{FlanT5}_{\mathrm{large}}$ . We plot the performance curve of the COCO caption in Fig. 12.
C.2 VPGTrans Enabling Training with Less Data under TaS
We empirically find that TaS can reduce the requirement for the amount of training data. By reducing the training data to only COCO, we find no obvious performance drop. However, we want to stress that the retained data should be of high quality, which means that if the same number of SBU data
Figure 12: Comparison of training FlanT5large using different methods.
Table 6: The speed-up rate of our VPGTrans compared with  $VPG$  inherit. The symbol "-" means VPGTrans can not achieve better performance than  $VPG$  inherit.
Transfer	COCO Caption	NoCaps	VQAv2	GQA	OKVQA
OPT125M→350M	1.1	2.7	10.0	6.0	6.0
OPT125M→1.3B	-	2.7	-	-	-
OPT350M→1.3B	-	1.7	1.0	2.0	1.0
OPT125M→2.7B	3.0	3.0	3.0	1.0	3.0
OPT350M→2.7B	3.3	4.5	1.0	1.0	1.0
OPT1.3B→2.7B	2.3	3.0	-	-	-
FlanT5base→XL	-	1.0	1.8	9.0	0.4
is retained, the performance especially for captioning will drop. The conclusion can also be found in Table 4. We refer the readers to the next subsection for more details.
C.3 Comparison between VPGTrans and VPG Inherit
As shown in Fig. 13, the green line (our VPGTrans) can be higher than the orange line (VPG inherit) for the majority of various conditions. Especially when considering the best performance of different tasks, our VPGTrans can achieve better performance than VPG inherit (non “-” in Table 6) for over  $74\%$  Transfer-Task variants. Moreover, among the variants that our VPGTrans can achieve better performance, our VPGTrans can also achieve a speed-up on  $69.2\%$  conditions.
C.4 Scale-up Experiment Implementation Details
The scale-up experiment refers to the results in Table 4. We try to imitate BLIP-2's pre-training data composition. First of all, two human-annotated datasets COCO and VG are used. SBU is also used. Then, BLIP-2 uses BLIP to generate captions for the 115M web images and rank them with CLIP ViT-L/14. We also adopt similar synthetic data from Laion-COCO. We report the concrete number of data we use in Table 4. For the stage-1 training, we keep the same as the previous validation experiments where COCO and SBU are used for warm-up with a 5 times the learning rate. Then, we use COCO, VG, and Laion-COCO for the stage-2 training. Note that we have tried to include Laion-COCO and VG for the stage-1 training, but found no obvious difference and thus use COCO and SBU for simplicity. For VL-Vicuna, to align with the conversation scenario, we further fine-tune our VL-Vicuna with MiniGPT-4's self-instruct data, which is 3,439 image-text pairs.
<sup>8</sup>https://laion.ai/blog/laion-coco
Figure 13: Comparison between different methods across 7 TaS variants on 5 tasks. Note that the model is directly evaluated after pre-training without further fine-tuning.
Table 7: Comparison between tuning only the projector (i.e. linear transfer) and the best results the VPGTrans achieve.
Transfer	COCO Caption	VQAv2
linear	best	linear	best
FlanT5base → OPT350M	110.8	136.5	40.4	44.2
FlanT5XL → OPT2.7B	132.1	139.3	50.4	50.3
OPT350M → FlanT5base	1.1	122.1	34.0	49.9
OPT2.7B → FlanT5XL	106.5	133.2	51.3	53.5
D Extended TaT Experiments
In this section, we mainly illustrate extending findings of TaT experiments (cf. §5). We plot a more complete version of Fig. 8 in Fig. 14.
D.1 Linear Transfer Gap between Different LLM's Visual Prompts
As illustrated in Section 5.2, it is more difficult to transfer between two small LLMs with our VPGTrans due to the weaker linear transferability between two small LLMs' visual prompts. To better support our results, we compare the results of tuning only the projector (i.e. linear transfer) and the best results the VPGTrans can achieve. As shown in Table 7, we can see that when conducting transfers between two large models  $\mathrm{(OPT_{350M}}$  and  $\mathrm{FlanT5_{base}}$ , the performance is typically far from the optimal results. However, when we transfer between  $\mathrm{OPT}_{2.7\mathrm{B}}$  and  $\mathrm{FlanT5_{XL}}$ , the linear performance is near to the optimal performance. There is a 25.7 points gap between  $\mathrm{FlanT5_{base}} \rightarrow \mathrm{OPT}_{350\mathrm{M}}$  's linear and best on COCO caption datasets but only 7.2 points gap between  $\mathrm{FlanT5_{XL}} \rightarrow \mathrm{OPT}_{2.7\mathrm{B}}$  's linear and best. If considering the transfer between the small to large LLMs under TaT, both the VPG's visual perception ability and transfer gap should be considered. We leave the systematic exploration for future works.
E Extended Results for MLLM Customization
We show more comparisons between VL-Vicuna and MiniGPT-4 in Fig. 15. We can see that our VL-Vicuna has better visual perception ability. For example, when MiniGPT-4 falsely recognizes the three people in the image as two in the first example, our VL-Vicuna can not only recognize the number of people but also tell their roles. Moreover, our VL-Vicuna can successfully link the vision content with external knowledge. In the third example, our VL-Vicuna can recognize Leonardo and link the content with his films like Titanic.
F Potential Impact and Limitations
Our VPGTrans is designed for building new MLLMs with lower computational cost, i.e., shorter training time and less training data. With an already pre-trained MLLM, VPGTrans enables fast VPG transfer to build either a larger MLLM or a MLLM with a different type of LLM. We hope VPGTrans can facilitate teams in LLM community to customize their MLLMs with reduced cost. There are also possible limitations of the current version of VPGTrans. The first one is that our VPGTrans should rely on some already-aligned VPGs. The second potential limitation is that the VPGTrans-built MLLMs still suffer from the common problems of content generation AI systems [45, 46, 48, 7]. For example, the VL-Vicuna may make up some sentences with falsely recognized visual facts, like what is shown in Fig. 16. It is worth exploring associating our VPGTrans with training safer models [5, 41, 38].