VPGTrans: Transfer Visual Prompt Generator across LLMs
Authors: Ao Zhang
Hao Fei
Yuan Yao
Wei Ji
Li Li
Zhiyuan Liu
Tat-Seng Chua
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Prompt Generator
Efficient Transfer
Multimodality
TL;DR: We propose a two-stage VPGTrans framework that can build new MLLMs (e.g., BLIP-2, PALM-E) by transferring visual perception modules of exsiting MLLMs to reduce the training cost (e.g., using only 10% cost).

Abstract: Since developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG component of the MLLM still incurs significant computational costs, such as thousands of GPU hours and millions of training data points. An alternative solution is transferring an existing VPG from one MLLM to the target MLLM. In this work, we investigate VPG transferability across LLMs for the first time, aiming to reduce the cost of VPG training.  Specifically, we explore VPG transfer across different LLM sizes (e.g., small-to-large) and types.  We identify key factors to maximize transfer efficiency, based on which we develop a simple yet highly effective two-stage transfer framework, called VPGTrans. Notably, it enables VPG transfer from BLIP-2 OPT 2.7B to BLIP-2 OPT 6.7B with less than 10% of the GPU hours using only 10.7% of the training data compared to training a VPG for OPT 6.7B from scratch. Furthermore, we provide a series of intriguing findings and discuss potential explanations behind them. Finally, we showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with recently released LLaMA and Vicuna LLMs.

VPGTrans: Transfer Visual Prompt Generator across LLMs
Ao Zhang
Hao Fei
Yuan Yao 2.
Wei Ji
Li Zhiyuan Liu
Tat-Seng Chua
NExT+ Lab, School of Computing, National University of Singapore Department Computer Science and Technology, Tsinghua University
zhanga6@outlook. com haofei37 edu.sg yaoyuanthu@163.c com
Project: https: gi github.r. io
Abstract
Since developing multimodal LLM (MLLM) by pre-training tremen- dous amount of image-text pairs from scratch is exceedingly resource-consuming connecting an existing LLM with comparatively lightweight visual prompt gen- erator (VPG) becomes a feasible paradigm. However, further tuning the VPG component of the MILMM still incurs significant computational costs, such thou- sands of GPU hours and millions of training data points. An alternative solution is transfer an existing VPG from one MLLM the target MILM.. In this work, we investigate VPG transferability across LLMs for the first time, aiming reduce the cost VPG transfer. Specifically, we explore VPG transfer across different LLM sizes (e.g., small-to-large) and types. We identify key factors to maximize the transfer efficiency, based on which we develop a simple yet highly effective two-stage transfer framework, called VPGTrans. Notably, it enables VPG trans- fer from BLIP-2 OPT2,7B to BLIP-2 OPT6.7B with less than 10% of GPU hours using only 10.7% of the training data compared to training VPG for OPT678 from scratch. Furthermore, we provide a series of intriguing findings and discuss potential explanations behind them. Finally, showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with the recently released LLaMA and Vicuna LLMs.
Transfer
VPG
VPG
Trawsfer
M.M.arc
LLMggt
VIG (1.28)
Projector (4M)
LLM (11B)
Projector
LLM size
LLMamai
Mange
Trainable (fally partially)
Trainable
Frozen
LLM
L.
LLM type
LLMrypaA
LLMoypal
SRC MLLM
TGT MLLM
(a) General MLLM architecture (representatively with 11B LLM)
(b) VPG transfer across LLMs
Figure 1: (a) The general architecture of MLLMs, e.g., 2 30 and PaLM-E [66], including a visual prompt generator (VPG), a linear projector and a backbone LLM. Typically, to tune the MILM,, only the VPG and the projector are updated, while the LLM is kept frozen. (b) This work investigates the VPG transferability across LLMs, including different LLM sizes and LLM types.
Introduction
Background. Recent years have witnessed a great rise in large-scale language models (LLMMs) in ushering the human-like artificial intelligence. Text- L.LMs [42 44 are further enhanced
"Corresponding Author: Hao Fei, Yuan Yao
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

by associating with other modalities such as vision, leading to the multimodal LLMs (MLLMs). such as BLIP-2 [30 Flamingo 2), GPT-4 g for multimodal dialog system, and PaLM-E [6 for embodied Al system. To construct a MILM, visual prompt generator (VPG) module (ef. Fig. aa)) that produces soft prompts for the input images/videos added for bridging the gap between vision and language modalities. Currently, such architecture has been frequently adopted by many popular MLLMs [30 277 For example, BLIP-2 pre-trains a CLIP- ViT [43] combined with a Q-Former as VPG. To obtain the final MLLM, the VPG needs to be tuned. Ideally, the vast LLM backbone can remain untouched, leaving only the relatively lightweight VPG module to be fully or partially
updated
Motivation. However, building a MLLM is inevitably computaaioon-expeniiiv, due to the huge overhead brought by the LLM. For example, training a BLIP-2 FlanT5xxI. needs over 600 A100-GPU hours over 100 million image-text pairs. Hopefully, transferring pre-trained VPG (which is the main body of trainable parts) from an existing MLLM to novel LLM instead of training from promising solution. Intuitively, all the MLLMs literally can share the same VPG infrastructure and utility which makes the VPG transfer theoretically feasible. In this work, we thus investigate the potential of transferring VPG across LLMs.
scratch offers Offfers
Proposal. Specifically, this paper examines the transferability of VPG across LLMs: 1) with different sizes (the same type), i.e. transfer across LLM sizes, and 2) across different LLM types, i.e. transfer across LLM rype, illustrated in Fig. [b).
[T (Trunsfer across LLM Sizes (TaS)). It has been a typical practice for LLM-related research { to validate the training strategy and the hyperparameter on smaller models (e-g.. 7P))7a and then scale up to larger ones (e.g., 67))))) It is thus worth exploring whether a VPG trained on a smaller LLM can be transferred to a larger LLM,, resulting in reduced computational costs & data, and maintaining comparable performance.
[Transfer across LLM Types (TaT)]. With a well-tuned VPG for a type of LLM, it is interesting see if VPG can be transferred other types of LLMs even with different architectures (e.g., decoder v.s. encoder-decoder), If the transfer can be achieved, how to make it more efficient?
We conduct series of exploratory analyses (cf. 43.1 identify the key factors for transfer efficiency. Based on our empirical study, we design a two-stage transfer learning framework (cf. (3.2). namely VPGTrans, that includes projector warm-up (stage-1) and vanilla fine-tuning (stage-2). For stage- I, we find that warming up the projector before VPG tuning can effectively reduce the training step for adapting pre-trained VPG to new LLM, and avoid the potential performance drop in the adaptation. To achieve an efficient warm-up, up, the projector will be well-initialized and then trained with extremely large learning rate (5 Ir). For 2t there vanilla fine-tuning of both the VPG and projector. Despite its simplicity, VPGTrans is able to significantly speed up the VPG-transfer process without harming the performance.
Results and Findings. Via extensive experiments on the transfer across LLM sizes and types (cf. (5) we gain the following key observations:
VPGTrans helps to avoid the performance drop caused by directly inheriting the VPG and achieves at most 10 times acceleration for the small-to-large transfer across LLMS in the same type.
VPGTrans can also achieve comparable better performance than training from scratch and achieve at most 5 times acceleration for the transfers between different model types.
Notably, our VPGTTans helps to achieve a BL.IP-2 ViT-G OPT 2.7B-66.77B transfer with less than 10% of GPU hours and 10.7% of training data required for the original model training.
Furthermore, our framework can even outperform the original BLIP-2 OPT on most of the evaluated datasets, with +2.9 improvement on VQAv2 and +3.4 improvement on OKVQA.
Our investigation further reveals some intriguing findings, for which we provide possible explanations:
Also including a linear projector for dimension matching.
"Note that Flamingo also inserts some tunable parameters into the LLM part, but recent works 30 16
found
that freezing LLM can be more efficient.
'It is not a rigorous expression, because the VPG is typically a pre-trained model, like CLIP 43 We use
for simplicity in this paper.
while the projector can not be shared due to the dimension mismatch.

---

BLIP-2 (VTT-G FlanT5
BLIP-2 (VTT-G + OPT..7B)
2386.
129.0
VPGTrams (Ours)
65.3
572
Train from scratch
39,0
241.2
15.8
hours (A.00))
Cost (USD)
Trxin datta (M)
VQAv2 (aco.)
dixta (M)
VQAv2 (acc.)
(A100)
(USD)
Figure 2: Comparing the cost between training VPG from scratch vs, transferring VPG via our VPGTrans strategy. Note the LLM via VPGTrans is Flan'S XL-XXXX and OPT 2.77-6..BB respectively.
When conducting TaS from LL to LLMgg the size of LLM not the larger the better. The transfer sometimes follows counterintuitive principle of "the smaller the LLM size, the more speed-up and better performance" (ef. (4.2).
When conducting TaT, efficient VPG transfer can not be achieved between two small LLMs with our VPGTrans, due the large gap between small LLMs' embedding space (cf. (5.
Contributions. In this study, we show for the first time that effective VPG transfer across LLMs can achieved under most conditions, suggesting that is possible build a new MLLM with considerably lower computational cost, as seen in Fig. 2 To summarize, we make the following key contributions:
Effective approach. We investigate the key factors for IPG-transfer efficiency and propose two-stage transfer framework VPGTrans. The approach helps achieve highly-efficient VPG transfer across LLMs with less training data and even task improvements.
Intriguing, findings. By exploring the VPG transfer across LLMs, reveal several intriguing findings and provide potential explanations that will shed light on further research.
Open source. We showcase how to customize a novel GPT-4-like MLLM with our VPGTrans (cf. } and release multimodal-version MLLMs: VL-LLaMA and VL-Vicuna. All codes and models is released at wtthhttt://// /estps:/githin httgtgittu....../// com /tttts/s//tttttb.c.//////T
2
Preliminary
This section will outlines the existing prevailing MILMM, and elaborates on the settings of the exploratory analyses of these MILMs.
2.1 MLLM
Architecture. As illustrated in Fig. Ta), current MILMs mostly adopt common architecture, including a visual prompt generator (VPG), projector, and backbone LLM. Typically, VPG takes images/videos as inputs, and encodes the visual input into a fixed length of soft prompts. Then, linear projector is employed to align the soft prompt's dimension to LLM's word embedding dimension. Finally, the LLM will generate sentences based on the information from the soft prompt. We list some of the recent representative MILMs in Table T
Table I: MI.LMSs architectures and pre-training paradigm, t: it GPT-2-like LLM with relative position embeddings.
MLLMSs
VPG
VPG Trainable
LLM
LLM Trainable
KOSMOS-
CLIP
Rand. Init. LM GPT-2-likef
All
Frozen
NE-ResNee-50
Ne-ResNet-50
No
Flamingo
N6Net-F6 Resampler
Resumpler
Chinchilla 18
Xattn-Dense
PaLM-E [6
ViT DS] OSRT
49
All
PaLM MI]
No
BLIP-2 30
EVA-CLIP 52
Q-Former
30
Q-Former
OPT
60
Flan-T5 5 12
No
Training Paradigm. Given a MILM, typically the VPG and linear projector will be trained, fully or partially. For example, PaLM-E updates all of the parameters of VPG in the pre-training stage, while BLIP-2 and Flamingo freeze the ViTs and tune their Q-Former and Resampler, respectively. As the main part of the whole architecture, the LLM is usually frozen during the training or tuned only small portion (e.g., 10B for Flamingo-80B) KOSMOS-I is an exception, which does not use pre-trained L.L trains the L.MM from scratch. Such training paradigm typically results in much longer training time and data (both multimodal and pure text corpus). Recent works 30 [6] show that

---

adopting an existing LLM and freezing all of its parameters can also achieve excellent performance with significantly reduced computational cost, which leads the trend of adapting frozen pre- rre-trained can achieve better zero-shot VQAv2 performance (65.0% in Acc.) compared with KOSMOS- 1 (51.0% Acc.) and Flamingo-800 (56.3% in Acc.). Thus, in this paper, we mainly focus on VPG transfer across frozen LLMs.
LLM. For example, BLIP-2 F5an (12.1B)
2.2 Experiment Settings
Architecture. We adopt BLIP-2's architecture and training paradigm. In our exploration experi- ments, we consider using the VPG that consists of CLIP ViT-L/14 [43], and a Q-Former that has already undergone a BLIP-like pre-training (the 1st stage pre-training in BLIP-2's paper (30]).
Training Data. For all of the exploration experiments, we adopt human-annotater COCO caption dataset (34) and web image-text pairs SBU dataset [40), which results in 1.4 million image-text pairs.
Transfer Direction. For the small-to-large model transfer among the same type of LLMs, we investigate: 1) OPT (60) (decoder-only) series including 125M, 350M, 1.3B, and 2.7B, and 2) FlanT5 12] (encoder-decoder) ranging base, large, and XI.. For the transfer across different types of LLMs, we consider the ones of OPT and FlanTT5 with similar sizes.
Evaluation. To evaluate the performance of MLLMs, we choose two caption datasets: (1) COCO caption [34 (2) NoCaps [I, and three VQA datasets: (3) VQAv2 4 (4) GQA 20 (5) OKVQA B7 We make evaluations after the pre-training without task-specific fine-tuning and report the CIDEr 55 for all caption tasks and accuracy for all VQA tasks.
Implementation Details. We follow the same implementation details of BLIP-2, via the open code 0 Concretely, we use FP16 and 1 for OPT and Flan'i5 respectively in the model training. For the learning rate, we first conduct a linear warm-up from 1e-6 to le-4, and then learning rate schedule with the minimal Ir=le-5 for 10 epochs. Due to the limited data amount, we slightly decrease the batch size, which find beneficial for the final performance. Specifically, we set the batch size of 1,728 and 1,152 for OPT and Flan'5-baase models, respectively.
cosine
3 Maximizing the Transfer Efficiency with a Two-stage Transfer Strategy
In this section, we first identify the key factors for maximizing transfer efficiency, based on which we then motivate our solution for better transfer.
3.1 Exploratory Analysis: Identifying Key Factors for VPG Transfer
Via selected experiments of small-to-large transfer among OPT models, we can obtain the following key observations. More systematical comparisons are conducted in the later section (ef $4)
Inheriting the trained VPG can accelerate training. To demonstrate this, we compare the convergence rates of VPG training on OPT from scratch, and inheriting VPG trained OPT 125M- The patterns are shown in Fig. 3 Overall, we find that inheriting VPG trained on OPT 125M accelerates convergence, particularly for two caption tasks. However, for datasets that require fine-grained visual perception such as VQ and GQA, directly conduct continue training with an inherited VPG will harm the performance. We hypothesize that tuning VPG with a randomly initialized projector will compromise the existing fine-grained visual perception ability of VPG. The possible reason can be that, the VPG is typically a pre-trained model with powerful visual perception ability, and thus updating based on the gradient passed through a random projector will mislead the VPG at the initial steps 23).
Warming up the linear projector can prevent performance drop and expedite VPG training. To verify this, we first conduct a warm-up training of the linear projector for 3 epochs, during which both VPG and LLM are frozen. Subsequently, we jointly train VPG and the projector and plot the performance curve in Fig. 4 (the warm-up process is not included in this figure). The results show that the performance drop observed in Fig 3 can be avoided in Fig. 4 Additionally, we observe that the warm-up training leads fewer training steps required for VPG and projector joint training. However, we must emphasize that warming up is costly step. In the case of a large LLM, such as 6. 7B, the trainable parameters of BLIP-2's VPG will account for less than 10% of the total parameters. where freezing VPG can only lead to a reduction of 5.4% of A 100 hours (36.9 out of 684.0 A100

---

COCO Caplion
NoCaps
VOAv2
GQA
CKVQA
34
20
130
GIDE
50
10
30
16
110
DVP neart 20 hun
70
LFG ihwd
bun scazhh
from soutch
Inw
14
11 Fsn scutch
DVF
opochs
apochs
opochs
opodra
apochs
Figure 3: Comparisons between i) inheriting VPG from OPT 125M and training it with randomly initialized projector for 350M and ii) training VPG and randomly initialized projector for OPT 350M from scratch.
hours), We will elaborate on how to accelerate the linear projector warm-up in our later discussion (cf 3.1
Initializing LLMgtss 's projector with the help of the word converter can accelerate the linear pro- jector warm-up. In fact, the VPG and projector trained on LLM have already learned how to map the visual content to L. 's understandable soft prompt [3]. If we can convert the LLM 's soft prompt to LLM 's soft prompt, we can directly get VPG suitable for LLM One natural idea is to leverage the word embeddings of both models as a proxy for the soft prompt [25]. The intuition behind the scene that, the soft prompt works in the same format as normal words.
CDCD Caption
VQ2A2
130
warm up-transter
warm up prattaneeer
scratch
20
spacts
Figure 4: First warming-up then transferring can avoid performance drop on VQAv2 and accelerate convergence for COCO caption.
To validate our hypothesis, we conduct an experiment on the transfer from 12T to OPT After training linear word embedding converter (ef. (3.2(()), we initialize the projector for OPT with the merged linear operation of the projector for OPT 125M and converter. As shown in Table 2 observe that the initialization can reduce the 3 epochs' warm-up to 2 epochs.
Linear projector warm-up enables faster convergence with an extremely large learning rate. To determine the most efficient transfer practice, we experiment with training the projector using different learning rates. Surprisingly, we find that the linear projector enables fast and stable convergence with an extremely large learning rate. Specifically, by setting the learning rate times of the original value, the COCO caption's CIDEr score can reach 133.1 with 1 epoch training, which is higher than the 3 epochs results of w/o init. as shown in Table
Table 2: Comparison between lin- projector warm-up with/without word embedding initialization. The metric is COCO caption's 's CIDEr.
Epoch
w/ init.
w/o init.
1
130.2
126.1
2
132.7
131.6
3
133.4
132.8
3.2 A Two-stage VPG Transfer Framework
By connecting all the dots as discussed above in 3.1 we now design our two-stage VPGTrans framework for more efficient VPG transfer. As shown in Fig. 5 the stage-1 of VPGTrans performs projector warm-up and the stage-2 carries out vanilla fine-tuning. Our results demonstrate that the VPGTrans is simple yet effective that can significantly speed up the transfer without compromising performance. Detailed results are given in the later sections (cf. & 5)
Stage-1: Projector Warm-up.
(a) Inherit VPG. We first initialize the VPG for LLMg with the VPG trained on LLM
(b) Projector Initialization. Then, we initialize the projector for LLMg merged from the projector of LLM and a linear word converter. Formally, we define the linear projector of LLM,,. as fz) = W.r + by, the linear projector for LLMgg as fe(x) = Wix + b,, and the word converter (x) W.r be.
The word converter is a linear layer trained with text-only caption data to convert the LLM 's word embeddings to I.MM''s 's word embeddings. We experiment with optimizing losses based on cosine similarity or Fuclidean distance, and observe no significant difference between the two losses. Thus simply cosine similarity in our experiments. In cases where LLM and LLMgt use different
5

---

SRC MLLM
Projector
L.M
Stage-1:
Projector Warm-up
laheriting VPG
(b) Initiating
word converter
Projector
VPG
Projector
LLM
(c) Tuning Projector (With apoch Skir)
Stage-2: Direct Fine-tuning
VPG
Projector
LLM
Overall fine fine-tuning (with n epoch)
TGT MLLM
Figure 5: Our two-stage VPGTrans framework. Stage-1 is first (a) inherit the VPG of LLM and (b) initialize the projector by merging the projector of LLM and word converter. (c) Then the projector will be warmed up for epoch with large learning rate. Stage-2 is (d) conduct vanilla fine-tuning for the VPG and projector for n epochs.
tokenization methods, we optimize based on the overlapped tokens. Formally, for every given token k, we denote its word embeddings of LLM and LLMgt as and rτ. Then, we minimize the loss:
L. = sirm(ge
Once we obtain the word converter ge(-). we can easily merge it with the projector of LLM as:
f((x) (g(((x) W. W. be) + ba. resulting in ft's weight and bias as W; W W. and b, = W.b. b
(c) Warm-up Training. Then, only train the projector in this stage with frozen VPG and LLM. Specifically, we train the projector for epoch with 5 times of the normal learning rate.
Stage-2: Vanilla Fine-tuning.
(d) Vanilla Fine-tuning. In the final step, we conduct a joint training of VPG and projector for epochs with normal learning rate.
Exp-I: Transfer across Different Model Sizes
In this section, we conduct experiments to systematically illustrate the effectiveness of our VPGTTans and analyze the relationship between transfer efficiency and model size. For simplicity, we use TaS represent the transfer across different model sizes.
4.1 Experimental Settings
In this part, we introduce baselines and transfer variants. For details about training data and imple- mentation details, please refer to the experiment settings in the Preliminary (cf. 2.2
Baselines. We mainly compare our VPGTrans with training from scratch (TFS) and VPG inheri- tance (VPG Inherit), where we report their performance on the aforementioned 5 tasks without further spec-sp fine-tuning. For our VPGTrans, the word converter training only requires updating linear layer on tokenized text data and typically takes less than 10 minutes on A100 GPU with less than 15G GPU memory. Meanwhile, freezing the VPG can lead to at least epoch. Therefore, we consider the whole stage-1 1 training as the 1st epoch for simplicity.
100 minutes speed-up
Transfer Variants. We conducted experiments on transfer learning using 1) the OPT model across four different sizes: 125M, 350M, 1.3B, and 2.7B, and the FlanT5 model across three sizes: base, large, and XL. However, we encountered significant instability during training with Flan 3 As result, we mainly present the transfer results between F5an hase and Flan
4.2 VPGTrans Enabling Faster Convergence without Performance Drop under TaS
First of all, shown in Fig. 6 our VPGTrans can consistently accelerate the model convergence. For COCO caption and NoCaps that require more training steps to converge, our VPGTrans (green line) can be higher than the other two lines (blue and orange lines). To give quantitative evaluation of the

---

19
COCO Capton
OFT NoGapa
CPT 20 VON2
OFT2IN-211 GOA
01 OKVOA
194
34
zan for scas-3
tan form sorech
tain fun scarct
OAR Eon sonzh
10x
VPoTar
VPOTrow
10
OPT)
0OCD Cartion
OFF
Capas
CPT
OFT:
60A
OPT OKVOA
33
4+
XPQ inst
VC0 Imwr:
(PG Plut
VPG mmu
VPOTEE
spocts
spotss
apoza
66.3
0DA
525
33
475
pan pcaich
fun
30
425
Figure 6: Comparison between different methods across 3 TaS variants on 5 tasks. Note that the model is directly evaluated after pre-training without further fine-tuning. Please refer to Appendix C for other transfer variants.
Table 3:
The speed-up rate of our VPGTrans compared with training from scratch (TFS). The symbol "_" means VPGTrans can not achieve better performance than TFS.
Transfer
COCO Caption
NoCaps
VQAv2
GQA
OKVQA
12T >350M
1.7
3,0
1.0
5.0
5.0
1P >1.38
9.0
10.0
9.0
2.0
3P7 v1.3B
4.5
5.0
9.0
2.0
2.0
12T >2.78
10.0
10.0
2.0
2.0
3.0
OPT +2.78
10.0
10.0
2.0
3,0
OPT aB -8.78
3.3
3.3
2.0
1.5
Flan base--XL
1.0
1.1
3.0
4.0
2.0
F5an OPT278
5.0
5.0
2.0
2.0
3.0
OPT2.BB > F5anT5xx
1.7
2.0
2.0
speed-up rate, we show the speed-up rate in Table 3 The speed-up rate is calculated by considering the number of epochs reduced to achieve the best TFS performance on particular dataset. Formally, given a dataset D, TFS obtains the best performance p on D at epoch eis. whereas VPGTrans first achieves better performance than p at epoch ev-. The speed-up rate on D is given by Sus eu According to Table our VPGTrans can achieve least 4 times speed-up on 40% of Transfer-Taas variants. Furthermore, for the two caption datasets, which take a long time to converge, our VPGTrans OPT 125 delivers 10 times speed-up.
Moreover, when compared with the VPG inherit in Fig. 6 our VPGTrans can achieve a higher speed-up rate on all of the variants caption tasks, and achieve better performance on most variants except for
We refer the readers to Appendix C.3
C.3 for more comparisons.
We provide interesting findings with respect to the efficiency transfer by VPGTrans in the following.
The smaller size of LLM the easier the transfer. In our OPT based experiments, notice an interesting phenomenon: when transferring to a given LLMgt. both the convergence rate and optimal performance are roughly inversely proportional to the size of LLM For example, as shown in Table the 1PT and OPT:50M-22.7 have much higher speed-up rate than OPT 3 on all of the datasets. Meanwhile, as demonstrated in Fig. 6 the optimal performance of OPT 125M is better than OPT13p -2 on 3 VQA tasks.
7

---

Table 4: Comparison between models built with our VPGTrans and the original BLIP-2 ViT-G 67 and BLIP-2 VIT-G Flan
VQAv2 val
GQA test-dev
OKVQA test
Models
GPU hours
training data
BLIP-2 ViT-G OPT6.7B
54.3
36.4
36.4
631.5
129M
BLIP-2 ViT-G 7PT2.7B-678 (ours)
57.2
36.2
39.8
59.0
13,8M
VI--LLaMA71 (ours)
58.1
37.5
37.4
67.1
13.8M
BLIP-2 ViT-G F5xx.
65.22
44.7
45,9
684.0
121.6M
BLIP-2 ViT-G FlanT5xx XX-XXXX (ours)
65.2
45.0
45.0
32.4
5.3M
OPT RanTSpus Sqare
OP Tssae Faan
OPT Funt
OPTsee PlanT
OPT
Flant
37.9
263
40
325
313
2T0 borm sczntch
from acraldh
Iran: from
Irar
From sontchh
VPG
VPCCTnaaa
VPC Tara
102
VPCT
opochs
opochs
opochs
epcots
opochs
Captin
NoGaps
VGNv2 FlantSon GPT,
GGA TlenTfcc DPT;
CKVVAA
110
T5
Fiwiihc
22
110
VPGT
turm scrtcch
VPCTTme
from scraddh
Far from scrrch VPGTTw
Irar scratee VPGTrar
VPGT
Foom sortich
opochs
opochs
000chs
opccts
spochs
Figure& 8: Comparison between different methods across TaT variants on 5 tasks. Note that the model is directly evaluated after pre-training without further fine-tuning. Please refer to Appendix D for other transfer variants.
We hypothesize that training VPG on larger OPT will have a worse influence on VPG's existing fine-grained perception ability, which might be caused by the enlarging embedding dimensions. To validate our hypothesis, we fix the VPG weight and only tune linear projectors to test VPGs trained on different L.LM are through cross-size transfer. The SPICE evaluate the VPG's visual perception ability, where SPICE is specifically designed for visual concept perception in captions. As shown in Fig. 7 for each row, given the LLM the performance of VPG trained on smaller LLM,re can outperform the larger ones in most conditions, which indicates a better visual perception ability of VPG trained on smaller LLM Therefore, adapting a VPG from a smaller OPT model which is less affected, is helpful to take fewer steps to reach the TFS's best performance and achieve even better performance.
metric on COCO caption is used to
4.3 Scale-up Experiments
oFT coPTaue CaTTce
0.0
23.1
21.9
21.5
To validate the effectiveness of our VPGTrans on the real-world application level, experiment on transferring from BLIP-2 ViT-G OPT2m to OPT,,7 and from BLIP-2 ViT-G Flan to F5 Please refer Appendix& for implementation details.
22.6
2.1.2
19.8
13
24.22
23.6
23.3
Speed-up with non-degenerated performances. As shown in Ta- 4 we can see that (1) OPT our VPGTrans achieves a 10.7 times speed-up with only 10.7% training data, while the perfor- mance on VQAv2 and OKVQA have over 2 points improvement. (2) Flan F -xx- VPGTrans can achieve 21.1 times speed-up with less than 5% training data while achieving the same performance on VQAv2, higher performance on GQA and slightly lower perfor- mance on OKVQA. Note that continuing training the Flan only shows improvement on VQAv2 and GQA. Thus, we do not show checkpoint with more training steps.
22.9
9
orion
LIM..
Figure 7: The confusion ma- trix. Only linear layers are trained for VPG evaluation. Models are tested on COCO caption with SPICE metric to compare the VPGs trained on different LLM
8

---

Table 5: Comparison between our VL-Vicuna and SOTA MLLMs for multimodal conversation. The
evaluation done by Multimodality Chatbot Arena platform via user voting.
Rank
Model
Elo Rating
LLaMA-Adapter v2
17
1023.0
LLaVA 35
1019,9
VL-Vicuna (ours)
1012.1
MiniGPT-4 [62
1011.9
InstructBLIP
999.5
mPLUG-OW/ 59
996.3
Otter [26]
981.5
BLIP-2 30
955.8
5 Exp-lI: Transfer across Different Model Types
In this section, further investigate the transfer across different model types. For simplicity, mark this type of transfer as TaT.
5.1
Experimental Settings
In this part, we introduce baselines and transfer variants. For details about training data and imple- mentation details, please refer to the experiment settings in the Preliminary (ef. 2.2
Baselines. We mainly compare VPGTrans with training from scratch (TFS), and report the performance on the aforementioned tasks. Other details (cf. 4.1) are totally the same with TaS experiments.
Transfer Variants. We conducted experiments transfer learning between 1) OPT35uM and FlanT5 and 2) OPT27B and FlanT5xL.
5.2 VPGTrans Enabling Faster Convergence only on Large LLMs under TaT
There no speed-up of TaT between two small LLMs. A finding is that on TaT our VPGTTrans does not show speed-up for small models, and even shows degeneration of training speed in the initial several epochs. As shown in Fig. 8 when transferring from 350 to FlanT5base the convergence speed of VPGTrans
slower than TFS in the initial several epochs.
Speed-up of VPGTrans happens in large LLMs. However, when moving to the large LLMs like OPT2,7 and FlanT5xx. there is an obvious speed-up. As shown Table we can see at least 2 times speed-up when transferring from Flan'T5x1. to 2PT We empirically find that the soft prompts for larger LLM are more linear transferrable among different LLM types. As shown in Fig. g when transferring between Flan'T5 and OPT 350M, the VGPTrans' st epoch results datasets are limited, where only a linear operation can be trained. The result of on the COCO caption is even near zero. By contrast, the result of Flan -PP5x with our VPGTrans are obviously higher than TFS. We hypothesize that larger LLM typically learned more generalizable text embeddings and share more similarity among relative word distances, which enables an easier VPG transfer.
caption
6 Customizing New MILMss with Any LLMs
Above, thoroughly certify the efficacy of our proposed VPGTrans approach for higher efficient transfer of VPG. In this section, we illustrate how to apply the VPGTrans framework for VPG transfer customize new MILMss with any LLMs.
By applying our VPGTrans, we can equip the recently released LLaMA 53 model with VPG trained on BLIP-2 OPT6.7 to perceive the visual information. As shown Table we see that VL-LLaMA can outperform the original BLIP-2 OPTs.7B all datasets.
VI.-LIaMA
VL-Vicunal An exciting application of our VPGTrans is to build GPT-4 g style multimodal conversation chatbot. To achieve our goal, we employ Vicuna [10] as our base LLM. Similarly, we transfer the VPG from BLIP-2 OP. and add an extra instruction tuning using MiniGPT-4's self- instruct data 622] We compare our model with MiniGPT-4 in Fig. 9 When compared MiniGPT-4, VL-Vicuna shows better visual perception ability. Please refer Appendix B for more cases. In

---

MiniGPT-4
VL-Vicuna
How people are in the image? 8
How many people are in the image? g
There three people in image: the batter, catcher, and the umpire.
people image.
they doing?
they doing?
The two people the baseball standing both holding One person is the pitcher, and the other the catcher. The pitcher holding the baseball bat preparing throw catcher standing bahind home plate and holding the giove, ready to catch the ball
The better is standing batter's holding bat, waiting pitsher throw ball. The catcher standing bahind the plate, holding the catcher's mitt, and ready to catch the ball umpire standing the catcher, hoiding the umpiie's is mask, and ready make call balls and strikee.
catcher.
Tell anything you know about the man the image.
Tell me anything you know about the man the image.
&
The man the image wearing gray sweater and altiing in front of fireplace. He has serious expression
The man the image an actor named Leonardoo DiCaprio, He
November 1974, his
the distance. The room appears to be dimly with shadows cast on the walls floor. There is sense stillness quiet
looking
Angeles, California, USA. roles in fimm such as Titanic, Inosption, and The Walf of Wall Street. DiCaprio acting, including Academy Award for Best Actor for his role in The Revenant.
several
Figure 9: Comparison between MiniGPT-4 and our VL- Vicuna.
addition, we also report the ranking and ELO ratings for our VL-Vieuna compared with the other 7 SOTA multimodal chatbots in Table 5 (The evaluation is done by Multimodality Chatbot Arena platform The results show the effectiveness of our VL-Vicuna compared with existing SOTA models.
Conclusion
this work, we conduct comprehensive investigation to the problem VPG transferability across LLMs. We first explore the key factors for maximizing the transfer efficiency under the VPG transfer across different LLM sizes and types. Based on the key findings, we propose a novel two-stage transfer framework, namely VPGTrans, which can help to achieve comparable better performance while significantly reducing the training costs. Moreover, a list of important findings and possible reasons behind them shown and discussed. Finally, demonstrate the practical value of VPGTrans, by customizing new MLLMs via VPG transfer from existing MI..Ms.
Acknowledgments
This research is supported by NExT+ Lab. Singapore Ministry of Education Academic Research Fund Tier 2 under MOE's official grant number T2EP20221-0023, and CCF-Baidu Open Fund.
References
[1] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In Proceedings of the
10

---

IEEE/CVF international conference compuler vision, pages 2019.
[2] Jean- can--Batiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23777 2022.
[3] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part V 14, pages 382-398. Springer, 2016.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick. and Devi Parikh, Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425 2433, 2015.
[5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2af::22200.822. 2022.
[6] Andy Brock, Soham De. Samuel L Smith, and Karen Simonyan. high-performance large-scale image recognition without normalization. International Conference on Machine Learning, pages 1059-1071. PMI.R, 2021.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamners. Advances in neural information processing systems, 33: 13:188 877-1901, 2020.
[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303. 12712, 2023.
[9] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, The Gan, Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer ViCi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer. 2020.
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL bttpp:tt////vcuua lmays. .mag
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[12] Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling preprint 100: 1416, 2022.
language models. arXiv
[13] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructhlip: Towards gencral-purpose vision-language models with instruction tuning. 2023.
[14] Mingkai Deng. Jianyu Wang. Cheng Ping Hsich, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint or 22v: 12548, 2022.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint MXX0::0010011922 2020.
[16] Danny Driess, Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery. Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.0333
2023.
[17] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui Hc, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-cfficient visual instruction model. arXiv preprint arXiv:2304.1 15010, 2023.
[18] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks. Johannes Welbl, Aidan Clark, et al. Training compute -atputae large language models. arXiv preprint 2rXiv:2203.15536. 2022.

---

[19] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprini arXiv:2302. 10.00..4400 14045, 2023.
[20] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[21] Andrew Jacgle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General pereeption with iterative attention. In International conference on machine learning, pages 4651- 4664. PMLR, 2021.
[22] Menglin Jía, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27. 2022, Proceedings, Purt XXXIII. pages 709-727. Springer, 2022.
[23] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Perey Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022.
[24] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0866 2021.
[25] Brian Lester, Joshua Yurtsever, Siamak Shakeri, and Noah Constant. Reducing retraining by recycling parameter-efficient prompts. arXiv preprint 20Xiv:200..05577, 2022.
[26] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.0377 2023.
[27] Juncheng Li. Kaihang Pan, Zhiqi Ge. Minghe Gao, Hanwang Zhang. Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, and Yueting Zhuang. Fine-tuning multimodal Ilms to follow zero-shot demonstrative instructions. arXiv preprint arXtv:2308.04 23...030..42222 04152, 2023.
[28] Junnan Li, Ramprasaath Selvaraju, Akhilesh Goumare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
[29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified lision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900 PMLR, 2022.
[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[31] Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. In Advances in neural information processing systems, 2023.
[32] Pandeng Li, Chen- Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge. Yun Zheng, Deli Zhao, and Yongdong Zhang. Progressive spatio-temporal prototype matching for text-video retrieval. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 4100-4110, 10, 2023.
[33] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning, arXiv preprint arXiv:2210.088:
2022.
[34] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuming. 2023.
[36] Jiasen La, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- lannguuge- tasks. Advances in neural information processing systems, 32, 2019.
[37] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK- VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/eyf conference on computer vision and pattern recognition, pages 1195-3204, 2019.
12

---

[38] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young. Lucy Giilinghm Geoffrey Irving, al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203. 11147,1111211 2022.
[39] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0933. 2021.
[40] Vicente Ordonez, Girish Kulkami, and Tamara Berg, 1m2text: photographs. Advances in neural information processing systems, 24, 2011.
Describing images using 1 million captioned
[41] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[42] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer lcarning with unified text-to-text transformer. The Journal of Machine Learning Reseurch, 21(1):5485-5511 2020.
[45] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tlya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831, PMLR, 2021.
[46] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint 2:040:200. 2022.
[47] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[48] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphacl Gontijo Lopes, Burcu Karagol Ayam, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3644 -64944, 2022.
[49] Mehdi SM Sajjadi. Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-9524, 2022.
[50] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: (eo::0908000333 2019.
[51] Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3949-3969, 2022.
[52] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 2023.
[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ama Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint ar0iv: 13971, 2023.
[54] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 200-212, 2021.
[55] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4566-4575, 2015.
13

---

[56] Wenhui Wang, Hangbo Bao. Li Dong. Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed. Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for all vision and vision-language tasks, arXiv preprint arXiv:2208. 10442. 2022.
[57] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Emergent abilities of large language models. arXiv preprint Xix::2206.076522 2022.
[58] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 2022.
[59] Qinghao Ye. Haiyang Xu, Guohai Xu. Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
[60] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diah, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205. 01068, 2022.
[61] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision risnun-language models. International Journal of Computer Vision, 130(9):2337 2348, 2022.
[62] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, MiniGPT-4; Enhancing vision-language understanding with advanced large language models. arXiv preprint wxXv:2304.1059/ 2023.
14

---

A Related Work
A.1 Vision and Language Models
aim at understanding visual and textual information with single model. Previously, the VLM mainly employs a pre-trained object detector as its feature extractor and conducts unsupervised training on huge amount of image-text pairs. For example, VL-Bert [50], ViL-Bert (36) and Uniter [9] adopt Faster-RCNN 47 to extract image information into object features and take advantage of the masked language model as their pre-training task. Later, due to the prevalence of vision transformer (ViT), the VLM paradigm is turned into end-to-end training with a VIT as the visual encoder. The representative works include ALBEF [28], BLIP [29), and BEIT-v3 [56), which show the state-of-the-arts supervised training performance wide range of downstream tasks.
Vision and language models (VLMs) 50
29 32
Recently, LLMs have shown their remarkable capability as zero/few-shot learners 7 and series of emergent abilities [57] like in-context learning 77], and thain-of-thought reasoning (58) A new paradigm, i.e., MLLMs is created by associating the VLM or pure vision encoders with LLMs. As we illustrated before, the VLM or visual encoders are typically able to convert the input vision signals into LLMM-uderstaandbble soft prompts, and thus we call them VPG. The MLLMS advance in inheriting the great potentials of the backbone LLMs, and thus are capable of achieving excellent zero/few-she performances 2 30 on downstream tasks or be equipped with visual planning ability [16]. However,
connecting the VPG to the existing LLMs with further tuning is costly. Even the BLIP-2 [30], targeted at efficient training, will take over 600 A100 GPU hours on over 100M image-text pairs for its largest model. With this regard, our proposed VPGTrans can effectively reduce the cost of building new MLLMs with the help of existing ones.
A.2 Prompt Transfer
In this paper, we investigate the VPG transfer, where the soft prompt is to represent the content of specific inputs like images and videos. In addition to the content prompt, the more explored soft prompt is the task prompt [24] 61 22]. where a sequence of soft prompts are tuned to assist the pre-trained models to achieve better performance on specific tasks. There have already been some works exploring the transferability of task prompts. For example, Su et al. [51] conducts a series of experiments to illustrate the transferability across tasks and models. Specifically, Su et al. (51) find that the transfer between similar tasks is beneficial for training speed-up and better performance. Lester et al. [25] proposes to recycle soft prompts across models with vocab-to- vocab transformations, or comainatmiinnioon transformations. Note that our word converter initialization is similar to the do not observe zero-shot transfer in our
idea of vocab-to- vo-ccb transformations. However, experiments like them, which indicates a potential difference between the content prompts and task prompts. Another way of soft prompt transfer [4] is conduct prompt tuning on discrete prompts, and thus the discrete prompts can be directly shared across models. Different from these task prompts transfer works, our VPG transfer scenario actually suffers from fewer limitations. For example, the task soft prompts transfer suffers from the dimension change problem, where the main body of the trainable parameters should be processed. However, our VPG (the main trainable parameters) can naturally be shared among LLMs with different embedding dimensions and leave the dimension change problem to a simple projector with ignorable parameters.
B Extended Findings in Exploratory Analysis
In this section, we show extended findings of exploratory analysis (cf. 3.1
1. Merely tuning the projector can not achieve the best performance. We want clarify that merely tuning the projector is insufficient for achieving the best performance. Notably, as shown in Tig.[0) significant performance gaps are observed between the "only linear" (green eurve) and "train from scratch" (orange curve) approaches for COCO caption and NoCaps. Therefore, if the goal is to build multimodal conversation robot using carefully collected dialog data, training only the linear projector is insufficient to align with the provided data.
15

---

COCO Capddn
NoCape
VOAW
044
ORYGA
150
VPQ frmm scrsion
VPO shetl taa from
imtel:
T987 szmtch
scrith
7081 lize
unly leuar
only
crty Iranr
only Trmar
onty
Figure 10: Comparisons between inheriting VPG from OPT 125M and training with randomly initialized projector for 350M and ii) training VPG and randomly initialized projector for OPT from scratch. iii) training only the projector.
Gcontainer
Gtable
Gflowers
EOS
Figure 11: Interpreting generated soft prompts for 12T with nearest words.
2. Word embedding converter can not replace a trained linear projector. As demonstrated in Fig. [T] we observe common pattern: the last token of soft prompts is closest to EOS, while the middle tokens represent the image content. Such phenomenon indicates similarity between soft prompts and word embeddings. However, they are not identical. For instance, the norm soft prompts is typically around 10 times the average norm of word embeddings. It is important to note that the linear projector initialization cannot replace the warm-up training. Using only the lincar projector initialization even yields random performance. We believe that a better understanding of how prompt works will further benefit the VPG's transfer learning.
3. The projector warm-up is robust to a larger learning rate, while VPG can not. The first thing we want to clarify is that the 5 times normal learning rate will result in a training crash for VPG. Additionally, we find that although increasing the learning rate to 10 times in the projector warm-up does not yield any additional acceleration, the projector can converge without crashing during training.
C Extended TaS Experiments
In this section, we first illustrate extending findings of TaS experiments (cf. $4. Then, we introduce the implementation details of scale-up experiments. We also plot more complete version of Fig. 6 Fig.
C.1 VPGTrans Enabling Stable Training under TaS
As illustrated before, FlanT5 training is extremely unstable. Even when the learning rate is adjusted to one-tenth of its original value, the model does not converge or shows a very slow convergence rate after 4 epochs. However, we find that by lowering the learning rate for stage-2 training, our VPGTrans can achieve stable training on Flan'5 We plot the performance curve of the COCO caption in Fig. I2
C.2 VPGTrans Enabling Training with Less Data under TaS
We empirically find that TaS can reduce the requirement for the amount of training data. By reducing the training data only COCO, we find no obvious performance drop. However, we want to stress that the retained data should be of high quality, which means that if the same number of SBU data
16

---

FlanT TaanTppaee large
COCO Caption
140
120
acc.
train from scratch
VPG inherit
20
VPGTrans
0
3
6
epochs
Figure 12: Comparison of training FlanT5 using different methods.
Table 6: The speed-up up rate of our VPGTrans compared with VPG inherit. The symbol "_" means VPGTrans can not achieve better performance than VPG inherit.
Transfer
COCO Caption
NoCaps
VQAv2
GQA
OKVQA
OPT 125M
125M-330M
1.1
2.7
10.0
6.0
6.0
OPT 125M
2.7
350
1.7
1.0
2.0
1.0
125M
3.0
3.0
3.0
1.0
3.0
3 350M +
3.3
4.5
1.0
1.0
1.0
OPT 1.3 +2.7B
2.3
3.0
Flan'l T ThanTTp base-xXL
1.0
1.8
9.0
0.4
retained, the performance especially for captioning will drop. The conclusion can also found in We refer the readers the next subsection for more details.
Table [
C.3
Comparison between VPGTrans and VPG Inherit
As shown in Fig. 13 the green line (our VPGTrans) can be higher than the orange line (VPG inherit) for the majority of various conditions. Especially when considering the best performance of different tasks, our VPGTrans can achieve better performance than VPG inherit (non "_" in Table (6) for over 74% Transfer-Tas variants. Moreover, among the variants that our VPGTrans can achieve better performance, our VPGTrans can also achieve a speed-up on 69.2% conditions.
C.4 Scale-up Experiment Implementation Details
The scale-up experiment refers to the results in Table 4 We try to imitate BLIP-2's pre-training data composition. First of all, two human-annotated datasets COCO and VG are used. SBU also used. Then, BLIP-2 uses BLIP to generate captions for the 1 115M web images and rank them with CLIP ViT-L/14. We also adopt similar synthetic data from Laion-COCo We report the concrete number of data use in Table For the stage-1 training, we keep the same the previous validation experiments where COCO and SBU are used for warm-up with a 5 times the learning rate. Then, COCO, VG, and Laion-COCO for the stage-2 training. Note that have tried include Laion-COCO and VG for the stage-1 training, but found no obvious difference and thus use COCO and SBU for simplicity. For VL- Vi-Vicuna. align with the conversation scenario, we further fine-tune our VL-Vicuna with MiniGPT-4's self self-instruct data, which is 3,439 image-text pairs.
http:::/haaoon.aa/llelllaaa
17

---

OPTTAM..W0M Caplin
OPT JwM NoCap
CWe VONe
135
120
30
0.2
ban cadda
bal Fire sietin
Talr sorete
Tile Servodal
VPCTars
VISTtera
COCO Cuptor
Offizie NoCapa
VOA2
Tase 1120 09A
OF Tarae
OW/GA
140
125
115
VPC imst
VTG Inwe:
VTG Innt:
30
VPG inat
VPO ihert
VTOTTer
WPOThers
spocta
spochs
speets
opocss
S NoGapa
cum.c. VOM2
L20 09A
Loom-1ar OR/GA
15a
190
139
sal Fux eassa
Tair
193
OPT 0P710-222 0000 Cuptin
OPT NaGara
OPI 12 VoAe
OFT UFT 00A
OPT OPT OXVGA
130
8
115
tat sosin
form
horvonatin
VP0 invtt
lewrt
140
OFT
00C0 Copt
OPT NoCaps
CPT VDA22
CF Q2A
OP OPTaoee... OR/GA
34
8
190
LPG inelt
VPQ
VPQ inwi
VPG
WesThers
apocht
apocte
apochs
4pocss
OFT, 222277
0000 Copt
OPT NcCapa
CPT VOAV
CFT, aP720-220 ODA
22:1 DIVGA
195
125
115
zat for scatch
zan for acaic
zair tos scurr
cair forn scruct VIG iewt
TET forvic
LPC Imst
VTG lihtt:
105
120
0000 Captio
BariTaue KeCaage
4.00
RattTurr.e Vole
Fiet 00A
Slitiarr DKVOA
120
139
5000
190 uret
VR0 stve
VTOTTer
spots
spochs
apoces
Figure 13: Comparison between different methods across 7 TaS variants on 5 tasks. Note that the model is directly evaluated after pre-training without further fine-tuning.
18

---

Table 7: Comparison between tuning only the projector (i.e. linear transfer) and the best results the VPGTrans achieve.
COCO Caption
VQAv2
Transfer
linear
best
linear
best
Flan hase OPT
110.8
136.5
40.4
44.2
Flan'T T5xL. -> OPT227
132.1
139.3
50.4
50.3
350M 7 FlanT5
1.1
122.1
34.0
49.9
2.7 + Fxa
106.5
133.2
51.3
53.5
D Extended TaT Experiments
In this section, we mainly illustrate extending findings of TaT experiments (cf. complete version of Fig. in Fig. 14
( We plot more
D.1 Linear Transfer Gap between Different LLM's Visual Prompts
As illustrated in Section 5.2 is more difficult transfer between two small LLMs with VPGTrans due to the weaker linear transferability between two small LLMs' visual prompts. To better support results, compare the results of tuning only the projector (i.e. linear transfer) and the best results the VPGTraas can achieve. As shown in Table we can see that when conducting transfers between large models (OP and FlanTShase the performance is typically far from the optimal results. However, when we transfer between OPT2 and Flan the linear performance is
the optimal performance. There is 25.7 points gap between Flan 3 s linear and best on COCO caption datasets but only 7.2 points gap between FlanT5xx OPT27B's linear and best. If considering the transfer between the small to large LLMs under TaT, both the VPG's visual perception ability and transfer gap should be considered. We leave the systematical exploration for future works.
E Extended Results for MILM Customization
We show more comparisons between VL- VL-Vicuna and MiniGPT-4 in Fig. 15 We can that VL- VL-Vicuna has better visual perception ability. For example, when MiniGPT-4 falsely recognizes the three people in the image as two in the first example, our VL- VL-Vicuna number of people but also tell their roles. Moreover, our VL-Vicuna can successfully link the vision content with external knowledge. In the third example, our VL- VL-Vicuna recognize Leonardo and link the content with his films like Titanic.
not only recognize the
Potential Impact and Limitations
Our VPGTrans is designed for building new MLLMS with lower computational cost, i.e., shorter training time and less training data. With an already pre-trained MILM, VPGTrans enables fast VPG transfer to build either a larger MLLM or MLLM with different type of LLM. We hope VPGTrans can facilitate teams in LLM communicty to customize their MLLMs with reduced cost. There are also possible limitations of the current version VPGTrans. The first one that our VPGTrans should rely on some already-aligned VPGs. The second potential limitation is that the VPGTrans-buile MLLMS still suffer from the common problems of content generation AI systems 45
46 48 77.
For example, the VL-Vicuna may make up some sentences with falsely recognized visual facts, like what shown in Fig. [6 is worth exploring associating our VPGTrans with training safer
models 5 41 38
19

---

COCO Gapdos
NaCaps
VGAV
GDA
CKOVOA
OPT
PlanTh
FantTuure OPTw
PaniTaa
134
100
122
g
115
7an Fam scratch
1281 from scrathh
Vain from seuzch VPGTT
trair from scraten VINGT
Tun fun soutch
VPG
VPGT
epords
spachs
moodhs
spodit
00C0 Capdio
NeCape
VoAve
GGA CPTas Flan Paace
OKVGA
DPT Laav Hlanl
OP Tasae - Han
CPT aac
Flan Daas
DPT sup - Plant
104
40.3
37.
205
175
20
from
VPOTiaro
VPCTions
VPCTiars
VPOTTun
25.
epodfu
spacta
020211
specht
0000 Capdon
NoCope
VoNV GP7).um
GGA
OKVVGA
FlarTEu:
22
258 Iom scratch
tran from scha
Faan from scruch VPCThws
tra- from scratee
T'an from sccatich VPCTaM
VPSTarn
VPCTTras
VPGTiare
25
spochs
spochs
spochs
CoCo Captas
NaCape
VOAU OPT
GDA OPT2Tc FluntSys
DKVIDA
OPT2.0 Faa175p
OPT..0 Par
Flan T5
OPT..u Fla175g
100
112
12
72
050dto
spochs
020dhs
050ors
spochs
Figure 14: Comparison between different methods across 4 TaS variants on 5 tasks. Note that the model is directly evaluated after pre-training without further fine-tuning.
20

---

MiniGPT-4
VL-Vicuna
Pike
Pike
What is written midcle sign?
g
written in middle sign?
The sign says "Pike Place" and points the direction building background.
The middle says "Pike Pl'
g
What does this mean?
What does this mean?
sign indicating which in likely a street or location the city. The arow the sign building in the background, which is likely a high- nse apartment or office building.
"Plke Pi" likely refarence the Intersection of and Streets downtown Seattle, Washington.
the direction the
How to cook this dish?
dish, you ingredients: broccol, olive oll, garlic, salt, pepper, and red pepper flakes. You will also need pan or pot cook the brocooli in.
following
To cook this dish, you will need the following olive oil garlic, pepper. Here are the steps to cock the dish:
ingredients:
Cut the broccoli into small florets. Heat some olive oll pan over medium heat.
you small florets and steam it for about 5-7 minutes, until tender. While the broccoli cooking, you can heat some olive add minced garlic and sait to Once the you add the garlic and oll and stir together. You can then add some dish some heat.
Add garlic pen sauté fragrant.
pan and
Add the broccoli to the and toss to coat garlic oll. Season with salt and pepper to taste. the
broccoli
sprinkle of and lemon juice on top.
flakes to the
Figure 15: Comparison between MiniGPT-4 and our VL-Vicuna.
Pleese describe the Image.
8
When now in the image?
The image is of a three-tiered wedding cake on a wooden lable. The cake decorated with while frosting and berree, table cluttered plates, silverware, and vase of flowers. The room appears reception
I'm sory, an language model, don't have the abilly to delermine the current time based on image you's provided. However, that the image appears be bar or restaurant with clock the wall.
tell you
banquet hall. The lighting is bright and wall-itt casting shadows the table and the cake.
Figure 16: Failure cases of our VL-Vicuna
21


Reviews:
[{"review_id": "HKTyaEo77v", "reviewer": "Reviewer_7KJM", "summary": "The paper presents a technique for VPG (Visual Prompt Generator) transferability across LLMs where the transferability can be between LLMs of different sizes (eg: OPT125M -> OPT2.7B) or across LLMs of different types (eg: OPT350M-> Flan T5base). This is achieved using a two stage strategy where a projector is learned by freezing VPG and LLM in the first stage and vanilla finetuning of VPG and projector in the second stage. The proposed technique significantly reduce training time (specifically when transfer from small to large LLMs).", "questions": "I would like authors to discuss the three points raised above : 1) Novelty, 2) Generalization, [3] Robustness", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The VPG transferability can significantly improve the training time/cost in some cases (from LLM_small->LLM_large)  can have many practical applications when training larger LLMs.\n2. The proposed technique consistently reduces the training time/cost compared to baseline (TFS) while achieving similar performance", "weaknesses": "Novelty: The proposed technique is a standard technique used in fine-tuning a model for a specific task where the newly added layer (head which is similar to projector here) is initially trained with backbone frozen and once trained, the whole network (backbone and head) is trained/fine-tuned on the target data. Please see [a, b] for references that talk about standard fine-tuning strategies. The authors extend this to VPG transferability between two LLMs. Although, the technique is good for practitioners, I don't see any technical insights in the proposed technique.\n\nGeneralization: Since the proposed technique uses a limited vocabulary from the datasets COCO, SBU-captions [1.4M image-caption pairs] for the word converter, the performance of the target LLM on the datasets with domain GAP could be affected. It wold be interesting to see the performance of the target LLM (and comparison with source LLM) on domain specific datasets such as RSVQA [c] and PathVQA [d] datasets. \n\nRobustness: To check the validity of the proposed technique, It would also be interesting to see the affect of transferability on Robustness of the target model which [target-LLM compared src-LLM] can be done by evaluating on datasets AdVQA [e], VQA-CE [f] or VQA-Rephrasing [g]\n\n\n[a] https://cv-tricks.com/keras/fine-tuning-tensorflow/\n\n[b] https://lightning-flash.readthedocs.io/en/stable/general/finetuning.html\n\n[c] https://rsvqa.sylvainlobry.com/#overview\n\n[d] https://paperswithcode.com/dataset/pathvqa\n\n[e] https://adversarialvqa.org/\n\n[f] https://paperswithcode.com/dataset/vqa-ce\n\n[g] https://facebookresearch.github.io/VQA-Rephrasings/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and the careful review. Your suggestions will definitely help improve our paper. Following, we present the response to address your concerns. And if you feel our responses effectively relieve your concerns, please kindly reconsider your evaluation.\n\n---\n**Q1: Novelty: The proposed technique is a standard technique used in fine-tuning... Although, the technique is good for practitioners, I don't see any technical insights in the proposed technique.**   \n**A:** While it is a fact that the transferring technique in our work coincides with the existing standard method for the practices of transfer learning, we would like to argue that our work is far more than the simply leverage of the model transfer itself.   \n- First, we have actually made certain technical improvements (simple yet effective) for the scenario of LLM transfer over the existing 'standard transfer learning'. For example, we devise the **word converter based initialization** (cf Sec. 3.2), without which, the warm-up of the projector will be 1.5 times slower. On the other hand, getting rid of complicated architectural design, we rather believe being simple to use to achieve prominent efficiency improvement is a merit of our method.\n- Second, compared to the technical innovation, our key contribution/novelty more lies in the experimental explorations. We first performed systematical and in-depth analyses to confirm the feasibility, before we decided to conduct the VPG transfer and propose our method. We note that, without such preliminary explorations, the direct transfer of VPG will be significantly less effective compared to ours.\n- Third, we for the first time presented a rich amount of intriguing findings/insights and meaningful explanations for the scenario of VPG transfer across LLMs, which will largely pave the way for all the furture explorations in this topic. For example, we show that, for OPT based VL-LLMs, the VPG trained on smaller src-LLM typically yield better transfer result.\n- Finally, our contributions to the community are substantial and evident, especially for the practical applications in production environment. Our VPGTrans enables building new VL-LLMs with much reduced cost (e.g. 10% cost), making it possible to **build new VL-LLMs with lab-level resources**. The VL-LLaMA and VL-Vicuna presented in our paper are good examples. For example, for some startup companies want to build their own VL-LLMs, our VPGTrans offers a valuable path to **save thousands and millions of dollars**.\n\n---\n**Q2: Generalization: Since the proposed technique uses a limited vocabulary from the datasets COCO, SBU-captions for the word converter, the performance of the target LLM on the datasets with domain GAP could be affected. It would be interesting to see the performance of the target LLM (and comparison with source LLM) on domain-specific datasets such as RSVQA and PathVQA datasets.**   \n**A:**  Thank you for carefully going through our paper. We want to clarify that our model is also trained on millions of web images (similar to BLIP-2's data composition) during **VPGTrans stage-2** (after the word converter training), i.e., the model sees data in multiple domains, which ensures a good generalizability over domains.\nMoreover, the fact is that boosting the amount of training data of word converter can be quite cheap and effortless. For example, using 100M data for 1 epoch training will only take **<40 min** consumption with one A100 (80G).\n\nDuring the rebuttal days, we run the experiments on domain-specific datasets for the LLMs. The experiments are with FlanT5-XL (src) and FlanT5-XXL (tgt):\n\n| Model | Data   | PathVQA  (yes&no acc. / F1) | RSVQA (acc.) |\n| :-------:|:--------:|:-----------------------------:|:--------------:|\n| src   | 121.6M | 50.6/27.2                   | 40.3         |\n| tgt   | 5.3M   | 51.2/26.9                   | 41.7         |\n| tgt   | 9.4M   | 51.3/27.0                   | 44.0         |\n\nAnd we find that our model can achieve even higher/comparable performance on these two datasets, compared with the source model.\nMoreover, we notice that including more data can further improve the VQA results on other domains.\nThanks for indicating this, and we will consider adding it into revision.\n\n---\n**Q3: Robustness: To check the validity of the proposed technique, It would also be interesting to see the effect of transferability on Robustness of the target model which [target-LLM compared src-LLM] can be done by evaluating on datasets AdVQA, VQA-CE or VQA-Rephrasing.**   \n**A:** Thanks for the suggestion, good idea.\nAlso, we run the experiment on the datasets you indicated, and the results are as follows. Likewise, the experiments are with FlanT5-XL (src) and FlanT5-XXL (tgt):\n\n| Model | Data   | AdVQA (acc.) | VQA-CE (acc.) | VQA-Reph. (acc.) |\n| :-------:|:--------:|:-----------------------------:|:--------------:|:--------------:|\n| src   | 121.6M | 37.2         | 36.7          | 60.2             |\n| tgt   | 5.3M   | 39.7         | 39.0          | 62.8             |\n| tgt   | 9.4M   | 40.3         | 39.4          | 63.2             |\n\nCompared with the source model, our tgt model can achieve better performance on all of three VQA datasets, demonstrating its capability on robustness.\nThis part will be added into revision, thanks again.\n"}, {"review_id": "YpIiKAMpxi", "reviewer": "Reviewer_GwBd", "summary": "* This paper presents an interesting study on how to effectively transfer the small VL model to the large VL model, which is very practical under the fact that the LLMs are very expensive to finetune.\n* The empirical study shows the effectiveness of the methods.", "questions": "* It is better to include the GPU hours and training data of a small model for a fair comparison\n* Is it possible to inject the Alignment loss during the tuning projector stage?\n", "limitations": "see above", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper proposes an effective word converter to better speed up the model training on a large model \n* The stage-wise training gives good results for the final performance.", "weaknesses": "* The paper writing is very complex to understand\n* This technical contribution is very limited. The core is word converter, which better aligns the knowledge between large LLMs and small LLMs. \n* The performance improvement is very obvious. Because the whole model training involves the small LLM, which contains more model parameters. \n* The whole training pipeline is very complex. There are multiple stage for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable suggestions. Following, we present the point-to-point response to address your concerns. And if you feel our responses are effective, please kindly raise your evaluation.\n\n---\n**Q1: The paper writing is very complex to understand.**  \n**A:** Thanks for your comment. We'd like to make a clarification on our paper organization.\nDifferent from the traditional _introduction-method-experiment_ structure, we adopt an **exploratory structure**.\nThe exploratory structure aims at deriving the method from a series of exploratory experiments over existing materials, which should be a merit for the topic of our focus.  \nAs admired by the Review#mi8b (Strengths 2), _'the paper reads like an investigation, from the shallower to the deeper, feels more organic and the lessons learned along the way are insightful'_.\n\nTo make it easier to follow for you, we list the outlines of the paper:\n- Sec. 1: Motivations and paper summary;  \n- Sec. 2: Descriptions/preliminaries of VL-LLMs;\n- Sec. 3: Exploratory analysis $\\to$ Our proposed method;  \n- Sec. 4: Experiments for cross LLM sizes transfer;  \n- Sec. 5: Experiments for cross LLM types transfer;  \n- Sec. 6: Building new VL-LLMs with our method.  \n\nWe will further revise the section titles to make each part clearer.\n\n---\n**Q2: This technical contribution is very limited. The core is word converter, which better aligns the knowledge between large LLMs and small LLMs.**  \n**A:**  We kindly re-emphasize our major contributions as to offer a thorough investigation to demonstrate the feasibility of VPG transfer for efficient accessibility of VL-LLMs, and devise a novel transfer method VPGTrans. In summary, our contributions are multifaceted:\n\n- We propose a VPGTrans framework, which significantly reduces the cost of building new VL-LLMs to lab-level resources.\n- We reveal intriguing insights and findings of the VPG transfer, and shed light on further research on this topic.\n- With VPGTrasn, we customize two novel VL-LLMs that are open-sourced for the community.\n\nThat being said, although not presenting ground-breaking technical novelty, we do contribute with innovations from the technical perspective of our proposed VPGTrans framework:\n- We proposed the _projector warm-up_ strategy, which helps achieve transfer with both high efficiency and high efficacy.    \n- We devised a novel _word converter_ method that can accelerate projector warm-up for over 1.5 times.  \n- We also proposed _training the projector with an extremely large learning rate_, via which we find can further reduce the warm-up consumption to 1/3.  \n\nMost importantly, we validate the feasibility of VPG transfer with systematical analysis via our VPGTrans system, where the findings help future researchers to obtain better practices in VPG transfer for building VL-LLMs.\n\nWe sincerely hope you can re-evaluate our contributions and the significance of this work.\n\n---\n**Q3: The performance improvement is very obvious. Because the whole model training involves the small LLM, which contains more model parameters.**  \n**A:** We respectfully disagree. The truth is, the **small and large models are trained and tested separately** in our scenario; but the scaling advantage you indicated in the comment mainly happens when conducting training and testing with one joint model.\nIn fact, it is non-trivial to conduct the transfer with high efficiency and efficacy. For example, the direct transfer (_VPG inherit_) even achieves worse performance than training from scratch.\n\nBut we think this is a wise question, and thanks for pointing this out. We will strengthen this point in the revision.\n\n---\n**Q4: The whole training pipeline is very complex. There are multiple stages for training.**  \n**A:** Our method actually contains two stages, and we don't think it is complex if we compare it with current VL-LLMs.\nRecent works on VL-LLMs like LLaVA (2 stages) and mPlug-Owl (2 stages) all entail 2 stages of training.\nIn addition, our source codes are made very easy and convenient for practitioners to implement.\nThus, there will be only very small cost to use our system and apply it into realistic applications.\n\n---\n**Q5: It is better to include the GPU hours and training data of a small model for a fair comparison.**  \n**A:** Thanks for the suggestion. The training costs are shown as follows:\n| Model                      | Src Model Cost (hours) | Transfer Cost (hours) |\n|:-|:-:|:-:|\n| BLIP-2 OPT 6.7B            | 631.5                  | 0                     |\n| VPGTrans OPT 6.7B (ours)   | 459.0                  | 59.0                  |\n| BLIP-2 FlanT5 XXL          | 684.0                  | 0                     |\n| VPGTrans FlanT5 XXL (ours) | 435.0                  | 32.4                  |\n\nThe total training data of the small model + transfer is the same as the larger ones.\nCompared with the one training from scratch, our VPGTrans can use even less time to train both the smaller and larger models.\nWe will add the table in the revision.\n\n---\n**Q6: Is it possible to inject the Alignment loss during the tuning projector stage?**  \n**A:** Thanks for the question, good idea!\nWe try to discuss different ways for the alignment loss.\nWe list the elements when conducting the transfer: (1) source VPG (2) source LLM (3) target LLM, where the alignment loss can be designed as these:\n- source VPG$\\to$source LLM: already aligned. \n- source VPG$\\to$target LLM: it is worth exploring the **alignment between visual soft prompts and word embeddings**.\nActually, current visual soft prompts (visual features) generated by VPG are not fully aligned with the word embedding.\nWhile the cosine similarity is high sometimes, the norm of two types of embeddings are quite different (Appendix line 519-520).\nThus, it will be interesting to explore the influence of alignment with different distance metrics.\n- source LLM$\\to$target LLM: it is what the word converter does.\nWe will discuss this in the revision."}, {"review_id": "iu01sTgfyz", "reviewer": "Reviewer_mi8b", "summary": "The paper discusses the transfer of a visual prompt generator (VPG) across different vision-language language models (VL-LLMs) to reduce computational costs. VL-LLMs include a VPG module that bridges the gap between vision and language, encoding visual inputs into fixed-length soft prompts. To reduce the cost of building new VL-LLMs, the authors propose a two-stage transfer learning framework, VPGTrans, for VPG transfer across different LLM sizes and types. VPGTrans includes a projector warm-up (stage-1) and vanilla fine-tuning (stage-2). They conduct exploratory analyses to determine key factors for transfer efficiency and show that VPG transfer across frozen LLMs is feasible and can lead to substantially lower computational costs. The authors demonstrate the practical value of VPGTrans by customizing two novel VL-LLMs with recently released LLMs. The paper highlights the need for further research into VPG transfer across LLMs to further reduce the cost of building new VL-LLMs. Overall, the paper's main contributions include identifying the feasibility and effectiveness of VPG transfer across LLMs, smaller LLM sizes leading to more speed-up and better performance during transfer across LLM sizes, and VPG trained on smaller LLMs outperforming larger ones in most conditions, and then propose a transfer learning framework with empirical evidence supporting its efficiency.", "questions": "See Weaknesses.", "limitations": "I do not foresee any potential for negative societal impact from this work.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The multimodal large language model (MM-LLM) is considered the most promising path towards achieving Artificial General Intelligence (AGI). However, a major challenge is the significant computational cost associated with training an MM-LLM. This obstacle hinders the widespread research and exploration of this topic. The presented paper offers an excellent pilot study that demonstrates the feasibility of VPG transfer. By showcasing the viability of VPG transfer, the paper provides valuable insights into mitigating the computational cost of training MM-LLMs. This finding has significant implications for future research in this field. The pilot study serves as a promising starting point, paving the way for further investigations and advancements in the development of MM-LLMs with higher efficiency.\n\n2. I quite enjoy the didactic/exploratory nature of this paper. I like that the paper reads like an investigation, and begins with a preliminary exp, as well as diagnosing reasons for the case, before then presenting the VPGTrans method. And then they delve into the transfer of VPGs under two settings. This, from the shallower to the deeper, feels more organic and the lessons learned along the way are insightful. Also the introduction with clear bullet point allows easy reading.\n\n3. The proposed method, VPGTrans, can be fairly straightforward but effective, which is a merit.\n\n4. The experimental work in this study is extensive and solid. It includes validation across a wide range of LLM sizes and types, providing thorough coverage.\n\n5. This work yields numerous interesting and meaningful findings, which will serve as important empirical evidence for future explorations in LLM/MM-LLM efficiency. The conclusions drawn from this research have the potential to guide and shape subsequent investigations in this field.\n\n6. Additionally, this paper makes good contributions by introducing two new state-of-the-art MM-LLMs in the community: VL-LLaMA and VL-Vicuna. Overall, I like this paper, and I believe this work will show important impact to the community.", "weaknesses": "1. The major possible limitation of this method is its reliance on the existence of a well-functioned VPG model, and without this assumption, this work is of less meaningfulness. How does the quality of the existing VPG influence the transfer efficacy and efficiency?\n\n2. This work evaluates on merely the captioning and VQA tasks. It would be interesting to see the performance on other different VL tasks and more benchmark datasets, to see the broad-coverage trends.\n\n3. On the other hand, the proposed VPGTrans, though straightforwardly simple, relies too much on the empirical tuning, e.g., when to inherit the raw VPG and when to tuning or fixing some modules. Are there any intuitions and theoretical supports to build up the method?\n\n4. When transferring the VPG, what has been changed of the visual soft prompts? Are the visual prompts of the raw VPG changed, to adapt to the new LLM backbone?\n\n5. The claim of ‘our VPGTrans helps achieve a BLIP-2 ViT-G OPT2.7B→6.7B transfer with less than 10% of the GPU hours and 10.7% training data required for the original model training’ can be a little misleading, as the comparison between VPGTrans and the training of the raw VL-LLM is not strict fair and counterpart; should compare the VPGTrans with the hard VPG transfer method and then draw the conclusion.\n\n6. Lack of exploration of potential limitations or drawbacks of the proposed method, which may limit its generalizability or practical usefulness. \n\n7. How did you choose the specific VL-LLMs for your experiments?\n\n8. In Section 4.2, you mention that the VPG trained on smaller LLMs may outperform larger ones in most conditions. Can you provide more explanation and insights for this observation of why it brought better task performance?\n\n9. Can you explain in more detail the warm-up training of the linear projector and how it can prevent performance drops?\n\n10. In Section 6, you mention that you customized two novel VL-LLMs with your VPGTransfer framework. How generalizable do you think this framework is to other novel VL-LLMs, and what steps would need to be taken to adapt it to different models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful that you acknowledge the strengths of our work so much. Your support definitely encourages us to improve the work further and push forward. Following we present the response to address your concerns.\n\n---\n**Q1: Reliance on the existence of a well-functioned VPG model.**  \n**A:** Thanks for pointing this out.\nWe want to clarify that the existence of VPG model is not a demanding requirement:\n- When building new MM-LLMs, it is common to first validate on small models and then scale up, where the small VPG models can be used for transfer.\n- There are already released VPG models like BLIP-2, which can be used.\n\n---\n**Q2: How does the quality of the existing VPG influence the transfer efficacy and efficiency?**  \n**A:** We discuss the question under TaS and TaT scenarios:\n- TaS: A better VPG can typically result in higher efficacy and efficiency under TaS.\nAs shown in Figure 7 of the main paper, the VPG trained on OPT 125M is the best performed one,\nwhich also achieves the highest acceleration rate in Table 3 and the best performances given the same target LLM.\n- TaT: When considering TaT, there will be a trade-off between the VPG ability and model gap.\nFor example, although the VPG trained on smaller OPT shows better results, it is also less transferable to FlanT5 models (cf Appendix D.1).\n\n---\n**Q3: More tasks and datasets.**  \n**A:** Thanks for the suggestion!\nPlease refer to the Reviewer#7KJM Q2 and Q3 for results on more datasets (medical, robustness VQA).\n\n---\n**Q4: Intuition of VPGTrans.**  \n**A:** We first dissect the success of VPGTrans into 3 parts and explain the intuitions one by one:\n- 1. Projector warm-up can accelerate normal pre-training (stage-2) and prevent performance drop.\nIntuitions: (1) projector warm-up can work as a good weight initialization for the stage-2 and thus accelerate. (2) the gradient passed through a randomly initialized projector will distort VPG's ability (performance drop), while the warm-up can avoid.\n- 2. Word converter initialization can accelerate projector warm-up. Intuition: a good initialization can lead to faster convergence. \n- 3. Projector enables fast convergence with extremely large learning rate. Intuition: projector has very limited parameters, and thus easier to train.\n\n---\n**Q5: Theoretical supports of VPGTrans.**  \n**A:** In this paper, we mainly focus on the empirical study. We try to discuss some potential theoretical analysis directions that may help:\n- Warm-up Aspect: [1] illustrates that training the classifier first and then fine-tuning can help the OOD performance. \nThe paper shows that gradient passed through a randomly initialized classifier will distort pretrained features.\nHowever, [1]'s theoretical analysis is limited to linear feature extractor assumption. \n- Visual Prompt Aspect: there are limited theoretical works on the mechanism of visual soft prompt.\nFor empirical study, [2] and our work try to interpret visual prompts with word embeddings (cf Appendix Fig. 11).\n\n---\n**Q6: When transferring the VPG, what has been changed of the visual soft prompts?**  \n**A:** Good question! We visualize the visual prompts (cf Appendix Fig. 11), where the last token is always nearest to EOS token of OPT,\nand other tokens represent visual contents like flower.  \nAfter transferring to larger OPT (e.g. OPT 2.7B), the last token is also nearest to EOS token. \nHowever, the nearest neighbour of each content token for a given image will be changed to other content (e.g. flower->table).\n\n**Q7: Comparison with hard VPG transfer.**  \n**A:** We show the result of hard VPG transfer with similar amounts of training data:\n\n| Model (OPT 6.7B)  | VQAv2 | GQA  | OKVQA | Data  |\n|-------------------|-------|------|-------|-------|\n| Hard VPG Transfer | 50.9  | 33.9 | 32.7  | 13.8M |\n| VPGTrans (ours)   | 57.2  | 36.2 | 39.8  | 13.8M |\n\nWe see that VPGTrans can achieve better performance with the same amount of training data.\nWe will also add it in the revision.\n\n---\n**Q8: Limitations and Drawbacks**  \n**A:** We discuss the social impact and limitations in the lines 584-594 of Appendix.\nThe limitation mainly includes the hallucination problem of constructed VL-LLMs.\n\n---\n**Q9: How did you choose the specific VL-LLMs for your experiments?**  \n**A:** For the VL-LLM architecture, we choose BLIP-2, as it is the most powerful open-sourced MM-LLM when we initialize this project. \nFor the LLM choices, we try to cover both encoder-decoder (FlanT5) and decoder-only (OPT) models across different sizes. \n\n---\n**Q10: VPG trained on smaller LLMs may outperform larger ones. Potential explanations.**  \n**A:** First of all, training VPG on LLMs will change the weight of the VPG, which will affect the pretrained VPG's ability. This phenomenon can be considered as a kind of VL alignment tax.\nThen, we empirically find that using larger OPT model usually takes more tax.\nThe conclusion can be validated by the confusion matrix (cf Figure 7), where the VPG is fixed to see which VPG is better.\nWe also check the absolute update of transfer sources (OPT 125M, 350M, 1.3B) and find that the VPG of OPT-1.3B has the largest absolute update in the 1st epoch. \nFinally, it is still unclear why the larger LLM will have more effect on VPG.\nWe hypothesize it might be caused by the more complicated mechanism, which may require more effort for VPG to align.\nWe will further explore this problem in the future.\n\n---\n**Q11: Why warm-up can prevent performance drop.**  \n**A:** Please refer to Q4-1-(2).\n\n---\n**Q12: Adaptions to different models.**  \n**A:** The adaptation will just be similar to the construction of VL-LLaMA and VL-Vicuna. \n- Find an existing VPG (e.g. ImageBind-LLM's VPG), and an LLM (e.g. LLaMA-2). \n- Then connecting them with VPGTrans' two-stage training.\n\n---\n**Q13: Potential Social Impacts.**  \n**A:** Please refer to Q8.\n\n---\n[1] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution.  \n[2] CLIPCap: CLIP Prefix for Image Captioning."}, {"review_id": "y4tHdUPTdb", "reviewer": "Reviewer_iPdz", "summary": "This paper mainly focuses on the transferability of visual prompt generator in VL-LLM. The authors conduct extensive experiments among different LLM types and sizes. By combining the experiment results they propose a new VPN training pipeline that engages projector warmup stage and vanilla finetuning stage.", "questions": "Please refer to the above weaknesses.", "limitations": "No obvious negative societal impact observed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written.\n2. The experiments are comprehensive.\n", "weaknesses": "1. I wonder if different structures for the projector, e.g. more layers, would lead to large performance gap.\n2. It is kind of weird to directly compare the training time between training the whole model and using the proposed transfer method, since you have to first get another VL-LLM. In this way the time used for training the original smaller VL-LLM should also be considered. It is inappropriate to claim the method is more efficient just because of other off-the-shelf models.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive reviews, which will definitely help consolidate our paper. We are also grateful that you acknowledge the strengths of our work. Following, we present the response to address your concerns.\n\n---\n**Q1: I wonder if different structures for the projector, e.g. more layers, would lead to large performance gap.**  \n**A:** Thanks for your suggestion.\nFirst of all, we want to clarify that existing VL-LLMs mainly adopt a linear projector and thus we just follow them in this paper.\nOf course, it is quite interesting to explore different projector structures.\nWe compare 3 types of projectors (linear, 3 layers MLP, 1 layer transformer) under OPT$_{\\text{125M} \\rightarrow \\text{1.3B}}$ scenario.\nTo evaluate the acceleration speed, we compare the performance of COCO caption (CIDEr) at different epochs as follows:\n\n|Projector Type| 1     | 2     | 3     | 4     | 5     | 6     | 7     | 8     | 9     | 10    |\n|-|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n|Linear| 131.5 | 133.1 | 134.1 | 135.1 | 135.7 | 135.9 | 136.1 | 136.2 | 137.0 | 136.8 |\n|3 Layers MLP| 96.8  | 120.9 | 125.5 | 126.7 | 128.3 | 129.4 | 130.4 | 129.7 | 131.5 | 131.9 |\n|Transformer| 132.3 | 134.7 | 134.7 | 134.8 | 135.6 | 135.4 | 135.8 | 135.0 | 135.9 | 135.7 |\n\nTo compare the visual perception ability, we show their best VQA performances:\n\n|Projector Type| VQAv2 (acc.) |\n|-|--------------|\n|Linear| 48.9         |         \n|3 Layers MLP| 49.6         |\n|Transformer| 49.2         |\n\nWe have the following observations:\n- Linear can achieve fast convergence, while the VQAv2 performance is the lowest.\n- 3 layers' MLP will result in much slower convergence and weak COCO caption performance, while achieves the best VQA results after convergence. \n- Transformer can achieve both good convergence speed and good performance on COCO caption and VQAv2,\nwhich may serve as a potential exploration direction for the projector structure design.  \n\nOverall, thanks for indicating this point. We will further explore it in the revision.\n\n---\n**Q2: It is kind of weird to directly compare the training time between training the whole model and using the proposed transfer method, since you have to first get another VL-LLM. In this way the time used for training the original smaller VL-LLM should also be considered. It is inappropriate to claim the method is more efficient just because of other off-the-shelf models.**  \n**A:** Thanks for your careful reading and for mentioning this aspect. Here we report the source models' GPU hours as follows:\n\n| Model                      | Src Model Cost (hours) | Transfer Cost (hours) |\n|-|-|-|\n| BLIP-2 OPT 6.7B            | 631.5                  | 0                     |\n| VPGTrans OPT 6.7B (ours)   | 459.0                  | 59.0                  |\n| BLIP-2 FlanT5 XXL          | 684.0                  | 0                     |\n| VPGTrans FlanT5 XXL (ours) | 435.0                  | 32.4                  |     \n\nCompared with training a large model from scratch, our VPGTrans enables training both the smaller and the larger models with even less time.\nWe will add the table in the revision.\n\nMoreover, we would like to re-emphasize that the existence of off-the-shelf models is not a demanding assumption.\nWhen building a new VL-LLM, it is common to validate on a smaller model first, which can serve as a transfer source.\nThere are also existing open-sourced VL-LLMs like BLIP-2, which can serve as the base for cross LLM types transfer.\n"}]

Author rebuttal:
# General Response to All Reviewers
Dear reviewers,

Thanks for all of your time to write valuable and constructive comments. Your feedback will definitely assist us in enhancing the quality of our paper, and thus we are committed to incorporating your suggestion in our revision process. 
Meanwhile, we feel encouraged that the reviewers find our method efficient and effective (Reviewer 7KJM, GwBd and mi8b), and our experiments solid and comprehensive (Reviewer iPdz and mi8b). Your support means a lot to us!  
At this juncture, we would like to re-emphasize the significance of this work.

The AI community has now entered the era of Large Language Models (LLMs), wherein multimodal Vision-Language LLMs (VL-LLMs) have demonstrated a powerful understanding across vision&text modalities, becoming a focal point of future LLM research. However, we recognize that obtaining a VL-LLM can be costly, especially when training from scratch. This has motivated our work: **exploring the model transfer learning approaches to significantly reduce the cost of acquiring VL-LLMs**, such as transitioning from existing smaller models to larger models.
With such background, this work contributes to the following key aspects:

1. We propose a VPGTrans framework, which significantly reduces the cost of building new VL-LLMs (e.g. in only 10% GPU hours) with the help of existing bases.
2. We reveal intriguing insights and findings of the VPG transfer across LLMs, and provide potential explanations to shed light on further research on this topic.
3. With VPGTrasn, we customize two novel VL-LLMs (i.e., VL-LLaMA and VL-Vicuna) that are open-sourced for the community.

Although this paper may not present ground-breaking technical novelty (pointed out by Reviewer#7KJM), we mainly contribute by offering a thorough investigation to demonstrate the feasibility of VPG transfer for efficient accessibility of VL-LLMs, and devise a novel transfer method **VPGTrans**.  
As recognized by Reviewer#mi8b, _'the pilot study serves as a promising starting point, paving the way for further investigations and advancements in the development of MM-LLMs with higher efficiency'_. 
Our VPGTrans enables more swift building of new VL-LLMs with lab-level resources, and thus will enable more researchers into this area.
We much believe VPGTrans will show a broad impact on the future research of LLMs. Thus, we will release all the codes and resources upon acceptance. 


In response to the reviewers' comments, we have thoroughly reviewed our paper, performed additional experiments, and prepared a comprehensive response.
We will fix all the typos and improve the manuscript according to your comments.
We hope that our paper adequately addresses your concerns.
We kindly request a **re-evaluation of our work** based on the updated information, and look forward to your recognition.

Best regards.


Meta-review: The paper received 6/9/6/4 ratings. Reviewers acknowledge that the experimental findings in VPG (Visual Prompt Generator) transfer from one VL-LLM to another VL-LLM may benefit the community. The score 4 review raised concerns regarding the presentation of the paper as well as limited technical contribution. The AC read the paper, the reviews as well as rebuttal discussions, and agreed with part of the concerns. Although it’s straightforward that initializing the VPG with a well-trained existing VPG can significantly reduce the training time when adopting to a new LLM (limited technical contribution), the study could be viewed as providing a best practice for this. Thus the AC recommends to accept the paper.
Decision: Accept (poster)