Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
Authors: Gen Luo
Yiyi Zhou
Tianhe Ren
Shengxin Chen
Xiaoshuai Sun
Rongrong Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: vision-language instruction tuning
multimodal LLM
efficient training

Abstract: Recently,  growing interest has been aroused in extending the multimodal capability of large language models (LLMs), e.g., vision-language (VL) learning, which is regarded as the next milestone of artificial general intelligence. However, existing solutions are prohibitively expensive, which not only need to optimize excessive parameters, but also require another large-scale pre-training before VL instruction tuning. In this paper, we propose a novel and  affordable  solution for the effective VL adaption of LLMs, called  Mixture-of-Modality Adaptation (MMA).  Instead of using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters, to  bridge the gap between LLMs and VL tasks, which also enables the joint optimization of the image and language models. Meanwhile, MMA is also equipped with a routing algorithm to help LLMs achieve an  automatic  shift between single- and multi-modal instructions without compromising their ability of natural language understanding.  To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision-language instructed model as LaVIN.  To validate MMA and LaVIN, we conduct extensive experiments  under two setups, namely  multimodal science question answering and multimodal dialogue. The experimental results not only demonstrate the competitive performance and the superior training efficiency  of LaVIN  than existing multimodal LLMs, but also confirm  its  great potential   as a general-purpose chatbot. More importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4 training hours with 3.8M trainable parameters, greatly confirming the effectiveness of MMA.   Our  code is anonymously released at:  https://anonymous.4open.science/r/LaVIN--1067.

Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
Gen Luo!3, Yiyi Zhou Tianhe Ren', Shengxin Chen', Xiaoshuai Sun 12, Rongrong I
'Key Laboratory of Multimedia Trusted Perception and Efficient Computing.
Ministry of Education China, School of Informatics, Xiamen University, 361005, P.R. China.
Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China.
3 Peng Cheng Laboratory, Shenzhen, 518000, China.
luugenn, chenshengxing rent innreianneeeeeee @stu. xmu edu..cu.c cn, (zhouylyi, xssun, rrji 0mmu. edu.cn
Abstract
Recently, growing interest has been aroused in extending the multimodal capability large language models (LLMss,, e.g. vision-language (VI.) learning, which is regarded as the next milestone of artificial general intelligence. However, existing solutions are prohibitively expensive, which not only need to optimize excessive parameters, but also require another large-scale pre-training before VL instruction tuning. this paper, we propose a novel and affordable solution for the effective VL adaption of LLMs, called ffixture-of-Modalie using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters, bridge the gap between LLMs and VL tasks, which also enables the joint optimization of the image and language models. Meanwhile, MMA is also equipped with routing algorithm to help LLMs achieve an automatic shift between single- and multi-modal instructions without compromising their ability of natural language understanding. To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision- language instructed model as LaVIN. To validate MMA and LaVIN, we conduct extensive experiments under two setups, namely multimodal science question answering and maltimodal dialogue. The experimental results not only demonstrate the competitive performance and the superior training efficiency of LaVIN than existing multimodal LLMs, but also confirm its great potential as a general-purpose chatbot. More importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4 training hours with 3.8M trainable parameters, greatly confirming the effectiveness of MMA. Our project is released at https: / lugen 9966 gi git...... lalinnn..
Adaptation (MMA). Instead
Introduction
In recent years, large language models (L.LMs) [3, 37, 5, 52, 38] have continuously pushed the upper limit of natural language understanding with ever increasing parameter sizes and pre-training data scales. The introduction of instruction tuning [30, 31, 35] also enables LLMs to engage in human-like conversations and handle various natural language processing (NLP) tasks [29, 44, 45], approaching artificial general intelligence, e.g., GPT-3.5 [33]. The next milestone is often regarded to extend these LLMs with multimodal capabilities, e.g., vision-language (VL) learning, making LLMs applicable to more real-war weal-world application scenarios. Such target has been recently realized by GPT-4 [34], which likely to adopt scale vision- -anguaee corpus directly train a multimodal GPT.
ccoresponniing author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input: Text VexioVision
Input: Text Text+ Vision
Input: Text VextoVision
Schome: No Optimization
Schome: Multi-stage Optimization
Scheme: Joint Optimization
Tunable Purarns: NEA
Tunable Params: Hundredss of Millions Billions
Tunable Params: 3-6 Millions
Modell
or
Image Encoder
Visual Adapter
Image Encuder
Visual Nock
LLM
Image Encoder
Visual Nock
I.LM
L.M
LLM
ModelN
1. Adapter
0 MM- Adanter
Stage-1: VI. Alignmeett
Stage-2: Instruction Tuning
(4) Expert System
(b) Modular Training Scheme
(c) Mixtureoof-Modality Adaptation
Figure I: Comparison of different multimodal adaptation schemes for L.MSs. In the expert system, L.MSs play a role of controller, while the ensemble of I.I.M and vision models is expensive in terms of computation and storage overhead. The modular training regime (b) requires an additional large neck branch and another large-scale pre-training for cross-modal alignment, which is inefficient in training and performs worse in previous NLP tasks. In contrast, the proposed Mixture-of-Modall Adaption (MMA) (c) an end-to-end optimization scheme, which is cheap in training and superior in the automatic shift between text-only and image-text instructions.
However, the training regime of GPT-4 [34] is prohibitively expensive, and recent endeavors [49, 50, 41, 21, 1, 8, 56, 4] are still keen to efficient VI. adaptions of LLMss. As shown in Fig. 1, the existing multimodal solutions for LLMs can be roughly divided into two main categories, Le., the expert system and the modular training ones, respectively. In the expert system solution [49, 50, 41], LLMs usually serve manager to interpret different natural language instructions, and then call the corresponding vision models to handle the input image, e,g. image captioning [18, 27], visual question answering [55, 28] or text-to-image generation [39]. The advantage of this solution is that it does not require the re-training of LLMs and can make full use of existing vision models. However, the ensemble of LLMs and various vision models still exhibits significant redundancy in terms of computation and parameters, leading to excessive memory footprints. Mcanwhile, the joint optimization of LLMSs and vision models is still an obstacle.
In this case, increasing attention has been paid to the modular training of LLMs [17, 21, 56, 15, 56]. As illustrated in Fig. 1, this paradigm often requires LLMs to deploy an additional neck branch to connect the visual encoders, and then performs another pre-training on numerous image-text pairs for cross-modal alignment. Afterwards, the neck branch and LLM are jointly tuned via VL instructions. Despite the effectiveness, the required VL pre-training is still expensive for a quick adaptation of LLMs. For instance. the pre- tr-iirriiin of BLIP2 [17] consumes more than 100 GPU hours 129 millions of image-text pairs. In addition, this paradigm often requires to update most parameters of LLM, limiting the efficiency of VL instruction tuning. For example, LLa 13B [21] fully fine-tunes the entire L.M during VL instruction tuning, resulting in significant increases in training time and intermediate storage overhead More importantly, these fine-tune schemes will inevitably undermine the NLP capabilities of LLMs due to the drastic changes in their parameter spaces. For instance, the existing multimodal LLMs, such as BLIP2 [17] and miniGPT4 [56], do not support text-only instructions, greatly hindering their applications.
In this paper, we propose novel and efficient solution for vasion-language instruction tuning, termed Mixture-of- Mfdoddlityyyy Adaptation (MMA). Different from existing modular training scheme [17, 2 MMA is end-to-end optimization regime. By connecting the image encoder and LLM with lightweight adapters, MMA can jointly optimize the entire multimodal LLM via a small number of parameters, saving more than thousands times of storage overhead compared with existing solutions [21, 56, 17]. To obtain quick shift between text-only and image-text instructions, MMA equips the inserted adapters with routing scheme, which can dynamically choose the suitable adaptation path for the inputs of different modalities, thereby well preserving the NLP capability of LLMs. To validate MMA, we apply it recently proposed LLM called LLaMA [43], and term this new large vision-language instructed model as VIV With the help of MMA, LaVIN can achieve cheap and quick adaptations on VI. tasks without the requirement of another large-scale pre-training.
To validate LAVIN, we first conduct quantitative experiments on ScienceQA [24]. Experimental results show that Va can achieve on-par performance with the advanced multimodal LLMs, e.g., LLaVA [21], while reducing up to 71.4% training time and 99.9% storage costs. Notably, fine-tuning "The checkpoints are often stored during training, and each of them takes up 26GB for storage.

---

LaVIN on ScienceQA only takes 1.4 hours with 8 A100 GPUs, and the updated parameters are only 3.8M. In addition, we also extend LaVIN to multimodal chatbot via tuning on 52k text-only instructions [42] and 152k text-image pairs [21]. The qualitative comparisons show that LaVIN can accurately execute various types of human instructions, e.g., coding, math and image captioning, while yielding superior vision-langune understanding than existing multimodal chatbots [56, 17, 50].
In summary, our contributions are three folds:
We present novel and efficient solution for vision-language instruction tuning, namely Mixture-of- Modaaity Adaptation (MMA), which does not require the expensive VI. pretrain- ing and can maintain the NLP capabilities of LLMs.
Based on MMA, we propose a new multimodal LLM, namely LaVIN. Experimental re- sults show the superior efficiency and competitive performance of LaVIN against existing multimodal LLMs, and also confirm its great potential as a general-purpose chatbot.
We release the source code and pre-trained checkpoints associated with this paper. We believe that our project can well facilitate the development of multimodal LLM.
2 Related Work
2.1 Cnrameter-Efficiee Transfer Learning
Since large language models have ever-increasing parameter sizes, parameter-efficient transfer learning (PETL) [13, 19, 25, 14, 22, 12] has gained increasing attention to reduce training and storage overhead of LLMs. PETL aims insert fine-tune small number of parameters into LLMs, thereby achieving the adaption on downstream tasks. In early efforts [13, 12], a small MLP network, known as Adapter [13], inserted into LLMs to project their hidden features to the semantic spaces of downstream tasks. Based on Adapter, numerous PETL methods [19, 46, 25, 14, 22, 12] have been proposed further enhance adaptation capabilities [19, 46, 25, 22, 12] and inference speed [14]. Among them, AdaMix [46] is a method relatively close to our MMA, which also includes set of candidate adapters for downstream task routing. However, AdaMix static and task-dependent, of which routing path is fixed after training. In contrast, our MMA is a dynamic method based on the input modality embeddings. Moreover, AdaMix still an unimodal module and hard to adaptively adjust the adaptions of different modalities. Driven by the great success in NLP, PETL has also achieved significant progresses in large vision models [26, 2, 54], e.g., ViT [7] and CLIP [36]. Despite the effectiveness, PETL for multimodal LLMs still lacks explorations. A very recent PETL method [51] is proposed for multimodal LLMs, but its performance still lags behind full fine-tuning.
2.2 Multimodal Instruction LLMs
Instruction tuning [30, 31, 31, 35, (7,447488 48] aims fine-tune LLMs natural language corpus describing diverse NLP tasks. This simple and effective method has been successfully applied to various well- known LLMs, such InstructGPT [35] and FLAN-TS [6], greatly improving their performance and generalization ability. Motivated by this success, numerous efforts have been devoted to constructing multimodal instructionfoolloiing LLMss. Existing works can be categorized into two groups, e.g., the expert systems [49, 50, 41] and modular training ones [17, 21, 56, 15, 56], respectively. The representative expert systems, such as Visual ChatGPT [49] and MMREACT [50], employ LLMs as the controller to invoke various vision models to accomplish the VL instructions. Despite the effec- tiveness, this heavy system also incurs non-negligible burdens in terms of storage and computation. Recently, modular training models [17, 21, 56, 15, 56] as proposed more efficient alternatives. Among them, Flamingo [1] is the first large-scale multimodal LLM that pre-trains on numerous image-text pairs, which demonstrates strong zero-shot ability on diverse tasks. The following works, including BLIP-2 [17], FROMAGe [16], PaLM-E [8], KOSMOS-I [15] and LLaVA [21], not only optimize the model architecture [17, 16, 8, 15] but also improve the quality of VI. instruction data [21]. Despite their effectiveness, most multimodal L.I.Ms require expensive training costs and perform worse text-only instructions.
3

---

Modality Token
Image:
Response: The image
features small brown dog running across grassy field, with its tongue out and ears flapping. The dog appears be enjoying itself as runs through the open space. In the background, there several scuttered across field, adding the natural setting.
Adapter
VIT
LLM
U
1
Visual Tokens
Mixture-of Modaltyy Adapter
Adapter
Multi-head heali-headdae Attention
Instruction:
Tokenizer
"describe this image.
Feed-forward Network
Textual Tokens
Lx Layers
Text-only Data
Sampling
Mosture-of-Modalll Truining
Text-image Data
Parameter Updates
Figure 2: The overview of the Mixture-ou Modaltyyy Adaptation (MMA) and the architecture of LaVIN. In LaVIN, the novel Mixture-ofModaaity Adapters employed process the instructions of different modalities. During instruction tuning, LaVIN is optimized by Mixture of Modality Training (MMT) in end-to-end manner.
Method
3.1
Mixture-of-Modality Adaptation
In this paper, we propose novel learning regime for the vision-language adaption of LLMs, which called
Adaptation (MMA). As shown in Fig. MMA includes two novel designs, namely Mixture-of-Modaae Adapter (MM-Adapter) and Mixture-of-Modallt Training (MMT). Specifically, MM-Adapter extends LLMs with multimodal abilities via lightweight adapters, which also realizes the automatic shift between single- and multi-modal instructions. Afterwards, the entire multimodal LLM is jointly optimized via MMT, which is cheap in training time and storage.
Mixture-of-Modalle Adapter (MM-Adapter), As shown in Fig. 2, we connect the LLM with the image encoder with a set of lightweight adaptation modules. In the image encoder, these modules can be the common adapters [13, 26]. In the LLM,, unimodal adaptation modules are inferior in handling single- and multi-moda instructions simultaneously.
In particular, we first introduce a modality token fm E IR to indicate the input modality, which is defined by
W
W
mEm
Router
Here, Em E 2xe is the modality embedding. m E IR22 is one-hot vector to represent the input modality. Based on the modality token by. MM-Adapter can dynamically adjust adaptations for the imput features Z E Raxe, In practice, Z can be the single- or multi-modal features, which will be introduced in Sec 3.2. Thus, MM-Adapter can be defined by
W
W
Routing Weights Generation
Figure 3: Illustration of the Mfxtureoof-Modaiity Adapter (MMA). MMA dynamically select the appropriate adapter according the input modalities.
 = Z + res.routterr.. (2), Sag fm
Here, fa, and Saz are RepAdapters [26] in our paper. is the scale factor, and router(-) routing function decide the routing path of two adapters. To further reduce the parameter costs, the downsampling projection of two adapters are shared.
As shown in Fig. 3, the key to realize the dynamic adaptations lies in the design of the routing function router(-). which is formulated as
router (Z), = un fa, (Z) + 1Z faa (Z), im Wm, + bm
(3)
where i (tm) softmax
7

---

Here, Wm E R and 4m E R² are the weight matrix and bias. respectively. ú denotes the routing weights, and is the temperature of the softmax. Based on Eq. 2 and 3. MM-Adapter can select the best adaption path according to the modalities of input instructions. More importantly, the process of MM-Adppter only introduces a few of additional parameters, which is still efficient. practice, MM-Adapter can be used as the unimodal adapter to improve the adaptation ability, thus we also apply it to the image encoder.
Training (MMT), Based on MM-Adapter, the target of MMT is to freeze the large image encoder and LLM, and only fine-tune the inserted adapters. In this case, the entire multimodal LLM can be jointly optimized in an end-to-end manner. Specifically, the end-to-end optimization objective can be formulated by
Mixture-of-Modaaee
arg min (Z), R; 0a).
(4)
Here, R and L(-) denote the ground-truth response [24] and the objective loss function, respectively, fo is the LLM, and 8, denotes the adaptation parameters. I E Raxaxa and T E IR' denote the input image and text instruction, respectively.
During training, we construct mini training batch randomly sampled from text-only and text-image instructions. In this case, the overall training objective L can be defined by
m S+1
-=ΣΣ>
log log R [Z',
0a).
(5)
Here, m denotes the batch size, and S is the length of the response. After MMT, the multimodal LLM can effectively execute the input instructions of different modalities.
In our training scheme, the number of optimized parameters is still kept at a very small scale, e.g., 3~5M, which greatly reduces the training time and the storage cost. Compared existing modular training paradigm, MMA does not require additional VL pre- pre-training and can optimize the entire model end-to-end, further improving the training efficiency.
3.2 Large Vision-language Instructed Model
To validate MMA, we apply it to an LLM called LLaMA [43] and adopt CLIP-VIT [36] as the image encoder. Here, we term this new large vision-languas instructed model as LaVIN.
Given the input image E hxxxxx3 we use the [cls] tokens from every fourth layer of VIT [7] as the visual feature, denoted as X E Igaxd In the image encoder, we insert the adapters before the multi-head attention modules. We represent the text instruction with word embeddings, denoted as Y E gx Then, simple visual adapter is used to transform the visual features to the same dimension with the LLM, which is defined by
(6) Here, Wa E Rdxd and W,, E dd denote the weight matrices, while Wa Rd and by E I are the bias terms. a is the SwiGLU activation function [40]. In practice, dy is much smaller than d and so the input of LLM can be defined by
X'= o + ba) + bu.
[lm: X', Y] [1 Y] text only.
text-image,
Z =
Here, [+] denotes the concatenation. Based on the multimodal input, LLM can predict the next token step by step, which can be formulated by
S+1
Pt = II (R, Ru:--;;0 0.)
(8)
Here, Ps E R denotes the probabilities of the predicted word and m is the length of the word embeddings. 0, and f,, denote the parameters of LLM and adaptation modules, respectively.
Compared with previous works [17,56, 21], the architecture of LaVIN is much simpler and more lightweight, which also easier to optimize. For example, the visual neck of LaVIN times smaller than that of LLaVA [21], but the performance of two models is close.
5

---

Subject
Context Modality IMG
Grade
Method
#T- -T-Param
LLM
NAT
SOC
LAN
TXT
NO
G1-6
G7-12
Average
Zero- seo methods
Human [24]
90.23
84.97
87.48
89.60
87.50
88,10
91.59
82.42
88,40
GPT-3,5 [24]
74.64
69.74
76.00
74.44
67.28
77.42
76.80
68.89
73.97
GPT-3.5 (CoT) [24]
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
75.17
GPT-4 [34]
84.06
73.45
87.36
81.87
70.75
90.73
84.69
79.10
82.69
Representotive & SoTA models
UnifiedQA [24]
223M
71.00
76.04
78.91
66.42
66.53
81.81
77.06
68.82
74,11
MM-CoT Roae
Rose [53]
223M
87.52
77.17
85.82
87.88
82.90
86.83
84.65
85.37
84.91
MM-CoT Large [53]
738M
95.91
82.00
90.82
95.26
88.80
92.89
92.44
90.31
91.68
LLaVA [21]
13B
90.36
95.95
88.00
89.49
88.00
90.66
90.93
90.90
90.92
Parameter-effiieett methods
LLuMA-Adupter [51]
18M
84.37
88.30
84.36
83.72
80.32
86,90
85,83
84.05
85,19
LaVIN-7B (ours)
3.8M
89.25
94.94
85.24
88.51
87.46
88.08
90.16
88.07
89.41
LaVIN-13B (ours)
90.32
94.38
87.73
89.44
87.65
90.31
91.19
89.26
90.50
LaVIN-138| (ours)
5.4M
89.88
94.49
89.82
88.95
87.61
91.85
91.45
89.72
90.83
Table 1: Comparison ScienceQA test set. Question classes: NAT natural science, SOC social science, LAN language science, TXT = text context, IMG = image context, NO = no context, G1-6 grades 1-6, G7-12 grades 7-12. i denotes that VIN is trained with 40 epochs. #T-Params
denotes that the number trainable parameters.
Settings
#T-Params
NAT
SOC
LAN
TXT
IMG
NO
GI-6
G7-12
Avg.
Text Only
1.8M
82.86
82.56
82.28
81.23
75.81
86.06
83.26
81.54
82.65.000
Vision Modality (MMT)
2.4M
85.97
90.66
83.55
84,90
83.59
88.14
83.06
86.32.557
Joint Opt. (MMT)
2.5M
86.59
94.71
82.91
85.63
84,98
86.41
88.62
85.04
87.34
Stronger Image Enc.
2.9M
88.01
94.94
83.64
87.15
86.81
87.04
89.87
85,56
88.33
MM-Adapter
3.8M
89.25
94.94
85.24
88.51
87.46
88.08
90.16
88.
89 89.41.608)
Larger LLM (13B)
5.4M
90.32
94.38
87.73
89.44
87.65
90.31
91.19
89.26
90.50.113)
Table 2: Ablation studies on ScienceQA test set. For the text-only baseline, we use the image caption to prompt the model. ViT-B/16 and LLaMA-7B are used as the default image encoder and L.M. "Joint Opt" denotes the joint optimization of image encoder and LLM. The Mixture-of-Modality Training (MMT) is ablated with the settings of "Vision Modality" and "Joint Opt.".
4
Experiments
4.1 Datasets and Metrics
ScienceQA. ScienceQA [24] is the large-scale multimodal dataset for science question answering. which covers various domains, including 3 subjects, 26 topics, 127 categories and 379 skills. Sci- enceQA consists of text-only and text-image examples in three splits namely train, val and text, with 12,726, 4,241 and 4,241 examples, respectively. We evaluate model using average accuracy.
Alphaca-52k & LLa VA-15kK Alphaca-52k [42] contains 52k text-only cnltructioo-followiin data dataset,
generated by - [3]. LLaVA-158k [21] is large-scale text-image instruction-follon where the answer is automatically generated by GPT-4 [34]. Following LLaVA [21], GPT-4 is employed to evaluate the quality of the chatbot's responses, which will assign higher scores to superior responses within a range of 10.
4.2
Implementation Details
We employ the ViT-L/14 [7] of the pre-trained CLIP [36] as the image encoder. The visual features consist of six [cls] ] tokens extracted from every fourth layer of ViT-L/14, For LLM, LLaMA- 7B [43] and LLaMA-13B [43] are used. The default dimension of the visual neck is set to 128. The dimension of MM-Adapter is 8, and the temperature is set to 10 for LaVIN-7B and 5 for LaVIN-13B. For text-only baseline, the image encoder is removed, and MM- MM-Adapter replaced with RepAdapter [26]. We adopt AdamW [23] as the optimizer, and train the model for 20 epochs with a cosine decay learning rate schedule. The batch size, learning rate and weight decay are set to 32, 9e-3 and 0.02, respectively. During the generation stage, the decoding uses top-p sampling with temperature of 0.1 and top-p value of 0.75, respectively. For the experiments of multimodal chatbot, all hyperparameters remain the same, except for the training epochs, which are reduced to 15.

---

4.3 Experimental Results
4.3.1 Quantitative Experiments
Results on ScienceQA. In Tab. 1, We first compare LaVIN with the state-of-the-art methods on ScienceQA. From this table, the first observation that the few-shot LLMs, such as GPT-4, still perform worse than human, suggesting the great challenge of ScienceQA. In contrast, ex- isting supervised methods [21, 51, 53] yield better results. In particular, MM-CoT [53] achieves the best performance, e.g,, 91.68. However, MM-CoT mainly focuses the multi- modal chain-of -ftthuuhtthhhtttthh for language models, of which contribution is orthogonal to our approach. In particular, LLaVA [21] is an end-to-end multimodal LLM, which is more close our work.
The results show that LLaVA remains compet-
itive performance against LMMCoTTargel [53], especially in the category of SOC. Despite the
Methods
#T-Params
Accuracy
LLaVA [21]
13B
85,81
effectiveness, its number of trainable parame-
LLaMA-Adapter [51]
1.8M
85.19
ters is still large, leading to higher training over- head. LLaMA-Adapter [51] adopts parameter- efficient scheme to reduce the training over- head, but its performance still greatly lags be- hind LLaVA. Compared to these approaches, La VIN achieves the better trade-offs between performance and training efficiency. For exam- ple, LaVIN-7B consumes a similar scale of trainable parameters as LLaMA-Adapter [51], while outperforming it by +4.22 gains. When scaling up to 13B, LaVIN can obtain more significant perfor- mance gains, i.e., +5.64. Compared LLaVA, LaVIN- also achieves comparable performance and even performs better in some question classes, e.g., LAN and NO. Considering the much lower training costs than LLaVA, such competitive performance greatly confirms the efficiency and designs of LaVIN.
LaVIN-7B
3.8M
89.41 (44.22)
LaVIN-13B
5.4M
90.83 (15.02)
Table 3: Results of VaN and existing multimodal LLMs without the pre-training stage. We report the average accuracy on ScienceQA test set.
In Tab. 3, we compare LaVIN with existing methods without VL pre- training. From this table, we observe that both LLaVA [21] and LLaMA- Adapter achieve the similar perfor- mance, iee, 85.81 vs. 85.19. In particular, LLaVA [21] and LLaMA- Adapter [51] freeze the image back- bone, and the entire multimodal LLM is not jointly optimized, which hin- ders the learning of visual content. Moreover, the adaptation module in LLaMA Adapter does not consider the modality gap in the input instructions, greatly limiting its performance upper bound. In contrast, with the help of MMA, LaVIN significantly outperforms these approaches, egg., +5.02 gains over LLaVA. These results validate the proposed MMA towards the effective and efficient VL adaption, and confirm the designs of LIVIN.
Methods
PT Data #T-Params
BLEU-4
CIDEr
ClipCap [32]
33.5
113.1
LLaMA-Adapter V2 [11]
14M
36.2
122.2
BLIP [18]
14M
583M
40.4
136.7
BLIP-2 [17]
129M
188M
43.7
145.3
LaVIN (ours)
5.4M
36.4
126.9
LaVIN (ours)
0.6M
5.4M
37.8
131.7
Table 4: Fine-tuning results of LaVIN and existing multi- modal LLMs on COCO captioning. We report performance on Karpathy test split.
Results on COCO Captioning. In Tab 4, we compare LaVIN with existing methods on the task of image captioning. From these results, we can still observe the competitive performance La As parameter-efficient tuning method, LaVIN outperforms alMA-Adapter v2 [11] by large margin, e.g., up to +9.5 of CIDEr. Compared with large-scale pre-training models, e.g., BLIP and BLIP-2, the performance of LaVIN is still comparable, while the expenditure is much cheaper. For instance, with only 0.6M pre- pre-traiming data and 5.4M updated parameters, LAVIN can achieve 131.7 CIDEr on COCO Captioning. Notably, our tuning only takes GPU hours on 8 A100s, while BLIP-2 requires more than 300 GPU hours on 16 A100s. These results further validate the effectiveness and training efficiency of MMA and LaVIN
Zero-shot evaluation on NLP and multimodal benchmarks. In Tab. 5, we evaluate the zero-shot ability of LaVIN and existing methods on TruthfulQA [20] and MME [10]. On TruthfulQA [20], we observe that the zero-shot performance of existing multimodal L.LMS is obviously inferior to the original LLaMA. In stark contrast, LaVIN can further improve the performance by +9.2%
7

---

than LLaMA-Bas [43] through its mfxture-of adaptation. On MME [10], challenging benchmark for multimodal evaluation, VaN still demonstrates competitive performance against existing multimodal L.M.s. Expect for BI.IP-2 [17], which is pre-trained on numerous data, the other methods perform similarly to or worse than LaVIN, e.g., 866.5 of MiniGPT-4 vs. 963.6 of LaVIN on MME-C. These results confirm the strong generalization ability of LaVIN, and also validate that the NLP capabilities are well preserved by MMA during VL instruction tuning.
Ablation study. To gain deep insights into MMA and LaVIN, we conduct comprehensive ablation studies in Tab. 2. From this table, we can see that each design of MMA and LaVIN greatly contributes to the final performance. As shown in Tab. 2, the mixture-of-modality train- ing (MMT) brings the most significant gains, e.g., +4.69. In MMT, the joint training with the vision modality provides up to +3.67 perfor- mance gains for LAVIN. With the joint optimiza- tion of the image encoder and LLM, the perfor- mance of LaVIN further boosts from 86.32 to 87.34, suggesting the significance of the joint optimization for multimodal LLMs. With the help of MMT, LaVIN already surpasses the isting parameter-efficient method, i.e,, LLaMA-Adapter. Additionally, the stronger image encoder, ie.. VIT-L/14, also improves the average accuracy by 0.99. An interesting observation is that better image encoder provides noticeable performance gains for both image- and text- bestd questions. When adopting MM-Adapter to LaVIN, we observe + +1.08 gains on average accuracy. Such an improvement only requires extra 0.9M parameters, which is very lightweight. Meanwhile the performance of LaVIN is significantly improved by MM-Adapter on more challenging metrics like G7-12, i.c., +2.51. After scaling up LLM to 13B, the performance of LaVIN is further improved by 1.09. Overall, these ablations well validate the significance of MMA in adapting multimodal LLM. and also confirm the effectiveness of LaVIN.
Methods
TruthfulQA MME-C MME-P
LLaMA-Base [43]
38.7
LLaMA-Adapter [1I]
24.4
972.6
248,9
LLaVA [21]
16.4
502.8
214.6
BLIP-2 [17]
1293.8
290.0
MiniGPT-4 [56]
866.5
292.1
LaVIN (ours)
47.9
963.6
249.6
Table 5: Zero-shot results on NLP and multimodal benchmarks. "Mcl_targets" setup is used on Truth- fulQA [20]. "MME-C" and "MME-P" PP" denote the splits of Cognition and Perception on MME bench- mark [10]. respectively.
Comparison of training efficiency. In Tab. 6, we compare the training expenditures of LaVIN, LLaVA [21] and BLIP2 [17]. The first observa- tion is that the pre-training cost of BLIP2 is ac- tually expensive, which requires more than 200 hours. Meanwhile, LLaVA cannot be trained on common machines with the default training settings'. Thus, it requires some GPU memory- saving techniques [9] to avoid out of memory (OOM). However, its training time and stor- age requirement are still significant. For ex- ample. it still takes up to 26GB space to store the updated parameters of the L.LM. contrast, LaVIN demonstrates superior training efficiency with the help of MMA. Compared to LLaVA. LaVIN-7B and LaVIN-13B reduce about 80% and 71.4% training time, respectively. In terms of GPU memory and storage cost, our approach can save more than 40% GPU memory and 99.9% disk storage. Overall, these results greatly confirm the training efficiency of MMA.
Methods
#T-Params Memory
Time
#Storage
BLIP2 [17]
188M
>00 hours
LLaVA [21]
13B
OOM
N/A
N/A
LLaVA; [21]
13B
36.8G
7 hours
26GB
LaVIN-7B
3.8M
33.9G
1.4 hours
15M
LaVIN-13B
5.4M
55.9G
hours
20M
Table 6: Training costs of LaVIN and existing multimodal LLMs on ScienceQA. & denotes that GPU memory-saving techniques are used. "OOM" denotes out of GPU memory. All results are evalu- ated on 8 A100 GPUs.
4.3.2 Qualitative Experiments
Examples of different instruction-followinn tasks. In Fig 4, we compare LaVIN with existing methods [51, 21] on single- and multi-modal instruction-folloiing tasks, e.g., math, coding and image captioning. Compared to LLaVA [21] and LLaMA-Adapter [51], LaVIN achieves overall better responses across multiple tasks. In Fig.4 (a), LaVIN correctly answers the math problem with a result of 28.8, whereas LLaMA-Adapter [43] provides an incorrect answer. In example (d), LaVIN generates accurate code for the request of "print prime numbers up to 100". In contrast, the
Phttss//gichbb..om/aaataan hhtpa//c..c..ooooaaiii/aaaaaaa.. liu/LLaVA
8

---

It's toy costs S24 and the price is increased by 20% what is the new price of the toy?
Trunslate the following
mix water with water, do gel thin water or thick water?
Write python code that: Print
Chinese into English : in B. PE22,9/E SEA0 2 19-3t.
out prime numbers up to LaVIN (ours):
In mespc),
LaVIN (ours):
LaVIN (ours):
LaVIN (ours):
altss 3 in for ranget2,100
The now price of the toy is $28.80.
Life is like summer flower.
You will get the some type of water, just mixture of the two.
petence)
death is like autumn leaf,
LLaMA-AAapter:
quiet and beautiful.
def orine_numberstng
range(2,
LLaM A-dappee::
L.aMA-Adapter:
break
The new price of the toy is $28.40.
Born like summer flowers, die like autumn leaves.
Thin water.
wlises
printlil prine, numbers(100)
(a)
(b) Translation
(c) Question Answering
(d) Coding
The boy's clothes in the image are wet, because
How many food containers are there in the image?
Descrübe the Imuge.
of the rain?
LaVIN (ours)
LLaVA
LaVIN (ours)
LaVIN (ours)
In the image, there two food containers on the dining table. One container holds sandwich, the other holds plate broccoli. The sandwich appears to be cut in half, and the broccoli is served separate plate.
There are three food containers in the image: bowl, a plate, sandwich.
The image features beach seene with large number of kites flying in the sky.
No. boy's clothes are
because he is surfing in the ocean,
LLaVA
LLaVA
image captures lively beach scene with several people enjoying kite flying near the occan.
Yos, is likely that the boy's clothes are wel due to the rain.
(e) Visual Question Answering
(i) Image Captioning
Multimodal Reasoning
Figure 4: Comparison between LaVIN-13B and existing methods on single- and multi-modal instructions. The noteworthy aspects of the responses are highlighted in green, whereas the illogical portions are marked in red. More tasks and examples are given in appendix.
code written by LLaMA-Adapter is to check prime numbers, which does not produce any output during execution. Meanwhile, LIV presents clear and concise coding behavior, acting more like a professional programmer. In Fig 4 (e)-(g), LaVIN demonstrates remarkable visual reasoning ability in accomplishing various multimodal tasks. In Fig.4 (e), LaVIN accurately answers the complex questions about the number of food containers in the image and provides a detailed description about the complex scene. The same observation can also be witnessed in Fig.4 (g), where LaVIN infers correct reason for the wetness of the boy's clothes. Overall, these examples show the superior reasoning ability of LaVIN in executing single- and multi-modal instructions, while also confirming the significance of MMA in adapting LLMs to multi-modal tasks.
Examples of multimodal dialogue In Fig. 5, we compare LaVIN with existing multimodal LLMs in multi-turn conversations. and use GPT4 [34] to evaluate the quality of their responses. From the results, we can see that LaVIN has higher GPT4 scores among all compared models, suggesting superior ability in multimodal dialogue. Meanwhile, we also observe different response styles of these multimodal LLMs. particular, BLIP2 [17] tends to produce brief responses, which lack detailed explanations. In contrast, the responses of MiniGPT4 [56] are the longest among all models, but their content is often redundant and repetitive. Compared to them, LaVIN and LLaVA [21] can generate more accurate responses. Particularly, LaVIN performs better than the other methods, mainly due to its more logical and detailed descriptions. As illustrated in the first question, LaVIN not only provides the correct answer, but also explains the reason behind it. In the second question, LaVIN and LLa VA are required to judge whether the man will get wet, and La VIN answers "yes" while LLaVA considers "no". It can be seen that the reason of LaVIN is more comprehensive, logical and persuasive than LAaVA, which considers the situation of "the overhand may not provide the complete protection". Overall, these examples confirm that MMA equips LLMs with excellent multi-modal ability, requiring no pre-training on large-scale image-text data.
5 Limitations and Broader Impact
We observe two primary limitations of LaVIN. Firstly, LaVIN may generate incorrect or fabricate responses, similar to existing multimodal LLMs. Secondly, LaVIN can not identify extremely fine-
9

---

Agh
What is the weather
A=
rains now, will the get wet?
deh:
What is the baby
Agh What color is the
in the image now?
doing in the image?
baby's clothes?
VIV (ours)
LaVIN (ours)
LaVIN (ours)
the will get wet. In the image, the man is holding a baby and petting a brown horsc, and they are both standing under an overhang. If it starts raining, the overhang will not provide complete protection from the rain, and the man will get wel as he continues to interact haby.
Yes, it rains
image, baby
The buby's clothes are white. In the image, the man is holding the haby while the horse is nazzling baby's face. The baby is wearing white onesie, which stands out against the man's brown shirt and the horse's color.
being held by man, who is
also holding horse's nose.
The baby is reaching out to touch the horse's nose, likely experiencing the horse for the first time. interaction between the buby and the hurse creales moment of connection and curiosity for the child.
LAVIN (ours)
The weather in the image oppears he sunny, the man is holding a baby in his arms outside, and there are no visible signs rain or stormy conditions. The presence of the
with the horse
suggests that the location might be farm or rural area, where sunny weather is more common. GPT4 Seore:
GPT4 Score: 8
GPT4 Score: 9
GPT4 Score: 9
BLIP2 Yes. GPT4 Store: 6
BLIP2 Petting the borse. GPT4 Score:
BI.IP2 Blue. GPT4 Score:
GPT4 Score: 7
BLIPP: It is sunny day.
it starts raining, as already
No. the man will not get wet if
ILLaVA
The baby touching horse, reaching out to pet it while being held by the man.
LLaVA
'The baby is wearing white clothes.
LLaVA
inside the stable with the hoese.
LLaVA
The stable provides sheller from the rain, keeping hoth the man and the horse dry.
TThe weather in the image sunny.
GPT4 Score:
GPT4 Score: 7
GPT4 Score: 8
GPT4 Score:
MiniGPT4
MiniGPT4
MiniGIT4
MiniGPT4
:It is not possible to determine the current weather location depicted in the image. the image does not provide
:tt is not possible to determine whether the man in the image would get as the image does not provide any information about the current weather conditions or the environment in the area. The image shows man holding baby, but it dees not provide any information about the weather or the environment in the
:In the image, the baby is reaching out to touch the horse's nose. The baby is held up by the man, who is standing
The baby the image is not wearing any clothes.
the
starts raining.
any information about the weather conditions. The image shows
the horse, and the baby is reaching out with one hand to touch the horse's nose. The baby is smiling and seems to be enjoying the interaction the horse.
horse standing baby. and of building with thatched roof, but it does
provide any information about the weather or the environment in the area.
GPT4 Score:
GPT4 Score:
GPT4 Score:
GPT4 Score:
Figure 5: Comparison of LaVIN- 13B and existing multimodal LLMs in multi-turn conversations. GPT-4 assigns score ranging from 1 to 10 to evaluate the quality of a response, with a higher score indicating superior performance. The noteworthy aspects of the responses are highlighted in green, whereas the illogical portions are marked in red.
grained visual content, such as text characters. We believe that the recognition ability of LaVIN still has a large room to improve, which will be left in our future work.
6 Conclusions
In this paper, we propose a novel and affordable solution for vision-language instruction tuning, namely Mixture-of-Modality Adaptation (MMA). Particularly, MMA is an end-to-end optimization regime, which connects the image encoder and LLM via lightweight adapters. With the help of MMA, the entire multimodal LLM can be jointly optimized via small number of parameters, greatly reducing the training costs. Meanwhile, we also propose a novel routing algorithm in MMA, which can help the model automatically shifts the reasoning paths for single- and multi- modal instructions. Based on MMA, we develop a large vision-language instructed model called LaVIN, which demonstrates superior reasoning ability than existing multimodal L.Ms in various instruction-folloving tasks.
Acknovledgements. This work was supported by National Key R&D Program of China No.22022ZD0118201) the National Science Fund for Distinguished Young Scholars (No.62025603)
the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), the Natural Science Foundation of Fujian Province of China (00.2021001002,
No. 20.200220000,, ), and the China Fundamental Research Funds for the Central Universities (Grant No. 20720220068). We also thank Dr. Mingbao Lin for his valuable suggestions.
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23777 2022.
[2] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. CoRR, abs/2205. 2hs//205.13335, 2022.
[3] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-suuprrvsed learners. Advances in neural information processing systems (NeurlPS), 33:22243-22255,, 2020.
[4] Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
[6] Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cneruction-finttune language models. arXiv preprint 22v: 22...22111100 1416, 2022.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[8] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter. Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0333 03378, 2023.
[9] FairScale authors. Fairscale: A general purpose modular pytorch library for high performance and large scale training. https: / thub. com/ fa cebook research, c fa rsca le, 2021.
[10] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin. Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. / 13394, 2023.
[11] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint ar0iv:2304. 15010, 2023.
[12] Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Kirg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In ICLR, 2022.
[13] Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML., 2019.
[14] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[15] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 1404220022444 2023.
[16] Jing Yu Koh Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multi- modal generation. arXiv preprint arXiv:2301. 13823, 2023.
[17] Jumnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12500 2023.
[18] Junnan Li, Dongxu Li. Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman image pre- training for unitied vision-langual understanding and generation. In International Conference on Machine Learning, pages 12888-12 12900. PMLR, 2022.

---

[19] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/JICNLP, 2021.
[20] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqu: Mcasuring how models mimic human falsehoods. arXiv preprint urX1:2109.07955 2021.
[21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08433 2023.
[22] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/2111..0660000 2021.
[23] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.51001. 2017.
arXiv preprint
[24] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 35:2507-255 2521, 2022.
[25] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[26] Gen Luo, Minglang Huang. Yiyi Zhou, Xiaoshuai Sun, Guanman Jiang, Zhiyu Wang, and Rongrong Ji. Towards efficient visual adaption via structural re-parameterizationn arXiv preprint arXiv:2302.00 006. 2023.
[27] Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang. and Rongrong
Ji. Towards lightweight transformer via group-wise transformation for dision-and-languaae tasks. IEFE Transactions on Image Processing, 31:3386-3398 2022.
[28] Gen Luo. Yiyi Zhou, Xiaoshuai Sun, Yongjian Wu, Yue Gao, and Rongrong Ji. Towards language-guided visual recognition via dynamic convolutions. International Journal of Computer Vision, pages 1-19, 2023.
[29] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
[30] Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannanch Hajishirzi. Reframing instructional prompts to gptk's 's language. ACL Findings, 2021.
[31] Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: suite fundamental yet challenging mathematical reasoning tasks. The 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
[32] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09733 2021.
[33] OpenAl. Chatgpt, 2023.
[34] OpenAL. Gpt-4 technical report, 2023.
[35] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretehen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, Proceedings of Machine Learning Research, 2021.
[37] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8)99 2019.
[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wai Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:14001--40::1 2020.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffiusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages
2022.
12

---

[40] Noam Shazeer. Glu variants improve transformer. arXiv preprint urXiv:2002.052022 2020.
[41] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends in huggingface. arXiv preprint S.Kiv::23337758000 2023.
[42] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instraction-ollloiing llama model. https: /ggthbb. com/tatsur haattsu-labbaae stanford__ddr... alaca, 2023.
[43] Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Maric-Anna Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 2023.
[44] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1.0.0:8804.0741,, 2018.
[45] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 845-854, 2017.
[46] Yaqing Wang. Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adapeee for parameter-efficient tuning of large language models. arXiv preprint arXiv:2205. 12410, 2022.
[47] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint wrwii:2212..1055 2022.
[48] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabsshi, Yeganch Kordi, Amirreza Mirzaci, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasckaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2022.
[49] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.0467) 2023.
[50] Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasa,, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint orX3v:2303. /381, 2023.
[51] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfci Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama- adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[52] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00) 00066, 2022.
[53] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint ar2iv:2302.00923, 2023.
[54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, 2022.
[55] Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Deyu Meng, Yue Gao, and Chunhua Shen. Plenty plaguc: Fine-grained learning for visual question answering. IEEE transactions on pattern analysis and machine intelligence, 44(2):977770,, 2019.
[56] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:3004.10500 10592, 2023.
13


Reviews:
[{"review_id": "cxmzKu8SE7", "reviewer": "Reviewer_3xnb", "summary": "The paper presents a novel method to do efficient vision language fine-tuning. Through a mixture of modality adaptation mechanism, the model can close the gap between different modalities. Additionally, the paper proposes a routing algorithm to switch between multiple \ntasks. The training cost of the proposed system is low as the number of total trainable parameters is less than 4M. The model has been evaluated on public benchmark of ScienceQA. ", "questions": "- The model is only tested on one dataset, which might not be convincing enough regarding the effectiveness of the model. Can the authors evaluate the model using more public datasets?\n- The proposed model utilizes two adapters and one routing function to decide which adapter to use. The features generated after the routing function is affected by scale factor, will the performance be sensitive with the scale factor? Additionally, the routing function is \ndefined as a weighted summation, is there any other option that performs better?\n- How to make it possible to extend the model to more modalities, e.g. video, audio signal.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is efficient. The total trainable parameters are less than 4M.\n- The mixture-of-modality adaptation mechanism provides a way to adapt the LLM to vision modalities without expensive VL pretraining.\n- The proposed method has been evaluated on public available benchmark, achieves comparable results, and for some of the evaluation metrics, it surpasses some existing models that have larger sizes", "weaknesses": "For weaknesses, please see my questions in below section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your time and effort in reviewing this paper, and also thanks for your constructive comments on our work. Below, we response to your  concerns point by point. \n\n**Comment#1：** \n\nThe model is only tested on one dataset, which ight not be convincing enough regarding the effectiveness of the model.   \n\n**Response:**\n\nThanks for your this comment.  Following your suggestion, we supplement a more comprehensive comparison of LaVIN  on  COCO Captioning [K],  VQA2.0 [L], TruthfulQA [E] and MME Benchmark [F]， and compare  LaVIN with three representative approaches, namely LLaMA-adapter V2 [B],  BLIP [16] and BLIP-2 [15].  The results are  reported in the following tables. \n\n| Image Captioning *Karpathy test* | Pre-training Data | Updated Params | BLEU@4 | Cider |\n| -------------------------------- | :---------------: | :------------: | :----: | :---: |\n| ClipCap [A]                      |         0         |       -        |  33.5  | 113.1 |\n| LLaMA-Adapter V2 [B]             |         0         |      14M       |  36.2  | 122.2 |\n| BLIP [16]                        |        14M        |      583M      |  40.4  | 136.7 |\n| BLIP-2 [15]                      |       129M        |      188M      |  43.7  | 145.3 |\n| LaVIN (ours)                     |         0         |      5.4M      |  36.4  | 126.9 |\n| LaVIN (ours)                     |       0.6M        |      5.4M      |  37.8  | 131.7 |\n\n| VQAv2 *val*          | Overall | Number | Yes/no | Other |\n| -------------------- | :-----: | :----: | :----: | :---: |\n| LLaMA-adapter V2 [B] |  67.22  | 49.34  | 84.83  | 56.59 |\n| LaVIN (ours)         |  68.74  | 51.12  | 87.67  | 59.01 |\n\n| Zero-shot ThruthfulQA [E] | Acc  |\n| ------------------------- | :--: |\n| LLama [37]                | 38.7 |\n| LLaVA [18]                | 16.4 |\n| LLaMA-Adapter V2 [B]      | 24.4 |\n| LaVIN (ours)              | 47.9 |\n\n| Zero-shot MME benchmark [F] | Cognition | Perception | Overall |\n| --------------------------- | :-------: | :--------: | :-----: |\n| BLIP2 [15]                  |  1293.84  |    290     | 1583.84 |\n| LLaMA-adapter v2 [B]        |  972.67   |   248.93   | 1221.6  |\n| MiniGPT-4 [48]              |  866.58   |   292.14   | 1158.72 |\n| LLaVA [18]                  |  502.82   |   214.64   | 717.46  |\n| LaVIN (ours)                |  963.61   |   249.64   | 1213.25 |\n\nFrom the above tables, we can observe some important findings. \n\n1.  As as an parameter-efficient tuning method, our LaVIN are consistently better than LLaMA-Adapter v2 under both supervised and zero-shot settings. For example, LaVIN outperforms LLaMA-adapter V2 by +1.52% on VQAv2 and +23.5% on zero-shot TruthfulQA, respectively.\n2.   Compared with BLIP and BLIP-2 pre-trained on large-scale VL data, our performance is still competitive, while the expenditure is much cheaper. For instance, with only 0.6M pre-training data and 5.4M updated parameters, LAVIN can achieve 131.7 CIDEr on COCO Captioning. **Notably, our tuning only takes 4 GPU hours on 8 A100s, while BLIP-2 requires more than 300 GPU hours on 16 A100s.** \n3.   Our  NLP ability also outperforms existing methods. As discussed in the main paper, the  NLP capabilities  of most existing multimodal LLMs are  often undermined during VL instruction tuning.   On TruthfulQA, we can see that  their zero-shot performance is  obviously  inferior to the original LLaMA.   In stark contrast,  LaVIN can further improve the performance by +8.3% than LLaMA through its mixture-of-modality adaptation.\n\nOverall, we believe that these results can further validate the effectiveness and generalization ability of LaVIN.  Following your suggestion, we will  supplement these results and discussions to our final version.   \n\n\n\n**Comment#2：**\n\nWill the performance be sensitive with the scale factor? \n\n**Response:**\n\nYes, the selection of scale factor will  somewhat  affect the performance.  According to  LoRA [12],   the value of scale factor is positively related to the learning rate, but it does not need to be specifically tuned.\n\n\n\n**Comment#3：** Is there any other option for routing functions that performs better?\n\n**Response:**\n\n Thanks for this insightful question.   In fact, we ever tried two alternative solutions for LaVIN, namely *mean-routing* and *hard-routing*.  In particular, mean-routing refers to the average output of the two adapters and is static for all examples. Compared with our MMA, hard-routing is also dynamic, but only selects one adapter at each inference step. This binary routing scheme is often more difficult to optimize [M].  \n\nAs shown in the following table, these  alternatives are obviously inferior to our MMA, *i.e.,* the weighed summation. \n\n| ScienceQA                 | Overall  Acc |\n| ------------------------- | :----------: |\n| Mean routing              |    87.93     |\n| Hard routing              |    74.18     |\n| Weighted Summation（ours) |    89.41     |\n\n\n\n**Comment#4：**\n\nHow to extend the model to more modalities, e.g. video, audio signal ?\n\n**Response:**\n\nYes, the extension of LaVIN can be summarized as the following steps:\n\n1. Add  the backbones of the new modalities  for feature  embedding.\n2. Mix the  training data of different modalities.\n3. Add more routing paths in LLM  to adapt  different modalities.\n4. Conduct multimodal training  via our mixture-of-modality training scheme.\n\nBased on these steps, we have successfully extended LaVIN to the video  data, which will be discussed  in our  next work. \n\n\n\n[A] ClipCap: CLIP Prefix for Image Captioning\n\n[B] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model\n\n[E] TruthfulQA: Measuring How Models Mimic Human Falsehoods\n\n[F] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models\n\n[K] Microsoft COCO: Common Objects in Context\n\n[L] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering\n\n[M] ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware"}, {"review_id": "eWc1PCcQ0I", "reviewer": "Reviewer_q2tp", "summary": "This paper proposes Mixture-of-Modality Adaptation (MMA), which adopts lightweight adapters to bridge the gap between LLMs and VL tasks. The adapter utilizes a router to automatic shift between single-modal and multi-modal instructions. When applying MMA to LLaMA and training on both single-modal and multi-modal data, their proposed approach achieves competitive performance with supervised methods on ScienceQA dataset. Besides, the training of MMA is efficient and cheap.", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The mixture-of-modality adapter could dynamically adjust the adaptations for single-modal inputs and uni-modal inputs, which help preserve the NLP capability of LLMs.\n2. Achieve competitive performance given small number of training parameters.\n3. Ablation studies support the effectiveness of mixture-of-modality training and mixture-of-modality adaptation. ", "weaknesses": "1. The adapter idea has been extensively explored in previous efficient VL training, and using adapter to efficiently bridge vision and LLM has been explored in LLaMA-Adapter.\n2. Missing discussion/ablation of how the router in the MMA helps with the LLM learn visual information while preserving NLP capability.\n3. This paper only evaluates on ScienceQA. Evaluation on other benchmarks like (COCO Caption) is needed for better comparison with LLaMA-Adapter. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your careful review and constructive suggestions for this paper. Below, we response to your key concerns point by point. \n\n**Comment#1：** \n\nThe adapter idea has been extensively explored in previous efficient VL training, and using adapter to efficiently bridge vision and LLM has been explored in LLaMA-Adapter.\n\n**Response:**\n\nThanks for this comment. Compared with existing adapters, the main difference of our mixture-of-modality adapter (MM-Adapter) lies in its dynamic routing property. Unlike previous works, MM-Adapter can automatically switch between single- and multi-modal tasks, which can not only retain the NLP ability of LLaMA but also facilitate the cross-modal instruction tuning , see table 2. \n\nIn this paper, we also explore the mixture-of-modality training and reveal the importance of end-to-end optimization, *e.g.,* adding adapters for the visual backbone, which are also the key differences to LLaMA-Adapter. \n\nWith these innovative designs, our LaVIN is obviously better than LLaMA-Adapter on various single- and multi-modal benchmarks， of which results are given in below.  Notably, to our best knowledge, our method is also the most efficient multimodal adaptation scheme for LLaMA, which only takes 1.4 hours to adapt to ScienceQA. \n\n| COCO Captioning      | Updated Params | BLEU@4 | Cider |\n| -------------------- | -------------- | ------ | ----- |\n| ClipCap [A]          | -              | 33.5   | 113.1 |\n| LLaMA-Adapter V2 [B] | 14M            | 36.2   | 122.2 |\n| LaVIN (ours)         | 5.4M           | 36.4   | 126.9 |\n\n| VQAv2 *val*          | Overall | Number | Yes/no | Other |\n| -------------------- | ------- | ------ | ------ | ----- |\n| LLaMA-adapter V2 [B] | 67.22   | 49.34  | 84.83  | 56.59 |\n| LaVIN (ours)         | 68.74   | 51.12  | 87.67  | 59.01 |\n\n| Zero-shot ThruthfulQA [E] | Acc  |\n| ------------------------- | ---- |\n| LLama [37]                | 38.7 |\n| LLaVA [18]                | 16.4 |\n| LLaMA-Adapter V2 [B]      | 24.4 |\n| LaVIN (ours)              | 47.9 |\n\n**Comment#2：** \n\nMissing discussion/ablation of how the router in the MMA helps with the LLM learn visual information while preserving NLP capability .\n\n**Response:**\n\nThanks for this constructive comment.  As discussed above, the router in MMA can automatically choose the specific adapters for the single or multi-modal inputs, thus our MM-Adapters can learn visual information without contradicting to the default NLP capability of LLaMA.\n\nIn MMA, we also adopt a mixture-of-modality training (MMT) regime to jointly train MM-Adapters on data of different modalities, which can further improve the NLP and VL capabilities of our LaVIN.  For instance, compared with  LLaMA-Adapter and LLaMA-Adapter v2, our LAVIN demonstrates consistently better performance on four benchmark datasets, including COCO captioning, VQAv2, ScienceQA and TruthfulQA.\n\nFollowing your suggestion, we will add the above discussions to our final submission.\n\n\n\n**Comment#3：** \n\nThis paper only evaluates on ScienceQA. Evaluation on other benchmarks like (COCO Caption) is needed for better comparison with LLaMA-Adapter.\n\n**Response:**\n\nThanks for this suggestion. The comparisons with LLaMA-adapter v2 on COCO captioning and VQA2.0 are given in the following tables. \nFrom these results, we can see that our LaVIN is consistently better than LLaMA-Adapter v2 in both performance and parameter efficiency. \n\n| COCO Captioning      | Updated Params | BLEU@4 | Cider |\n| -------------------- | :------------: | :----: | :---: |\n| ClipCap [A]          |       -        |  33.5  | 113.1 |\n| LLaMA-adapter V2 [B] |      14M       |  36.2  | 122.2 |\n| LaVIN (ours)         |      5.4M      |  36.4  | 126.9 |\n\n| VQAv2 *val*          | Overall | Number | Yes/no | Other |\n| -------------------- | ------- | ------ | ------ | ----- |\n| LLaMA-adapter V2 [B] | 67.22   | 49.34  | 84.83  | 56.59 |\n| LaVIN (ours)         | 68.74   | 51.12  | 87.67  | 59.01 |\n\n[A] ClipCap: CLIP Prefix for Image Captioning\n\n[B] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model\n\n[E] TruthfulQA: Measuring How Models Mimic Human Falsehoods"}, {"review_id": "kTT5uoPKbT", "reviewer": "Reviewer_mWDr", "summary": "> Update: I bumped up my rating to 6 after rebuttal\n\nThis paper proposes, LaVIN, an efficient and effective vision-language instruction tuning scheme to adapt LLMs. Specifically, the authors utilize parameter-efficient modules to adapt the LLaMA LM – they insert several adapters to the image encoder and mixture of modality adapters to the LM, the LM is expected to automatically select and route through the adapters of different modalities. LaVIN can be trained in an end-to-end fashion. Because only adapters are learned during training, LaVIN training is much more efficient than LLaVA that uses full tuning. Instruction tuning in LaVIN includes both text-image data and text-only data in a multi-task fashion (but with better separation due to adapters) to enable text-image instructions and text-only instruction at inference time. Strong results are achieved on ScienceQA.\n", "questions": "As mentioned in the weakness section, I think LLaMA-adapter should be described early in the paper and a fair comparison with it is needed.\n", "limitations": "The authors have discussed the limitations of the proposed method in Section 5.\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is simple and sound. By the use of a bunch of adapters, the model can naturally learn different-modality instructions in an end-to-end fashion.  \n2. LaVIN is much more efficient than LLaVA.  \n3. Empirical results on ScienceQA are strong.  \n4. The paper is well-written.\n", "weaknesses": "1. I think the proposed method resembles LLaMA-adapter a lot, maybe the authors should better note their difference early in the paper – currently LLaMA-adapter is not described until the experiment.  \n2. The experimental comparison with LLaMA-Adapter is not an apple-to-apple comparison because  LLaMA-Adapter and LaVIN use different instruction tuning datasets – the performance gap may simply originate from the instruction tuning datasets. A better baseline should be using LLaMA-Adapter on the same instruction tuning datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your time and effort in reviewing this paper, and also thanks for  positive rating and beneficial feedback. Below, we response to your key concerns point by point. \n\n**Comment#1：**  \n\nI think the proposed method resembles LLaMA-adapter a lot, maybe the authors should better note their difference early in the paper – currently LLaMA-adapter is not described until the experiment.\n\n**Response:**\n\nThanks for your suggestion.   The main differences between our work and LLaMA-adapter lies in the following aspects:\n\n1.   LLaMA-adapter is a static multi-modal approach, while our MM-Adapter can dynamically adjust its adaption across single-and multi-modal tasks.   This  dynamic routing design can not only help LLMs retain the NLP ability  but also facilitate the cross-modal instruction tuning, see the following table.. \n2.  In our mixture-of-modality adaptation (MMA), we  reveal the importance of end-to-end optimization for multi-modal LLMs,  *i.e.,* adding more adapters into visual backbone,  and also adopts the mixed modality training to improve the cross-modal training.   These innovative designs are also greatly different from LLaMA-adapter.\n\nBased on these differences, we think that our contribution is orthogonal to LLaMA-adapter.   Meanwhile, to our best knowledge, our MMA is currently the most efficient transfer learning scheme for LLaMA, which only takes 1.4 GPU hours to adapt to ScienceQA.\n\nFollowing your suggestion, we will highlight the above differences in our new version. \n\n| Zero-shot ThruthfulQA [E] | Acc  |\n| ------------------------- | :--: |\n| LLama [37]                | 38.7 |\n| LLaVA [18]                | 16.4 |\n| LLaMA-Adapter V2 [B]      | 24.4 |\n| LaVIN (ours)              | 47.9 |\n\n| Zero-shot MME benchmark [F] | Cognition | Perception | Overall |\n| --------------------------- | :-------: | :--------: | :-----: |\n| BLIP2 [15]                  |  1293.84  |    290     | 1583.84 |\n| LLaMA-adapter v2 [B]        |  972.67   |   248.93   | 1221.6  |\n| MiniGPT-4 [48]              |  866.58   |   292.14   | 1158.72 |\n| LLaVA [18]                  |  502.82   |   214.64   | 717.46  |\n| LaVIN (ours)                |  963.61   |   249.64   | 1213.25 |\n\n**Comment#2：**  \n\nThe experimental comparison with LLaMA-Adapter is not an apple-to-apple comparison because LLaMA-Adapter and LaVIN use different instruction tuning datasets – the performance gap may simply originate from the instruction tuning datasets. A better baseline should be using LLaMA-Adapter on the same instruction tuning datasets.\n\n**Response:**\n\nThanks for this  suggestion.    In fact, the experimental comparison on ScienceQA and Multi-modal Chatbot are relatively fair  for LLaMA-Adapter and our LaVIN.   Particularly, both models are trained from scratch on ScienceQA and use the same text instruction data on Multi-modal Chatbot.  \n\nTo further address your concerns, we also supplement the comparisons on COCO Captioning and VQA2.0, of which experimental settings are the same for both methods. It can be seen that LaVIN is consistently better than LLaMA-Adapter in both performance and efficiency. \n\n| COCO Captioning      | Updated Params | BLEU@4 | Cider |\n| -------------------- | -------------- | ------ | ----- |\n| ClipCap [A]          | -              | 33.5   | 113.1 |\n| LLaMA-adapter V2 [B] | 14M            | 36.2   | 122.2 |\n| LaVIN (ours)         | 5.4M           | 36.4   | 126.9 |\n\n| VQAv2                | Overall | Number | Yes/no | Other |\n| -------------------- | ------- | ------ | ------ | ----- |\n| LLaMA-adapter V2 [B] | 67.22   | 49.34  | 84.83  | 56.59 |\n| LaVIN (ours)         | 68.74   | 51.12  | 87.67  | 59.01 |\n\n[A] ClipCap: CLIP Prefix for Image Captioning\n\n[B] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model"}, {"review_id": "Hh7AfN8jIU", "reviewer": "Reviewer_mD9H", "summary": "This paper presents a cost-efficient method to fine-tune LLMs thus enabling their multimodal reasoning capabilities. The main technical contribution includes using Mixture-of-Modality Adapation, which adopts lightweight adapters to bridge the gap between modality gaps. In the meanwhile, MMA also allows automatic routing such that the model can process both multimodal prompts and text-only prompts. Experiments show superior results on ScienceQA and training efficiency, in terms of both training time and number of trainable parameters.", "questions": "NA", "limitations": "Limitations are discussed.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method may help significantly reduce the fine-tuning cost of LLMs with multimodal inputs. \n- The proposed method is conceptually concise and generic. It could be potentially incorporated into different models/systems thus harvesting the development of both LLMs and vision encoders.\n- Results on ScienceQA is promising, surpassing some strong competitors such as GPT-4.", "weaknesses": "- Evaluation is not sufficiently convincing. Only quantitative results on ScienceQA are presented. This leads to a narrowed view of the multimodal capabilities of the fine-tuned model. For example, does the model perform equally well on image captioning datasets such as NoCaps, COCO; how does the model perform on cross-modal retrieval / other VQA benchmarks? Missing these results makes it difficult to understand whether the proposed fine-tuning paradigm is actually bridging the modality gap as good as some previous works.\n- In L50 and also Figure 1(b), authors claim \"this paradigm often requires to update most parameters of LLM, limiting the efficiency of VL instruction tuning\", \"these fine-tune schemes will inevitably undermine the NLP capabilities of LLMs due to the drastic changes in their parameter spaces\" and \"existing multimodal LLM do not support text-only instructions, greatly hindering their applications\". I wouldn't say these are correct. For example, BLIP-2/MiniGPT4 both keep LLMs frozen without updating their parameters, thus one can always prompt its LLMs with text-only prompt without degradation in language generation quality. I am therefore not fully convinced by the motivation of having automatic routing, instead of having just frozen LLMs speaking of to preserve the text generation capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts spent in this paper.  Below, we response to your concerns point by point. \n\n**Comment#1：**  \n\nEvaluation is not sufficiently convincing. Results on image captioning and VQA should be reported.\n\n**Response:**\n\nThanks for this  constructive comment.  Following your suggestion, we conduct more experiments on other VL benchmarks, including your suggested COCO Captioning and VQA2.0, and  zero-shot multimodal evaluation and zero-shot question answering.    For a fair comparison, we follow the settings of previous works[B,18,37] on these benchmarks. \n\n| Image Captioning *Karpathy test* | Pre-training Data | Updated Params | BLEU@4 | Cider |\n| -------------------------------- | :---------------: | :------------: | :----: | :---: |\n| ClipCap [A]                      |         0         |       -        |  33.5  | 113.1 |\n| LLaMA-Adapter V2 [B]             |         0         |      14M       |  36.2  | 122.2 |\n| LaVIN (ours)                     |         0         |      5.4M      |  36.4  | 126.9 |\n| BLIP [16]                        |        14M        |      583M      |  40.4  | 136.7 |\n| BLIP-2 [15]                      |       129M        |      188M      |  43.7  | 145.3 |\n| LaVIN (ours)                     |       0.6M        |      5.4M      |  37.8  | 131.7 |\n\n| VQAv2 *val*          | Overall | Number | Yes/no | Other |\n| -------------------- | :-----: | :----: | :----: | :---: |\n| LLaMA-adapter V2 [B] |  67.22  | 49.34  | 84.83  | 56.59 |\n| LaVIN (ours)         |  68.74  | 51.12  | 87.67  | 59.01 |\n\n| Zero-shot ThruthfulQA [E] | Acc  |\n| ------------------------- | :--: |\n| LLama [37]                | 38.7 |\n| LLaVA [18]                | 16.4 |\n| LLaMA-Adapter V2 [B]      | 24.4 |\n| LaVIN (ours)              | 47.9 |\n\n| Zero-shot MME benchmark [F] | Cognition | Perception | Overall |\n| --------------------------- | :-------: | :--------: | :-----: |\n| BLIP2 [15]                  |  1293.84  |    290     | 1583.84 |\n| LLaMA-adapter v2 [B]        |  972.67   |   248.93   | 1221.6  |\n| MiniGPT-4 [48]              |  866.58   |   292.14   | 1158.72 |\n| LLaVA [18]                  |  502.82   |   214.64   | 717.46  |\n| LaVIN (ours)                |  963.61   |   249.64   | 1213.25 |\n\nFrom the above tables, we can observe some important findings. \n\n1. As as an parameter-efficient tuning method, our LaVIN are consistently better than LLaMA-Adapter v2 under both supervised and zero-shot settings. For example, LaVIN outperforms LLaMA-adapter V2 by +1.52% on VQAv2 and +23.5% on zero-shot TruthfulQA, respectively.\n2. Compared with BLIP and BLIP-2 pre-trained on large-scale VL data, our performance is still competitive, while the expenditure is much cheaper. **Notably, our tuning only takes 4 GPU hours on 8 A100s, while BLIP-2 requires more than 300 GPU hours on 16 A100s.** \n3. Our  NLP ability also outperforms existing methods.   On TruthfulQA, we can see that  their zero-shot performance is  obviously  inferior to the original LLaMA.   In stark contrast,  LaVIN can further improve the performance by +8.3% than LLaMA through its mixture-of-modality adaptation.\n\nOverall, we believe that these results can further validate the effectiveness and generalization ability of LaVIN.  Following your suggestion, we will  supplement these results and discussions to our final version.   \n\n\n\n**Comment#2：**  \n\nThe claiming about  \"the NLP capabilities declines in fully tuned multimodal LLM'' are not correct, since BLIP-2/MiniGPT4 both keep LLMs frozen without degradation of NLP ability.  I am therefore not fully convinced by the motivation of automatic routing, instead of having just frozen LLMs.\n\n**Response:**\n\nThanks for your careful review. We agree that MiniGPT-4 and BLIP-2 are inappropriate  examples  to support the argument of  \"full VL  fine-tuning\".    Nevertheless, their shortcomings lie in their expensive parameter and training costs, as discussed in Line 49-50 of our paper. For instance, BLIP-2 requires a deep neck branch to connect the vision and language models, which needs to be pre-trained on massive VL data for over 200 hours on 16 A100s.  In stark contrast, our LaVIN can only update 5.4M parameters and require only about 2 hours on 8 A100s. \n\nAside from the inappropriate examples, our argument about full VL tuning still stands. Full VL tuning is still a popular paradigm to adapt LLMs to VL tasks [18,G,H,I,J], and it does greatly undermine the NLP capabilities of the used LLMs. For instance, after being fully tuned on VL data, the performance of LLaVA greatly declines on the zero-shot TruthfulQA, a NLP benchmark, especially compared with the default LLaMA. \n\nMeanwhile, the common adapter-based schemes are also sub-optimal. When using the adapter  VL data to handle examples of mixed modalities, its performance is still greatly affected on NLP evaluations, see the following table.  Besides, the performance of existing adapter-based methods like LLaMA-Adapter significantly lags behind that of fully tuned approach, *i.e.,* LLaVA, see our comparison on ScienceQA. \n\nCompared with the above paradigms, our MM-Adaption  provides an elegant  yet effective way to automatically handle the inputs of mixed modalities, *i.e.,* dynamic routing.  **Notably,  on zero-shot TruthfulQA, our NLP performance is even obviously better than the frozen LLaMA.**\n\n\n\n\n[A] ClipCap: CLIP Prefix for Image Captioning\n\n[B] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model、\n\n[E] TruthfulQA: Measuring How Models Mimic Human Falsehoods\n\n[F] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models\n\n[G] Otter: A Multi-Modal Model with In-Context Instruction Tuning \n\n[H] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic\n\n[I] LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding \n\n[J] GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest \n"}, {"review_id": "QWMMIdvd06", "reviewer": "Reviewer_4c4u", "summary": "This paper presents a novel method to connect vision-language instruction tuning with large language models.\nTo implement this, the authors introduce the adapters rather than heavy bottlenecks between vision-language input tokens and LLM.\nA routing mechanism is designed to adaptively choose the right direction for different modalities of inputs.\nAfter applying this method to the recent strong LLaMa model, the proposed method achieves efficient instruction tuning and close the performance gap with that of the original large language model.", "questions": "see the weaknesses above", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and most parts of this paper are easy to follow.\n- The proposed method achieves significant performance improvement pertaining to instruction tuning while using a very small magnitude of trainable parameters.\n- The proposed method demonstrates favorable results on both single- and multi-modal instructions.", "weaknesses": "- My biggest concern lies in the routing mechanism.\n  - Why do we need the routing between different kinds of inputs? Can we just use determined inputs to the modality-aware adapters?\n  - What the role of ```s``` is is not well explained.\n  - It seems the router output is simply a weighted sum of the two adapters. How should we explain this?\n\n- It seems the authors also introduce adapters to the image encoder. This should also be explained.\n\n- For the image input features, the authors use the ```[CLS]``` of every fourth layer of CLIP-ViT, are there any rationales behind this?\n\n- I'm confused about Sec. 3.2. It seems that Sec. 3.2 can be cohesively organized with that of MMT. Are there more considerations about this section? If not, the writing should be better organized.\n\n- Some typos:\n  - Line 156, ```m``` and ```n``` are in reverse order;\n  - Line 170, the dimension definition is not correct.\n\n \n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your careful review for this paper. Your beneficial feedback and valuable suggestions indeed improve our paper a lot.  Below, we response to your key concerns point by point. \n\n**Comment#1：**  \n\nWhy do we need the routing between different kinds of inputs? Can we just use determined inputs to the modality-aware adapters?\n\n**Response:**\n\n Thanks for this insightful question. Compared to  our MM-adapter,  the shortcomings of modality-specific adapters  lies in two aspects: \n\n1. **Inference flexibility.**  Modality-specific adapters require to manually set up the inference adapters according to the input modalities, while MM-adapter can automatically adjust its inference paths, which is more convenient. \n2. **Training efficiency.** Modality-specific adapters can not be benefited from the mixture-of-modality training.  In modality-specific adapters,  each adapter is separately optimized by one kind of input, greatly reducing the training efficiency. Under the same training epoch, the performance of modality-specific adapters on ScienceQA is 83.76,  obviously inferior to 89.41 of MM-adapter.\n\nOverall, the proposed MM-adapter is still a better solution than modality-specific adapters in practice.\n\n\n\n**Comment#2：**  \n\nWhat the role of s  is not well explained.\n\n**Response:**\n\n''s''  is the scale factor to adjust the  numeric magnitude [12] for stable training, which is  set to 1 in our experiments. We will add more explanations in our final version.\n\n\n\n**Comment#3：**  \n\nIt seems the router output is simply a weighted sum of the two adapters. How should we explain this?\n\n**Response:**\n\n Thanks for this comment.  Weighted summation is a simple  yet effective way to  accomplish the adapter routing in LLMs. As shown in our appendix, the routing weights become shaper as the the network goes deeper.   We have also tried  two alternative schemes,  namely *mean routing* and *hard routing*, which achieves inferior performance than the weighted summation on ScienceQA.\n\n| ScienceQA          | Overall  Acc |\n| ------------------ | ------------ |\n| Mean routing       | 87.93        |\n| Hard routing       | 74.18        |\n| Weighted Summation | 89.41        |\n\n**Comment#4：**  \n\nIt seems the authors also introduce adapters to the image encoder. This should also be explained.\n\n**Response:**\n\n Thanks for this insightful comment. This setting is also an important finding of our paper. To explain,  the adapters in image encoder can help to narrow the domain gap between the pre-trained and down-stream images.   For example, ScienceQA  includes a large number of synthetic images, which are  barely seen  by the pre-trained image encoder before, *i.e.,* CLIP-ViT. \n\n\n\n**Comment#5：**  \n\nFor the image input features, the authors use the `[CLS]` of every fourth layer of CLIP-ViT, are there any rationales behind this?\n\n**Response:**\n\n Thanks for this careful review.   Using [CLS] features at different scales  can provide richer semantic information for multimodal tasks.  To explain, semantic information of visual features usually varies at different depths of the network. For example, shallow layers typically encode low-level semantic, such as texture and color. In this case, the use of visual features from different layers will facilitate the learning of attribute words.\n\n\n\n**Comment#6：**  \n\nI'm confused about Sec. 3.2. It seems that Sec. 3.2 can be cohesively organized with that of MMT. Are there more considerations about this section? If not, the writing should be better organized.\n\n**Response:**\n\nThanks for your suggestion. Sec. 3.2 mainly describes the model details of LaVIN, which can be combined together  with the descriptions of MMT. Following your suggestion, we will re-organize this section to improve the readability.\n\n\n\n**Comment#7：**  \n\n- Some typos:\n  - Line 156, `m` and `n` are in reverse order;\n  - Line 170, the dimension definition is not correct.\n\n**Response:**\n\nThanks for your careful review  and  we will revise these typos in our final version."}]

Author rebuttal:
Dear Reviewers:



We thank all reviewers for their valuable and encouraging comments on the novelty and technical contributions of our paper, such as *"it surpasses some existing models that have larger size"*, *"achieve competitive performance given small number of training parameters "*, *"LaVIN is much more efficient than LLaVA"* and *" favorable results "*.   During the rebuttal phrase, our main responses include:  

1.   Additional evaluations on new VL benchmarks, including COCO captioning, VQAv2, TruthfulQA and MME Benchmark, which  further validate the effectiveness and generalization  of  our method.

2. The extensive comparisons with LLaMA-Adpater on four additional benchmarks, which well confirm our superiority over this similar PETL methods. 

3. The active discussion about the role of our cross-modality routing design, which is more convenient and effective than existing solutions towards handling the inputs of mixed modalities. 

In addition, the key concerns of all reviewers are point-by-point addressed in each rebuttal.

   







Lastly, the codes of new experiments in our response are anonymously released at: https://anonymous.4open.science/r/LaVIN--1067, and the technical details can be found in the attachment. 

 

 

Best,

The Authors

Meta-review: This paper proposes a novel dynamic routing method with the mixture of adapters for efficient adaption of LLM for VL instruction model while maintaining the original NLP capability. Overall, there are two main concerns raised by reviewers: i) the lack of analysis on the proposed mixture of adapters and routing mechanism, which lead to efficient adaptation with such small number of parameters and even without additional VL training, ii) insufficient benchmark evaluation for VL and Language-only understanding tasks. During the rebuttal period, the authors have addressed these two concerns with more detailed explanation and additional experimental results, and 4 out of 5 reviewers have inclined to positive ratings. I agree with these reviewers that the concerns are somewhat resolved, and I recommend this paper to be accepted. However, the authors still need to address these two concerns more thoroughly, for example, more in-depth analysis (not just the final metric) on the benefits from the mixture and routing especially with the connection to the resulting efficiency, and more empirical validation on VL and Language-only understanding tasks. At this moment, it seems that this paper overstates the unnecessity of additional VL training by considering inferior results compared to BLIP-2.
Decision: Accept (poster)