SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The results show that LLaVA remains competitive performance against MM-CoTLarge[53], especially in the category of SOC. Despite the effectiveness, its number of trainable parameters is still large, leading to higher training overhead. LLaMA-Adapter [51] adopts a parameter-efficient scheme to reduce the training overhead, but its performance still greatly lags behind LLaVA. Compared to these approaches, LaVIN achieves the better trade-offs between performance and training efficiency. For exam
ple, LaVIN-7B consumes a similar scale of trainable parameters as LLaMA-Adapter [51], while outperforming it by  $+4.22$  gains. When scaling up to 13B, LaVIN can obtain more significant performance gains, i.e.,  $+5.64$ . Compared to LLaVA, LaVIN-13B also achieves comparable performance and even performs better in some question classes, e.g., LAN and NO. Considering the much lower training costs than LLaVA, such competitive performance greatly confirms the efficiency and designs of LaVIN.
Methods	#T-Params	Accuracy
LLaVA [21]	13B	85.81
LLaMA-Adapter [51]	1.8M	85.19
LaVIN-7B	3.8M	89.41 (+4.22)
LaVIN-13B	5.4M	90.83 (+5.02)
In Tab. 3, we compare LaVIN with existing methods without VL pretraining. From this table, we observe that both LLaVA [21] and LLaMA-Adapter achieve the similar performance, i.e., 85.81 vs. 85.19. In particular, LLaVA [21] and LLaMA-Adapter [51] freeze the image backbone, and the entire multimodal LLM is not jointly optimized, which hinders the learning of visual content. Moreover, the adaptation module in LLaMA-Adapter does not consider
the modality gap in the input instructions, greatly limiting its performance upper bound. In contrast, with the help of MMA, LaVIN significantly outperforms these approaches, e.g., +5.02 gains over LLaVA. These results validate the proposed MMA towards the effective and efficient VL adaption, and confirm the designs of LaVIN.
Table 3: Results of LaVIN and existing multimodal LLMs without the pre-training stage. We report the average accuracy on ScienceQA test set.
Methods	PT Data	#T-Params	BLEU-4	CIDEr
ClipCap [32]	0	-	33.5	113.1
LLaMA-Adapter V2 [11]	0	14M	36.2	122.2
BLIP [18]	14M	583M	40.4	136.7
BLIP-2 [17]	129M	188M	43.7	145.3
LaVIN (ours)	0	5.4M	36.4	126.9
LaVIN (ours)	0.6M	5.4M	37.8	131.7
Table 4: Fine-tuning results of LaVIN and existing multimodal LLMs on COCO captioning. We report performance on Karpathy test split.
Results on COCO Captioning. In Tab 4, we compare LaVIN with existing methods on the task of image captioning. From these results, we can still observe the competitive performance of LaVIN. As a parameter-efficient tuning method, LaVIN outperforms LLaMA-Adapter v2 [11] by a large margin, e.g., up to +9.5 of CIDEr. Compared with large-scale pre-training models, e.g., BLIP and BLIP-2, the performance of LaVIN is still comparable, while the expenditure is much cheaper. For instance, with only 0.6M pre-training data and 5.4M updated parameters, LAVIN can achieve 131.7 CIDEr on COCO Captioning. Notably, our tuning only takes 4 GPU hours on 8 A100s, while BLIP-2 requires more than 300 GPU hours on 16 A100s. These results further validate the effectiveness and training efficiency of MMA and LaVIN.
Zero-shot evaluation on NLP and multimodal benchmarks. In Tab. 5, we evaluate the zero-shot ability of LaVIN and existing methods on TruthfulQA [20] and MME [10]. On TruthfulQA [20], we observe that the zero-shot performance of existing multimodal LLMs is obviously inferior to the original LLaMA. In stark contrast, LaVIN can further improve the performance by  $+9.2\%$
than LLaMA-Base [43] through its mixture-of-modality adaptation. On MME [10], a challenging benchmark for multimodal evaluation, LaVIN still demonstrates competitive performance against existing multimodal LLMs. Expect for BLIP-2 [17], which is pre-trained on numerous data, the other methods perform similarly to or worse than LaVIN, e.g., 866.5 of MiniGPT-4 vs. 963.6 of LaVIN on MME-C. These results confirm the strong generalization ability of LaVIN, and also validate that the NLP capabilities are well preserved by MMA during VL instruction tuning.
Ablation study. To gain deep insights into MMA and LaVIN, we conduct comprehensive ablation studies in Tab. 2. From this table, we can see that each design of MMA and LaVIN greatly contributes to the final performance. As shown in Tab. 2, the mixture-of-modality training (MMT) brings the most significant gains, e.g., +4.69. In MMT, the joint training with the vision modality provides up to +3.67 performance gains for LaVIN. With the joint optimization of the image encoder and LLM, the performance of LaVIN further boosts from 86.32 to 87.34, suggesting the significance of the joint optimization for multimodal LLMs. With the help of MMT, LaVIN already surpasses the ex
Methods	TruthfulQA	MME-C	MME-P
LLaMA-Base [43]	38.7	-	-
LLaMA-Adapter V2 [11]	24.4	972.6	248.9
LLaVA [21]	16.4	502.8	214.6
BLIP-2 [17]	-	1293.8	290.0
MiniGPT-4 [56]	-	866.5	292.1
LaVIN (ours)	47.9	963.6	249.6
Table 5: Zero-shot results on NLP and multimodal benchmarks. "Mc1_targets" setup is used on TruthfulQA [20]. "MME-C" and "MME-P" denote the splits of Cognition and Perception on MME benchmark [10], respectively.
isting parameter-efficient method, i.e., LLaMA-Adapter. Additionally, the stronger image encoder, i.e., ViT-L/14, also improves the average accuracy by 0.99. An interesting observation is that a better image encoder provides noticeable performance gains for both image-based and text-based questions. When adopting MM-Adapter to LaVIN, we observe  $+1.08$  gains on average accuracy. Such an improvement only requires extra 0.9M parameters, which is very lightweight. Meanwhile, the performance of LaVIN is significantly improved by MM-Adapter on more challenging metrics like G7-12, i.e., +2.51. After scaling up LLM to 13B, the performance of LaVIN is further improved by  $+1.09$ . Overall, these ablations well validate the significance of MMA in adapting multimodal LLM, and also confirm the effectiveness of LaVIN.
Comparison of training efficiency. In Tab. 6, we compare the training expenditures of LaVIN, LLaVA [21] and BLIP2 [17]. The first observation is that the pre-training cost of BLIP2 is actually expensive, which requires more than 200 hours. Meanwhile, LLaVA cannot be trained on common machines with the default training settings<sup>3</sup>. Thus, it requires some GPU memory-saving techniques [9] to avoid out of memory (OOM). However, its training time and storage requirement are still significant. For example, it still takes up to 26GB space to store the updated parameters of the LLM. In contrast, LaVIN demonstrates superior training efficiency with the help of MMA. Compared to LLaVA,
Methods	#T-Params	Memory	Time	#Storage
BLIP2 [17]	188M	-	>200 hours	-
LLaVA [21]	13B	OOM	N/A	N/A
LLaVA‡ [21]	13B	36.8G	7 hours	26GB
LaVIN-7B	3.8M	33.9G	1.4 hours	15M
LaVIN-13B	5.4M	55.9G	2 hours	20M
Table 6: Training costs of LaVIN and existing multimodal LLMs on ScienceQA.  $\ddagger$  denotes that GPU memory-saving techniques are used. "OOM" denotes out of GPU memory. All results are evaluated on 8 A100 GPUs.
LaVIN-7B and LaVIN-13B reduce about  $80\%$  and  $71.4\%$  training time, respectively. In terms of GPU memory and storage cost, our approach can save more than  $40\%$  GPU memory and  $99.9\%$  disk storage. Overall, these results greatly confirm the training efficiency of MMA.
4.3.2 Qualitative Experiments
Examples of different instruction-following tasks. In Fig 4, we compare LaVIN with existing methods [51, 21] on single- and multi-modal instruction-following tasks, e.g., math, coding and image captioning. Compared to LLaVA [21] and LLaMA-Adapter [51], LaVIN achieves overall better responses across multiple tasks. In Fig.4 (a), LaVIN correctly answers the math problem with a result of 28.8, whereas LLaMA-Adapter [43] provides an incorrect answer. In example (d), LaVIN generates accurate code for the request of "print prime numbers up to 100". In contrast, the
<sup>3</sup>https://github.com/haotian-liu/LLaVA
If a toy costs $24 and the price is increased by 20%, what is the new price of the toy?
LaVIN (ours):
The new price of the toy is $28.80.
LLaMA-Adapter:
The new price of the toy is \(28.40.
Translate the following
Chinese into English :生如夏花之绚烂死如秋叶之静美.
Life is like a summer flower, death is like a autumn leaf, quiet and beautiful.
Born like summer flowers, die like autumn leaves.
(a) Math
How many food containers are there in the image?
There are three food containers in the image: a bowl, a plate, and a sandwich.
I mix water with water, do I get thin water or thick water?
u will get the same type of water, just a mixture of the 0.
Thin water.
(b) Translation
(c) Question Answering
Describe the Image.
(f) Image Captioning
LaVIN (ours)
The image features a beach scene with a large number of kites flying in the sky.
LLaVA
: The image captures a lively beach scene with several people enjoying kite flying near the ocean.
Write a python code that: Print out all prime numbers up to 100
1 for i in range(1,101):
2 if all(i % j)
3 print(j)
1 def prime(numbers(n):
2 for j in range(2, n):
if n % i == 0;
11 H 1
5 else:
6 print(i)
7 primeNumbers(100)
(d) Coding
(g) Multimodal Reasoning
Figure 4: Comparison between LaVIN-13B and existing methods on single- and multi-modal instructions. The noteworthy aspects of the responses are highlighted in green, whereas the illogical portions are marked in red. More tasks and examples are given in appendix.
The boy's clothes in the image are wet, is it because of the rain?
LaVI
:No, the boy's clothes are wet because he is surfing in the ocean.
: Yes, it is likely that the boy's clothes are wet due to the rain.
code written by LLaMA-Adapter is to check prime numbers, which does not produce any output during execution. Meanwhile, LaVIN presents a clear and concise coding behavior, acting more like a professional programmer. In Fig 4 (e)-(g), LaVIN demonstrates remarkable visual reasoning ability in accomplishing various multimodal tasks. In Fig.4 (e), LaVIN accurately answers the complex questions about the number of food containers in the image and provides a detailed description about the complex scene. The same observation can also be witnessed in Fig.4 (g), where LaVIN infers a correct reason for the wetness of the boy's clothes. Overall, these examples show the superior reasoning ability of LaVIN in executing single- and multi-modal instructions, while also confirming the significance of MMA in adapting LLMs to multi-modal tasks.
Examples of multimodal dialogue In Fig. 5, we compare LaVIN with existing multimodal LLMs in multi-turn conversations, and use GPT4 [34] to evaluate the quality of their responses. From the results, we can see that LaVIN has higher GPT4 scores among all compared models, suggesting superior ability in multimodal dialogue. Meanwhile, we also observe different response styles of these multimodal LLMs. In particular, BLIP2 [17] tends to produce brief responses, which lack detailed explanations. In contrast, the responses of MiniGPT4 [56] are the longest among all models, but their content is often redundant and repetitive. Compared to them, LaVIN and LLaVA [21] can generate more accurate responses. Particularly, LaVIN performs better than the other methods, mainly due to its more logical and detailed descriptions. As illustrated in the first question, LaVIN not only provides the correct answer, but also explains the reason behind it. In the second question, LaVIN and LLaVA are required to judge whether the man will get wet, and LaVIN answers "yes" while LLaVA considers "no". It can be seen that the reason of LaVIN is more comprehensive, logical and persuasive than LLaVA, which considers the situation of "the overhand may not provide the complete protection". Overall, these examples confirm that MMA equips LLMs with excellent multi-modal ability, requiring no pre-training on large-scale image-text data.
5 Limitations and Broader Impact
We observe two primary limitations of LaVIN. Firstly, LaVIN may generate incorrect or fabricate responses, similar to existing multimodal LLMs. Secondly, LaVIN can not identify extremely fine-
: What is the weather in the image now?
LaVIN (ours)
:If it rains now,will the man get wet?
LaVIN (ours)
What is the baby doing in the image?
LaVIN (ours)
BLIP2: Petting the horse.  
GPT4 Score: 7
LLaVA
: What color is the baby's clothes?
LaVIN (ours)
BLIP2:Blue. GPT4 Score:2
LLaVA
The weather in the image appears to be sunny, as the man is holding a baby in his arms outside, and there are no visible signs of rain or stormy conditions. The presence of a horse in the scene suggests that the location might be a farm or a rural area, where sunny weather is more common. GPT4 Score: 9
BLIP2: It is a sunny day.  
GPT4 Score: 7
MiniGPT4
BLIP2:Yes, GPT4 Score:6
LLaVA
MiniGPT4
MiniGPT4
MiniGPT4
Figure 5: Comparison of LaVIN-13B and existing multimodal LLMs in multi-turn conversations. GPT-4 assigns a score ranging from 1 to 10 to evaluate the quality of a response, with a higher score indicating superior performance. The noteworthy aspects of the responses are highlighted in green, whereas the illogical portions are marked in red.
: The weather in the image is sunny.
GPT4 Score: 7
It is not possible to determine the current weather in the location depicted in the image, as the image does not provide any information about the weather conditions. The image shows a man, a baby, and a horse standing in front of a building with a thatched roof, but it does not provide any information about the weather or the environment in the area. GPT4 Score: 4
: Yes, if it rains now, the man will get wet. In the image, the man is holding a baby and petting a brown horse, and they are both standing under an overhang. If it starts raining, the overhang will not provide complete protection from the rain, and the man will get wet as he continues to interact with the horse and the baby.
GPT4 Score: 8
: No, the man will not get wet if it starts raining, as he is already inside the stable with the horse. The stable provides shelter from the rain, keeping both the man and the horse dry.
It is not possible to determine whether the man in the image would get wet if it starts raining, as the image does not provide any information about the current weather conditions or the environment in the area. The image shows a man holding a baby, but it does not provide any information about the weather or the environment in the area.
GPT4 Score: 5
In the image, the baby is being held by a man, who is also holding a horse's nose. The baby is reaching out to touch the horse's nose, likely experiencing the horse for the first time. This interaction between the baby and the horse creates a moment of connection and curiosity for the child.
GPT4 Score: 9
: The baby is touching the horse, reaching out to pet it while being held by the man.
In the image, the baby is reaching out to touch the horse's nose. The baby is held up by the man, who is standing next to the horse, and the baby is reaching out with one hand to touch the horse's nose. The baby is smiling and seems to be enjoying the interaction with the horse.
The baby's clothes are white. In the image, the man is holding the baby while the horse is nuzzling the baby's face. The baby is wearing a white one'sie, which stands out against the man's brown shirt and the horse's color.
: The baby is wearing white clothes.
The baby in the image is not wearing any clothes.
GPT4 Score: 2
grained visual content, such as text characters. We believe that the recognition ability of LaVIN still has a large room to improve, which will be left in our future work.
In this paper, we propose a novel and affordable solution for vision-language instruction tuning, namely Mixture-of-Modality Adaptation (MMA). Particularly, MMA is an end-to-end optimization regime, which connects the image encoder and LLM via lightweight adapters. With the help of MMA, the entire multimodal LLM can be jointly optimized via a small number of parameters, greatly reducing the training costs. Meanwhile, we also propose a novel routing algorithm in MMA, which can help the model automatically shifts the reasoning paths for single- and multimodal instructions. Based on MMA, we develop a large vision-language instructed model called LaVIN, which demonstrates a superior reasoning ability than existing multimodal LLMs in various instruction-following tasks.
Acknowledgements. This work was supported by National Key R&D Program of China (No.2022ZD0118201), the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), the Natural Science Foundation of Fujian Province of China (No.2021J01002, No.2022J06001), and the China Fundamental Research Funds for the Central Universities (Grant No. 20720220068). We also thank Dr. Mingbao Lin for his valuable suggestions.