SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gen Luo $^{13}$ , Yiji Zhou $^{12}$ , Tianhe Ren $^{1}$ , Shengxin Chen $^{1}$ , Xiaoshuai Sun $^{12}$ , Rongrong Ji $^{123*}$
<sup>1</sup> Key Laboratory of Multimedia Trusted Perception and Efficient Computing,
Ministry of Education of China, School of Informatics, Xiamen University, 361005, P.R. China.
$^{2}$ Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China.
3 Peng Cheng Laboratory, Shenzhen, 518000, China.
{luogen,chenshengxin,rentianhe}@stu.xmu.edu.cn,
{zhouyiyi, xssun, rrji}@xmu.edu.cn
Recently, growing interest has been aroused in extending the multimodal capability of large language models (LLMs), e.g., vision-language (VL) learning, which is regarded as the next milestone of artificial general intelligence. However, existing solutions are prohibitively expensive, which not only need to optimize excessive parameters, but also require another large-scale pre-training before VL instruction tuning. In this paper, we propose a novel and affordable solution for the effective VL adaption of LLMs, called Mixture-of-Modality Adaptation (MMA). Instead of using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters, to bridge the gap between LLMs and VL tasks, which also enables the joint optimization of the image and language models. Meanwhile, MMA is also equipped with a routing algorithm to help LLMs achieve an automatic shift between single- and multi-modal instructions without compromising their ability of natural language understanding. To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision-language instructed model as LaVIN. To validate MMA and LaVIN, we conduct extensive experiments under two setups, namely multimodal science question answering and multimodal dialogue. The experimental results not only demonstrate the competitive performance and the superior training efficiency of LaVIN than existing multimodal LLMs, but also confirm its great potential as a general-purpose chatbot. More importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4 training hours with 3.8M trainable parameters, greatly confirming the effectiveness of MMA. Our project is released at https://luogen1996.github.io/lavin.
In recent years, large language models (LLMs) [3, 37, 5, 52, 38] have continuously pushed the upper limit of natural language understanding with ever increasing parameter sizes and pre-training data scales. The introduction of instruction tuning [30, 31, 35] also enables LLMs to engage in human-like conversations and handle various natural language processing (NLP) tasks [29, 44, 45], approaching artificial general intelligence, e.g., GPT-3.5 [33]. The next milestone is often regarded to extend these LLMs with multimodal capabilities, e.g., vision-language (VL) learning, making LLMs applicable to more real-world application scenarios. Such a target has been recently realized by GPT-4 [34], which is likely to adopt a large-scale vision-language corpus to directly train a multimodal GPT.
* corresponding author
Input: Text  $\bullet$  Text+Vision
Scheme: No Optimization Tunable Params: N/A
(a) Expert System
Input: Text  $\text{喜}$  Text+Vision
Scheme: Multi-stage Optimization
Tunable Params: Hundreds of Millions ~ Billions
Stage-1: VL Alignment
Stage-2: Instruction Tuning
(b) Modular Training Scheme
Scheme: Joint Optimization
Tunable Params: 3~6 Millions
(c) Mixture-of-Modality Adaptation
Figure 1: Comparison of different multimodal adaptation schemes for LLMs. In the expert system, LLMs play a role of controller, while the ensemble of LLM and vision models is expensive in terms of computation and storage overhead. The modular training regime (b) requires an additional large neck branch and another large-scale pre-training for cross-modal alignment, which is inefficient in training and performs worse in previous NLP tasks. In contrast, the proposed Mixture-of-Modality Adaption (MMA) (c) is an end-to-end optimization scheme, which is cheap in training and superior in the automatic shift between text-only and image-text instructions.
However, the training regime of GPT-4 [34] is prohibitively expensive, and recent endeavors [49, 50, 41, 17, 21, 1, 8, 56, 4] are still keen to efficient VL adoptions of LLMs. As shown in Fig. 1, the existing multimodal solutions for LLMs can be roughly divided into two main categories, i.e., the expert system and the modular training ones, respectively. In the expert system solution [49, 50, 41], LLMs usually serve as a manager to interpret different natural language instructions, and then call the corresponding vision models to handle the input image, e.g., image captioning [18, 27], visual question answering [55, 28] or text-to-image generation [39]. The advantage of this solution is that it does not require the re-training of LLMs and can make full use of existing vision models. However, the ensemble of LLMs and various vision models still exhibits significant redundancy in terms of computation and parameters, leading to excessive memory footprints. Meanwhile, the joint optimization of LLMs and vision models is still an obstacle.
In this case, increasing attention has been paid to the modular training of LLMs [17, 21, 56, 15, 56]. As illustrated in Fig. 1, this paradigm often requires LLMs to deploy an additional neck branch to connect the visual encoders, and then performs another pre-training on numerous image-text pairs for cross-modal alignment. Afterwards, the neck branch and LLM are jointly tuned via VL instructions. Despite the effectiveness, the required VL pre-training is still expensive for a quick adaptation of LLMs. For instance, the pre-training of BLIP2 [17] consumes more than 100 GPU hours on 129 millions of image-text pairs. In addition, this paradigm often requires to update most parameters of LLM, limiting the efficiency of VL instruction tuning. For example, LLaVA-13B [21] fully fine-tunes the entire LLM during VL instruction tuning, resulting in significant increases in training time and intermediate storage overhead<sup>2</sup>. More importantly, these fine-tune schemes will inevitably undermine the NLP capabilities of LLMs due to the drastic changes in their parameter spaces. For instance, the existing multimodal LLMs, such as BLIP2 [17] and miniGPT4 [56], do not support text-only instructions, greatly hindering their applications.
In this paper, we propose a novel and efficient solution for vision-language instruction tuning, termed Mixture-of-Modality Adaptation (MMA). Different from existing modular training scheme [17, 21], MMA is an end-to-end optimization regime. By connecting the image encoder and LLM with lightweight adapters, MMA can jointly optimize the entire multimodal LLM via a small number of parameters, saving more than thousands times of storage overhead compared with existing solutions [21, 56, 17]. To obtain a quick shift between text-only and image-text instructions, MMA equips the inserted adapters with a routing scheme, which can dynamically choose the suitable adaptation path for the inputs of different modalities, thereby well preserving the NLP capability of LLMs. To validate MMA, we apply it to a recently proposed LLM called LLaMA [43], and term this new large vision-language instructed model as LaVIN. With the help of MMA, LaVIN can achieve cheap and quick adaptations on VL tasks without the requirement of another large-scale pre-training.
To validate LaVIN, we first conduct quantitative experiments on ScienceQA [24]. Experimental results show that LaVIN can achieve on-par performance with the advanced multimodal LLMs, e.g., LLaVA [21], while reducing up to  $71.4\%$  training time and  $99.9\%$  storage costs. Notably, fine-tuning
2The checkpoints are often stored during training, and each of them takes up 26GB for storage.
LaVIN on ScienceQA only takes 1.4 hours with 8 A100 GPUs, and the updated parameters are only  $3.8M$ . In addition, we also extend LaVIN to a multimodal chatbot via tuning on  $52k$  text-only instructions [42] and  $152k$  text-image pairs [21]. The qualitative comparisons show that LaVIN can accurately execute various types of human instructions, e.g., coding, math and image captioning, while yielding superior vision-language understanding than existing multimodal chatbots [56, 17, 50].
In summary, our contributions are three folds:
- We present a novel and efficient solution for vision-language instruction tuning, namely Mixture-of-Modality Adaptation (MMA), which does not require the expensive VL pretraining and can maintain the NLP capabilities of LLMs.
- Based on MMA, we propose a new multimodal LLM, namely LaVIN. Experimental results show the superior efficiency and competitive performance of LaVIN against existing multimodal LLMs, and also confirm its great potential as a general-purpose chatbot.
- We release the source code and pre-trained checkpoints associated with this paper. We believe that our project can well facilitate the development of multimodal LLM.
2.1 Parameter-Efficient Transfer Learning
Since large language models have ever-increasing parameter sizes, parameter-efficient transfer learning (PETL) [13, 19, 25, 14, 22, 12] has gained increasing attention to reduce training and storage overhead of LLMs. PETL aims to insert or fine-tune a small number of parameters into LLMs, thereby achieving the adaption on downstream tasks. In early efforts [13, 12], a small MLP network, known as Adapter [13], is inserted into LLMs to project their hidden features to the semantic spaces of downstream tasks. Based on Adapter, numerous PETL methods [19, 46, 25, 14, 22, 12] have been proposed to further enhance adaptation capabilities [19, 46, 25, 22, 12] and inference speed [14]. Among them, AdaMix [46] is a method relatively close to our MMA, which also includes a set of candidate adapters for downstream task routing. However, AdaMix is static and task-dependent, of which routing path is fixed after training. In contrast, our MMA is a dynamic method based on the input modality embeddings. Moreover, AdaMix is still an unimodal module and hard to adaptively adjust the adaptations of different modalities. Driven by the great success in NLP, PETL has also achieved significant progresses in large vision models [26, 2, 54], e.g., ViT [7] and CLIP [36]. Despite the effectiveness, PETL for multimodal LLMs still lacks explorations. A very recent PETL method [51] is proposed for multimodal LLMs, but its performance still lags behind full fine-tuning.
2.2 Multimodal Instruction-following LLMs
Instruction tuning [30, 31, 35, 47, 48] aims to fine-tune LLMs on natural language corpus describing diverse NLP tasks. This simple and effective method has been successfully applied to various well-known LLMs, such as InstructGPT [35] and FLAN-T5 [6], greatly improving their performance and generalization ability. Motivated by this success, numerous efforts have been devoted to constructing multimodal instruction-following LLMs. Existing works can be categorized into two groups, e.g., the expert systems [49, 50, 41] and modular training ones [17, 21, 56, 15, 56], respectively. The representative expert systems, such as Visual ChatGPT [49] and MMREACT [50], employ LLMs as the controller to invoke various vision models to accomplish the VL instructions. Despite the effectiveness, this heavy system also incurs non-negligible burdens in terms of storage and computation. Recently, modular training models [17, 21, 56, 15, 56] as proposed as more efficient alternatives. Among them, Flamingo [1] is the first large-scale multimodal LLM that pre-trains on numerous image-text pairs, which demonstrates strong zero-shot ability on diverse tasks. The following works, including BLIP-2 [17], FROMAGE [16], PaLM-E [8], KOSMOS-1 [15] and LLaVA [21], not only optimize the model architecture [17, 16, 8, 15] but also improve the quality of VL instruction data [21]. Despite their effectiveness, most multimodal LLMs require expensive training costs and perform worse on text-only instructions.
Figure 2: The overview of the Mixture-of-Modality Adaptation (MMA) and the architecture of LaVIN. In LaVIN, the novel Mixture-of-Modality Adapters are employed to process the instructions of different modalities. During instruction tuning, LaVIN is optimized by Mixture of Modality Training (MMT) in an end-to-end manner.
3.1 Mixture-of-Modality Adaptation
In this paper, we propose a novel learning regime for the vision-language adaption of LLMs, which is called Mixture-of-Modality Adaptation (MMA). As shown in Fig. 2, MMA includes two novel designs, namely Mixture-of-Modality Adapter (MM-Adapter) and Mixture-of-Modality Training (MMT). Specifically, MM-Adapter extends LLMs with multimodal abilities via lightweight adapters, which also realizes the automatic shift between single- and multi-modal instructions. Afterwards, the entire multimodal LLM is jointly optimized via MMT, which is cheap in training time and storage.
Mixture-of-Modality Adapter (MM-Adapter). As shown in Fig. 2, we connect the LLM with the image encoder with a set of lightweight adaptation modules. In the image encoder, these modules can be the common adapters [13, 26]. In the LLM, unimodal adaptation modules are inferior in handling single- and multi-modal instructions simultaneously.
In particular, we first introduce a modality token  $t_m \in \mathbb{R}^c$  to indicate the input modality, which is defined by
$$
t _ {m} = m E _ {m}. \tag {1}
$$
Here,  $E_{m} \in \mathbb{R}^{2 \times c}$  is the modality embedding.  $m \in \mathbb{R}^2$  is a one-hot vector to represent the input modality. Based on the modality token  $t_{m}$ , MM-Adapter can dynamically adjust the adaptations for the input features  $Z \in \mathbb{R}^{n \times c}$ . In practice,  $Z$  can be the single- or multi-modal features, which will be introduced in Sec 3.2. Thus, MM-Adapter can be defined by
$$
Z ^ {\prime} = Z + s \cdot \operatorname {r o u t e r} \left(f _ {a _ {1}} (Z), f _ {a _ {2}} (Z); f _ {w} \left(t _ {m}\right)\right). \tag {2}
$$
Here,  $f_{a_1}$  and  $f_{a_2}$  are RepAdapters [26] in our paper.  $s$  is the scale factor, and router(\cdot) is a routing function to decide the routing path of two adapters. To further reduce the parameter costs, the downsampling projection of two adapters are shared.
Figure 3: Illustration of the Mixture-of-Modality Adapter (MMA). MMA can dynamically select the appropriate adapter according to the input modalities.
As shown in Fig. 3, the key to realize the dynamic adaptations lies in the design of the routing function router(\cdot), which is formulated as
$$
\begin{array}{l} r o u t e r \left(f _ {a _ {1}} (Z), f _ {a _ {2}} (Z)\right) = \hat {w} _ {0} \cdot f _ {a _ {1}} (Z) + \hat {w} _ {1} \cdot f _ {a _ {2}} (Z), \\ \text {w h e r e} \quad \hat {w} = f _ {w} \left(t _ {m}\right) = \operatorname {s o f t m a x} \left(\frac {t _ {m} W _ {m} + b _ {m}}{\tau}\right). \tag {3} \\ \end{array}
$$
Here,  $W_{m} \in \mathbb{R}^{c \times 2}$  and  $b_{m} \in \mathbb{R}^{2}$  are the weight matrix and bias, respectively.  $\hat{w}$  denotes the routing weights, and  $\tau$  is the temperature of the softmax. Based on Eq. 2 and 3, MM-Adapter can select the best adaption path according to the modalities of input instructions. More importantly, the process of MM-Adapter only introduces a few of additional parameters, which is still efficient. In practice, MM-Adapter can be used as the unimodal adapter to improve the adaptation ability, thus we also apply it to the image encoder.
Mixture-of-Modality Training (MMT). Based on MM-Adapter, the target of MMT is to freeze the large image encoder and LLM, and only fine-tune the inserted adapters. In this case, the entire multimodal LLM can be jointly optimized in an end-to-end manner. Specifically, the end-to-end optimization objective can be formulated by
$$
\arg \min  \mathcal {L} \left(f _ {\phi} (Z), R; \theta_ {a}\right). \tag {4}
$$
Here,  $R$  and  $\mathcal{L}(\cdot)$  denote the ground-truth response [24] and the objective loss function, respectively.  $f_{\phi}$  is the LLM, and  $\theta_{a}$  denotes the adaptation parameters.  $I \in \mathbb{R}^{h \times w \times 3}$  and  $T \in \mathbb{R}^l$  denote the input image and text instruction, respectively.
During training, we construct a mini training batch randomly sampled from text-only and text-image instructions. In this case, the overall training objective  $\mathcal{L}$  can be defined by
$$
\mathcal {L} = \sum_ {i = 1} ^ {m} \sum_ {s = 1} ^ {S + 1} \log p \left(R _ {s} ^ {i} \mid Z ^ {i}, R _ {0: s - 1} ^ {i}; \theta_ {a}\right). \tag {5}
$$
Here,  $m$  denotes the batch size, and  $S$  is the length of the response. After MMT, the multimodal LLM can effectively execute the input instructions of different modalities.
In our training scheme, the number of optimized parameters is still kept at a very small scale, e.g.,  $3\sim 5\mathrm{M}$ , which greatly reduces the training time and the storage cost. Compared to existing modular training paradigm, MMA does not require additional VL pre-training and can optimize the entire model end-to-end, further improving the training efficiency.
3.2 Large Vision-language Instructed Model
To validate MMA, we apply it to an LLM called LLaMA [43] and adopt CLIP-ViT [36] as the image encoder. Here, we term this new large vision-language instructed model as LaVIN.
Given the input image  $I \in \mathbb{R}^{h \times w \times 3}$ , we use the [c1s] tokens from every fourth layer of ViT [7] as the visual feature, denoted as  $X \in \mathbb{R}^{n \times d}$ . In the image encoder, we insert the adapters before the multi-head attention modules. We represent the text instruction with word embeddings, denoted as  $Y \in \mathbb{R}^{l \times c}$ . Then, a simple visual adapter is used to transform the visual features to the same dimension with the LLM, which is defined by
$$
X ^ {\prime} = \sigma \left(X W _ {d} + b _ {d}\right) W _ {u} + b _ {u}. \tag {6}
$$
Here,  $W_{d} \in \mathbb{R}^{d \times d_{h}}$  and  $W_{u} \in \mathbb{R}^{d_{h} \times c}$  denote the weight matrices, while  $W_{d} \in \mathbb{R}^{d_{h}}$  and  $b_{u} \in \mathbb{R}^{c}$  are the bias terms.  $\sigma$  is the SwiGLU activation function [40]. In practice,  $d_{h}$  is much smaller than  $d$  and  $c$ , so the input of LLM can be defined by
$$
Z = \left\{ \begin{array}{l l} \left[ t _ {m}, X ^ {\prime}, Y \right] & \text {t e x t - i m a g e}, \\ \left[ t _ {m}, Y \right] & \text {t e x t o n l y}. \end{array} \right. \tag {7}
$$
Here,  $[\cdot ]$  denotes the concatenation. Based on the multimodal input, LLM can predict the next token step by step, which can be formulated by
$$
p _ {t} = \prod_ {s = 1} ^ {S + 1} p \left(R _ {s} \mid Z, R _ {0: s - 1}; \theta_ {l}, \theta_ {a}\right) \tag {8}
$$
Here,  $p_t \in \mathbb{R}^m$  denotes the probabilities of the predicted word and  $m$  is the length of the word embeddings.  $\theta_l$  and  $\theta_a$  denote the parameters of LLM and adaptation modules, respectively.
Compared with previous works [17, 56, 21], the architecture of LaVIN is much simpler and more lightweight, which is also easier to optimize. For example, the visual neck of LaVIN is 6 times smaller than that of LLaVA [21], but the performance of two models is close.
Method	#T-Param	LLM	Subject	Context Modality	Grade	Average
NAT	SOC	LAN	TXT	IMG	NO	G1-6	G7-12
Zero- & few-shot methods
Human [24]	-	X	90.23	84.97	87.48	89.60	87.50	88.10	91.59	82.42	88.40
GPT-3.5 [24]	-	✓	74.64	69.74	76.00	74.44	67.28	77.42	76.80	68.89	73.97
GPT-3.5 (CoT) [24]	-	✓	75.44	70.87	78.09	74.68	67.43	79.93	78.23	69.68	75.17
GPT-4 [34]	-	✓	84.06	73.45	87.36	81.87	70.75	90.73	84.69	79.10	82.69
Representative & SoTA models
UnifiedQA [24]	223M	X	71.00	76.04	78.91	66.42	66.53	81.81	77.06	68.82	74.11
MM-CoTBase [53]	223M	X	87.52	77.17	85.82	87.88	82.90	86.83	84.65	85.37	84.91
MM-CoTLarge [53]	738M	X	95.91	82.00	90.82	95.26	88.80	92.89	92.44	90.31	91.68
LLaVA [21]	13B	✓	90.36	95.95	88.00	89.49	88.00	90.66	90.93	90.90	90.92
Parameter-efficient methods
LLaMA-Adapter [51]	1.8M	✓	84.37	88.30	84.36	83.72	80.32	86.90	85.83	84.05	85.19
LaVIN-7B (ours)	3.8M	✓	89.25	94.94	85.24	88.51	87.46	88.08	90.16	88.07	89.41
LaVIN-13B (ours)	5.4M	✓	90.32	94.38	87.73	89.44	87.65	90.31	91.19	89.26	90.50
LaVIN-13B† (ours)	5.4M	✓	89.88	94.49	89.82	88.95	87.61	91.85	91.45	89.72	90.83
Table 1: Comparison on ScienceQA test set. Question classes: NAT = natural science, SOC = social science, LAN = language science, TXT = text context, IMG = image context, NO = no context, G1-6 = grades 1-6, G7-12 = grades 7-12. † denotes that LaVIN is trained with 40 epochs. #T-Params denotes that the number of trainable parameters.
Settings	#T-Params	NAT	SOC	LAN	TXT	IMG	NO	G1-6	G7-12	Avg.
Text Only	1.8M	82.86	82.56	82.28	81.23	75.81	86.06	83.26	81.54	82.65(+0.00)
+ Vision Modality (MMT)	2.4M	85.97	90.66	83.55	84.90	83.59	86.41	88.14	83.06	86.32(+3.67)
+ Joint Opt. (MMT)	2.5M	86.59	94.71	82.91	85.63	84.98	86.41	88.62	85.04	87.34(+4.69)
+ Stronger Image Enc.	2.9M	88.01	94.94	83.64	87.15	86.81	87.04	89.87	85.56	88.33(+5.68)
+ MM-Adapter	3.8M	89.25	94.94	85.24	88.51	87.46	88.08	90.16	88.07	89.41(+6.76)
+ Larger LLM (13B)	5.4M	90.32	94.38	87.73	89.44	87.65	90.31	91.19	89.26	90.50(+7.85)
Table 2: Ablation studies on ScienceQA test set. For the text-only baseline, we use the image caption to prompt the model. ViT-B/16 and LLaMA-7B are used as the default image encoder and LLM. "Joint Opt" denotes the joint optimization of image encoder and LLM. The Mixture-of-Modality Training (MMT) is ablated with the settings of "Vision Modality" and "Joint Opt".
4.1 Datasets and Metrics
ScienceQA. ScienceQA [24] is the large-scale multimodal dataset for science question answering, which covers various domains, including 3 subjects, 26 topics, 127 categories and 379 skills. ScienceQA consists of text-only and text-image examples in three splits namely train, val and test, with 12,726, 4,241 and 4,241 examples, respectively. We evaluate our model using average accuracy.
Alphaca-52k & LLaVA-158k. Alphaca-52k [42] contains 52k text-only instruction-following data generated by GPT-3.5 [3]. LLaVA-158k [21] is a large-scale text-image instruction-following dataset, where the answer is automatically generated by GPT-4 [34]. Following LLaVA [21], GPT-4 is employed to evaluate the quality of the chatbot's responses, which will assign higher scores to superior responses within a range of 1 to 10.
4.2 Implementation Details
We employ the ViT-L/14 [7] of the pre-trained CLIP [36] as the image encoder. The visual features consist of six [cls] tokens extracted from every fourth layer of ViT-L/14. For LLM, LLaMA-7B [43] and LLaMA-13B [43] are used. The default dimension of the visual neck is set to 128. The dimension of MM-Adapter is 8, and the temperature is set to 10 for LaVIN-7B and 5 for LaVIN-13B. For text-only baseline, the image encoder is removed, and MM-Adapter is replaced with RepAdapter [26]. We adopt AdamW [23] as the optimizer, and train the model for 20 epochs with a cosine decay learning rate schedule. The batch size, learning rate and weight decay are set to 32, 9e-3 and 0.02, respectively. During the generation stage, the decoding uses top-p sampling with a temperature of 0.1 and a top-p value of 0.75, respectively. For the experiments of multimodal chatbot, all hyperparameters remain the same, except for the training epochs, which are reduced to 15.
4.3 Experimental Results
4.3.1 Quantitative Experiments
Results on ScienceQA. In Tab. 1, We first compare LaVIN with the state-of-the-art methods on ScienceQA. From this table, the first observation is that the few-shot LLMs, such as GPT-4, still perform worse than human, suggesting the great challenge of ScienceQA. In contrast, existing supervised methods [21, 51, 53] yield better results. In particular, MM-CoTLarge [53] achieves the best performance, e.g., 91.68. However, MM-CoT mainly focuses on the multimodal chain-of-thought for language models, of which contribution is orthogonal to our approach. In particular, LLaVA [21] is an end-to-end multimodal LLM, which is more close to our work.