LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation
Authors: Fangxun Shu
Yue Liao
Lei Zhang
Le Zhuo
Chenning Xu
Guanghao Zhang
Haonan Shi
Long Chan
TaoZhong
Zhelun Yu
Wanggui He
Siming Fu
Haoyuan Li
Si Liu
Hongsheng Li
Hao Jiang
Venue: ICLR 2025 Poster (2025)
Keywords: MLLM
MoE
Distillation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We introduce LLaVA-MoD, a novel framework designed to enable the efficient training of small-scale Multimodal Language Models ($s$-MLLM) distilling knowledge from large-scale MLLM ($l$-MLLM). Our approach tackles two fundamental challenges in MLLM distillation. First, we optimize the network structure of $s$-MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the language model, striking a balance between computational efficiency and model expressiveness. Second, we propose a progressive knowledge transfer strategy for comprehensive knowledge transfer. This strategy begins with mimic distillation, where we minimize the Kullback-Leibler (KL) divergence between output distributions to enable $s$-MLLM to emulate $s$-MLLM's understanding. Following this, we introduce preference distillation via Preference Optimization (PO), where the key lies in treating $l$-MLLM as the reference model. During this phase, the $s$-MLLM's ability to discriminate between superior and inferior examples is significantly enhanced beyond $l$-MLLM, leading to a better $s$-MLLM that surpasses $l$-MLLM, particularly in hallucination benchmarks.
Extensive experiments demonstrate that LLaVA-MoD surpasses existing works across various benchmarks while maintaining a minimal activated parameters and low computational costs. Remarkably, LLaVA-MoD-2B surpasses Qwen-VL-Chat-7B with an average gain of 8.8\%, using merely $0.3\%$ of the training data and 23\% trainable parameters. The results underscore LLaVA-MoD's ability to effectively distill comprehensive knowledge from its teacher model, paving the way for developing efficient MLLMs.

Published as a conference paper at ICLR 2025
ALAVA-MOD) MAKING LLAVA TINY VIA MoE- KNOWLEDGE DISTILLATION
Fangxun Shu"
Yue Liao?* Lei Zhang Le Zhuo
Chenning Xu
Guanghao Zhang'
Haonan Shi!
Long Chen'
Wanggui He Si Liu Hongsheng Li Hao Jiang
Tao Zhong'
Siming Fu! Haoyuan Li'
Zhelun Yu
!Alibaba Group 'hhe Chinese University of Hong Kong
University of California, San Diego 'Beihang University
ABSTRACT
We introduce LLaVA-MoD, novel framework designed enable the efficient training of small-scale Multimodal Language Models (s-MLLM) distilling knowl- edge from large-scale MLLM (--MLLM). Our approach tackles two fundamental challenges in MLLM distillation. First, we optimize the network structure of s- MILM by integrating a sparse Mixture of Experts (MoE) architecture into the language model, striking a balance between computational efficiency and model capability. Second, we propose a progressive knowledge transfer strategy for comprehensive knowledge transfer. This strategy begins with mimic distilla- tion, where we minimize the Kullback-Leibler (KL. divergence between output distributions enable s-MLLM to emulate I-MLLM's understanding. Follow- ing this, we introduce preference distillation via Preference Optimization (PO), where the key lies in treating 1-MLLM as the reference model. During this phase, the s-MLLM's ability to discriminate between superior and inferior ex- amples is significantly enhanced beyond /-MLLM, leading better s-MLLM that surpasses 1-MILM, particularly in hallucination benchmarks. Extensive ex- periments demonstrate that I.L. VA-MoD surpasses existing works across vari- ous benchmarks while maintaining minimal activated parameters and low com- putational costs. Remarkably, LLaVA-MoD-2B surpasses Qwen-VL-Chat-7B with an average gain of 8.8%, using merely 0.3% of the training data and 23% trainable parameters. The results underscore -LoD'ss capability to effec- tively distill comprehensive knowledge from its teacher model, paving the way for the development of efficient MILMs. The code is available https: / //github.
thub. com/ wttp://gun////.aaaaaaaaa--
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Bai al., 2023b; Liu et al., 2024; Lin et al., 2024b; Chen et al., 2023b; Shu et al., 2023; Lu et al., 2024) have demonstrated promising performance in multimodal tasks by integrating visual encoders with Large Language Models (LLMs) (Achiam et al., 2023; Bai et al., 2023a; Team et al., 2023; Touvron et al., 2023). However, the consider- able size of these models and their reliance on vast training data pose significant computational challenges. For instance, the largest version of LLaVA-NeXTT utilizes the Qwen-1.5. 1.5-1 model, requiring 128 H800 GPUs for 18 hours of training. Furthermore, the extensive number of parame- ters necessitates advanced hardware, resulting in slow inference speeds that complicate real-world deployment, particularly on mobile devices. Consequently, is essential to explore small-scale MLLMs (s-MLLMs) that strike balance between performance and efficiency.
Previous works on M-MLM (Zhou et al., 2024a; Yuan et al., 2023; Shao et al., 2024; He et al., 2024; Chu et al., 2023; Yao et al., 2024) have focused on high-quality data collection and filtering proto- cols. While effective, they are inherently limited by the model capacity. Knowledge Distillation
'Equal Contribution
1 CCore Member
1 'Corresonning Author

---

Published as a conference paper at ICLR 2025
TwntO
1500
Training Stmphee (M)
23% trinatie parand -..5P Galls unagee
733
C0.
092
05
Grvan-LL-Cral
VILA-3
1ep40
DespBook VL-1
CogyiM
VusiCN
POPE
Trainable Paramotera (6)
(4) Comparison of training samples, parametens, and GPU hours
(b) Comparison of performance various tasks
Figure 1: Comparisons of training cost and performance. LLaVA-MoD achieves comparable perfor- mance with advanced MLLMs using significantly lower training costs while outperforming current small-scale MLLMs by a large margin.
(KD) from large-scale MILM (/MLLLM offers promising yet unexplored strategy for enhancing s-MLLM performance. By aligning small models' output distributions with large models, KD allows s-MLLM to leverage the rich knowledge embedded in /-MLLM. We explore distillation framework for MLLM and consider two primary challenges. The first challenge is to design a lightweight archi- tecture for s-MLLM while still maintaining strong learning and expressive capabilities to effectively absorb complex knowledge from /-MIL.M. The second challenge is to ensure an effective transfer of complex, multi-task knowledge from /-MLLM s-MLLM. We present VA-MooD address the challenges through Mixture-of-Expert (MoE) and Knowledge Distillation.
To address the first challenge, one straightforward approach is create small model by simply scaling down 1-MLLM. However, this direct reduction in size can significantly diminish the model's expressive capabilities, resulting in a decline in performance for handling complex multi-modal tasks. Drawing inspiration from the recent success of MoE (Dai et al., 2024; Jiang et al., 2024) language modeling, we design an MoE structure into M-MILMM to balance scale reduction while maintaining the ability to capture complex multimodal knowledge through sparsely activated experts during distillation. Specifically, we equip M-MLLM with multiple feedforward networks (FFNs) and a linear gate within the LLM block. Each FFN serves edge from 1-MLLM, while the gate dynamically selects the top-k experts to identify the optimal knowledge transfer pathway and maintain the training and inference efficiency.
expert to capture fine-grained knowl-
To address the second challenge, we propose a progressive distillation strategy. The process begins by aligning the vision encoder with the LLM using a learnable adapter, thereby initializing dense s-MLLM. Subsequently, we employ two consecutive distillation stages, where s-MLLM evolves from mimicking and approximating [-MLLMM ultimately surpassing it: Mimic Distillation. This stage is divided into two steps, i.e., dense-to-dense (D2D) and dense-to-sparse (D2S), The rationale for this two-step process to faciliittte the transfer of complex knowledge, where the first step targets general knowledge to build solid foundation, while the second step targets specialized knowledge to handle multi-task tasks. In D2D, we employ standard KD loss to align the output logits distribution between the initialized dense 5-MLLM and 1-MLLM, using general captioning and conversation datasets. Next, D2S, M-MLLM first transformed from dense to sparse. We then employ the standard KD loss and LM loss to distill the MoE block, using the complex multi- task datasets. Preference Distillation. In this stage, /-MLLM provides knowledge regarding what constitutes "good" and "bad" samples, establishing foundational reference for the student model. The s-MLLM leverages this knowledge to adjust its probability distribution, ensuring that good samples have a higher probability than those from J-MLLM, while bad samples assigned lower probability. This process enhances the M-MLL''s ability to mitigate hallucinations by improving its judgment capabilities beyond those of 1-MLLM.
LL VAMMoD exhibits impressive performance on various multimodal benchmarks while maintain- ing minimal activated parameters and low computational resources. As illustrated in Figure 1, LLaVA-MoD-2B exceeds Qwen-VL-CCat-77B by an average of 8.8% on these benchmarks, uti-
MoD denotes Mixture-of-Fxppne Knowledge Distillation for MI.IMss
2

---

Published as a conference paper at ICLR 2025
lizing only 0.3% of the training data and 23% trainable parameters. Furthermore, it matches the performance of RLHF-based methods with 7B and 13B parameters on several hallucination bench- marks. Specifically, LLaVA-MoD-20 surpasses RLHF-V (Yu et al., 2024a) by 8.2% in response- level hallucination rate and by 21 21.3% in mention-level hallucination rate on the Object HalBench. The impressive results demonstrate the effectiveness of our distillation framework in transferring knowledge from /-MLLM to s-MLLM.
2 RELATED WORK
Multimodal Large Language Models. LLMs have greatly advanced the field of natural language processing, Connecting visual information LLM is crucial for promoting the unified compre- hension of vision and language. BLIP-2 (Li et al., 2023b) adds additional intermediary structures to adapt visual features LLM. Flamingo (Alayrac et al., 2022) incorporates additional cross attsentttnn modules into LLM to handle arbitrarily interleaved multimodal sequences. Recently, models like LLaVA (Liu et al., 2024) and MiniGPT-4 (Zhu et al., 2023) have tried to enhance the models' instructionffolloiin
ability through visual instruction tuning. In addition, some works focus on a stronger vision encoder (Chen et al., 2023b; Li et al., 2024) or more powerful fine- lineggrinnd vi- sual understanding capabilities (Bai et al., 2023b; Wang et al., 2024). Unlike these approaches, method does not need to significantly enlarge the model. We aim to combine distillation and MoE to improve computational and storage efficiency while maintaining advanced performance.
Knowledge Distillation. The enormous sizes and high inference costs of LLMs limit their applica- tion in low-resource environments. Knowledge distillation (Hinton et al., 2015) uses large model as the teacher to transfer its advanced knowledge to a smaller student model, which plays a critical role in compressing model size and enables smaller models to self-improve. MiniLLM (Gu al., 2023) minimizes the reverse KL divergence to prevent students from overestimating low-probability regions in the teacher distribution, while GKD (Tan et al., 2023) introduces generalized knowledge distillation and promotes the integration of distillation with RLHF Additionally, some works adopt context distillation (Huang et al., 2022) and CoT (Chain-of (Chain-of-Thought) distillation (Mukherjee al., 2023; Li et al., 2022; Ho et al., 2022) to enhance specific skills of small models. Our approach innovatively distills the knowledge of MILM into a smaller sparse MoE architecture, significantly enhancing the multimodal processing capabilities of small models minimal cost.
Mixture-of-Expers The mixture of experts (MoE) architecture, introduced by (Jacobs et al., 1991), enhances performance by leveraging independent experts to handle diverse samples. In transformer- based architecture, the feed-forward neural network (FFN) layers serve as the experts, sparsely ac- tivated through gating strategy (Lepikhin et al., 2020; Fedus et al., 2022). This design effectively enhances model capacity while keeping computational overhead low. Additionally, the sparse up- cycling method (Komatsuzaki et al., 2022) which initializes experts using those from well-trained dense model is proposed to further mitigate training costs. MoE has shown great potential not only in language models (Jiang al., 2024; Dai et al., 2024) but also in vision models (Riquelme al., 2021) and vision- linion-language models (Lin et al., 2024a; Shen et al., 2023). Our approach integrates MoE with knowledge distillation techniques to provide stronger signals for sparse training, which remarkably decreases the training cost associated with sparse models.
3 METHOD
We introduce LLaVA-MoD, a novel framework for building efficient s-MLLM using mixture-of- experts (MoE) and knowledge distillation. Our framework consists of two main components: (a). Architecture Design of S-MLLM: As shown in Fig. 3, we design a sparse s- s-MLLM with MoE, en- hancing the ability to acquire specialized expert knowledge while maintaining training and inference efficiency. (b). Distillation Mechanism: We design a progressive distillation mechanism as shown in Fig. to transfer knowledge from I-MILMM to sparse s-MLLM. This process involves two stages: mimic distillation followed by preference distillation.
3.1 ARCHITECTURE DESIGN OF SPARSE s- M-MLLM
In this section, we describe the architecture design of our sparse s-MLLM, which serves as the student model in the distillation process.
3

---

Published as a conference paper at ICLR 2025
P
KL Loss
Tr
Loss
Y
PO Loss
Teacher
Student
Teacher
Student
f
f
How many clephamis are thene tte image?
What the cat doing?
A:: The black white car looking at the camera.
A: Ther are swo deppants, one sdblt daphant and one young calf.
Ahe sitting rable.
(a). Mimic Distillation
(b). Preference Distillation
Figure 2: Progressive Distillation of LLaVA-MoD. (a). Mimic Distillation: Aligning the student's response probabilities ps with those of the teacher pr, via the Bellbackk-eibble (KL.) loss. (b)- Preference Distillation: Increasing the student's positive response probabilities P5 to surpass those of the teacher PT. while decreasing the student's negative response probabilities Pg to fall below those of the teacher Pr, via the preference-Optimizatie (PO) loss.
s-MLLM Definition. As illustrated in Fig. 3, the basic architecture of s-MLLM consists of three primary components: vision encoder, large language model (LLM), and vision-language (VL) adaptor. Given multimodal instruction conversation (xr, y). we define our s-MLLM to process response y as follows:
y = LLMs (Proj., (ViT (x.)),
(1)
where Ze is the input image, and æ, is the text instruction. The input image is resized 336 336 and patched into 576 image tokens, each of size 14 14. ViT, is the CLIP vision encoder with parameters which first extracts image features from Ty. Proj., is the vision-language adaptor with parameters w, serving as the vision tokenizer to align the image features with the word embedding space. LLMg is the large language model with parameters Φ which produces the response y based on the multimodal tokens of r [x,,,,]]
Sparsify S-MLLM. The principle of building 5-MLLM is downsizing the LLM while leaving the vision encoder and vision-language adaptor unmodified. To achieve this downsiz- ing goal, we sparsify the dense M-MLLM by in- corporating an MoE architecture. Specifically, Fig. illustrates the process, where we apply sparse upcycling technique (Komatsuzaki et al., 2022) to replicate N feedforward net- works (FFNs) as the expert modules. Addition- ally, we introduce a linear layer the router, which dynamically activates the appropriate ex- perts by predicting the probability of expert signment. Given each token in the sequence, first compute the routing value of N experts:
Suall-wale
Teleater
Lave News
Mall
Viace
Lige
Lagas hoon
Sparno LLM
Figure 3: Sparsification of s-MLLM. The VL
Adaptor and Vision Encoder remain unchanged, while the LLM is upcycled from dense sparse.
r r= Softmax(x W.).
(2)
where W, denotes the weight matrix of the router and each element in represents the probability of activating the i-th expert. After that, we apply the Top-k strategy to determine the activated experts with the highest & routing values:
ri ifik 0, otherwise
F Top-k(r)
where the routing values for the inactivated experts are set to zero, effectively excluding them from contributing the final output. The output y calculated by aggregating the contributions of the
4

---

Published as a conference paper at ICLR 2025
activated experts, weighted by their corresponding routing values:
y
7. E((x).
km1
3.2 PROGRESSIVE DISTILLATION
Our progressive distillation consists of two distinct stages, i.e., mimic distillation (Fig. 2 (a)) and preference distillation (Fig. 2 (b)). In the mimic distillation stage, the s-MLLM Ts imitates the general and specific knowledge from the /-MLLM mт. In the preference distillation stage, as gains
the preference knowledge of Tr to further refine its output and reduce hallucinations. Both æş and ar are from the same LLM family. This ensures a consistent vocabulary space, which is essential for accurate imitation.
Initialization. Before distillation, we first align the vision encoder with the LLM via a learnable adapter, aiming to obtain well cell-initialized dense version of sT. The LLM, and ViT x maintained frozen since their pre-trained parameters have already captured rich visual and language knowledge. Only Proj.. is optimized to bridge the gap between the vision and language domain. For the ini- tialization, we utilize common image-caption pairs from a widely used and curated dataset, which covers diverse range of subjects and visual entities. The training objective is minimize the cross-entropy of the generated tokens. The objective function is:
(Ts) = - Es (ecc.)) <s(k I [ck.
(S)
where IT (y& yeck, 2) represents the probability of the predicted token ys conditioned on and the previous tokens V<ck = 1/.......... -
Mimic Distillation. We decompose the comprehensive knowledge within कт into general and spe- cialized aspects to address the challenges posed by their structural differences, which can complicate simultaneous learning, Subsequently, we conduct general tol-ttoooooot to-specialized mimic distillation, which consists of two steps: dense-to-dense (D2D) and dense-to-sparse (D2S) distillation, to transfer the knowledge to wy. This two-step approach balances the transfer of general and specialized knowl- edge through the progressive distillation, thereby enhancing overall performance. As illustrated in Fig. utilize the dense structure of πī during D2D to acquire general knowledge and transform it into a sparse structure during D2S to acquire complex specialized knowledge. Throughout both stages, Ty remains unchanged.
a). Dense-to-Dense Distillation. In this stage, we aim to replicate the general knowledge of I- MLLM. Acquiring general knowledge first is crucial, as it establishes a broad foundation across various scenarios, enabling s-MLLM to develop a basic framework before advancing to multiple specialized tasks. To achieve this, we maintain ViTx frozen, and jointly optimize LLM, and Proj.- with trainable parameters 0 = (w, Φ). We leverage common image-caption pairs and conversation datasets. The training objective is to minimize the Kullback-Leible divergence (KLD) between the output logits of s-MLLM and 1-MLLM. The objective function is:
TT (yk #ck,
(TS TT) -Be log
(6)
ITS (yk Vck,
where denotes the vocabulary, while is (yn yeck, x) and TT(8k Beck, denote the probability of the predicted tokens for s-MLLM and /-MLLM.
b). Dense-to-Sparse Distillation. In this stage, our focus shifts to transfer the specialized knowl- edge of 1-MLLM into M-MLLM to obtain advanced capabilities and achieve superior performance in complex tasks. However, directly learning this knowledge in the dense form of s-MLLM could be insufficient due to the diverse knowledge structure of different tasks. Therefore, we sparsify the dense M-MLLM by introducing multiple experts. feedforward networks (FFNs) within LLM, and add an MLP layer as router, forming the experts with parameters de. This sparse architecture enables s-MLLM to activate the most relevant experts based different inputs selectively, offering significant advantages in emulating the specialized knowledge of /-MLLM. For training. we leverage multi-task data, updating only the experts and the adaptor. We employ Top-k routing strategy to select the experts. The trainable parameters (w, d.). Similar to the previous stage, we adopt the KLD as the training objective. Addition- ally, include the standard next-token prediction objective, i.e., minimizing the cross-entropy loss
detailed in Section 3.1, we replicate N
5

---

Published as a conference paper at ICLR 2025
of generated tokens by s-MLLM, to inject supervision from ground truth data which can reduce the existing bias in I-MLLM. The objective function is:
AT (yk
Yick, x)
L m (... [log (y( (.ck:)]] (x. )
log
(7)
aT Mick xr)
Preference Distillation. In this stage, our goal is to distill the preference knowledge from I-MLLM to guide s-MLLM towards generating not only accurate but also reasonable responses, which is crucial in reducing hallucinations. For the training process, we effectively use preference data, which comprises meticulously paired positive responses y" and negative responses y for the identical prompt x. Our preference distillation strategy is inspired by recent advancements in Preference Optimization (PO) (Rafailov et al., 2024; Ethayarajh et al., 2024), which bypasses the need for training reward model by directly training on an offline preference dataset. Our key insight is to treat /-MLLM as the reference model to provide insights on what constitutes "good" and "bad", thereby establishing a fundamental reference for s-MLLM.
Specifically, the training objective is to optimize 5-MLLM to assign higher probabilities to positive responses and lower probabilities to negative ones compared to I-MILM. This training process involves two key optimization aspects: First, s-MLLM aims to align with the teacher model in distinguishing positive from negative responses. Second, s-MLLM seeks to surpass 1-MLLM by assigning higher probabilities to positive responses and lower probabilities to negative responses. We only train the experts and adaptor in s-MLLM and employ a Top-k routing strategy to select the experts. The trainable parameters are 0 [w, d. The objective function is:
(y x))
TT x क xx) - )
LPD(Asi πT) = -Bxt.,t.. log a $log
3 log
(8)
TTly+ x)
where ITS (y I) and Ts( I) denote the probabilities of positive and negative responses for s-MLLM, and TT my + x) and (T a)) denote the same for 1-MLLM.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Implementation Details. We employ the "ViT-MLP-LLM" architecture to demonstrate the effec- tiveness of LLaVA-MoD. A pre-trained ELIP-VVT-L//IE utilized as the vision encoder and a 2-layer MIP is utilized the adaptor. Qwen- 1.522 with different sizes are utilized as the LLM for [-MLLM and M-MLLM. Specifically, /-MILM configured with 7B parameters, while s-MLLM is config- ured with 1.8B and 0.5B parameters. The performance of (-MILM on multimodal benchmarks is presented in Tab. 1. We use the same series LLMs for distillation, i.e. employing Qwen- 7B to distill 1wen-1.5 1.8B and 1. 0.5B. Each stage uses distinct training configurations. The detailed training strategy and hyperparameter are illustrated in Appendix A.1.
Training Datasets. The training data consists of 5M samples from the open-source datasets, with each training stage utilizing a distinct dataset. During Initialization, 0.6M general captioning sam- ples used to bridge the gap between visual and language modalities. In mimic distillation, 2.4M general captioning and conversation samples are used to distill general knowledge from 1-MLLM, and 1.4M multi-tasks data, including VQA, documents, science, and OCR, are used distill spe- cialized knowledge from /-MLLM. For preference distillation, 80K preference data samples are used transfer preference knowledge from 1-MLLM. The detailed dataset of each training stage is illustrated in Appendix A.2.
Evaluation Benchmarks. 2023c), and CM Each encompasses various sub-tasks. enabling comprehensive evaluation of multimodal understanding and reasoning capabilities. Additionally, we carry out experiments across broad spectrum of VQA tasks, which include general VQA, teientree VQA, and science VQA. Specifically, for general VQA tasks, we use VizWiz (Gurari et al., 2018) and GQA (Hudson Man- ning, 2019) to test general visual understanding and relational reasoning. TextVQA (Singh et al., 2019) is employed for text-oriented VQA tasks, focusing on fine-grate grained visual recognition and understanding of text within images. ScienceQA (Lu al., 2022b) is utilized to measure scien- tific knowledge. Moreover. we conduct experiments on several hallucination benchmarks such as POPE (Li et al., 2023c), Object HalBench (Yu et al., 2024a), MMHal-Benn (Sun et al., 2023).
We conduct experiments on MME (Fu et al., 2023). MMB (Liu et al.,

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with state-of-the-art MILMS the commonly-used multimodal benchmarks for MLLMs. #Sample: Training data sample. #Param: Trainable parameters. SQA': ScienceQA test, VQAT: ; TextVQA val, MME: MME Benchmark, normalized to percentage, MMB: MMBench dev, MMBCN MalBench-Chinesse dev. The best result for model sizes 1B/2B is shown in bold, and the second-bes result is underlined. Our LLa VA-MoD achieves the best average result for both.
Method
LLM
#Sample #Param GQA
VisWiz SQA!
VQAT
MME
MMB
MMB
AVG
Teacher MLLM
Qwen- 1.57-
SM
62.3
40.7
70.9
55.3
72.1
68.5
64,9
62.1
~B
Teacher MIL.M
Qwen-2-7B
5M
62.5
37.9
73.2
57.2
78.0
71.6
71.1
64.5
BLIP-2
Vicuna- 13B
129M
41.0
19.6
61.0
42.5
64.7
VILA-7B
LLaMA-7B
50M
62.3
57.8
68.2
64.4
76.7
68.9
61.7
65.7
CLg
Vicuna-7B
1500M
64.9
65.6
78.2
71.8
63.7
53.8
InstructBLIP
Vicuna- 13B
130M
49.5
33.4
63.1
50.7
60.6
27
Qwen-VL-e VL-Chat
Qwen-7B
1450M
57.5
38.9
68.2
61.5
74.4
60.6
56.7
56.7
Deepseek- VL-7B
DLLM-7B
2000M
61.3
49.9
74,0
73.4
72.8
67.2
LLLVA--L5--B
Vicuna-1.5-78
1.2M
62.0
50.0
66.8
75.5
58.3
62.1
LLaVA-NeXT
Vicuna-1.5-13B
1.3M
65.4
60.5
73.6
67.1
78.7
70.0
64.4
68.5
1mp-3B
Phi-2-2.7B
1.6M
63.5
54.1
72.8
72.3
72.9
46.7
Bunny-3B
Phi-2-2.7B
2.7M
62.5
70.9
74.4
37.2
VIIA--B
LLaMA-2.7B
5IM
61.5
$3.5
72.1
52.7
61.8
MobileVLM
MLLaMA-2.7B
1.3M
59.0
61.0
47.5
64.4
59.6
MobileVLM
MLLaMA-2.7B
3.6M
61.1
70.0
72.0
-B
MoB-LLaVA-3B
Phi-2-2.7B
2.2M
61.4
68.5
51.4
41.8
57.6
MiniCPM-V
MiniCPM-2.4B
570M
51.5
50.5
74.4
56.6
68.9
64.0
62.7
61.2
MiniCPM-V.2
MiniCPM-2.4B
570M
52.1
60.2
76.3
73.2
70.5
68.5
67.2
66.9
Imp-2B
Qwen-1.5-1.88
1.6M
61.2
58.9
Bunny-2B
Qwen-I.5-1.8R
2.7M
59.6
64.6
53.2
65.0
59.1
58.5
56.3
Mini- Gemini-nmiii2
Gemma-2B
2.7M
60.7
41.5
63.1
56.2
67.0
59.8
51.3
57.1
MoB-LLaVA-25
Qwen-1.5-11.8
2.2M
61.5
32.6
63.1
48.0
64.6
59.7
57.3
55.3
DxepSeck-LL-1.333
DLLM-1.3B
2000M
59.3
36.8
64.2
58.4
65.3
64.6
61.0
58.5
LLaVA-MoD-2BB
Qwen-1.5-1.88
5M
58.7
39.2
68.0
58.5
67.6
66.3
61.9
59.9
LLaVA-MoD-2R
Qwen-2-1.5B
SM
58.8
40.4
69.2
59.9
69.2
68.9
64,4
61.6
SPHINX-Tiny
TLLaMA-L.IB
15M
58.0
49.2
21.5
57.8
63.1
56.6
37.8
49.2
L.AVA-MoD-IB
Qwen-1.5-0.5B
5M
~BB
56.2
31.6
62.8
53.9
65.3
58.8
50.4
54.1
LLaVA-MoD-IB
Qwen-2-0.5B
5M
56.6
35.1
61.1
57.1
67.0
58.7
54.1
55.7
4.2 MAIN RESULTS
In this section, we conduct experiments of LLaVA-MoD) to highlight its advantages in two aspects: performance and efficiency. For performance, we evaluate comprehension-oe oriintted benchmarks (Tab. I) and hallucination-oorinnee benchmarks (Tab. 2). For efficiency, we present comparison in terms of data samples and model size. The performance is obtained under the same series LLMs, i.e. employing Qwen-1.5 7B to distill Qwen-1.5 1.8B and Qwen- 1.5 0.5B.
Comprehennion--Orienee Benchmarks. As indicated Tab. 1, LLaVA-MoD achieves SOTA average results among the models of 1B and 2B size comprehension-oriente 2B-sized I.LaVA-MoD surpasses Mini-Gemini-255 (Li et al., 2024) by 8.1 %, while using lower image resolution (336 vs. 768). The IB-sized LLaVA-MoD surpasses SPHINX-Tiny (Gao al., 2024) by 13.2%. using fewer data samples (5M vs. 15M). Furthermore, LLaVA-MoD-2B matches and even surpasses the performance of large-scale MLLMs. The 2B-sized LLaVA-MoD surpasses Qwen-VL- VL-ChLt--tt (Bai et al., 2023b) by 8.8% and matches the performance of VILA-3B (Lin al., 2024b) and MiniCPM-V (Yao et al., 2024). These results highlight that our approach trains small-scale MILMS effectively by distilling the sparse MoE architecture from large-scale MLLMss
benchmarks. The
Hallucination--rriented Benchmarks. As indicated Tab. VLL shows remarkable per- formance in mitigating hallucination, even beating its teacher model. It can be attributed to prefer- ence distillation from two aspects: Firstly, by assigning a higher probability for the positive response, preference distillation encourages LLa VA-MoD to focus providing correct and relevant informa- tion. Secondly, by assigning a lower probability for the negative response, preference distillation discourages incorrect unsubstantiated information. By using the teacher model reference to adjust response probabilities, preference distillation enables LLaVA-MoD to handle hallucination issues accurately and reliably, thereby surpassing its teacher model. Moreover, LLaV

---

Published as conference paper at ICLR 2025
Table Comparison with state-of-the-art MILMS on the hallucination benchmarks. We compare LLLa with SFT-based works and RI.FF-based works Hall: Hallucination Rate Resp:
response-level hallucination rate, Ment: mention-level hallucination rate. The best result in bold, and the second-best result is underlined.
Object HalBench
POPE
HMMHal-Bench
Model
LLM
#Param
Resp
Ment
F1
Score t
Hall +
Teacher MLLM
Owen-1.5-77 Qwen-2-7B
7B
50.1
24.8
84.9
2.60
20.7
Teacher MLLM
7B
29.7
23.4
85.7
2.88
14.5
Qwen-VL--Cha VhaChat
Qwen-7B
9.6B
40.4
20.7
74.9
2.76
38.5
LLaVA-1.5-7B
Vicuna-7B
53.6
25.2
86.1
2.36
51.0
VCD
Vicuna- 1.5-7B
7B
48.8
24.3
84.5
2.12
54.2
OPERA
Vicuna- 1. 1.5-7B
7B
45.1
22.3
85.4
2.15
54.2
HA-DPO
Vicuna- 1.5-7B
7B
39.9
19.9
86.8
1.98
60.4
POVID
Vicuna- 1.5-7B
7B
48.1
24.4
86.3
2.08
56.2
LLaVA-RLHF
Vicuna- 1.5-13B
13B
38.1
18.9
82.7
2.02
62.5
LURE
Vicuma- 1.5-7B
7B
27.7
17.3
1.64
60.4
RLHE-V
Vicuna- 13uma-133
13B
12.2
7.5
86.2
2.45
51.0
RLAIF-Y
Vicuna- 1.5-7
7B
8.5
4.3
3.06
29.2
MiniCPM-V
MiniCPM-2 2.44
2,8B
21.6
11.5
79.5
3.70
24.9
MiniCPM
MiniCPM-2..4
2.8B
14.5
7.8
86.3
4.09
18.2
Mini-Gemint 2B
Gemma-2 2B
2B
29.7
21.1
85.6
2.83
18.8
Bunny-2B
Qwen- Owen-1.5-1.88
2.2B
50.2
23.4
85.8
2.72
19.3
LLAVA-MeD-22
Qwen-1.5-1.80 1.5-1.8B
2.2B
11.4
7.2
87.0
2.76
17.2
LIAVA-MoD-2B
Qwen-2-1.5B
1.9B
11.2
5.9
87.2
2.91
13.8
even surpasses recent RLHF-based models (Sun et al., 2023; Zhou et al., 2024b; Huang et al., 2024; Leng et al., 2024; Zhou al., 2023). On the Object HalBench, it outperforms RLHF-V (Yu al., 2024a) by 8.2% in response-level hallucination rate and by 21.3% mention-level hallucination rate, This shows that preference distillation is effective task in minimizing hallucination.
Efficiency Comparison. As indicated in Tab. 18, LLaVA-MoD achieves significant efficiency in both training and inference. Compared to Qwen VL-Cha--- our 2B model achieves 8.8% higher accuracy with only 0.3% of its training data (5M samples) and 23% of activated parameters (2.2B total), while significantly reducing computational costs: on single A100-80G GPU, demon- strates 2.5x faster decoding speed, 26% FLOPs, and 38% memory consumption. This efficiency advantage extends to other SOTA models like MiniCPM-V (2.8B parameters), where I.LaVA-Mo) outperforms using only 1. 6% of training data despite similar model size. Therefore, LLaVA- MoD establishes a new efficiency frontier deploying high-performance multimodal models in resource-constrained scenarios.
4.3 ABLATION STUDY
In this section, thoroughly explore the effect of the distillation strategy and model design. Firstly, we investigate the preference distillation on comprehension and hallucination benchmarks. Since focus is the training strategy and model design, carry out subsequent ablation experiments specifically on mimic distillation. We use CLIP- ViT-L/14 as the default vision encoder, and Qwen- 1.5-1.8B and -wen-11.-7B- the default student and teacher LLMs respectively.
4.3.1 IMPACT OF PREFERENCE DISTILLATION
Preference Distillation Mitigates Hallucination. We explore the effect of preference distillation on comprehension capability and hallucination mitigation. shown in Tab. 3 and Tab. 4, preference distillation remarkably reduces the hallucination of s-MLLM, while it does not yield consistent im- provements in comprehension capability. This observation aligns with that in LLMs (Achiam et al., 2023), where preference optimization tends to prioritize reducing hallucinations, often resulting in a certain degree of decline comprehension gains.
KTO Stabilizes Preference Distillation. We investigate the training stability associated with various preference-opiie -ppiriiiainoo losses, specifically employing KTO (Ethayarajh et al., 2024) and DPO (Rafailov al., 2024). As demonstrated in Appendix B.2, both methods contribute im-
8

---

Published as a conference paper at ICLR 2025
Table The impact of preference distillation on the comprehension--rrintted benchmarks.
Distillation
GQA
VizWiz
SQA!
VQA
MME
MMB
MMB
AVG
Mimic 59.3
40.0
68.4
58.7
68.4
65.1
61.5
60.2
Mimic+Preference
58.7
39.2
68.0
58.5
66.7
66.3
61.9
59.9
Table 4: The impact of preference distillation on the hallucinatiin--oriented benchmarks.
Object HalBench
POPE
MMHal-Berse
Distillation
Resp
Ment 4
FI
Score
Hall
Mimic
39.1
22.6
86.7
2.75
17.8
Mimic+Preference
11.4
7.2
87.0
2.76
17.2
proved hallucination mitigation; however, KTO exhibits significant advantage over DPO. This finding suggests that KTO, which categorizes samples either good or bad, offers more effective signal for guiding the s-MLLM to outperform
1-MLLM, contrast to DPO, which relies on comparing samples to determine superiority. Additionally, Knowledge Distillation methods, including KD (Hinton et al., 2015) and GKD (Tan et al., 2023), as detailed in Appendix B.3, highlight the advantages of proposed approach.
compare our MoD with existing
4.3.2 IMPACT OF TRAINING STRATEGY
KD Facilitates MoE Training. We explore the effect of knowledge distillation (KD) in MoE train- ing by comparing it with supervised fine-tuning (SFT). The ablation experiments are carried out using the same sparse configuration of E4T2, where four experts are initialized and the top two ex- perts are activated. As shown Tab. 5, KD surpasses SFT on all benchmarks, achieving an 8. % average gain over SFT. Additionally, it achieves notable gains in complex multi-task scenarios, such as a +8.2% gain on MMB and +10.0% gain on MME. These results indicate that KD provides better optimization signal for effectively training the MoE architecture.
Table 5: Comparison between KD and SFT. The MoE architecture is E4T2.
Training
GQA
VizWiz
SQA
VQA
MME
MMB
MMB
AVG
SFT
58.6
31.2
66.2
55.6 58.7
63.2
59.2
56.1
55.7
KD
59.3
40.0
68.4
68.4
65.
61.5
60.2
General-to-Speeiiaizee KD Boosts Performance. We explore the effect of the general-to- specialized approach in mimic distillation by comparing the proposed D2D+D2S with D2S. The D2D+D2S involves a two-phase process. Firstly, it distills a dense S-MLLM using general data. Subsequently, transforms the s-MLLM from dense to sparse and further distills using multi- task data. The D2S directly distills the sparse s-MLLM using a combined of general and multi-task dataset. Tab. shows that D2D+D2S surpasses D2S with an average gain of 4.0%. The result indi- cates that D2D+D2S enables effective knowledge transfer, striking a balance between general and specialized competencies. Additionally, D2D+D2S is more computationally efficient, consuming only 62.5% of the GPU hours required by D2S.
Table 6: Comparison between different distillation strategies in mimic distillation.
Strategy
#GPU
GQA
VizWiz
SQA!
VQA
MME
MMB
CMB
AVG
D2S
1536
57.9
36.5
66.3
57.5
65.7
61.4
60.0
57.9
D2D+D2S
59.3
40.0
58.7
68.4
65.1
61.5
60.2
Focus on Response Distillation Improves Generalization. We explores the effect of the distilla- tion target by comparing response distillation with response+instruction distillation. Tab. 7 shows that response distillation surpasses responsetinstruction distillation with average gain of 3.1 %. The result indicates that focusing on response is more effective for knowledge distillation auto- regressive modeling. A possible reason for this is that mimicking additional instructions can lead to

---

Published as a conference paper at ICLR 2025
overfitting the specific instruction patterns in the training data, potentially reducing the generaliza- tion to unseen instructions.
Table 7: Comparison between different distilled tokens. Response indicates distilling solely on the
response tokens. Response+Instructtin incorporates additional distillation of instruction tokens.
Distilled Tokens
GQA
VizWiz
SQA'
VQA
MME
MMB
MMB
AVG
Response + Instruction
58.5
35.0
66.8
58.3
68.5
61.8
59.0
58.4
Response
59.3
40.0
68.4
58.7
68.4
65.1
61.5
60.2
4.3.3 IMPACT OF MODEL. ARCHITECTURE
Sparse Architecture Facilitates Knowledge Transfer. We explore the effect of sparse architec- ture distillation by comparing with dense architecture. The configuration is E4T1 where four experts are initialized and the top-1 expert is activated to ensure that the activated parameters are the same as those of the dense architecture. Tab. 8 shows that sparse architecture surpasses dense archi- tecture with an average gain of 3.7%. The superior performance is prominent in complex multi-task benchmarks, such as MME and MMB. The result indicates that sparse architecture leverages MoE to effectively transfer diverse knowledge from 1-MLLM while maintaining computational efficiency. We also conduct a comprehensive investigation into the performance and computational trade-offs associated with various MoE configurations, including the number of experts, the routing parameter k, and the routing strategies employed Appendix B.I.
Table 8: Comparison between sparse and dense architecture within the distillation.
Architecture
GQA
VizWiz
SQA'
VQA
MME
MMB
MMB
AVG
Dense
57.6
32.7
67.3
56.8
65.2
61.8
58.2
57.1
Sparse
58.7
36.9
67.9
58.2
66.1
64.5
61.7
59.2
Teacher Capacity Matters in Knowledge Distillation. We explore the effect of the teacher capac- ity in the distillation. We employ Qwen-1.5-7B the strong teacher and Qwen-1.5-4B the weaker one. The MoE configuration is set as E4T2. Tab. 9 shows that a well-suited teacher can boost per- formance. For Qwen-1.5-1..8 using 7B-sized teacher yields an average gain of 2.2% compared to 4B-sized teacher. However, when the student and teacher have a large capacity disparity, it can hinder effective knowledge transfer. For Qwen-1.5-0.55h using 7B-sized teacher yields marginal average gain compared to 4B-sized teacher. Utilizing a "middle teacher" with an intermediate capacity can bridge the gap, facilitating smooth knowledge transfer.
Table 9: Comparison between the strong and weak teachers. We set the strong teacher the LLM is Qwen-1.5-7B and the weak teacher as the LLM is Qwen- 1.5-4B.
Student
Teacher
GQA
VizWiz
SQA
VQAT
MME
MMB
MMB
AVG
Qwen-1.5-4B
56.0
25.3
64.7
53.8
63.3
62.2
50.8
53.7
Qwen-1.5-0.52
Qwen-1.5-7B
56.1
29.8
63.8
54.5
64.2
58.5
49.7
53.8
Qwen-1.5-4B
58.7
34.6
67.9
57.7
67.6
64.9
60.7
58.9
Qwen-1.55 1,5-1.8B
Qwen-1.5-7B
59.3
40.0
68.4
58.7
68.4
65.1
61.5
60.2
5 CONCLUSION
In this paper, we introduce LLaVA-MoD, novel framework for efficient training of small-scale multimodal language models via knowledge distillation from large-scale ones. It addresses two key challenges in MLLM distillation: enhancing S-MLLM architecture with MoE design for efficiency and expressiveness balance and implementing progressive knowledge transfer strategy. Extensive experiments show that LLaVA-MoD outperforms existing models with low activation parameters and computational costs. Notably, it outperforms Qwen- -i-en-t-- activation parameters, 0.3% training data, and 23% trainable parameters, highlighting its effective- ness in distilling knowledge and driving more efficient MILM development.
8.8% with only 2 billion
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023b.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In COLING, pp. 1511-15200 2022.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang. Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision- lanounlanguage model. arXiv preprint 240. 2024.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-moda models with better captions. arXiv preprint arXiv:2311.12799. 2023a.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint 33..2.222..442 2023b.
Chu, Qiao, X Lin, Xu, Y Yang, Y Hu, Wei, X Zhang, B Zhang, X Wei, et al. Mo- bilevlm: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312. 16886, 2023.
Christopher Clark and Matt Gardner. Simple and effective multi- multi-paragraph reading comprehension. In ACL, pp. 845-855, 2018
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.06066 2024.
P Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:24422011306, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39,
2022.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.

---

Published as a conference paper at ICLR 2025
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family multi-modal large language models. arXiv preprint arXiv:2402.05595. 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stue Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6904-6913, 2017.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. In The Twelfth International Conference on Learning Representations, 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018.
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint 24t:::2002.113300 2024.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255: 2015.
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212. 2022.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and carropection-allocation. In Proceedings of the IEEE/CVF Conference 2024.
on Computer Vision and Pattern Recognition, pp.
Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distilla- tion: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.10670 2022.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991. doi: 10. 11t.///co..99........
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.0444 2024.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- izations via question answering. CVPR, pp. 5648-5656, 2018
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. diagram is worth a dozen images. In ECCV, pp. 235-251, 2016.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document un- derstanding transformer. In ECCV, 2022.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp. Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upeycling: Training mixture-of-experts from dense checkpoints. arXiv preprint arXiv:2212.050553 2022.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen. Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 1 123:32-73, 2017.
12

---

Published as a conference paper at ICLR 2025
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- fion, pp. 13872-13882, 2024.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint (xXi:23006.05225 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023. 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. html.
PMLR, 2023b. URL wwwptttss/ppptoeeeeeeecete iggs ngs ml wwt.g..m.....ss............ /visss/v2222
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. Explanations from large language models make small reasoners better. CoRR, abs/2210.06722 2022. doi: 10.48550ARXXIV22100. 06726. URL /ttpp:////o...r//.......... 4...0..655//arra 2210.06722
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini Mining the potential of multi-modality vision language models. CoRR, 115p/24433.888144 2024. doi: 10.48550/ARXIV.4400.. 18814. URL https: /doi org/ 10. 10.48550/aa2 aa.55////xivv.// 2403. 11.4401.8888 18814.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- vasion-language models. arXiv preprint arXiv:2305.10355 2023c.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024a.
Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. (6689-26699,, 2024b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.0377 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: your multi-modal model all-around player? arXiv preprint arXiv:2307.0628/ 2023c.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vi: towards real-world vision-language understanding. arXiv preprint urXiv:2403.05525 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 35:2507-2521, 2022a.
13

---

Published as a conference paper at ICLR 2025
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chum Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference Neural Information Processing Systems (NeurIPS), 2022b.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, pp. 11-20, 2016.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, pp. 3195-3204, 2019.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. In ACL, pp. 2263- 2022.
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pp. 947-952, 2019.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4. CORR, abs/2306.02707, 2023. doi: 10.45500ARXXW.2306002000 URL https: /tw.//dii/////////////../ orgg/0. 48550/arXiv 2306 02707.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-855 8595, 2021.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, pp- 146-162, 2022.
Zhenwei Shao, Zhou Yu, Jun Yu, Xuecheng Ouyang, Lihao Zheng, Zhenbiao Gai, Mingyang Wang, and Jiajun Ding, Imp: Highly capable large multimodal models for mobile devices. arXiv preprint arXiv:240S.12107, 2024.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226. 2023.
Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual I1m for video understanding. arXiv preprint arXhv:2312.06722 2023.
Oleksüi Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, pp. 742-758, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the JEFF/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Lia Gui, Yu-Xiong Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint wxXiv:2309.14522 2023.
Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Yang Yang, Hongyin Tang, Keqing He. Jiahao Liu, Jingang Wang. Shu Zhao, al. Gkd: A general knowledge distillation framework for large-scale pre-trained language model. arXiv preprint crXiv:2306.06622 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud. Yonghui Wu, Jean-Baptist Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 /8805 2023.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXhv:2302.13971, 20233
Junke Wang, Lingchen Meng. Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gan Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning, arXiv preprint arXiv:2311.07574, 2023.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng. Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: gpt-4v level milm on your phone, arXiv preprint 00:::2408.08000 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, pp. 69-85, 2016.
Tianyu Yu. Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- ain-ggrined correctional human feedback. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13807-13816, 2024a
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 17220, 2024b.
Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint (xXv:23322.16862 2023.
Bo Zhao, Boya Wu, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04080 04087, 2023.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 1428899..4429, 2024a.
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754 2023.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402. 1411, 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.1059,, 2023.
A IMPLEMENTATION DETAILS
A.1 TRAINING STRATEGY AND HYPERPARAMETERS
We first freeze the vision encoder and LLM while optimizing the VI. adaptor to align image tokens with the word embedding space during initialization. This stage employs cross-entropy loss with a batch size 512 and a learning rate of lc-4. In mimic distillation, the vision encoder remains frozen, while the LLM and VL adaptor are co-optimized to distill general knowledge from 1-MLLM in dense-to- tense-to-dense distillation. Then, the FFN in the LLM is first transformed into a sparse archi- tecture with mixture of FFNs co-optimized with the VL adaptor distill specialized knowledge from 1-MLLM in dense-to-sparse to-sse distillation. This stage employs a KL-divergence loss, with cross- entropy loss added for dense-to-sparse distillation. The batch size is 256, and the learning rate is
15

---

Published as a conference paper at ICLR 2025
adjusted to 2c-5. In preference distillation, the model inherits from the mimic distillation. The vi- sion encoder remains frozen, and a mixture of FFN experts is co-optimized with the VL adaptor to distill preference knowledge from the teacher MLLM. This stage employs KTO (Ethayarajh al., 2024) loss to optimize the probability of 5-MLLM for positive responses to be greater than that of 1-MLLM and the probability of s-MLLM for negative responses to be lower than that of I-MILM. The batch size is 256, and the learning rate is adjusted to 2c-6. Throughout all stages, we employ the adam optimizer (Diederik, 2014) and train on 16 NVIDIA A100 GPUs for one epoch each, totaling approximately 960 GPU hours. The detailed training hyperparameters are shown in Tab. 10.
Table 10: Training hyperparameters of each stage.
Configuration
Initialization
Mimic Distillation
Preference Distillation
LLM
X
VL Aduptor
ViT
X
LLM init.
Qwen-1.5 1.5--.8B MLP
Qwen-1.5-1.88 1st-stage CLIP-Large 336
2nd-stage 2nd-stage
VL Adaptor init.
VIT init.
Image resolution
336 x 336 576 2048
VIT sequence length
LLM sequence length
Optimizer
Adam W 0.9, 8g 0.98 2e Cosine decay 0.0
Optimizer hyperparameter
Learning
le
2e
Learning rate schedule
Weight decay
Training epoch
Warm-up ratio
0.03
Global hatch size
256
128
128
Numerical precision
B
Model parallelism
Zero2
Zero2 offload
Zero2 offload
Table 11: Training dataset of each stage. #Sample means the training samples
Stage
Task
Dataset
Initialization
Captioning
LLa Praetan.5-Pretrain (Liu al., 2023b)
ALLaVA-Caption-4V (Chen et al., 2024).
Captioning
Dense-to-Dense Distillation
ShareGPT4V-PT (Chen et al., 2023a)
MIMIC-IT (li et al., 2023a), LVIS (Wang et al., 2023),
Conversation
IRV (Liu et al., 2023a), SVTT (Zhao et al., 2023)
ShareGPT4V-100k (Chen al., 2023a).
Captioning
TexiCaps (Sidorov al., 2020)
Conversation
LLaVA-I55-nsstuuet (Liu et al., 2023b)
GQA (Hudson & Manning, 2019), VQAv2 (Gnyal et al., al, 2017),
General QA
Dense-to-Sparse Distillation
OKVQA (Marino et al., 2019)
Visual Genome (Krishna et al., 2017),
Grounding
RefCOCO (Yu et al., 2016; Mao et al., 2016)
Science
A12D (Kembhavi et al., 2016), ScienceQA (Lu et al., 2022a)
DVQA (Kafle et al., 2018), ChartQA (Masry et al., 2022),
Chart & Doc
DocQA (Clark & Gardner, 2018)
OCR
OCRVQA (Mishra et al., 2019). SynthDoG-EN (Kim et al., 2022)
Knowledge
A-OKVQA (Schwenk al., 2022), GeoQA+ (Cao & Xiao, 2022)
Preference Distillation
Preference
RLAIF-V (Yu et al., 2024b)
A.2 TRAINING DATASETS
We curate comprehensive open-source dataset of 5 million samples encompassing both general and expert tasks. the Initialization stage, we utilize 0.6M general captioning samples from LLaVA- 1.5-pretrain dataset (Liu al., 2023b) to bridge the gap between visual and language modalities. In the mimic distillation stage, we utilize 2.4M general captioning and conversation samples to distill general knowledge from the teacher MIL.M and 1.4M multi-tasks data samples, including VQA,
16

---

Published as a conference paper at ICLR 2025
documents, science, and OCR to distill specialized knowledge from the teacher MILM. In the pref- erence distillation stage, we utilize 80K preference data samples to transfer preference knowledge from the teacher MLLM. The detailed dataset of each training stage is illustrated in Tab. 11.
B ADDITIONAL EXPERIMENTS
B. MOE CONHGURRATONS
We thoroughly explore the performance and computational trade-offs across different MoE. configu- rations, focusing on the number of experts, the routing parameter and routing strategies. As shown in Tab 12, increasing the number of experts from to 8 does not yield an improvement in perfor- mance. Conversely, Tab 13 illustrates that enhancing the routing parameter from 1 to 2 results in noticeable performance boost. This observation suggests that adjusting the routing parameter K seems to facilitate more stable training and consistent performance improvements compared to sim- ply increasing the number of experts. One explanation for this is that adding more experts without increasing the complex knowledge of data results each expert being insufficiently optimized. Fur- thermore, investigate various routing strategies in Tab 14, comparing Top-K with RSample and Jitter. The results indicate that Top-K routing achieves the highest average performance, particularly demonstrating significant advantages in the MMB, which encompasses 20 sub-tasks. This superi- ority can be attributed to Top-K's ability to dynamically activate the most relevant experts based on the input, thereby showcasing its robustness across diverse tasks.
Table 12: Ablations with the expert number.
Expert
GQA
SQA'
VQA
MME
MMB
AVG
59.3
68.4
58.7
68.4
65.1
64.0
58.3
67.4
58.4
69.1
64.6
63.6
Table 13: Ablations with the routing parameter k.
Top-K
GQA
SQA!
VQAT
MME
MMB
AVG
58.2
67.2
58.3
66.1
64.7
62.6
59.3
68.4
58.7
68.4
65.1
64.0
Table 14: Ablations with the routing strategy.
Routing Strategy
GQA
SQA!
VQA
MME
MMB
AVG
Top-K(ours)
58.8
69.2
59.9
69.2
68.9
65.2
RSample
57.9
67.5
60.1
68.9
67.3
64.3
Jitter
58.1
68.4
60.0
69.1
67.7
64.6
B.2 PREFERENCE PREFREECCE-PTIIIZATTO Loss
We explore the training stability of various preference-optimizzaioo losses. We employ KTO (Etha- yarajh et al., 2024) and DPO (Rafailov et al., 2024). As shown in Tab. 15, and Tab. 16 while both methods have an impact on reducing hallucination, KTO is significantly superior to DPO. This is because KTO, which indicates whether sample is good bad, offers better signal for the MLLM to surpass the /-MLLM compared to DPO, which determines relative quality by comparing two samples.
Table 15: Comparison between different preference-optimizzaioo loss on comprehension-orieee benchmarks.
PO Loss
GQA
VizWiz
SQA'
VQA
MME
MMB
CM
AVG
DPO
58.5
37.3
68.2
58.6
67.8
65.4
61.5
59.6
KTO
58.7
39.2
68.0
58.5
67.6
66.3
61.9
59.9
17

---

Published as a conference paper at ICLR 2025
Table 16: Comparison between different preference-oppimizzaion loss on hallucination-orientaa
benchmarks.
Object HalBench
POPE
MMHal-Bench
PO Loss
Resp
Ment
F1 f
Score
Hall
DPO
20.0
10.5
86.8
2.68 2.76
18.9
KTO
11.4
7.2
87.0
17.2
B.3 DISTILLATION STRATEGY
We conducted experiments to compare our method with standard Knowledge Distillation (KD) and Generalized Knowledge Distillation (GKD). To ensure fair comparison, utilized the same teacher and student models: a 7B dense teacher model and 1.8B MoE student model, config- ured with activated experts and a routing parameter k=2. The results in Tab. demonstrate that approach outperforms both KD and GKD across the comprehensive understanding benchmarks MMB and MME, as well as ballucination benchmark Object- Halbench. This enhanced per- formance be attributed to the distinctive features of our progressive distillation method, which employs a two-phase approach: it begins with dense-to-dense general distillation and is followed by dense-to- to-sparse multi-task distillation. This strategy facilitates thorough assimilation of the teacher's 8 comprehension capabilities. Moreover, our preference distillation contributes to hallucina- tion mitigation by providing more reliable preference information derived from the robust teacher model. These advantages are absent in standard KD and GKD approaches, underscoring the addi- tional value offered by our proposed method.
Table 17: Comparison with different distillation strategies.
Method
MME
MMB
ObjHal
KD
65.0
61.8
53.4
GKD
66.7
63.2
52.8
MoD
69.2
68.9
11.2
B.4 TRAINING AND INFERENCE EFFICIENCY
We also conduct detailed analysis of the training and inference costs of LLaVA-MoD-288 with comparison to Wwen-VL-Chht-72B of 3B parameters, with 2.2B activated during training and inference. For training costs, we report training days and the size of the dataset. ALLVA--MD-a using only 0.3% of the training data and 23% of the activated parameters compared to Qwen-Vi VL- Chat-7B. For inference costs, our evaluation on a single A A100-80G GPU indicates that LLaVA- MoD-2B is 2.5 faster in decoding speed, consumes 26% of FLOPs, and uses 38% of the memory compared to Qwen-VL--Cha- VL.-Chat-7B, These results demonstrate L LLoVA-Mo---B'ss superior efficiency while maintaining high performance.
Tab. 18. Our results show that LLaVA-MoD-2B has a total
achieves superior performance while
Table 18: Comparison with training and inference costs.
Metries
LLa VA-MoD-2B
Qwen- VL-Chat-7B
Total Params (B)
3
9.6
Activated Params (B)
2.2
9.6
Dataset Size (Million)
5
1450
Training Time (Day)
2
Memory Usage (GB)
7.8G
20.4G
Inference FLOPs (TFLOPs)
2.3
8.8
Decoding Speed (Tokens/s)
38.5
15.6
Average Performance
61.6
56.7
18

---

Published as a conference paper at ICLR 2025
C LIMITATIONS AND FUTURE WORKS
LLaVA-MoD requires that both s-MLLM and /-MLLM belong to the same series of LLMs to ensure consistency in the vocabulary space. Future research can explore distillation techniques involving heterogeneous model families. Moreover, the requirement to load both s-MLLM and /-MLLM leads substantial memory consumption. To enable efficient distillation, possible solution could be to pre-extract the logits of the I-MLLM. This would allow only the s-MLLM to be loaded during training. reducing memory requirements and potentially speeding up the training process.
19


Reviews:
[{"review_id": "3O8t85Vrcr", "reviewer": "Reviewer_k5Mn", "summary": "LLaVA-MoD introduces a framework for creating efficient small-scale multimodal language models through knowledge distillation from larger models. The approach tackles two key challenges: optimizing network structure through sparse Mixture of Experts (MoE) architecture, and implementing a progressive knowledge transfer strategy. This strategy combines mimic distillation, which transfers general and specialized knowledge, with preference distillation to reduce hallucinations. The framework represents an advancement in making multimodal language models more efficient and practical for real-world applications.", "questions": "Why does increasing the number of experts from 4 to 8 fail to improve performance, and is there a fundamental limitation in the MoE architecture?\n\nHow can the framework be modified to enable distillation between different model families, rather than requiring the same LLM family for student and teacher?\n\nWhat specific mechanisms enable preference distillation to reduce hallucinations more effectively than RLHF-based methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The technical design features a well-constructed progressive distillation strategy that combines mimic distillation for knowledge transfer with preference distillation for reducing hallucinations. The integration of MoE architecture into small-scale models is thoughtfully implemented, striking an effective balance between computational efficiency and model performance.\n\nThe empirical validation is thorough. The authors present comprehensive evaluations across multiple benchmarks, showing that LLaVA-MoD-2B not only outperforms larger models while using fewer resources, but also achieves better results in hallucination reduction compared to existing methods. The ablation studies demonstrate the contribution of each component to the overall performance.\n\nThe work also shows strong practical relevance, addressing the challenge of deploying efficient multimodal language models in resource-constrained environments. The reduction in required training data and parameters while maintaining or improving performance makes this work valuable for real-world applications.", "weaknesses": "The most significant technical constraint is the requirement that student and teacher models must belong to the same LLM family to ensure vocabulary space consistency. This limitation restricts the framework's flexibility and broader applicability. Additionally, the training process demands substantial memory resources as it requires loading both student and teacher models simultaneously.\n\nThe experimental evaluation, while comprehensive in many aspects, lacks detailed analysis in key areas. The paper could benefit from a more thorough examination of inference costs and computational requirements in real-world deployment scenarios. There is also limited exploration of training stability across different configurations and minimal discussion of failure cases or error patterns.\n\nThe architectural exploration could be more extensive, particularly regarding different MoE configurations and alternative approaches to reducing memory requirements during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHfz2LAd1w", "reviewer": "Reviewer_vZAr", "summary": "This paper presents LLaVA-MoD, a framework for efficiently training a small-scale multimodal language model (MLLM) by distilling knowledge from a larger, more capable MLLM (I'll denote this as L for brevity). The main contributions are twofold: introducing a sparse MoE architecture for the student (I'll denote this as S for brevity) and developing a progressive distillation strategy. The MoE architecture adds multiple FFN heads with a linear routing mechanism, for efficiency and model expressiveness. The progressive distillation approach occurs in two stages: (1) mimic distillation, which minimizes KL divergence to align S’s outputs with the L’s outputs, and (2) preference distillation, which enhances the student’s ability to distinguish preference between examples, thus reducing hallucination rates. Compared to existing approaches, LLaVA-MoD is notably efficient both in terms of training data and the number of trainable parameters.", "questions": "- Since the paper emphasizes data efficiency, could you clarify how the other methods in the comparison were trained? For instance, is there any early-stopping strategy, or specific training limits set for them? And for LLaVA, how do you know that “5M” samples are enough?\nDo the S and L need to be the same architecture, or is there some flexibility here? As I understand it, L is distinct from S, where S is initially trained as a dense model before introducing sparsity. Is that correct?\n- For Table 1, could you clarify the comparison parameters? Are the final model sizes (S) similar across comparisons? And do the models differ in their teacher models, or is it consistent in that regard?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides extensive experimental comparisons with prior work, especially highlighting gains in data efficiency and trainable parameters, which is a strong advantage for low-resource training.\n- I found the preference distillation interesting in that it reduces hallucination. This aspect could benefit from a deeper explanation of the mechanism behind this reduction, but it’s a notable strength.\n- Given the complexity of the architecture, the authors include an extensive ablation study to evaluate each component’s effectiveness, which helps to validate the design choices.", "weaknesses": "- While training sample efficiency and the number of trainable parameters are discussed, it would be helpful to see more detail on the total parameters required for training and inference. Efficiency in trainable parameters doesn’t always translate directly to lower memory or computational costs-full backpropagation through the LLM is still necessary even if only adapter and FFN heads are updated. Information on actual memory usage or wall-clock time would give a more complete picture of the framework’s practical efficiency.\n- The architecture is fairly complex, and although the authors do a good job with the explanation, some details remain a bit unclear. For example, the architectural differences between S and L could be more explicitly described to clarify distinctions. See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wSwdFmmGd", "reviewer": "Reviewer_EM2J", "summary": "This work introduces a framework called LLaVA-MoD for building efficient small-scale multimodal large language models (MLLMs) using a sparse Mixture-of-Experts (MoE) architecture. The framework specifically applies MoE to MLP layers within transformer blocks. To preserve the capabilities of larger MLLMs while maintaining efficiency, the authors developed a specialized distillation approach combining KL divergence loss and Preference Optimization (PO) loss between the student (small) and teacher (large) models.", "questions": "1. There are more small-scale MLLM coming out these days with higher performance and comparable size (0.5B) [1]. Could the sparse MoD architecture and distillation strategy be applied to these newer models to further enhance their performance?\n\n2. I am wondering about the performance of Preference Distillation vs DPO in the second stage. Does it show advantages using distillation instead of direct DPO, in terms of model performance or training efficiency?\n\n\n[1] Li, Bo, et al. \"Llava-onevision: Easy visual task transfer.\" arXiv preprint arXiv:2408.03326 (2024).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This work introduces a new framework for better tuning small-scale MLLM. This framework provides solutions in real world problems by offering a more efficient approach to model inference and deployment across diverse application scenarios.\n\nThe experiment is comprehensive, covering a wide range of tasks and models, on both general comprehension benchmarks and hallucination benchmarks. The design of s-MLLM is efficient and effective, achieving superior performance compared to other models of similar size across multiple benchmark tests. The distillation pipeline is clear and well defined. The paper is clearly written.", "weaknesses": "This work  can also be related to inference efficiency and inference acceleration. It would be good if it included inference measurements like FLOPs and inference latency showing the work’s practical benefits in deployment scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vfc6pfi22m", "reviewer": "Reviewer_2roi", "summary": "The paper introduces LLaVA-MoD, a framework aimed at distilling knowledge from large multimodal language models (MLLMs) to create smaller, efficient multimodal language models (s-MLLMs) that retain high performance with reduced computational requirements. LLaVA-MoD achieves this by employing a sparse Mixture of Experts (MoE) architecture to maintain expressive power while minimizing parameters. The distillation process is twofold: Mimic Distillation aligns the smaller model with the larger one through Kullback-Leibler (KL) divergence, and Preference Distillation enhances the smaller model’s ability to judge quality, outperforming the larger model on tasks prone to hallucination. Experimental results highlight LLaVA-MoD's efficiency, achieving significant performance improvements with minimal data and trainable parameters.", "questions": "--What's the computational complexity?\n--How do you address the limitation of the teacher's model?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. LLaVA-MoD achieves high performance with only a small fraction of the data and computational resources required by traditional large MLLMs. This makes it more suitable for real-world applications, especially in low-resource environments.\n2. Reduced Hallucination: By incorporating preference distillation, LLaVA-MoD significantly reduces hallucination rates, making it more reliable for applications where factual accuracy is critical.\n3. Flexible Architecture with MoE: The use of a sparse MoE architecture allows the model to selectively activate different \"experts\" for specific tasks, ensuring computational efficiency while retaining expressive power.\n4. Superior Performance: LLaVA-MoD surpasses larger models like Qwen-VLChat-7B in various benchmarks, demonstrating the effectiveness of its knowledge distillation approach.", "weaknesses": "--The progressive distillation method, which includes dense-to-sparse transitions and multi-task training, could be complex to implement and tune effectively.\n--While preference distillation enhances hallucination mitigation, it does not consistently improve the model’s comprehension capabilities, which might limit its applicability in understanding complex instructions.\n--The effectiveness of LLaVA-MoD is heavily reliant on the quality and capacity of the teacher model. If the teacher model is limited, the distilled s-MLLM might inherit those limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RavkpwRP6N", "reviewer": "Reviewer_vgkT", "summary": "The paper introduces LLaVA-MoD, which is a framework that distills large multimodal models into efficient, small-scale ones using a sparse MoE and progressive knowledge transfer. By optimizing both model architecture and distillation strategy, LLaVA-MoD achieves high performance with minimal data and parameters, even surpassing larger models on multimodal tasks. This framework enables cost-effective deployment of high-performing models in resource-limited settings", "questions": "See above in weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper combines MoE with knowledge distillation in a novel way, optimizing small multimodal models efficiently without sacrificing performance.\n- The proposed method is robust, with a progressive distillation process that includes preference-based fine-tuning to mitigate hallucinations.\n- The paper is well-written and clearly explains the framework.\n- Experiments were done on a comprehensive set of evaluations.", "weaknesses": "- The paper does not provide an analysis of how different MoE configurations (e.g., number of experts, routing strategies, parameter k from top-k routing) impact performance and computational trade-offs. Including such an analysis could provide further valuable insights for the model.\n- The results presented in Table 2 is relatively modest in improvement. LLaVA-MoD-2B only outperforms the other 2B baselines (Imp-2B) by 2.7% on average while using twice as many data samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f96ldRmpMV", "reviewer": "Reviewer_uFgm", "summary": "The paper proposes a framework LLaVA-MoD for distilling knowledge from large multimodal models to smaller models. The method incorporates mixture-of-experts for enhancing the expressiveness of small multimodal models. Additionally, the method uses two stage distillation -- Mimic Distillation for general knowledge alignment and Preference Distillation to enhance the student model's discriminatory capabilities for superior generalization. The experimental results are quite interesting -- The 2-billion parameter LLaVA-MoD outperforms larger models like Qwen-VL-Chat-7B by 8.8%, using only 23% of trainable parameters and 0.3% of the training data. LLaVA-MoD performs significantly well on 7 multimodal tasks and hallucination benchmarks.", "questions": "See the above comments.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a scalable approach for distilling large multimodal models to smaller one, which is a very relevant topic in recent time. The proposed methodology is somewhat novel and addresses key challenges faced by the existing small multimodal models. The experiments are extensive and performance improvement shown by the proposed methodology is significant.", "weaknesses": "1. The experimental setup is not very convincing to me. The authors claim their method to be an alternative distillation method without much empirical comparison. In order to establish the effectiveness of their method, the authors must compare some of the existing KD methods -- KD and GKD, to show why their distillation method is needed. \n\n2. The dense-to-sparse loss is very similar to the competitive loss proposed by Sengupta et al., 2023 [1]. The authors should show how their proposed losses differ from these losses.\n\n3. The authors should elaborately discuss why dense vs sparse architecture works. The authors mostly report the numbers with little explanation of the how part.\n\n4. The lower performance of Mimic+preference than mimic in Table 4 should be discussed. It is counterintuitive.\n\n5. The authors claim that their method is efficient. More detailed analysis with training/inference time and memory should be discussed. \n\n**References**\n\n[1] A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation - ICLR'2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)