LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation

## Abstract

Abstract We introduce LLaVA-MoD, a novel framework designed to enable the efficient training of small-scale Multimodal Language Models ( s -MLLM) distilling knowledge from large-scale MLLM ( l -MLLM). Our approach tackles two fundamental challenges in MLLM distillation. First, we optimize the network structure of s -MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the language model, striking a balance between computational efficiency and model expressiveness. Second, we propose a progressive knowledge transfer strategy for comprehensive knowledge transfer. This strategy begins with mimic distillation, where we minimize the Kullback-Leibler (KL) divergence between output distributions to enable s -MLLM to emulate l -MLLM’s understanding. Following this, we introduce preference distillation via Preference Optimization (PO), where the key lies in treating l -MLLM as the reference model. During this phase, the s -MLLM’s ability to discriminate between superior and inferior examples is significantly enhanced beyond l -MLLM, leading to a better s -MLLM that surpasses l -MLLM, particularly in hallucination benchmarks.
Extensive experiments demonstrate that LLaVA-MoD surpasses existing works across various benchmarks while maintaining minimal activated parameters and low computational costs. Remarkably, LLaVA-MoD-2B surpasses Qwen-VL-Chat-7B with an average gain of 8.8%, using merely 0.3 % percent 0.3 0.3\% 0.3 % of the training data and 23% trainable parameters. The results underscore LLaVA-MoD’s ability to effectively distill comprehensive knowledge from its teacher model, paving the way for developing efficient MLLMs.
The code is available at https://github.com/shufangxun/LLaVA-MoD .

## 1 Introduction

Multimodal Large Language Models (MLLM) (Bai et al., ; Liu et al., ; Lin et al., ; Chen et al., ; Shu et al., ; Lu et al., ) have shown promising results in multimodal tasks by integrating visual encoders with Large Language Models (LLM) (Achiam et al., ; Bai et al., ; Team et al., ; Touvron et al., ). However, the large model size and extensive training data present significant computational challenges. For example, the largest version of LLaVA-NeXT employs Qwen-1.5-110B and trains with 128 H800 GPUs for 18 hours. Additionally, the substantial number of parameters requires advanced hardware and results in slow inference speed, complicating real-world deployment, especially on mobile devices. Therefore, exploring a small-scale MLLM (*s*-MLLM) that balances performance and efficiency is a crucial topic.

Figure: Figure 1: Comparisons of training cost and performance. LLaVA-MoD achieves comparable performance with advanced MLLMs using significantly lower training costs while outperforming current small-scale MLLMs by a large margin.
Refer to caption: x1.png

Previous works on *s*-MLLM (Zhou et al., ; Yuan et al., ; Shao et al., ; He et al., ; Chu et al., ; Yao et al., ) have focused on high-quality data collection and filtering protocols. While effective, they are inherently limited by the model capacity. Knowledge Distillation (KD) from large-scale MLLM (*l*-MLLM) offers a promising yet unexplored strategy for enhancing *s*-MLLM performance. By aligning small models’ output distributions with large models, KD allows *s*-MLLM to leverage the rich knowledge embedded in *l*-MLLM. We explore a distillation framework for MLLM and consider two primary challenges. The first challenge is to design a lightweight architecture for *s*-MLLM while still maintaining strong learning and expressive capabilities to effectively absorb complex knowledge from *l*-MLLM. The second challenge is to ensure an effective transfer of complex, multi-task knowledge from *l*-MLLM to *s*-MLLM. We present LLaVA-MoD(^1^11MoD denotes Mixture-of-Expert Knowledge Distillation for MLLMs) to address the challenges through Mixture-of-Expert (MoE) and Knowledge Distillation.

To address the first challenge, one straightforward approach is to create a small model by simply scaling down *l*-MLLM. However, this direct reduction in size can significantly diminish the model’s expressive capabilities, resulting in a decline in performance for handling complex multimodal tasks.
Drawing inspiration from the recent success of MoE (Dai et al., ; Jiang et al., ) in language modeling, We design an MoE structure into *s*-MLLM to balance the scale reduction while maintaining the ability to capture complex multimodal knowledge through sparsely activated experts during distillation.
Specifically, we equip *s*-MLLM with multiple feedforward networks (FFNs) and a linear gate within the LLM block. Each FFN serves as an expert to capture fine-grained knowledge from *l*-MLLM, while the gate dynamically selects top-$k$ experts to identify the optimal knowledge transfer pathway and maintain the training and inference efficiency.

To address the second challenge, we propose a progressive distillation strategy. The process begins by aligning the vision encoder with the LLM using a learnable adapter, thereby initializing a dense *s*-MLLM. Subsequently, we employ two consecutive distillation stages, where *s*-MLLM evolves from mimicking and approximating *l*-MLLM to ultimately surpassing it:
*Mimic Distillation.* This stage is divided into two steps, *i.e.*, dense-to-dense (D2D) and dense-to-sparse (D2S). The rationale for this two-step process is to facilitate the transfer of complex knowledge, where the first step targets general knowledge to build a solid foundation, while the second step targets specialized knowledge to handle multi-task tasks.
In D2D, we employ standard KD loss to align the output logits distribution between the initialized dense *s*-MLLM and *l*-MLLM, using general captioning and conversation datasets. Next, in D2S, *s*-MLLM is first transformed from dense to sparse. We then employ the standard KD loss and LM loss to distill the MoE block, using the complex multi-task datasets.
*Preference Distillation.* In this stage, *l*-MLLM provides knowledge regarding what constitutes “good” and “bad” samples, establishing a foundational reference for the student model. The *s*-MLLM leverages this knowledge to adjust its probability distribution, ensuring that good samples have a higher probability than those from *l*-MLLM, while bad samples are assigned a lower probability. This process enhances the *s*-MLLM’s ability to mitigate hallucinations by improving its judgment capabilities beyond those of *l*-MLLM.

LLaVA-MoD exhibits impressive performance across various multimodal benchmarks while maintaining minimal activated parameters and low computational resources. As illustrated in Figure 1, LLaVA-MoD-2B exceeds Qwen-VL-Chat-7B by an average of $8.8\%$ on these benchmarks, utilizing just $0.3\%$ of the training data and $23\%$ trainable parameters. Furthermore, it matches the performance of RLHF-based methods with 7B and 13B parameters on several hallucination benchmarks. Specifically, LLaVA-MoD-2B surpasses RLHF-V (Yu et al., ) by $8.2\%$ in response-level hallucination rate and by $21.3\%$ in mention-level hallucination rate on the Object HalBench. The impressive results demonstrate the effectiveness of our distillation framework in transferring knowledge from *l*-MLLM to *s*-MLLM.

## 2 Related Work

Multimodal Large Language Models.
LLMs have greatly advanced the field of natural language processing.
Connecting visual information to LLM is crucial for promoting the unified comprehension of vision and language.
BLIP-2 (Li et al., ) adds additional intermediary structures to adapt visual features to LLM.
Flamingo (Alayrac et al., ) incorporates additional cross-attention modules into LLM to handle arbitrarily interleaved multimodal sequences.
Recently, models like LLaVA (Liu et al., ) and MiniGPT-4 (Zhu et al., ) have tried to enhance the models’ instruction-following ability through visual instruction tuning.
In addition, some works focus on a stronger vision encoder (Chen et al., ; Li et al., ) or more powerful fine-grained visual understanding capabilities (Bai et al., ; Wang et al., ).
Unlike these approaches, our method does not need to significantly enlarge the model. We aim to combine distillation and MoE to improve computational and storage efficiency while maintaining advanced performance.

Knowledge Distillation.
The enormous sizes and high inference costs of LLMs limit their application in low-resource environments. Knowledge distillation (Hinton et al., ) uses a large model as the teacher to transfer its advanced knowledge to a smaller student model, which plays a critical role in compressing model size and enables smaller models to self-improve. MiniLLM (Gu et al., ) minimizes the reverse KL divergence to prevent students from overestimating low-probability regions in the teacher distribution, while GKD (Agarwal et al., ) introduces generalized knowledge distillation and promotes the integration of distillation with RLHF. Additionally, some works adopt context distillation (Huang et al., ) and CoT (Chain-of-Thought) distillation (Mukherjee et al., ; Li et al., ; Ho et al., ) to enhance specific skills of small models. Our approach innovatively distills the knowledge of MLLM into a smaller sparse MoE architecture, significantly enhancing the multimodal processing capabilities of small models at a minimal cost.

Mixture-of-Experts
The mixture of experts (MoE) architecture, introduced by (Jacobs et al., ), enhances performance by leveraging independent experts to handle diverse samples. In transformer-based architecture, the feed-forward neural network (FFN) layers serve as the experts, sparsely activated through a gating strategy (Lepikhin et al., ; Fedus et al., ). This design effectively enhances model capacity while keeping computational overhead low. Additionally, the sparse upcycling method (Komatsuzaki et al., ) which initializes experts using those from a well-trained dense model is proposed to further mitigate training costs. MoE has shown great potential not only in language models (Jiang et al., ; Dai et al., ) but also in vision models (Riquelme et al., ) and vision-language models (Lin et al., ; Shen et al., ). Our approach integrates MoE with knowledge distillation techniques to provide stronger signals for sparse training, which remarkably decreases the training cost associated with sparse models.

## 3 Method

Figure: Figure 2: Progressive Distillation of LLaVA-MoD. (a). Mimic Distillation: Aligning the student’s response probabilities $p_{S}$ with those of the teacher $p_{T}$, via the Kullback–Leibler (KL) loss. (b). Preference Distillation: Increasing the student’s positive response probabilities $p^{+}_{S}$ to surpass those of the teacher $p^{+}_{T}$, while decreasing the student’s negative response probabilities $p^{-}_{S}$ to fall below those of the teacher $p^{-}_{T}$, via the Preference-Optimization (PO) loss.
Refer to caption: x2.png

We introduce LLaVA-MoD, a novel framework for building efficient *s*-MLLM using mixture-of-experts (MoE) and knowledge distillation. Our framework consists of two main components: *(a). Architecture Design of *s*-MLLM*: As shown in Fig. 3, we design a sparse *s*-MLLM with MoE, enhancing the ability to acquire specialized expert knowledge while maintaining training and inference efficiency. (b). *Distillation Mechanism*: We design a progressive distillation mechanism as shown in Fig. 2 to transfer knowledge from *l*-MLLM to sparse *s*-MLLM. This process involves two stages: mimic distillation followed by preference distillation.

### 3.1 Architecture Design of Sparse s -MLLM

In this section, we describe the architecture design of our sparse *s*-MLLM, which serves as the student model in the distillation process.

*s*-MLLM Definition. As illustrated in Fig. 3, the basic architecture of *s*-MLLM consists of three primary components: a vision encoder, a large language model (LLM), and a vision-language (VL) adaptor. Given a multimodal instruction conversation $(x,y)$, we define our *s*-MLLM to process response $y$ as follows:

$$ $y=\text{LLM}_{\phi}(\text{Proj}_{\omega}(\text{ViT}_{\chi}(x_{v})),x_{i}),$ (1) $$

where $x_{v}$ is the input image, and $x_{i}$ is the text instruction. The input image is resized to 336 $\times$ 336 and patched into 576 image tokens, each of size 14 $\times$ 14. $\text{ViT}_{\chi}$ is the CLIP vision encoder with parameters $\chi$ which first extracts image features from $x_{v}$. $\text{Proj}_{\omega}$ is the vision-language adaptor with parameters $\omega$, serving as the vision tokenizer to align the image features with the word embedding space. $\text{LLM}_{\phi}$ is the large language model with parameters $\phi$, which produces the response $y$ based on the multimodal tokens of $x=[x_{v},x_{i}]$.

Figure: Figure 3: Sparsification of *s*-MLLM. The VL Adaptor and Vision Encoder remain unchanged, while the LLM is upcycled from dense to sparse.
Refer to caption: x3.png

Sparsify *s*-MLLM. The principle of building our *s*-MLLM is downsizing the LLM while leaving the vision encoder and vision-language adaptor unmodified. To achieve this downsizing goal, we sparsify the dense *s*-MLLM by incorporating an MoE architecture. Specifically, Fig. 3 illustrates the process, where we apply the sparse upcycling technique (Komatsuzaki et al., ) to replicate $N$ feedforward networks (FFNs) as the expert modules. Additionally, we introduce a linear layer as the router, which dynamically activates the appropriate experts by predicting the probability of expert assignment. Given each token $x$ in the sequence, we first compute the routing value of $N$ experts:

$$ $r=\text{Softmax}(x\cdot W_{r}),$ (2) $$

where $W_{r}$ denotes the weight matrix of the router and each element $r_{i}$ in $r$ represents the probability of activating the $i$-th expert. After that, we apply the Top-$k$ strategy to determine the activated experts with the highest $k$ routing values:

$$ $\tilde{r}=\text{Top-}k({r})=\begin{cases}r_{i},&\text{if }i\in k\\ 0,&\text{otherwise}\\ \end{cases}$ (3) $$

where the routing values for the inactivated experts are set to zero, effectively excluding them from contributing to the final output. The output $y$ is calculated by aggregating the contributions of the activated experts, weighted by their corresponding routing values:

$$ $y=\sum_{i=1}^{N}\tilde{r}\cdot E_{i}(x).$ (4) $$

### 3.2 Progressive Distillation

Our progressive distillation consists of two distinct stages, *i.e.*, mimic distillation (Fig. 2 (a)) and preference distillation (Fig. 2 (b)).
In the mimic distillation stage, the *s*-MLLM $\pi_{\text{S}}$ imitates the general and specific knowledge from the *l*-MLLM $\pi_{\text{T}}$.
In the preference distillation stage, $\pi_{\text{S}}$ gains the preference knowledge of $\pi_{\text{T}}$ to further refine its output and reduce hallucinations.
Both $\pi_{\text{S}}$ and $\pi_{\text{T}}$ are from the same LLM family. This ensures a consistent vocabulary space, which is essential for accurate imitation.

Initialization. Before distillation, we first align the vision encoder with the LLM via a learnable adapter, aiming to obtain a well-initialized dense version of $\pi_{\text{S}}$. The $\text{LLM}_{\phi}$ and $\text{ViT}_{\chi}$ are maintained frozen since their pre-trained parameters have already captured rich visual and language knowledge. Only $\text{Proj}_{\omega}$ is optimized to bridge the gap between the vision and language domain. For the initialization, we utilize common image-caption pairs from a widely used and curated dataset, which covers a diverse range of subjects and visual entities. The training objective is to minimize the cross-entropy of the generated tokens. The objective function is:

$$ $\mathcal{L}_{\text{Init}}(\pi_{\text{S}})=-\mathbb{E}_{(y_{k}\mid y_{<k},x) \sim\pi_{\text{S}}}\left[\log\pi_{\text{S}}(y_{k}\mid y_{<k},x)\right],$ (5) $$

where $\pi_{\text{S}}(y_{k}\mid y_{<k},x)$ represents the probability of the predicted token $y_{k}$ conditioned on $x$ and the previous tokens $y_{<k}=(y_{1},y_{2},\dots,y_{k-1})$.

Mimic Distillation. We decompose the comprehensive knowledge within $\pi_{\text{T}}$ into general and specialized aspects to address the challenges posed by their structural differences, which can complicate simultaneous learning. Subsequently, we conduct a general-to-specialized mimic distillation, which consists of two steps: dense-to-dense (D2D) and dense-to-sparse (D2S) distillation, to transfer the knowledge to $\pi_{\text{S}}$. This two-step approach balances the transfer of general and specialized knowledge through the progressive distillation, thereby enhancing overall performance.
As illustrated in Fig. 3, we utilize the dense structure of $\pi_{\text{S}}$ during D2D to acquire general knowledge and transform it into a sparse structure during D2S to acquire complex specialized knowledge. Throughout both stages, $\pi_{\text{T}}$ remains unchanged.

*a). Dense-to-Dense Distillation.*
In this stage, we aim to replicate the general knowledge of *l*-MLLM. Acquiring general knowledge first is crucial, as it establishes a broad foundation across various scenarios, enabling *s*-MLLM to develop a basic framework before advancing to multiple specialized tasks.
To achieve this, we maintain $\text{ViT}_{\chi}$ frozen, and jointly optimize $\text{LLM}_{\phi}$ and $\text{Proj}_{\omega}$, with trainable parameters $\theta=\{\omega,\phi\}$.
We leverage common image-caption pairs and conversation datasets. The training objective is to minimize the Kullback-Leibler divergence (KLD) between the output logits of *s*-MLLM and *l*-MLLM. The objective function is:

$$ $\mathcal{L}_{\text{D2D}}(\pi_{\text{S}};\pi_{\text{T}})=-\mathbb{E}_{(x,y_{k}) \sim\pi_{\text{T}}}\left[\log\frac{\pi_{\text{T}}(y_{k}\mid y_{<k},x)}{\pi_{ \text{S}}(y_{k}\mid y_{<k},x)}\right],$ (6) $$

where $V$ denotes the vocabulary, while $\pi_{\text{S}}(y_{k}\mid y_{<k},x)$ and $\pi_{\text{T}}(y_{k}\mid y_{<k},x)$ denote the probability of the predicted tokens for *s*-MLLM and *l*-MLLM.

*b). Dense-to-Sparse Distillation.* In this stage, our focus shifts to transfer the specialized knowledge of *l*-MLLM into *s*-MLLM to obtain advanced capabilities and achieve superior performance in complex tasks. However, directly learning this knowledge in the dense form of *s*-MLLM could be insufficient due to the diverse knowledge structure of different tasks. Therefore, we sparsify the dense *s*-MLLM by introducing multiple experts. As detailed in Section 3.1, we replicate $N$ feedforward networks (FFNs) within $\text{LLM}_{\phi}$ and add an MLP layer as a router, forming the experts with parameters $\phi_{e}$. This sparse architecture enables *s*-MLLM to activate the most relevant experts based on different inputs selectively, offering significant advantages in emulating the specialized knowledge of *l*-MLLM.
For training, we leverage multi-task data, updating only the experts and the adaptor. We employ a Top-$k$ routing strategy to select the experts. The trainable parameters are $\theta=\{\omega,\phi_{e}\}$. Similar to the previous stage, we adopt the KLD as the training objective. Additionally, we include the standard next-token prediction objective, *i.e.*, minimizing the cross-entropy loss of generated tokens by *s*-MLLM, to inject supervision from ground truth data which can reduce the existing bias in *l*-MLLM. The objective function is:

$$ $\mathcal{L}_{\text{D2S}}(\pi_{\text{S}};\pi_{\text{T}})=-\mathbb{E}_{(x,y_{k}) \sim\pi_{\text{S}}}\left[\log\pi_{\text{S}}(y_{k}\mid y_{<k},x)\right]-\mathbb {E}_{(x,y_{k})\sim\pi_{\text{T}}}\left[\log\frac{\pi_{\text{T}}(y_{k}\mid y_{< k},x)}{\pi_{\text{S}}(y_{k}\mid y_{<k},x)}\right].$ (7) $$

Preference Distillation. In this stage, our goal is to distill the preference knowledge from *l*-MLLM to guide *s*-MLLM towards generating not only accurate but also reasonable responses, which is crucial in reducing hallucinations. For the training process, we effectively use preference data, which comprises meticulously paired positive responses $y^{+}$ and negative responses $y^{-}$ for the identical prompt $x$. Our preference distillation strategy is inspired by recent advancements in Preference Optimization (PO) (Rafailov et al., ; Ethayarajh et al., ), which bypasses the need for training a reward model by directly training on an offline preference dataset. Our key insight is to treat *l*-MLLM as the reference model to provide insights on what constitutes “good” and “bad”, thereby establishing a fundamental reference for *s*-MLLM.

Specifically, the training objective is to optimize *s*-MLLM to assign higher probabilities to positive responses and lower probabilities to negative ones compared to *l*-MLLM.
This training process involves two key optimization aspects: First, *s*-MLLM aims to align with the teacher model in distinguishing positive from negative responses. Second, *s*-MLLM seeks to surpass *l*-MLLM by assigning higher probabilities to positive responses and lower probabilities to negative responses. We only train the experts and adaptor in *s*-MLLM and employ a Top-$k$ routing strategy to select the experts. The trainable parameters are $\theta=\{\omega,\phi_{e}\}$. The objective function is:

$$ $\mathcal{L}_{\text{PD}}(\pi_{\text{S}};\pi_{\text{T}})=-\mathbb{E}_{(x,y^{+},y ^{-})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\text{S}}(y^{+} \mid x)}{\pi_{\text{T}}(y^{+}\mid x)}-\beta\log\frac{\pi_{\text{S}}(y^{-}\mid x )}{\pi_{\text{T}}(y^{-}\mid x)}\right)\right],$ (8) $$

where $\pi_{\text{S}}(y^{+}\mid x)$ and $\pi_{\text{S}}(y^{-}\mid x)$ denote the probabilities of positive and negative responses for *s*-MLLM, and $\pi_{\text{T}}(y^{+}\mid x)$ and $\pi_{\text{T}}(y^{-}\mid x)$ denote the same for *l*-MLLM.

## 4 Experiments

### 4.1 Experimental Settings

Implementation Details. We employ the “ViT-MLP-LLM” architecture to demonstrate the effectiveness of LLaVA-MoD. A pre-trained CLIP-ViT-L/14 is utilized as the vision encoder and a 2-layer MLP is utilized as the adaptor. Qwen-1.5/2 with different sizes are utilized as the LLM for *l*-MLLM and *s*-MLLM. Specifically, *l*-MLLM is configured with 7B parameters, while *s*-MLLM is configured with 1.8B and 0.5B parameters. The performance of *l*-MLLM on multimodal benchmarks is presented in Tab. 1. We use the same series of LLMs for distillation, *i.e.* employing Qwen-1.5 7B to distill Qwen-1.5 1.8B and Qwen-1.5 0.5B. Each stage uses distinct training configurations. The detailed training strategy and hyperparameter are illustrated in Appendix A.1.

**Table 1: Performance of *l*-MLLM on multimodal benchmarks.**
| LLM | VisEnc | GQA | VisWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen-1.5-7B | CLIP-L | 62.3 | 40.7 | 70.9 | 55.3 | 72.1 | 68.5 | 64.9 | 62.1 |
| Qwen-2-7B | CLIP-L | 62.5 | 37.9 | 73.2 | 57.2 | 78.0 | 71.6 | 71.1 | 64.5 |

Training Datasets. The training data consists of 5M samples from the open-source datasets, with each training stage utilizing a distinct dataset. During Initialization, 0.6M general captioning samples are used to bridge the gap between visual and language modalities. In mimic distillation, 2.4M general captioning and conversation samples are used to distill general knowledge from *l*-MLLM, and 1.4M multi-tasks data, including VQA, documents, science, and OCR, are used to distill specialized knowledge from *l*-MLLM. For preference distillation, 80K preference data samples are used to transfer preference knowledge from *l*-MLLM. The detailed dataset of each training stage is illustrated in Appendix A.2.

Evaluation Benchmarks. We conduct experiments on MME (Fu et al., ), MMB (Liu et al., ), and MMB${}^{\text{CN}}$. Each encompasses various sub-tasks, enabling comprehensive evaluation of multimodal understanding and reasoning capabilities. Additionally, we carry out experiments across a broad spectrum of VQA tasks, which include general VQA, text-oriented VQA, and science VQA. Specifically, for general VQA tasks, we use VizWiz (Gurari et al., ) and GQA (Hudson & Manning, ) to test general visual understanding and relational reasoning. TextVQA (Singh et al., ) is employed for text-oriented VQA tasks, focusing on fine-grained visual recognition and understanding of text within images. ScienceQA (Lu et al., ) is utilized to measure scientific knowledge. Moreover, we conduct experiments on several hallucination benchmarks such as POPE (Li et al., ), Object HalBench (Yu et al., ), MMHal-Bench (Sun et al., ).

**Table 2: Comparison with state-of-the-art MLLMs on the commonly-used multimodal benchmarks for MLLMs. #Sample: Training data sample. #Param: Trainable parameters. SQA${}^{\text{I}}$: ScienceQA test, VQA${}^{\text{T}}$: TextVQA val, MME: MME Benchmark, normalized to percentage, MMB: MMBench dev, MMB${}^{\text{CN}}$: MMBench-Chinese dev. The best result for model sizes 1B/2B is shown in bold, and the second-best result is underlined. Our LLaVA-MoD achieves the best average result for both.**
| Method | LLM | #Sample | #Param | GQA | VisWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLIP-2 | Vicuna-13B | 129M | $\geq$7B | 41.0 | 19.6 | 61.0 | 42.5 | 64.7 | - | - | - |
| VILA-7B | LLaMA-7B | 50M | 62.3 | 57.8 | 68.2 | 64.4 | 76.7 | 68.9 | 61.7 | 65.7 |  |
| CogVLM | Vicuna-7B | 1500M | 64.9 | - | 65.6 | 78.2 | 71.8 | 63.7 | 53.8 | - |  |
| InstructBLIP | Vicuna-13B | 130M | 49.5 | 33.4 | 63.1 | 50.7 | 60.6 | - | - | - |  |
| Qwen-VL-Chat | Qwen-7B | 1450M | 57.5 | 38.9 | 68.2 | 61.5 | 74.4 | 60.6 | 56.7 | 56.7 |  |
| Deepseek-VL-7B | DLLM-7B | 2000M | 61.3 | 49.9 | 74.0 | 64.7 | 73.4 | 74.1 | 72.8 | 67.2 |  |
| LLaVA-1.5-7B | Vicuna-1.5-7B | 1.2M | 62.0 | 50.0 | 66.8 | 58.2 | 75.5 | 64.3 | 58.3 | 62.1 |  |
| LLaVA-NeXT | Vicuna-1.5-13B | 1.3M | 65.4 | 60.5 | 73.6 | 67.1 | 78.7 | 70.0 | 64.4 | 68.5 |  |
| Imp-3B | Phi-2-2.7B | 1.6M | $\sim$3B | 63.5 | 54.1 | 72.8 | 59.8 | 72.3 | 72.9 | 46.7 | 63.2 |
| Bunny-3B | Phi-2-2.7B | 2.7M | 62.5 | 43.8 | 70.9 | 56.7 | 74.4 | 68.6 | 37.2 | 59.2 |  |
| VILA-3B | LLaMA-2.7B | 51M | 61.5 | 53.5 | 69.0 | 60.4 | 72.1 | 63.4 | 52.7 | 61.8 |  |
| MobileVLM | MLLaMA-2.7B | 1.3M | 59.0 | - | 61.0 | 47.5 | 64.4 | 59.6 | - | - |  |
| MobileVLM^v2 | MLLaMA-2.7B | 3.6M | 61.1 | - | 70.0 | 57.5 | 72.0 | 63.2 | - | - |  |
| MoE-LLaVA-3B | Phi-2-2.7B | 2.2M | 61.4 | 43.9 | 68.5 | 51.4 | 71.1 | 65.2 | 41.8 | 57.6 |  |
| MiniCPM-V | MiniCPM-2.4B | 570M | 51.5 | 50.5 | 74.4 | 56.6 | 68.9 | 64.0 | 62.7 | 61.2 |  |
| MiniCPM-V-2 | MiniCPM-2.4B | 570M | 52.1 | 60.2 | 76.3 | 73.2 | 70.5 | 68.5 | 67.2 | 66.9 |  |
| Imp-2B | Qwen-1.5-1.8B | 1.6M | $\sim$2B | 61.9 | 39.6 | 66.1 | 54.5 | 65.2 | 63.8 | 61.2 | 58.9 |
| Bunny-2B | Qwen-1.5-1.8B | 2.7M | 59.6 | 34.2 | 64.6 | 53.2 | 65.0 | 59.1 | 58.5 | 56.3 |  |
| Mini-Gemini-2B | Gemma-2B | 2.7M | 60.7 | 41.5 | 63.1 | 56.2 | 67.0 | 59.8 | 51.3 | 57.1 |  |
| MoE-LLaVA-2B | Qwen-1.5-1.8B | 2.2M | 61.5 | 32.6 | 63.1 | 48.0 | 64.6 | 59.7 | 57.3 | 55.3 |  |
| DeepSeek-VL-1.3B | DLLM-1.3B | 2000M | 59.3 | 36.8 | 64.2 | 58.4 | 65.3 | 64.6 | 61.0 | 58.5 |  |
| LLaVA-MoD-2B | Qwen-1.5-1.8B | 5M | 58.7 | 39.2 | 68.0 | 58.5 | 67.6 | 66.3 | 61.9 | 59.9 |  |
| LLaVA-MoD-2B | Qwen-2-1.5B | 5M | 58.8 | 40.4 | 69.2 | 59.9 | 69.2 | 68.9 | 64.4 | 61.6 |  |
| SPHINX-Tiny | TLLaMA-1.1B | 15M | $\sim$1B | 58.0 | 49.2 | 21.5 | 57.8 | 63.1 | 56.6 | 37.8 | 49.2 |
| LLaVA-MoD-1B | Qwen-1.5-0.5B | 5M | $\sim$1B | 56.2 | 31.6 | 62.8 | 53.9 | 65.3 | 58.8 | 50.4 | 54.1 |
| LLaVA-MoD-1B | Qwen-2-0.5B | 5M |  | 56.6 | 35.1 | 61.1 | 57.1 | 67.0 | 58.7 | 54.1 | 55.7 |

### 4.2 Main Results

In this section, we conduct experiments of LLaVA-MoD to highlight its advantages in two aspects: performance and efficiency. For performance, we evaluate comprehension-oriented benchmarks (Tab. 2) and hallucination-oriented benchmarks (Tab. 3). For efficiency, we present a comparison in terms of data samples and model size. The performance is obtained under the same series of LLMs, *i.e.* employing Qwen-1.5 7B to distill Qwen-1.5 1.8B and Qwen-1.5 0.5B.

Comprehension-Oriented Benchmarks. As indicated in Tab. 2, LLaVA-MoD achieves SOTA average results among the models of 1B and 2B size on comprehension-oriented benchmarks. The 2B-sized LLaVA-MoD surpasses Mini-Gemini-2B (Li et al., ) by 8.1%, while using a lower image resolution (336 vs. 768). The 1B-sized LLaVA-MoD surpasses SPHINX-Tiny (Gao et al., ) by 13.2%, using fewer data samples (5M vs. 15M). Furthermore, LLaVA-MoD-2B matches and even surpasses the performance of large-scale MLLMs. The 2B-sized LLaVA-MoD surpasses Qwen-VL-Chat-7B (Bai et al., ) by 8.8% and matches the performance of VILA-3B (Lin et al., ) and MiniCPM-V (Yao et al., ). These results highlight that our approach
trains small-scale MLLMs effectively by distilling the sparse MoE architecture from large-scale MLLMs.

**Table 3: Comparison with state-of-the-art MLLMs on the hallucination benchmarks. We compare LLaVA-MoD with SFT-based works and RLHF-based works . Hall: Hallucination Rate Resp: response-level hallucination rate, Ment: mention-level hallucination rate. The best result is in bold, and the second-best result is underlined.**
| Model | LLM | #Param | Object HalBench | POPE | MMHal-Bench |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Resp $\downarrow$ | Ment $\downarrow$ | F1 $\uparrow$ | Score $\uparrow$ | Hall $\downarrow$ |  |  |  |
| Teacher MLLM | Qwen-1.5-7B | 7B | 50.1 | 24.8 | 84.9 | 2.60 | 20.7 |
| Teacher MLLM | Qwen-2-7B | 7B | 29.7 | 23.4 | 85.7 | 2.88 | 14.5 |
| Qwen-VL-Chat | Qwen-7B | 9.6B | 40.4 | 20.7 | 74.9 | 2.76 | 38.5 |
| LLaVA-1.5-7B | Vicuna-7B | 7B | 53.6 | 25.2 | 86.1 | 2.36 | 51.0 |
| VCD | Vicuna-1.5-7B | 7B | 48.8 | 24.3 | 84.5 | 2.12 | 54.2 |
| OPERA | Vicuna-1.5-7B | 7B | 45.1 | 22.3 | 85.4 | 2.15 | 54.2 |
| HA-DPO | Vicuna-1.5-7B | 7B | 39.9 | 19.9 | 86.8 | 1.98 | 60.4 |
| POVID | Vicuna-1.5-7B | 7B | 48.1 | 24.4 | 86.3 | 2.08 | 56.2 |
| LLaVA-RLHF | Vicuna-1.5-13B | 13B | 38.1 | 18.9 | 82.7 | 2.02 | 62.5 |
| LURE | Vicuna-1.5-7B | 7B | 27.7 | 17.3 | - | 1.64 | 60.4 |
| RLHF-V | Vicuna-13B | 13B | 12.2 | 7.5 | 86.2 | 2.45 | 51.0 |
| RLAIF-V | Vicuna-1.5-7B | 7B | 8.5 | 4.3 | - | 3.06 | 29.2 |
| MiniCPM-V | MiniCPM-2.4B | 2.8B | 21.6 | 11.5 | 79.5 | 3.70 | 24.9 |
| MiniCPM-V-2 | MiniCPM-2.4B | 2.8B | 14.5 | 7.8 | 86.3 | 4.09 | 18.2 |
| Mini-Gemini-2B | Gemma-2B | 2B | 29.7 | 21.1 | 85.6 | 2.83 | 18.8 |
| Bunny-2B | Qwen-1.5-1.8 | 2.2B | 50.2 | 23.4 | 85.8 | 2.72 | 19.3 |
| LLaVA-MoD-2B | Qwen-1.5-1.8B | 2.2B | 11.4 | 7.2 | 87.0 | 2.76 | 17.2 |
| LLaVA-MoD-2B | Qwen-2-1.5B | 1.9B | 11.2 | 5.9 | 87.2 | 2.91 | 13.8 |

Hallucination-Oriented Benchmarks. As indicated in Tab. 3, LLaVA-MoD shows remarkable performance in mitigating hallucination, even beating its teacher model. It can be attributed to preference distillation from two aspects: Firstly, by assigning a higher probability for the positive response, preference distillation encourages LLaVA-MoD to focus on providing correct and relevant information. Secondly, by assigning a lower probability for the negative response, preference distillation discourages incorrect or unsubstantiated information. By using the teacher model as a reference to adjust response probabilities, preference distillation enables LLaVA-MoD to handle hallucination issues more accurately and reliably, thereby surpassing its teacher model. Moreover, LLaVA-MoD even surpasses recent RLHF-based models (Sun et al., ; Zhou et al., ; Huang et al., ; Leng et al., ; Zhou et al., ). On the Object HalBench, it outperforms RLHF-V (Yu et al., ) by 8.2% in response-level hallucination rate and by 21.3% in mention-level hallucination rate. This demonstrates that preference distillation is an effective task to mitigate hallucination.

Efficiency Comparison. We compare the efficiency of LLaVA-MoD with recent SOTA MLLMs in terms of training data and trainable parameters. As indicated in Tab. 2, The 2B-sized LLaVA-MoD requires only 5M samples and activates merely a subset of its total 2.2B parameters during training and inference. It achieves 8.8% higher performance than Qwen-VL-Chat, using only 23% trainable parameters and 0.3% of training data samples. Moreover, compared to the small-sized model MiniCPM-V with 2.8B trainable parameters, LLaVA-MoD also achieves superior performance with just 1.6% of the data samples and 2B parameters, underscoring its efficiency. LLaVA-MoD enables more efficient utilization of data, parameters, and computational resources. This not only results in efficient training and inference but also provides a practical solution for deploying high-performance models in resource-constrained environments.

### 4.3 Ablation Study

In this section, we thoroughly explore the effect of the distillation strategy and model design. Firstly, we investigate the preference distillation on comprehension and hallucination benchmarks. Since our focus is on the training strategy and model design, we carry out subsequent ablation experiments specifically on mimic distillation. We use CLIP-ViT-L/14 as the default vision encoder, and Qwen-1.5-1.8B and Qwen-1.5-7B as the default student and teacher LLMs respectively.

#### 4.3.1 Impact of Preference Distillation

**Table 4: The impact of preference distillation on the comprehension-oriented benchmarks.**
| Distillation | GQA | VizWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mimic | 59.3 | 40.0 | 68.4 | 58.7 | 68.4 | 65.1 | 61.5 | 60.2 |
| Mimic+Preference | 58.7 | 39.2 | 68.0 | 58.5 | 66.7 | 66.3 | 61.9 | 59.9 |

**Table 5: The impact of preference distillation on the hallucination-oriented benchmarks.**
| Distillation | Object HalBench | POPE | MMHal-Bench |  |  |
| --- | --- | --- | --- | --- | --- |
| Resp $\downarrow$ | Ment $\downarrow$ | F1 $\uparrow$ | Score $\uparrow$ | Hall $\downarrow$ |  |
| Mimic | 39.1 | 22.6 | 86.7 | 2.75 | 17.8 |
| Mimic+Preference | 11.4 | 7.2 | 87.0 | 2.76 | 17.2 |

Preference Distillation Mitigates Hallucination. We explore the effect of preference distillation on comprehension capability and hallucination mitigation. As shown in Tab. 4 and Tab. 5, preference distillation remarkably reduces the hallucination of *s*-MLLM, while it does not yield consistent improvements in comprehension capability.
This observation aligns with that in LLMs (Achiam et al., ), where preference optimization tends to prioritize reducing hallucinations, often resulting in a certain degree of decline in comprehension gains.

KTO Stabilizes Preference Distillation. We explore the training stability of various preference-optimization losses. We employ KTO (Ethayarajh et al., ) and DPO (Rafailov et al., ). As shown in Appendix B.2, both methods enhance hallucination mitigation. However, KTO has a significant advantage over DPO. The result implies that KTO, which indicates whether a sample is good or bad, provides a better signal for *s*-MLLM to surpass *l*-MLLM compared to DPO, which determines which one is better by comparing two samples.

#### 4.3.2 Impact of Training Strategy

KD Facilitates MoE Training. We explore the effect of knowledge distillation (KD) in MoE training by comparing it with supervised fine-tuning (SFT). The ablation experiments are carried out using the same sparse configuration of E4T2, where four experts are initialized and the top two experts are activated. As shown in Tab. 6, KD surpasses SFT on all benchmarks, achieving an 8.1% average gain over SFT. Additionally, it achieves notable gains in complex multi-task scenarios, such as a +8.2% gain on MMB and a +10.0% gain on MME. These results indicate that KD provides a better optimization signal for effectively training the MoE architecture.

**Table 6: Comparison between KD and SFT. The MoE architecture is E4T2.**
| Training | GQA | VizWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SFT | 58.6 | 31.2 | 66.2 | 55.6 | 63.2 | 59.2 | 56.1 | 55.7 |
| KD | 59.3 | 40.0 | 68.4 | 58.7 | 68.4 | 65.1 | 61.5 | 60.2 |

General-to-Specialized KD Boosts Performance.
We explore the effect of the general-to-specialized approach in mimic distillation by comparing the proposed D2D+D2S with D2S. The D2D+D2S involves a two-phase process. Firstly, it distills a dense *s*-MLLM using general data. Subsequently, it transforms the *s*-MLLM from dense to sparse and further distills it using multi-task data. The D2S directly distills the sparse *s*-MLLM using a combined of general and multi-task dataset. Tab. 7 shows that D2D+D2S surpasses D2S with an average gain of 4.0%. The result indicates that D2D+D2S enables effective knowledge transfer, striking a balance between general and specialized competencies. Additionally, D2D+D2S is more computationally efficient, consuming only 62.5% of the GPU hours required by D2S.

**Table 7: Comparison between different distillation strategies in mimic distillation. #GPU indicates the training GPU hours.**
| Strategy | #GPU | GQA | VizWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| D2S | 1536 | 57.9 | 36.5 | 66.3 | 57.5 | 65.7 | 61.4 | 60.0 | 57.9 |
| D2D+D2S | 960 | 59.3 | 40.0 | 68.4 | 58.7 | 68.4 | 65.1 | 61.5 | 60.2 |

Focus on Response Distillation Improves Generalization. We explores the effect of the distillation target by comparing response distillation with response+instruction distillation.
Tab. 8 shows that response distillation surpasses response+instruction distillation with an average gain of 3.1%. The result indicates that focusing on response is more effective for knowledge distillation in auto-regressive modeling. A possible reason for this is that mimicking additional instructions can lead to overfitting the specific instruction patterns in the training data, potentially reducing the generalization to unseen instructions.

**Table 8: Comparison between different distilled tokens. Response indicates distilling solely on the response tokens. Response+Instruction incorporates additional distillation of instruction tokens.**
| Distilled Tokens | GQA | VizWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Response + Instruction | 58.5 | 35.0 | 66.8 | 58.3 | 68.5 | 61.8 | 59.0 | 58.4 |
| Response | 59.3 | 40.0 | 68.4 | 58.7 | 68.4 | 65.1 | 61.5 | 60.2 |

#### 4.3.3 Impact of Model Architecture

##### Sparse Architecture Facilitates Knowledge Transfer.

We explore the effect of sparse architecture in distillation by comparing it with dense architecture. The configuration is E4T1 where four experts are initialized and the top-1 expert is activated to ensure that the activated parameters are the same as those of the dense architecture.
Tab. 9 shows that sparse architecture surpasses dense architecture with an average gain of 3.7%, The superior performance is prominent in complex multi-task benchmarks, such as MME and MMB. The result indicates that sparse architecture leverages MoE to effectively transfer diverse knowledge from *l*-MLLM while maintaining computational efficiency.

**Table 9: Comparison between sparse and dense architecture within the distillation.**
| Architecture | GQA | VizWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dense | 57.6 | 32.7 | 67.3 | 56.8 | 65.2 | 61.8 | 58.2 | 57.1 |
| Sparse | 58.7 | 36.9 | 67.9 | 58.2 | 66.1 | 64.5 | 61.7 | 59.2 |

Teacher Capacity Matters in Knowledge Distillation. We explore the effect of the teacher capacity in the distillation. We employ Qwen-1.5-7B as the strong teacher and Qwen-1.5-4B as the weaker one. The MoE configuration is set as E4T2. Tab. 10 shows that a well-suited teacher can boost performance. For Qwen-1.5-1.8B, using a 7B-sized teacher yields an average gain of 2.2% compared to a 4B-sized teacher.
However, when the student and teacher have a large capacity disparity, it can hinder effective knowledge transfer. For Qwen-1.5-0.5B, using a 7B-sized teacher yields a marginal average gain compared to a 4B-sized teacher.
Utilizing a “middle teacher” with an intermediate capacity can bridge the gap, facilitating smooth knowledge transfer.

**Table 10: Comparison between the strong and weak teachers. We set the strong teacher as the LLM is Qwen-1.5-7B and the weak teacher as the LLM is Qwen-1.5-4B.**
| Student | Teacher | GQA | VizWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen-1.5-0.5B | Qwen-1.5-4B | 56.0 | 25.3 | 64.7 | 53.8 | 63.3 | 62.2 | 50.8 | 53.7 |
| Qwen-1.5-7B | 56.1 | 29.8 | 63.8 | 54.5 | 64.2 | 58.5 | 49.7 | 53.8 |  |
| Qwen-1.5-1.8B | Qwen-1.5-4B | 58.7 | 34.6 | 67.9 | 57.7 | 67.6 | 64.9 | 60.7 | 58.9 |
| Qwen-1.5-7B | 59.3 | 40.0 | 68.4 | 58.7 | 68.4 | 65.1 | 61.5 | 60.2 |  |

## 5 Conclusion

In this paper, we introduce LLaVA-MoD, a novel framework for efficient training of small-scale multimodal language models via knowledge distillation from large-scale ones. It addresses two key challenges in MLLM distillation: enhancing s-MLLM architecture with MoE design for efficiency and expressiveness balance and implementing a progressive knowledge transfer strategy. Extensive experiments show that LLaVA-MoD outperforms existing models with low activation parameters and computational costs. Notably, it outperforms Qwen-VL-Chat-7B by 8.8% with only 2 billion activation parameters, 0.3% training data, and 23% trainable parameters, highlighting its effectiveness in distilling knowledge and driving more efficient MLLM development.

## Appendix A Implementation Details

### A.1 Training Strategy and Hyperparameters

We first freeze the vision encoder and LLM while optimizing the VL adaptor to align image tokens with the word embedding space during initialization. This stage employs a cross-entropy loss with a batch size of 512 and a learning rate of 1e-4.
In mimic distillation, the vision encoder remains frozen, while the LLM and VL adaptor are co-optimized to distill general knowledge from *l*-MLLM in dense-to-dense distillation. Then, the FFN in the LLM is first transformed into a sparse architecture with a mixture of FFNs co-optimized with the VL adaptor to distill specialized knowledge from *l*-MLLM in dense-to-sparse distillation. This stage employs a KL-divergence loss, with cross-entropy loss added for dense-to-sparse distillation. The batch size is 256, and the learning rate is adjusted to 2e-5.
In preference distillation, the model inherits from the mimic distillation. The vision encoder remains frozen, and a mixture of FFN experts is co-optimized with the VL adaptor to distill preference knowledge from the teacher MLLM. This stage employs a KTO (Ethayarajh et al., ) loss to optimize the probability of *s*-MLLM for positive responses to be greater than that of *l*-MLLM and the probability of *s*-MLLM for negative responses to be lower than that of *l*-MLLM. The batch size is 256, and the learning rate is adjusted to 2e-6.
Throughout all stages, we employ the adam optimizer (Diederik, ) and train on 16 NVIDIA A100 GPUs for one epoch each, totaling approximately 960 GPU hours.
The detailed training hyperparameters are shown in Tab. 11.

**Table 11: Training hyperparameters of each stage.**
| Configuration | Initialization | Mimic Distillation | Preference Distillation |
| --- | --- | --- | --- |
| LLM | ✗ | ✓ | ✓ |
| VL Adaptor | ✓ | ✓ | ✓ |
| ViT | ✗ | ✗ | ✗ |
| LLM init. | Qwen-1.5-1.8B | Qwen-1.5-1.8B | 2nd-stage |
| VL Adaptor init. | MLP | 1st-stage | 2nd-stage |
| ViT init. |  | CLIP-Large@336 |  |
| Image resolution |  | 336 $\times$ 336 |  |
| ViT sequence length |  | 576 |  |
| LLM sequence length |  | 2048 |  |
| Optimizer |  | AdamW |  |
| Optimizer hyperparameter |  | $\beta_{1}=0.9,\beta_{2}=0.98$ |  |
| Learning rate | 1e^-4 | 2e^-5 | 2e^-5 |
| Learning rate schedule |  | Cosine decay |  |
| Weight decay |  | 0.0 |  |
| Training epoch |  | 1 |  |
| Warm-up ratio |  | 0.03 |  |
| Global batch size | 256 | 128 | 128 |
| Numerical precision |  | Bfloat16 |  |
| Model parallelism | Zero2 | Zero2 offload | Zero2 offload |

**Table 12: Training dataset of each stage. #Sample means the training samples**
| Stage | Task | Dataset |
| --- | --- | --- |
| Initialization | Captioning | LLaVA-1.5-Pretrain (Liu et al., ) |
| Dense-to-Dense Distillation | Captioning | ALLaVA-Caption-4V (Chen et al., ),<br>ShareGPT4V-PT (Chen et al., ) |
| Conversation | MIMIC-IT (Li et al., ), LVIS (Wang et al., ),<br>LRV (Liu et al., ), SViT (Zhao et al., ) |  |
| Dense-to-Sparse Distillation | Captioning | ShareGPT4V-100K (Chen et al., ),<br>TextCaps (Sidorov et al., ) |
| Conversation | LLaVA-1.5-Instruct (Liu et al., ) |  |
| General QA | GQA (Hudson & Manning, ), VQAv2 (Goyal et al., ),<br>OKVQA (Marino et al., ) |  |
| Grounding | Visual Genome (Krishna et al., ),<br>RefCOCO (Yu et al., ; Mao et al., ) |  |
| Science | AI2D (Kembhavi et al., ), ScienceQA (Lu et al., ) |  |
| Chart & Doc | DVQA (Kafle et al., ), ChartQA (Masry et al., ),<br>DocQA (Clark & Gardner, ) |  |
| OCR | OCRVQA (Mishra et al., ), SynthDoG-EN (Kim et al., ) |  |
| Knowledge | A-OKVQA (Schwenk et al., ), GeoQA+ (Cao & Xiao, ) |  |
| Preference Distillation | Preference | RLAIF-V (Yu et al., ) |

### A.2 Training Datasets

We curate a comprehensive open-source dataset of 5 million samples encompassing both general and expert tasks.
In the Initialization stage, we utilize 0.6M general captioning samples from LLaVA-1.5-pretrain dataset (Liu et al., ) to bridge the gap between visual and language modalities.
In the mimic distillation stage, we utilize 2.4M general captioning and conversation samples to distill general knowledge from the teacher MLLM and 1.4M multi-tasks data samples, including VQA, documents, science, and OCR to distill specialized knowledge from the teacher MLLM.
In the preference distillation stage, we utilize 80K preference data samples to transfer preference knowledge from the teacher MLLM. The detailed dataset of each training stage is illustrated in Tab. 12.

## Appendix B Additional Experiments

### B.1 The Effect of Expert and Router Capacity

We explore the effects of expert and router capacity on the results. Specifically, we utilize CLIP-ViT-L/14 as the vision encoder and the Qwen-1.5-1.8B as the LLM. We experiment with experts of 4 and 8, and router strategies of Top-1 and Top-2. All experiments are conducted under the same datasets and training hyperparameters to ensure a fair comparison. Tab. 14 shows that scaling experts from 4 to 8 doesn’t improve performance. Tab. 14 shows that scaling the router strategy from Top-1 to Top-2 improves performance. This indicates that increasing the capacity of the MoE block could introduce training difficulty. As the capacity grows, the model might struggle more with convergence and optimization, potentially making it harder to effectively learn from the given data. Thus, a careful balance must be achieved between capacity and training efficiency to ensure optimal performance.

**Table 13: Ablations with the number of experts. We set the gating strategy as Top-2. The number of experts is set to 4 and 8.**
| Expert | GQA | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | AVG |
| --- | --- | --- | --- | --- | --- | --- |
| 4 | 59.3 | 68.4 | 58.7 | 68.4 | 65.1 | 64.0 |
| 8 | 58.3 | 67.4 | 58.4 | 69.1 | 64.6 | 63.6 |

### B.2 The Effect of Preference-Optimization Loss in Preference Distillation

We explore the training stability of various preference-optimization losses. We employ KTO (Ethayarajh et al., ) and DPO (Rafailov et al., ). As shown in Tab. 15, while both methods have an impact on reducing hallucination, KTO is significantly superior to DPO. This is because KTO, which indicates whether a sample is good or bad, offers a better signal for the *s*-MLLM to surpass the *l*-MLLM compared to DPO, which determines relative quality by comparing two samples.

**Table 15: Comparison between different preference-optimization loss on comprehension-oriented benchmarks.**
| PO Loss | GQA | VizWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | MME | MMB | MMB${}^{\text{CN}}$ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DPO | 58.5 | 37.3 | 68.2 | 58.6 | 67.8 | 65.4 | 61.5 | 59.6 |
| KTO | 58.7 | 39.2 | 68.0 | 58.5 | 67.6 | 66.3 | 61.9 | 59.9 |

**Table 16: Comparison between different preference-optimization loss on hallucination-oriented benchmarks.**
| PO Loss | Object HalBench | POPE | MMHal-Bench |  |  |
| --- | --- | --- | --- | --- | --- |
| Resp $\downarrow$ | Ment $\downarrow$ | F1 $\uparrow$ | Score $\uparrow$ | Hall $\downarrow$ |  |
| DPO | 20.0 | 10.5 | 86.8 | 2.68 | 18.9 |
| KTO | 11.4 | 7.2 | 87.0 | 2.76 | 17.2 |

## Appendix C Limitations and Future Works

LLaVA-MoD requires that both *s*-MLLM and *l*-MLLM belong to the same series of LLMs to ensure consistency in the vocabulary space. Future research can explore distillation techniques involving heterogeneous model families. Moreover, the requirement to load both *s*-MLLM and *l*-MLLM leads to substantial memory consumption. To enable efficient distillation, a possible solution could be to pre-extract the logits of the *l*-MLLM. This would allow only the *s*-MLLM to be loaded during training, reducing memory requirements and potentially speeding up the training process.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Why does increasing the number of experts from 4 to 8 fail to improve performance, and is there a fundamental limitation in the MoE architecture?

How can the framework be modified to enable distillation between different model families, rather than requiring the same LLM family for student and teacher?

What specific mechanisms enable preference distillation to reduce hallucinations more effectively than RLHF-based methods?
soundness: 3
strengths: The technical design features a well-constructed progressive distillation strategy that combines mimic distillation for knowledge transfer with preference distillation for reducing hallucinations. The integration of MoE architecture into small-scale models is thoughtfully implemented, striking an effective balance between computational efficiency and model performance.

The empirical validation is thorough. The authors present comprehensive evaluations across multiple benchmarks, showing that LLaVA-MoD-2B not only outperforms larger models while using fewer resources, but also achieves better results in hallucination reduction compared to existing methods. The ablation studies demonstrate the contribution of each component to the overall performance.

The work also shows strong practical relevance, addressing the challenge of deploying efficient multimodal language models in resource-constrained environments. The reduction in required training data and parameters while maintaining or improving performance makes this work valuable for real-world applications.
summary: LLaVA-MoD introduces a framework for creating efficient small-scale multimodal language models through knowledge distillation from larger models. The approach tackles two key challenges: optimizing network structure through sparse Mixture of Experts (MoE) architecture, and implementing a progressive knowledge transfer strategy. This strategy combines mimic distillation, which transfers general and specialized knowledge, with preference distillation to reduce hallucinations. The framework represents an advancement in making multimodal language models more efficient and practical for real-world applications.
weaknesses: The most significant technical constraint is the requirement that student and teacher models must belong to the same LLM family to ensure vocabulary space consistency. This limitation restricts the framework's flexibility and broader applicability. Additionally, the training process demands substantial memory resources as it requires loading both student and teacher models simultaneously.

The experimental evaluation, while comprehensive in many aspects, lacks detailed analysis in key areas. The paper could benefit from a more thorough examination of inference costs and computational requirements in real-world deployment scenarios. There is also limited exploration of training stability across different configurations and minimal discussion of failure cases or error patterns.

The architectural exploration could be more extensive, particularly regarding different MoE configurations and alternative approaches to reducing memory requirements during training.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Since the paper emphasizes data efficiency, could you clarify how the other methods in the comparison were trained? For instance, is there any early-stopping strategy, or specific training limits set for them? And for LLaVA, how do you know that “5M” samples are enough?
Do the S and L need to be the same architecture, or is there some flexibility here? As I understand it, L is distinct from S, where S is initially trained as a dense model before introducing sparsity. Is that correct?
- For Table 1, could you clarify the comparison parameters? Are the final model sizes (S) similar across comparisons? And do the models differ in their teacher models, or is it consistent in that regard?
soundness: 3
strengths: - The paper provides extensive experimental comparisons with prior work, especially highlighting gains in data efficiency and trainable parameters, which is a strong advantage for low-resource training.
- I found the preference distillation interesting in that it reduces hallucination. This aspect could benefit from a deeper explanation of the mechanism behind this reduction, but it’s a notable strength.
- Given the complexity of the architecture, the authors include an extensive ablation study to evaluate each component’s effectiveness, which helps to validate the design choices.
summary: This paper presents LLaVA-MoD, a framework for efficiently training a small-scale multimodal language model (MLLM) by distilling knowledge from a larger, more capable MLLM (I'll denote this as L for brevity). The main contributions are twofold: introducing a sparse MoE architecture for the student (I'll denote this as S for brevity) and developing a progressive distillation strategy. The MoE architecture adds multiple FFN heads with a linear routing mechanism, for efficiency and model expressiveness. The progressive distillation approach occurs in two stages: (1) mimic distillation, which minimizes KL divergence to align S’s outputs with the L’s outputs, and (2) preference distillation, which enhances the student’s ability to distinguish preference between examples, thus reducing hallucination rates. Compared to existing approaches, LLaVA-MoD is notably efficient both in terms of training data and the number of trainable parameters.
weaknesses: - While training sample efficiency and the number of trainable parameters are discussed, it would be helpful to see more detail on the total parameters required for training and inference. Efficiency in trainable parameters doesn’t always translate directly to lower memory or computational costs-full backpropagation through the LLM is still necessary even if only adapter and FFN heads are updated. Information on actual memory usage or wall-clock time would give a more complete picture of the framework’s practical efficiency.
- The architecture is fairly complex, and although the authors do a good job with the explanation, some details remain a bit unclear. For example, the architectural differences between S and L could be more explicitly described to clarify distinctions. See questions

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. There are more small-scale MLLM coming out these days with higher performance and comparable size (0.5B) [1]. Could the sparse MoD architecture and distillation strategy be applied to these newer models to further enhance their performance?

2. I am wondering about the performance of Preference Distillation vs DPO in the second stage. Does it show advantages using distillation instead of direct DPO, in terms of model performance or training efficiency?


[1] Li, Bo, et al. "Llava-onevision: Easy visual task transfer." arXiv preprint arXiv:2408.03326 (2024).
soundness: 4
strengths: This work introduces a new framework for better tuning small-scale MLLM. This framework provides solutions in real world problems by offering a more efficient approach to model inference and deployment across diverse application scenarios.

The experiment is comprehensive, covering a wide range of tasks and models, on both general comprehension benchmarks and hallucination benchmarks. The design of s-MLLM is efficient and effective, achieving superior performance compared to other models of similar size across multiple benchmark tests. The distillation pipeline is clear and well defined. The paper is clearly written.
summary: This work introduces a framework called LLaVA-MoD for building efficient small-scale multimodal large language models (MLLMs) using a sparse Mixture-of-Experts (MoE) architecture. The framework specifically applies MoE to MLP layers within transformer blocks. To preserve the capabilities of larger MLLMs while maintaining efficiency, the authors developed a specialized distillation approach combining KL divergence loss and Preference Optimization (PO) loss between the student (small) and teacher (large) models.
weaknesses: This work  can also be related to inference efficiency and inference acceleration. It would be good if it included inference measurements like FLOPs and inference latency showing the work’s practical benefits in deployment scenarios.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: --What's the computational complexity?
--How do you address the limitation of the teacher's model?
soundness: 2
strengths: 1. LLaVA-MoD achieves high performance with only a small fraction of the data and computational resources required by traditional large MLLMs. This makes it more suitable for real-world applications, especially in low-resource environments.
2. Reduced Hallucination: By incorporating preference distillation, LLaVA-MoD significantly reduces hallucination rates, making it more reliable for applications where factual accuracy is critical.
3. Flexible Architecture with MoE: The use of a sparse MoE architecture allows the model to selectively activate different "experts" for specific tasks, ensuring computational efficiency while retaining expressive power.
4. Superior Performance: LLaVA-MoD surpasses larger models like Qwen-VLChat-7B in various benchmarks, demonstrating the effectiveness of its knowledge distillation approach.
summary: The paper introduces LLaVA-MoD, a framework aimed at distilling knowledge from large multimodal language models (MLLMs) to create smaller, efficient multimodal language models (s-MLLMs) that retain high performance with reduced computational requirements. LLaVA-MoD achieves this by employing a sparse Mixture of Experts (MoE) architecture to maintain expressive power while minimizing parameters. The distillation process is twofold: Mimic Distillation aligns the smaller model with the larger one through Kullback-Leibler (KL) divergence, and Preference Distillation enhances the smaller model’s ability to judge quality, outperforming the larger model on tasks prone to hallucination. Experimental results highlight LLaVA-MoD's efficiency, achieving significant performance improvements with minimal data and trainable parameters.
weaknesses: --The progressive distillation method, which includes dense-to-sparse transitions and multi-task training, could be complex to implement and tune effectively.
--While preference distillation enhances hallucination mitigation, it does not consistently improve the model’s comprehension capabilities, which might limit its applicability in understanding complex instructions.
--The effectiveness of LLaVA-MoD is heavily reliant on the quality and capacity of the teacher model. If the teacher model is limited, the distilled s-MLLM might inherit those limitations.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above in weaknesses.
soundness: 3
strengths: - The paper combines MoE with knowledge distillation in a novel way, optimizing small multimodal models efficiently without sacrificing performance.
- The proposed method is robust, with a progressive distillation process that includes preference-based fine-tuning to mitigate hallucinations.
- The paper is well-written and clearly explains the framework.
- Experiments were done on a comprehensive set of evaluations.
summary: The paper introduces LLaVA-MoD, which is a framework that distills large multimodal models into efficient, small-scale ones using a sparse MoE and progressive knowledge transfer. By optimizing both model architecture and distillation strategy, LLaVA-MoD achieves high performance with minimal data and parameters, even surpassing larger models on multimodal tasks. This framework enables cost-effective deployment of high-performing models in resource-limited settings
weaknesses: - The paper does not provide an analysis of how different MoE configurations (e.g., number of experts, routing strategies, parameter k from top-k routing) impact performance and computational trade-offs. Including such an analysis could provide further valuable insights for the model.
- The results presented in Table 2 is relatively modest in improvement. LLaVA-MoD-2B only outperforms the other 2B baselines (Imp-2B) by 2.7% on average while using twice as many data samples.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the above comments.
soundness: 3
strengths: The paper proposes a scalable approach for distilling large multimodal models to smaller one, which is a very relevant topic in recent time. The proposed methodology is somewhat novel and addresses key challenges faced by the existing small multimodal models. The experiments are extensive and performance improvement shown by the proposed methodology is significant.
summary: The paper proposes a framework LLaVA-MoD for distilling knowledge from large multimodal models to smaller models. The method incorporates mixture-of-experts for enhancing the expressiveness of small multimodal models. Additionally, the method uses two stage distillation -- Mimic Distillation for general knowledge alignment and Preference Distillation to enhance the student model's discriminatory capabilities for superior generalization. The experimental results are quite interesting -- The 2-billion parameter LLaVA-MoD outperforms larger models like Qwen-VL-Chat-7B by 8.8%, using only 23% of trainable parameters and 0.3% of the training data. LLaVA-MoD performs significantly well on 7 multimodal tasks and hallucination benchmarks.
weaknesses: 1. The experimental setup is not very convincing to me. The authors claim their method to be an alternative distillation method without much empirical comparison. In order to establish the effectiveness of their method, the authors must compare some of the existing KD methods -- KD and GKD, to show why their distillation method is needed. 

2. The dense-to-sparse loss is very similar to the competitive loss proposed by Sengupta et al., 2023 [1]. The authors should show how their proposed losses differ from these losses.

3. The authors should elaborately discuss why dense vs sparse architecture works. The authors mostly report the numbers with little explanation of the how part.

4. The lower performance of Mimic+preference than mimic in Table 4 should be discussed. It is counterintuitive.

5. The authors claim that their method is efficient. More detailed analysis with training/inference time and memory should be discussed. 

**References**

[1] A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation - ICLR'2024.

Meta-review: This work introduces LLaVA-MoD, a framework for training compact multimodal language models through knowledge distillation from larger models. The study addresses two main challenges: (1) optimizing the network structure using a sparse Mixture of Experts (MoE) architecture, and (2) implementing a progressive knowledge transfer strategy to acquire both general and specialized knowledge via preference optimization / alignment. The experimental results effectively demonstrate the benefits of the proposed framework.

Overall, five out of six reviewers recognize the soundness and contributions of the proposed techniques, as well as the experimental improvements. Only one reviewer raised concerns regarding the complexity of the techniques and the limitations inherited from the teacher model. The authors have committed to providing detailed documentation and guidelines to facilitate implementation and tuning, which addresses the concern about complexity.

Regarding the limitations inherited from the teacher model, the authors introduced mimic distillation to capture the strong comprehension capabilities of the teacher model and preference distillation to address the teacher model’s inherent limitations in mitigating hallucinations. These efforts are partially validated by Table 3, where the student model even outperforms the teacher model in certain metrics.

Given these contributions and the steps taken to address the identified weaknesses, we conclude that the strengths of the work outweigh its limitations, many of which have been partially addressed. Therefore, we recommend accepting this submission. But for authors, please follow the reviewers' feedback and your promise to revise the paper.

Decision: Accept (Poster)