$\gamma-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models
Authors: Yaxin Luo
Gen Luo
Jiayi Ji
Yiyi Zhou
Xiaoshuai Sun
Zhiqiang Shen
Rongrong Ji
Venue: ICLR 2025 Poster (2025)
Keywords: mixture of depths; multimodal large language models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Despite the significant progress in multimodal large language models (MLLMs), their high computational cost remains a barrier to real-world deployment. Inspired by the mixture of depths (MoDs) in natural language processing, we aim to address this limitation from the perspective of ``activated tokens''. Our key insight is that if most tokens are redundant for the layer computation, then can be skipped directly via the MoD layer. However, directly converting the dense layers of MLLMs to MoD layers leads to substantial performance degradation. To address this issue,  we propose an innovative MoD adaptation strategy for existing MLLMs called $\gamma$-MoD.  In $\gamma$-MoD,   a novel metric is proposed to guide the deployment of MoDs in the MLLM, namely rank of attention maps (ARank). Through ARank, we can effectively identify which layer is redundant and should be replaced with the MoD layer.  Based on ARank,  we further propose two novel designs to maximize the computational sparsity of MLLM while maintaining its performance, namely  shared vision-language router and  masked routing learning.   With these designs, more than 90% dense layers of the MLLM can be effectively converted to the MoD ones. To validate our method, we apply it to three popular MLLMs, and conduct extensive experiments on 9 benchmark datasets.  Experimental results not only validate the significant efficiency benefit of $\gamma$-MoD to existing MLLMs but also confirm its generalization ability on various MLLMs.  For example, with a minor performance drop,  i.e., -1.5%, $\gamma$-MoD can reduce the training and inference time of LLaVA-HR by 31.0% and 53.2%, respectively.

Published as a conference paper at ICLR 2025
Y Y --MoDD: EXPLORING MIXTURE-OF-DEPTI ADAPTA- TION FOR MULTIMODAL LARGE LANGUAGE MODELS
Yaxin Luo', Gen Luo²t, Jiayi Ji3.4, Yiyi Zhou², Xiaoshuai Sun Zhiqiang Shen', Rongrong Ji
MBZUAL
2OpenGVLab, Shanghai Laboratory *National University of Singapore
3 Xiaaen University
Project Page: Gamma-MOD
ABSTRACT
Despite the significant progress in multimodal large language models (MLLMs), their high computational cost remains barrier to real- weal-world deployment. Inspired by the mixture of depths (MoDs) in natural language processing, we aim to address this limitation from the perspective of "activated tokens". Our key insight that if most tokens are redundant for the layer computation, then can be skipped directly via the MoD layer. However, directly converting the dense layers of MLLMs to MoD layers leads substantial performance degradation. To address this issue, we propose an innovative MoD adaptation strategy for existing MILMs called M-Moo. In >-MoD, a novel metric proposed to guide the deployment of MoDs in the MLLM, namely rank of attention maps (ARank). Through ARank, we can effectively identify which layer is redundant and should be replaced with the MoD layer. Based on ARank, we further propose two novel designs to maximize the computational sparsity MILM while maintaining its performance, namely shared vision-languuge router and musked routing learning. With these designs, more than 90% dense layers of the MI..M can be effectively converted to the MoD ones. To validate method, apply it to three popular MLLMs, and conduct extensive experiments on benchmark datasets. Experimental results not only validate the significant efficiency benefit of 7-MoD to existing MLLMs but also confirm its generalization ability on various MLLMs. For example, with a minor performance drop, i.e., -0.9%, M-MoD reduce the training and inference time of LLaVA-HR by 31.0% and 53.2%, respectively.
INTRODUCTION
Recent years have witnessed the great success of large language models (LLMs) in natural language
processing (NLP) (Achiam et al. 2023 Touvron et al. 2023 Cai et al. 2024b), which attracts
increasing attentions in extending LLMs vision-language (VL) tasks. Despite the progress, recent
multimodal large language models (MLLMs) (Luu et al., 2024d с Chen et al. 2024a Alayrac et al. 2022) are often criticized by their expensive computational costs. For example, the inference speed of existing MILMs like VA-HHR (Luo et al., 2024) is still far from practical requirements, e.g., 4.7 samples per second. Driven by the progress
NL.., recent advances have employed the mixture-of- experts (MoEs) (Lin et al.) 2024a| [Jiang et al. 2024) to MLLMs to reduce the "activated parameters", thus achieving trade-off between efficiency and performance.
Orthogonal to MoEs, we aim to tackle the efficiency bottleneck of MLLMs from the perspective of "activated tokens". As shown in Fig. (a). a large number tokens are less important in the computation, such visual background and prepositional words. However, existing MoEs still allocate the same experts to all input tokens, leading to redundant computational costs. A promising solution this issue is the recently proposed mixture-of-deptths (MoDs) in NL.P (Raposo al. 2024), which equips each token with a router to determine whether a module should be computed.
+Corresponding author.

---

Published as a conference paper at ICLR 2025
Tayer-I (High Rank)
Layer-4 (Low Rank)
Mixture-of-Eaee Expert (MoE)
Mixture-of- (MoD)
Feed-forward Network
Expert
Expert
Layer-24 (Low Rank)
Layer-31 (High Rank)
Multi head Attention
Router
Multi-head Attention
Router
(a) Attention maps of different layers in LLLVA-HR
(b) Comparison of MoE and MoD
Figure 1: Visualization of attention maps in the MLLM and comparison of MoE with MoD. (a) Lower-rank layers often exhibit redundancy in their attention computation. (b) Different from MoE, MoD achieves the computational sparsity from the perspective of "activated token", where the computational budget is dynamically allocated to each token.
However, recent MoDs (Raposo et al. 2024) typically require pre-training LLMs from scratch, and their employment on MLLMs still remains under-explored
In this paper, focus the efficient adaptation of MoDs to existing MLLMs. In particular, our goal is to maximize the computational sparsity of MLLMs while maintaining competitive performance. However, directly converting all dense layers of MILMs MoD) layers leads significant perfor- mance degradation, e.g., -33.3% of LLaVA-HR (Luo et al., 2024) on TextVQa (Singh et al.] (019). In practice, we observe that such issue is mainly caused by two aspects. Firstly, the deployment of MoDs lacks a practical guidance to measure the layer redundancy, thus undermining the necessary dense layers. As illustrated in Fig. (a), attention patterns vary significantly across layers, and some layers exhibit less redundancy. Additionally, the setting of MLLMs. e.g., input modality, differs substantially from that of LLMs, making the direct adaptation of MoDs suboptimal.
To overcome these limitations, we first propose a novel metric to guide the deployment of MoDs in MI..Ms, called the rank of attention maps (ARank). Our key insight is that lower-rank attention maps indicate that fewer tokens are necessary for computation. As shown in Fig. (a), most of tokens of Layer-4 are assigned small attention weights, contributing minimally to the final output. This provides a valuable hint for us replace the redundant layer with the MoD one under the guidance of ARank. In practice, the calculation of ARank is both efficient and flexible. Empirically, we find that the average ARank always keeps the similar despite the change of samples. Therefore, randomly sampling a small amount of data can already accurately estimate the ARanks.
Based the ARank, we propose an innovative MoD adaptation strategy for existing MILMs, called y-MoD. Specifically, 7-MoD is into current MLLMs via instruction tuning. In M-MoD, two novel designs are adopted to maximize its benefits to MLLMs, namely shared vision-language router and masked routing learning. The shared vision-language router performs routing on the entire multimodal sequence and uses a weight-sharing strategy to facilitate optimization. Then, masked routing learning is introduced to prevent critical tokens from being skipped during training, i.e., instruction tokens. With these designs, over 90% of dense layers can be converted to MoD layers with minimal performance sacrifice, resulting in even larger computational sparsity than the native MoD-based L.LM (Raposo et al. 2024).
adaptation approach that can be seamlessly integrated
To validate M-Mo we apply it to two popular MLLMs and conduct extensive experiments on 9 vision-language benchmarks. Experimental results show that MMoD significantly improves the training and inference efficiency of existing MLLMs while keeping their performance competitive. For example, M-MoD reduces 51.6% Flops, 31% training time and 53.2% inference time for LLaVA- HR (Luo et al. 2024). but its average performance decline is only -1.5%. More importantly, the great generalization ability of MoD is also witnessed on different MLLM structures and parameter sizes. Overall, the contribution of the paper can be summarized in three folds:
2

---

Published as a conference paper at ICLR 2025
We present novel mixture-of-depth (MoD) framework for the sparse computation of existing MLLMs, namely y-MoD, which can seamlessly convert most dense layers in MLLMs to the sparse MoD layers.
We propose an innovative metric to measure the layer redundancy, namely rank of attention maps (ARank). With ARank, we can best determine that which dense layer should be convert to the MoD one.
We carefully explore the design of y-MoD in existing MLLMs, including the shared vision- language router and the masked routing learning, which can achieve up 51.6% computa- tional sparsity with minor performance sacrifice. Extensive experiments also confirm the generalization ability of y-MoD.
RELATED WORK
2.1 MULTIMODAL LARGE LANGUAGE MODELS
Large language models (LLMs) Achiam et al.
2023 Touvron et al., 2023 Jiang et al. 2024
Almazrouei et al., 2023 Cai al.
2024b] Abdin al.
2024; Shen al. (2023) have proven their
strong capabilities in various natural language processing tasks (Paperno et al., 2016 Fyodorov
ct al.
2000 Reddy et al. 2019| Ziegler et al. 2019). Motivated by this, numerous efforts (Liu et al.
2024d
Bai et al. 2023a Ye et al. 2023 Dai et al. 2023 Chen et al. 2024b] [.i et al.) [024bb] Tong
ct al.
2024: Rasheed et al. 2024; Dong et al. 2023 Xie et al. 2024 Zhou et al.] [2024] [Chen et al.
2023
Alayrac et al. 2022 Sun al. 2024) have been devoted into extending LLMs to multimodal
large language models (MLLMs). Among them, the most representative work is LLaVA (Liu et al.
2024d). which uses lightweight project connect visual encoder and an LLM. This simple framework has now become the de-facto paradigm in the community, empowering a set of MLLMs like Mini-Gemini (Li et al. 2024b) and InternVL (Chen et al.] 2024b). Recently, researchers have shifted their attentions to high-resolution MILMs. For example, LLaVA-NexT (Liu et al. 2024c) and InternVL-1.5 (Chen et al. 2022a) adopt the dynamic image slicing strategy for high-resolution adaptation. LLaVA-HR (Luo et al. 2024) further propose a dual-branch structure to reduce the cost of high-resolution MLLMs. Despite the effectiveness, existing high-resolution MILMs (Liu et al. 2024c] [ii et al. 2024a) will produce a much longer input tokens, resulting in prohibitively expensive computational costs. this paper, the proposed MMD can greatly overcome the efficiency bottleneck of existing MLLMs, which is significant for their practical applications.
2.2 SPARSE COMPUTATION FOR LLMs
Recently, influx of attentions have been focused the sparse computation of LLMs. Specifically, the mixture of experts (MoEs) are the most popular technology in the community (McKinzie et al. 2024 Cai al. 2024a Xue et al., 2024), which dynamically activates part of expert networks for each Token, thereby achieving trade-oft's between capability and efficiency. For instance, MoË-LLa VA (Lin) al. 2024a) proposed novel approach convert dense MLLM to a mixture-of-expert structure.
However, these methods often require additional training costs to realize the adaptation to MLLMs.
Orthogonal MoE, Raposo et al. (2024) proposed the mixture of depths (MoDs) to dynamically allocate computations for each token. Compared to MoE, the main principle of MoD is to reduce the "activated tokens" instead of the "activated parameters". This paradigm has shown great potentials for the sparse computation of L.Mss, but its potential on MLLM is still under exploration. Recently, token-basee based pruning methods have emerged as a new promising solution. The most representative one is the FastV (Chen et al. 2025), which directly deletes the unimportant visual tokens according to their attention scores, thus achieving significant computational savings without compromising performance. Orthogonal to these works, we are the first explore MoDs on MLLMs, which can seamlessly realize sparse computations of exiting MLLMs on both visual and textual tokens.
3 PRELIMINARIES
We first recap the mechanism of Mixture of Experts (MoEs) and Mixture of Depths (MoDs).
3

---

Published as a conference paper at ICLR 2025
rankert0
Dense Transformer
Redundancy
Dense Transformer
Layer
Estimation
Layer
ranks110
Dense Transformer
Redundancy
MoD Transformer
feed-forward Network
Layer
Estimation
Layer
Multi-head Attention
MLP layer
laycr
Queries
Keys
Visual Encoder
Deseribe the image." of stuff the snow."
pile
Visual Encoder
"How many people are there the pieture?"
VI. Router
Scaled-Dot
Attention
maps
Calubate
Masked Routing Leamingg
Stage-1: Vision-Language Alignment
Stage-2: Instruction Tuning
Illustration of our >-MoD adaptation on LLaVA-HR, 7-MoD is a plug-and-play approach that can be directly applied in existing MILMs. After vision-language alignment, y-Mol) can replace most redundant layers with MoD ones via the rank-based redundancy estimation.
Figure 2:
Mixture of experts. In particular, the main principle of MoE is reduce the "activated parameters"
in dense models. Existing based LLMs (Dai et al.) 2024 Liu et al. 2024a Lin ct al. 2024a
Jiang et 2024) and MLLMs (Luo et al. 2024| 2024|Chen Chen al. 2024a] Liu et al. (024d) often contain muluple FFN modules in their layers, also termed experts. During training and inference, only few experts are activated to participate in computations, thus retaining the trade-offs hetween performance
and efficiency. Given input features x E I MoE mechanism can be defined by
& x= == Σ (x) 1=1
Here, D(.) denotes the expert layer, Le., FFN, is the number of activated experts, and (.(.) the corresponding routing function. In practice, top-k experts are selected according to their routing scores, where & much smaller than the total number of experts K.
Mixture of depths. Different from MoEs, MoDs aim to improve the model efficiency via the reduction of "activated tokens". Compared to MoBs, the routing mechanism MoDs performs on input tokens, and most tokens will directly skip the dense layer in MLLMs. Thus, MoDs can be written as
σ + ((I,) Ty
R(x;) 2 8,. if R(x,) < 6,,
(2)
where zj E IRd denotes the token vector in and 8, is routing threshold. As defined in Eq. inactive tokens will directly skip the layer (-) to save the computational cost.
Discussion. In existing MLLMs (Lin et al. 2024a). MoE is typically used to efficiently scale up the model size, while its computations are not directly reduced. In contrast, MoD can perform as plug-and-play module to save the cost of common dense layer, which is more significant to the efficient scenario. Unfortunately, the adaptation of MoD to existing MI.LMs is still under-explored, and its practical use in LLMs also requires expensive pretraining.
METHOD
4.1 OVERVIEW
In this paper, we propose a novel method to efficiently deploy MoDs to existing MLLMs, namely MoD. The core principle of M-D is to identify redundant MILLM layers via novel metric called
4

---

Published as a conference paper at ICLR 2025
GQA
100
50A
TextVOA
MMMU
24
32
6
16
24
32
Figure 3: Visualization of ARank based different tasks (left) and sample sizes (right). The horizontal axis represents the layer index of LLaVA-HR, The darker color indicates the larger ARank.
rank of attention maps (ARank) and replace them with the proposed MoD layer. Therefore, the deployment of 7-MoD in the given MILM,, i.e., FMLLM((). can be formulated by
FMLLM = - o 91 o G2... o In.
(D: Di if (D,)) 2 8
where Gi
S if r 67-
Here, G(-) denotes the layer of the MILM, where S(-) and D(-) indicate the dense layer and its MoD alternative, respectively. r(.) is a function to estimate the redundancy of the given dense layer D; and 6, is threshold. Given the architecture in Eq. 3 7-MoD aims maximize the sparsity while maintaining the performance. Thus, the optimization objective of y-MoD can be written as:
k
arg min Lobi (FMLLM(Z) + 0,0,
Σ
Lang (R(x';0,))
1
k
Here, Coby and Laug denote the regr-segre loss and the routing loss for the router R(-), respec- tively. x² is the input tokens of i-th layer, and is the pre-defined sparse target. [(()))< {0, 1) is the indicator function, which is equal to 1 when R(xt) < 6g. And k is the number of layers, d denotes the number of tokens per layer.
4.2 RANK-BASED BASED REDUNDANCY ESTIMATION
The key challenge of 7-MoD how identify the dense layer that should be converted to the MoD one. The original MoD-based LLM (Raposo et al. 2024) overcomes this issue by the hand-craft attempt, which still sub-optimal and time-consuming. However, in existing MLLMs, the LLM is already pre-trained on large scale of corpus, which can intuitively provide sufficient knowledge to achieve the process automatically.
Motivated by this, we propose an innovative metric to estimate the token-wise redundancy of layer in MILM. namely rank of attention maps (ARank). particular, given tokens x' g of i-th layer, ARank is defined by the average rank of attention maps:
7(x', D.)
1
1
Σ
Rh
rank (An):
nh
(5)
where A = (x (c
Here, rank(-) denotes the rank calculation. ny. is the number of attention heads. As E B is the attention map in h-th head, and W [ad and W _dx 1 are the corresponding weights.
Theoretical analysis of ARank. In Eq. 5 attention map A can well reflect the contribution of different tokens. Thus, A,, with a low rank suggests that most tokens are less informative. To validate this, conduct a SVD GoulD & C.Reinsch) 1971) analysis for An, which is written
A
Σ
(6)
5

---

Published as a conference paper at ICLR 2025
where r is the rank of A, and r' < r is constant value. de, 0; and v2 denote the i-th single value, left single vector and right single vector of A,, respectively. As shown in 6 A,, can be deposed to matrix of rank r' and additional information, i.e., Ei-r4a ".u,oo Therefore, lower-rank attention map suggests higher redundancy, which implies that MoD can be deployed to skip most tokens.
Pracfical calculation of ARakk. As defined in Eq. it is still challenging accurately calculate the ARank due to the variance of individual samples. Inspired by HRank (Lin et al.] 2020). we estimate Aaak using its expectation on batch of samples. Different from HRank, we aim to estimate the layer redundancy by the rank of their attention maps, thus guiding the deployment of MoD. Specifically, ARank estimates layer redundancy based on the rank of attention maps, enabling its use in guiding the deployment of MoD. As shown in Fig. [] we visualize the average ARank values of L.L.VA-HR (Luo et al., 2024) across different input samples. These results demonstrate that the expected ARank remains largely consistent across tasks, indicating that a small batch size sufficient for reliable computation. In our experiments, we set the sample size to 50 to balance computational efficiency and accuracy.
4.3 MIXTURE-OF-DEPTH ADAPTATION
To maximize the effectiveness of MoDs to existing MLLMs, we carefully investigate the micro design of MoDs, including the shared vision- Saiinnla router and the masked routing learning.
Shared vision- vision-language router. Conventional MoDs (Raposo et al., 2024) are designed for LLMs, so their routing is only performed on textual tokens. In MLLMs, such a strategy sub-optimal due to the large redundancy of visual tokens (Jin et al. 2024 Kim et al. (024). Therefore, the router of y ie., R(-), aims to skip both visual and textual tokens, which is defined by
R(x) softmax(=Wx bx),
where x x= {g. a, t denotes the vasion-language tokens, which consist of question tokens E IR xd image tokens R/axd and textual response tokens t R4 Wn R and bn E R are the weights and bias, respectively. Notably, we use a binary softmax function to produce the routing probability, where R(x) denotes the probability of skipping. Based Eq. 7 we further share the router parameters for all MoD layers, which is significant for the stable optimization. To explain, the shared router receives more gradients from different layers, greatly facilitating its convergence the beginning of training.
Masked routing learning. During VL training, not all tokens contribute equally the optimizing process. In particular, the skip of key tokens in the question, egg., subject, will greatly hurt the generative training as the answer relies on these conditional elements. Therefore, we introduce a masked routing learning strategy to prevent these tokens from being dropped during training. In this case, the objective of the routing learning can be defined by
(a) log (R(x)* Mg) R + log (1 R-)(()) (1) R). (R-
(8)
Here, Mg E RX denotes the binary mask, where the question tokens are assigned to 0. E is the one-hot vector, where the position with top-k routing scores are assigned to I.
The training scheme. Typically, MILM training is divided into two stages: linguae (VL) alignment and instruction tuning. M-Mo is plug-and-play adaptation method, which is deployed in the instruction tuning stage. Therefore, we can skip the VL alignment by directly using the well pre-trained projector. Then, MoD then evaluates layer redundancy using the ARank metric and replaces redundant layers with MoD layers. During instruction tuning, the routing parameters are jointly optimized via the routing and task objectives. Importantly, all other training configurations can remain consistent with the original MLLM setup, ensuring seamless integration of M-MoD.
5 EXPERIMENTS
5.1 DATASETS AND METRICS
We evaluate our M-D on five MLLM benchmarks, which includes POPE (i et al. 2023). MME (Fu
al. 2024), MMB (Liu et al. 2024e), MMMU (Yue et al. 2024) and Met (Yu al. 2023 We

---

Published as a conference paper at ICLR 2025
Table Comparison different y-MoD configurations on LLaVA-HR. The default setting used
in the table is colored in gray. "Q" and "A" refer to question and answer tokens.
GQA
SQA
MMMU
TextVQa
Average TFlops
Methods
Acc.
Skip
Acc.
Skip
Acc.
Skip
Acc.
Skip
Acc.
Skip
LL- (Luo et al. 2024
64.2
0%
67.9
0%
34.6
0%
67.1
0%
58.5
19.2
0%
Mol) layer:
All layers
45.9
38.2%
42.6
33.7%
25.9
32.8%
33.8
34.1%
37.1
12.3
34.7%
1 MoD per 2 layers
57.8
19.1%
52.3
16.5%
26.9
16.6%
54.0
17.9%
47.8
16.1
17.5%
2 MoDs per 3 layers
38.1
26.8%
24.6% 46.5
24.3
24.4%
42.1
24.9%
37.8
15.9
25.2%
ARank-based deployment
40.7% 63.7
35.9% 68.5
35.6
36.8%
65.3
38.2%
58.3
12.6
37.9%
Masked token:
None
63.2
52.0%
66.8
46.9%
33.9
47.0%
64.7
49.8%
57.2
10.7
48.9%
Q
63.7
40.7%
68.5
35.9%
35.6
36.8%
65.3
38.2%
58.3
12.6
37.9%
A
62.8 38.8%
68.6
30.5%
34.7
35.4%
62.0
37.2%
57.0
13.0
35.5%
Shared router:
Not Share
60.6
55.8%
64.5
48.2%
32.1
48.9%
58.4
52.9%
53.9
10.3
51.5%
Share
63.1 60.3%
67.9
56.9%
34.7
56.6%
64.9
57.1 %
57.6
9.3
57.7%
Routing ratio:
17%
63.6 18.9%
68.9
15.5%
34.7
14.7%
66.1
16.5%
58.3
16.3
16.4%
34%
63.7 40.7%
68.5
35.9%
35.6
36.8%
65.3
38.2%
58.3
12.6
37.9%
51%
63.1
60.3%
67.9
56.9%
34.7
56.6%
64.9
57.1% 1%
57.6
9.3
57.7%
68%
59.1
77.8%
70.1
73.5%
33.7
71.8%
58.4
74.1% %
55.3
6.5
74.3%
Table 2: Ablation study of 7-MoD on LLRVA-HR. "Parami,, "Acc." and "Skip" indicate the parameter, accuracy, and skip ratio, respectively.
GQA
SQA
MMMU
TexiVQa
Average TFlops
Methods
Param
Acc.
Skip
Acc.
Skip
Acc.
Skip
Acc.
Skip
Acc.
Skip
LLaVA-HR (Luo et al.
2024
7.4B
64.2
0%
67.9
0%
34.6
0%
67..
0%
18.5
19.2
0%
Defiult MoD (Rapsso et al. 2024
7.4B
45.9
38.2%
42.6
33.7%
25.9
32.8%
33.8
34.1%
37.1
12.3
34.7%
-RRak--baeed deployment (ours)
7.4B
63.2
52.0%
66.8
46.9%
33.9
47.0%
64.7
49.8%
57.2
10.7
48.9%
MMskked routing learning (ours)
7.4B
63.1
60.3%
67.9
56.9%
34.7
56.6%
64.9
57.1%
57.6
57.7%
report all the results in their default settings. addition, we evaluate y-Mol) on six image question answering benchmarks: VQAv2 (Goyal et al. (017), VizWiz (Gurari et al. 2018). TextVQA (Singh iet al. 2019), SQA (Lu et al. 2022), GQA (Hudson & Manning (219)) and SEED (Ge et al. (023).
We report all the results in their default settings. For MME, we report the perception score.
5.2 IMPLEMENTATION DETAILS
For all models, pre-training is conducted on LCS-558K dataset (Liu et al.) 2024b), which includes high-quality 558k image-text pairs. For instruction tuning, we follow LLaVA-1.5 (Liu et al. 2024b) to use 665k vision-language instruction data. To deploy 7-MoD to MLLMs, ARank is calculated to identify redundant layers after the pre- mreitrannin stage. For all models, the fourth largest ARank value is used as the threshold for converting dense layers to MoD ones. During instruction tuning, the coefficient for the routing loss is set to 0.01. The remaining settings are kept the same with LLaVA-HR (Luo et al. 2024) and LLaVA (Liu I et all 2024b), including learning rate, training epochs, optimizer and datasets, elc.
5.3
EXPERIMENTAL RESULTS
5.3.1
QUANTITATIVE ANALYSIS
Comparison with different MoD configurations. In Tab. we first compare different settings of MoD on LLaVA-HR (Luo et al. 2024). From this table, the first observation is that directly converting all layers to MoD ones leads to worse results, e.g., 33.8% TextVQA. Besides, although the hand-crafi strategy performs much better. its performance declines are still obvious, e.g.. -10.7% of MoD per 2 layers average. These results confirm the challenges of adopting MoDs ML.MM..
7

---

Published as a conference paper at ICLR 2025
Table Results of y-MoD on different MLLM architectures and model scales. 2-MoD-0.3 and 7-MoD-0.5 denote the routing ratio of 30% and 50%, respectively.
GQA
SQA
MMMU
TexIVQA
Average TFlops
Methods
Param
Acc.
Skip
Acc.
Skip
Acc.
Skip
Acc.
Skip
Acc.
Skip
MLLM architecture:
LLaVA
7B
62.0
66.8
34.3
0%
58.2
0%
55.3
10.7
+7-MoD-0.3
7B
61.1
34.1%
1%
64.7
29.4%
35.4
29.8%
56.3
30.7%
54.4
7.7
31.0%
+1-MoD-0.5
7B
41.4
60.9%
62.3
54.8%
31.0
53.6%
42.9
56.2%
44.4
5.3
56.4%
LLaVA-HR
7B
64.2
0%
67.9
0%
34.6
0%
67.1
0%
58.5
19.2
+1-MoD-0.3
7B
63.7
40.7%
68.5
35.9%
35.6
36.8%
38.2%
58.3
12.6
37.9%
+7-MoD-0.5
7B
63.1
60.3%
67.9
56.9%
34.7
56.6%
57.1%
57.6
9.3
57.7%
Mini-Gemini-HD
7B
62.9
0%
69.6
0%
36.8
0%
66.5
0%
59.0
60.2
+--MoD-0.3
7B
62.1
37.1%
69.0
34.6%
34,1
36.4%
66.4
36.6%
$7.9
39.4
36.2%
+-MoD-0055
7B
62.2
59.2%
70.4
56.8%
33.9
58.6%
67.0
57.7%
58.4
27.8
58.1%
Model scales:
LLaVA-HR
7B
64.2
0%
67.9
0%
34.6
0%
67.1
0%
58.5
19.2
0%
+1-MoD-0.3
7B
63.7
40.7%
68.5
35.9%
35.6
36.8%
38.2%
58.3
12.6
37.9%
+--oDD-0.5
7B
63.1
60.3%
67.9
56.9%
34.7
56.6%
64.9
57.1%
57.6
9.3
57.7%
LLaVA-HR
13B
64.8
0%
68.1
0%
36.7
0%
68.1
0%
59.4
37.1
0%
+7-MoD-0.3
13B
64.5
38. 1%
70.5
33.1%
37.8
32.5%
67.0
36.0%
60.0
25.1
34.9%
+7-MoD-0.5
13B
64.8
58.8%
69.5
52.2%
35.8
53.8%
66.8
55.4%
59.2
18.4
55.1%
Table Training and inference efficiency of q-MoD on LLaVA-HR, The inference efficiency is tested on an NVIDIA A100 GPU, which is the average value of GQA, SQA, MMMU, and TextVQA.
Training Time
Inference
Inference
Inference TFlops
Avg.
Methods
Throughput
Memory
Acc.
LLaVA-HR
20.7 h
4.7 samples/s
19.
19.2
58.5
+o-MoD-0.3
15.4
samples/s
G
12.6
58.3
-MDo--0
14.3 h
7.2 samples/s
14 G
9.3
57.6
Gains
-3100%
+53.2%
-26.3%
-51.6%
-0.9%
However, after employing our ARank-based strategy, the efficiency of LLaVA-HR is greatly increased while the performance is well maintained.
In Tab. D we also validate different micro-designs for deploying MoD on MILM,, including the masked routing learning, the shared router and the routing ratio. From these comparisons, we first see that the masked learning strategy much beneficial the optimization of M-MoD, providing up to +1.7% gains SQA. In addition, also find that the router sharing strategy plays significant role y-MoD. After removing this strategy, model performance will obviously drop on TextVQA by -6.5%. For routing threshold, we observe that the adaptive thresholds perform better while the default one more efficient. Finally, we validate the impact of different routing ratio on LLaVA-HE From results we can see that model performance can be retained under relatively small routing ratios, ie., 17% and 34%. When routing ratio is increased to 51 %, model performance drops slightly from 58.3% to 57.6% average. However, the benefit of efficiency is still notable, ie., -51.5% Flops
Ablation studies. To validate contributions of each design in 7-MoD, we conduct ablation study in Tab. 2 From this table, we can see that the default MoD will cause obvious performance degeneration, deployment, the average performance of LLaVA A-H is improved from 37.15 % to 57.6%, and the computational sparsity also boosts from 34.7% to 48.9 98.9% Such comparison confirms that not all layers can be converted to MoD layers, and ARank critical identify the redundant ones. addition, the use of masked routing learning can further benefit the model training, providing +0.8% on MMMU and +0.2% on TextVQA, respectively. These results further confirm the effectiveness of y-MoD.
resulting to -25.3% on SQA. In stark contrast, with our ARank-
5.3.2 COMPARISON WITH EXISTING MILMS
Generalizations of Moo different MLLMs. In Tab. also evaluate the generalization ca- pability of M-D across different MLLM architectures and model scales. In particular, 7-MoD with 30% routing ratio demonstrates great trade-off between performance and efficiency LLaVA. When
8

---

Published as a conference paper at ICLR 2025
Table Comparison with quantization and pruning methods. "Speed", "Prefilling" and "Next- token" indicate the throughput (samples/s), prefilling time (seconds) and next-token time (seconds), respectively. For MMMU, models predict one option without the need of the next-token time.
MMMU
MM- Prefillingg
Methods
Acc.
Speed
Prefilling
Acc.
Speed
Next-token
LLaVA-v115-7B Liu et al.
2024b
34.3
9.1
0.11
30.5
0.53
0.20
1.8
+WWQ-biit Lin al.
2024b
34.8
11.1
0.09
26.5
0.57
0.16
1.6
+Fas\V(==2, Chen +.-MoD-0.3
2025
33.9
11.6
0.09
28.8
0.68
0.17
1.3
35.4
12.5
0.08
29.1
0.76
0.19
1.1
Table 6: Comparison with existing dense and sparse MLLMs on 9 benchmarks. Speed is the average samples per second of GQA, SQA, MMMU, and TextVQA.
Image Question Answering
Benchmark Toolkit
Methods
Param.
Speed
TexVVQA VQA GQA SQA'
POPE MME MMB MMMU MetVet
Dense Model:
1-80B (Laurençon el 2024
65B
60.0
45.2
54.5
InstrucLBLIP (Dat al
2023
14B
50.7
49.5
78.9
1212.8
25.6
VILA (Lin
7B
64.4
79.9
62.3
68.2
85.5
1533.0
68.9
34.9
Qwen-VI. VL
Bai al. 2023b
10B
63.8
78.8
59.3
67.1
1487.6
38.2
4.6
LLaVA-1.5
Liui al
202446
7B
58.2
78.5
62.0
66.8
85.9
1510.7
64.3
34.3
30.5
8,1
LLRVA-HR
Lio 2t al.
21724
7B
67.1
81.9
64.2
67.9
87.6
1554.9
66.8
35.2
31.2
4.7
LLaVA-HR
al.
13B
68.1
82.3
87.8
1540.9
36.3
34.8
Sparse Model:
MoE-LLaVA (Lin al.
2024a
3B
50.1
76.7
60.3
62.6
85.7
1318.2
60.2
26.9
8.5
MoE-LLaVA (Lin al 2024a
5B
51.4
77.6
61.4
68.5
86.3
1423.0
65.2
34.3
5.6
M-Mo--LLaVAtun
7B
56.3
77.6
61.1
64.7
86.0
1342.1
59.4
35.4
29.8
10.3
7- MoD--LLaV (A-Hurs
7B
64.9
80.6
63.1
67.9
87.3
1516.0
63.4
34.7
31.5
7.2
MMD-LLLaVA MAD-LLLVA--R(our))
13B
66.8
82.0
64.8
69.5
86.7
1515.4
65.2
35.8
34.0
4.8
the routing ratio increases to 51%, the performance of LLaVA decreases significantly, suggesting its relatively low tolerance to high routing ratio. For LL- the 7-MoD-0.3 configuration maintains high accuracy 63.7% on GQA and 65.3% on TexVVQA while reducing TFlops by 34% and skipping 37.9% of tokens. When the routing ratio increases 51 %. the token skip rate improves 57.7%, though slight drop in accuracy is observed e.g., -0.6% on GQA. These comparisons also reflect that nigh-resolution MLLMs often have a higher token redundancy than low-resolution ones. Similar observations can also be witnessed on Mini-Gemini-HD. Li et al. (2024b). When scaling to larger models, such as the I.L.VA-HI configuration yields 38.1 5% skip rate and 25.1 T'llps with minimal accuracy loss, suggesting that larger models are better suited to handle higher skip rates while maintaining performance. Even increasing the routing ratio to 51% the competitive accuracy is still maintained, e.g., 64.8% on GQA and 66.8% on TextVQA.
our method continues to perform strongly. The 3-MoD--0.3
compare the training and inference efficiency of 7-MoD LLaVA-HR. From these results, we observe comprehensive advantages of y-Mo terms training and inference inference. In particular, 7-MolD-0.3 already achieves obvious improvement in efficiency, i.e., -26% training time and -35% T'lops. However, the performance drops of -MoD--0. can be almost ignorable, i.e., -0.2% average accuracy. When increasing the routing ratio 50% tokens, the inference throughput of 7-MoD-0.5 further improves by up to +53.2%. Despite the significant efficiency gains, the performance drop of 7-MoD is still acceptable, i.e., -1.5% average accuracy. These results well validate the obvious benefits of M-MoD in efficiency.
Efficiency analysis. In Tab.
Comparison with existing methods. In Tab. 6 we compare MLLMS deployed by y-MoD with both dense and sparse models on benchmarks. From it we can see M-MoD can maintain the competitive performance on all benchmarks, while achieving significant efficiency gains on LLaVA and LLaVA- HR. Specifically, 7-MoD-LLaVA- while outperforming the latter on multiple benchmarks, e.g., +3.0% on MMVet. In addition, compared to existing sparse models, i.e., MoE-LLaVA (Lin et al. 2024a), our approaches also achieve better trade-off between performance and efficiency. In particular, 7 M-MoD-LLaAA
(13B) can reach similar inference speed as LLaVA-HR (7B)
(7B) outperforms MoE-LLaVA (5B) on 5 of 8 benchmarks, e.g., 93 scores MME, while still maintaining better efficiency, Le., +28% gains on inference speed. worth noting that although the parameter scale of MoF-ILLaVA is smaller, its routing calculation often leads to higher latency.

---

Published as a conference paper at ICLR 2025
Routing Results
Routing Resalts
Q: Describe this image. The image features white haized dog standing in the grasses 2 holding
Who is this Give detailed renson.
A: The purson in the image is Eamous historicaa statue. The man is knight, and he is riding. Ho is knight in shining.
16
24
Overview of routing results LLaVA-HI
M-D-Le
MoD-Layer-28
MoD-Layer-1
MoD-Layer-28
MoD-Layer-1
MoD--ay
Q: Whene
Where
What square equal to?
Whot theisqure equal to?
Q: now? Q: Whar
A: You are in the sitt of New York
A: You are the cityiof New York
A: The time now is 11:30 A: The time now 11:30
The aquare equals 89
A: The square equual to
MoD-Layer-
MoD-Layenn228
MoD Layer:28
MoD-Layer-
MoD-Layer-25
Q Which states have most Q: Winchh states hove hoopital per
Which function lins
Which function highest growth rase? A: The highest growts ishown thaired line
Which different each other
Which diffiorent
hospital
highest growth rate?? A: The highist growill
from other blue realistic realistc representarion
Ahehe with mosi
Arele states with the most
The blue
hospital beds Califerns
Califoeniii rateis shown by the
realistic representation
Texas and Flonda
Texas, and lorida
line
a car
car
Illustration of skipped contents in different MoD layers
Figure 4: Visualization of routing results for different MoD layers. "Q", "T" and "A" denote the question, image and response, respectively. The skipped tokens in sub-figure (b) are colored gray.
In Tab. 5, also compare 7 M-MoD with common inference-time acceleration methods Chen et al. (2055 Lin et al. (2024b). Compared to these methods, 7-MoD can better maintain the model performance on MMMU and MMVet. In terms of efficiency. 7-MoD shows greater advantages in accelerating the next-ton eokt-token generation for MLLMs, providing up to +31 % speedups on MM Overall, these comparisons further confirm the effectiveness and efficiency of 7-MoD.
5.3.3 QUALITATIVE ANALYSTS
In Fig. 4 we visualize the routing ratio and the skipped content in both images and the corresponding conversations. The first observation from Fig. (a) is that question, image, and response tokens routed in a consistent pattern: question tokens are mostly kept, while image tokens are the most redundant, and thus routed the most. In Fig. (b). texts. The gray portions of the images represent tokens that are skipped by the router, indicating that many regions in the images, such background pixels, are redundant and do provide critical information for understanding. Routing out these tokens allows the model to focus more on the white portions, which highlight the image regions text parts that the model pays closer attention to. For example, in the middle of the first row with the IQ test example, the model can concentrate and spending more computations on the arithmetic and geometric aspects of the image.
visualize the skipped content images and
CONCLUSION
In this paper. we aim to overcome the efficiency problem in multimodal large language models (MLLMs) from the perspective of "activated token". In particular, we present y-MoD, novel mixture-of-depth adaptation strategy for computationally efficient MLLM. In 7-MoD, an innovative metric introduced to identify the redundant layers for MoD deployment, namely rank of attention maps (ARank). Moreover, 7-MoD also maximizes its benefit to MLLMs via two designs called shared vision-language router and masked routing learning. With these novel designs, MMMoD obviously reduce computational costs of existing MLLMs while maintaining their performance. Extensive experiments on 9 multimodal benchmarks validate the efficiency and effectiveness. Besides, the great generalization ability of M-Mo is also validated across different MIL MI..Ms.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. This work was supported by the National Natural Science Foundation of China (No. 623B2088) and the China Postdoctoral Science Foundation (No. 2024M761548).
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint Stti:22044.422199 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar 0tt.::2333.88777 2023.
Jean- Mann-Batisste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 2022.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojo- caru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311 166.67771111167, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.166000 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar 2338 wtt.::230.//966.66 /2966, 2023b.
Weilin Cai, Juyong Jiang, Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprini 20Xiv:2407.06244, 2024a.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403. 102.00..72900 17297, 2024b.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.094 09.49778 2023.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2025.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao. Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang. Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far the gap to commercial multimodal models with open-source suites, 2024a.
gpt-4v? closing
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic iisual-linguise tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24195 24198, 2024b.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:240I 200i::200000066.. 2024.
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-lane languae models with instruction tuning, 2023.
11

---

Published as a conference paper at ICLR 2025
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint 2xXv:23309.11499 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. A natural logic inference system. In Proceedings of the 2nd workshop on inference in computational semantics (ICoS-2), 2000.
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model, 2023.
G.H. and C.Reinsch. Singular value decomposition and least squares solutions. In Handbook for Automatic Computation: Volume II: Linear Algebra, pp. 134 151. Springer, 1971.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018.
Drew A. Hudson and Christopher D. Manning, Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.04000 2024.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint arXiv:240S.10739, 2024.
Minchul Kim, Shangqian Gao, Ven-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1383-1392, 2024.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. -lava-ooevissio: Easy visual task transfer. arXiv preprint arXiv:2408.08 03326, 2024a.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 188/4, 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large languaee models. 2023.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models, 2024a.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for IIm compression and acceleration, 2024b.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi. and Song Han. Vila: On pre-training for visual language models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 26689-266999 2024c.
12

---

Published as a conference paper at ICLR 2025
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1529-15338 2020.
Aixin Liu, Bei Feng, Bin Wang. Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprini arXiv:2405. 044444 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024d.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang. Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024c.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024.
Brandon McKinzie, Zhe Gan, -han-philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal IIm pre-training, arXiv preprint 2403v:2403.096/1 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv:1606.0603: 2016.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer- based language models. arXiv preprint arXhv:2404.02255. 2024.
Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad S. Khan. Llava++: Extending visual capabilities with Ilama-3 and phi-3, 2024.
Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
Zhiqiang Shen, Tianhua Tao, Liqun Ma. Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-de: Understanding data combinations for 1lm training. arXiv preprint arXiv:2309. 10818, 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14398-14409, 2024.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1 1: A fully open, vision-centric exploration of multimodal lms.. arXiv preprint arXiv:2406. 16860, 2024.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXtv:2302.13971, 2023.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528. 2024.
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun. Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ar Xiv: www::1i99000050000 2019.
14


Reviews:
[{"review_id": "WxjBQlPQJE", "reviewer": "Reviewer_1ATp", "summary": "This paper proposes a novel Mixture-of-Depth (MoD) adaptation method, $\\gamma$-MoD, designed to improve computational efficiency for multimodal large language models (MLLMs). By introducing ARank, a metric to identify redundant layers, $\\gamma$-MoD selectively replaces dense layers with MoD layers to reduce computational costs while maintaining performance. The model is validated across several benchmarks, showing promising efficiency gains.", "questions": "How does $\\gamma$-MoD ensure that ARank’s redundancy metric accurately reflects the importance of layers across diverse multimodal tasks, especially given the unique challenges posed by visual information?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. $\\gamma$-MoD significantly reduces training and inference time by efficiently identifying and replacing redundant layers.\n2. The use of ARank as a layer redundancy metric is novel and provides a structured approach to layer selection in MLLMs.\n3. The shared vision-language router and masked routing learning modules are thoughtfully designed for multimodal alignment, enhancing the model's performance in cross-modal tasks.", "weaknesses": "1. ARank’s accuracy heavily depends on the diversity and representativeness of the input samples, which may lead to inaccurate redundancy assessment, especially in varied multimodal tasks.\n2. ARank’s reliance on low-rank attention maps to identify redundancy may overlook the contextual importance of low-rank layers, which could contain compressed but critical information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pT7HyXELKN", "reviewer": "Reviewer_6GUt", "summary": "This article introduces the Mixture-of-Depth (MoD) scheme into multimodal large language model (MLLM) architectures. By analyzing the rank of attention maps (ARank) of existing MLLMs, the authors identify a model-agnostic and data-agnostic pattern, revealing significant redundancy in certain layers of MLLMs. Building on this insight, the paper proposes the $\\gamma$-MoD method, which (1) incorporates a shared vision-language router into existing MLLMs, transforming them into the MoD structure, and (2) employs masked routing learning to adapt MLLM architectures to the MoD scheme by refining the instruction tuning process used during MLLM training. Extensive experiments on multiple multimodal benchmarks and MLLM models demonstrate that $\\gamma$-MoD effectively reduces both training and inference time, while largely preserving performance on benchmark tasks.", "questions": "Things that WILL affect my overall rating towards this article:\n\n1. Regarding Weakness 1, since the authors consider the disadvantage of MoD to be demanding training from scratch (Line 49-50), the authors should demonstrate how to apply $\\gamma$-MoD to existing published MLLM checkpoints, rather than pre-training one from scratch like in the experiment section.\n2. Regarding Weakness 2, I would like to see more experiments on other MLLM models beside the LLaVA series. Considering the time of rebuttal and your reported time of training in Table 4, I would like to see this point resolved.\n3. Regarding Weakness 3:\n   - What is the applied value of the pre-defined sparse target in Equation 4? What is the relationship between the sparse target $\\alpha$ and the \"routing ratio\" and the \"skip ratio\" in the experiment section?\n   - Normally, the sparse target factor $\\alpha$ in Equation 4 satisfies $\\alpha\\in[0, 1]$ , but $\\alpha$ is actually the sum of multiple indicator functions with values $\\\\{0,1\\\\}$. The left side (value of the sum of indicator function) should be normalized.\n   - In Equation 8, the authors put one-hot vectors $\\hat{R}$ and $1-\\hat{R}$ inside $\\log$. This obviously doesn't seem right. Moreover, it is hard to understand what does the routing objective (Equation 8) do in $\\gamma$-MoD training. The authors should provide a more detailed explanations of that learning objective.\n4. Regarding Weakness 4: Since both works lead to inference acceleration of MLLMs, the authors should provide a detailed comparison with [1] and demonstrate the exclusive advantage of $\\gamma$-MoD.\n\nThings that WILL NOT affect my overall rating towards this article:\n\n1. Figure 2 indicates that transformer layers with $\\text{Rank}<110$ would be considered as redundant layers. The authors could provide a more detailed analysis on the rank distributions of tested models, instead of giving a figure without quantitative statistics like Figure 3.\n\n[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (ECCV 2024)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Originality: I appreciate the authors' effort in implementing the MoD scheme in MLLMs in a plug-and-play manner. In my view, the significance of this work lies in maintaining the pre-training stage unchanged, which contrasts with most approaches in the field of sparse LLMs/MLLMs (e.g., MoE structures) that focus on pre-training models from scratch. While there are some aspects that warrant further consideration (see weaknesses & questions), this approach could inspire future research into converting existing models into sparse structures in an off-the-shelf manner.\n2. Clarity: This article is mostly well-written. The motivation is clearly articulated and the solutions towards solving the existing problems are logically sound.\n3. Experiments: This article conducts extensive experiments across multiple MLLM structures and benchmark datasets to demonstrate the generalizability of $\\gamma$-MoD. Their results strongly suggest that $\\gamma$-MoD successfully reduces training and inference cost on MLLM models. Abundant analysis experiments also demonstrated that $\\gamma$-MoD leads MLLMs to focus on informative tokens and skip the redundant ones.", "weaknesses": "1. Method designs: In the introduction (Line 87), this article suggests that $\\gamma$-MoD is a *plug-and-play* method that can be integrated to current MLLMs. Existing MLLMs follow a 2-stage pre-training and instruction tuning training pipeline, and $\\gamma$-MoD modifies the instruction tuning pipeline to convert LLaVA models in $\\gamma$-MoD-LLaVA structure. In my opinion, this design is arguably *plug-and-play*, since open-source MLLM weights are mostly instruction tuned, so that $\\gamma$-MoD cannot be directly applied on published MLLM checkpoints. To evaluate the validity of $\\gamma$-MoD, the authors also pre-trains MLLMs on the LCS-558K dataset first. This hinders the applicability of $\\gamma$-MoD, as training the $\\gamma$-MoD model also requires conducting the complete pre-training pipeline.\n2. Further generalizability of $\\gamma$-MoD: The experiments in this article primarily focus on LLaVA structures, likely due to the availability of open-source pre-training and instruction tuning datasets, which are necessary for $\\gamma$-MoD as it modifies the instruction tuning process. However, as a general method for introducing the MoD scheme to MLLM models, additional experiments on other MLLMs are needed to further demonstrate the generalizability of $\\gamma$-MoD.\n3. Some ambiguities in the methodology: Key information are missing from the article & Some statements are quite confusing. See Questions for more details.\n4. Significance when compared to existing works: In the experiments section, this article finds that most image tokens can be skipped in MLLM inference. This discovery corresponds with [1], in which image tokens are simply pruned from off-the-shelf MLLMs without further training. experiment results in [1] indicates that this simple procedure reduces the FLOPs in MLLMs by over 50% without introducing performance drops, similar to that reported in $\\gamma$-MoD. If so, pre-train MLLMs from scratch and modify the instruction tuning methods would be too costly for accelerating an MLLM. \n\n[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (ECCV 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tqXCyO5bF6", "reviewer": "Reviewer_wmRq", "summary": "This paper proposes γ-MoD, a novel mixture-of-depth (MoD) adaptation strategy for existing multimodal large language models (MLLMs). The goal is to maximize computational sparsity while maintaining competitive performance. γ-MoD achieves this by identifying redundant layers using the rank of attention maps (ARank) metric and replacing them with MoD layers. The Key contributions of this paper include: **1.** γ-MoD Framework: A novel MoD framework for sparse computation of existing MLLMs, seamlessly converting most dense layers to sparse MoD layers. **2.** ARank Metric: An metric to measure layer redundancy, guiding the selection of layers for MoD adaptation. **3.** Shared Vision-Language Router and Masked Routing Learning: Designs to maximize the benefits of MoDs in MLLMs, achieving up to 51.6% computational sparsity with minimal performance sacrifice.", "questions": "1. In this paper, the binary mask Mq assigned the question tokens to 0. Therefore, during the training process, the value of Mq should be fixed. But for different tasks or datasets, the location of the question tokens may be different. How did the author determine and design Mq on different tasks or datasets?\n2. I would like to know how the \"All layer\" in Table 1 is specifically implemented.\n3. How sensitive is ARank to the choice of hyperparameters, such as the batch used for estimation?\n4. What are the criteria for determining the threshold for converting dense layers to MoD layers based on ARank values? Is there a more principled way to set this threshold?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. While MoD has been explored in LLMs, this is the first work to adapt them to MLLMs, which face unique challenges due to the combination of visual and textual modalities.\n2. This paper proposes a metric, rank of attention maps (ARank), to identify redundant layers in MLLMs, addressing the lack of guidance in existing MoD approaches.\n3. The paper introduces two designs to maximize the benefits of MoD for MLLMs, The Masked Routing Learning prevents critical tokens, such as question subjects, from being skipped during training, preserving the model’s ability to generate accurate answers.\n4. This paper provides comprehensive theoretical analysis, empirical validations. And it is well-structured and clearly explains the motivations, methodology, and experimental results.", "weaknesses": "1. The metric ARank seems to be a repurposed version of HRank [1] specifically for attention maps. This connection should be discussed in detail.\n2. The ablation study of the routing mechanisms is not enough. This paper only explores a binary routing mechanism (skip or not skip). Exploring more sophisticated routing strategies, such as weighted routing or adaptive routing thresholds and so on, could make the article more comprehensive.\n3. While the paper compares γ-MoD with dense and MoE-based MLLMs, it lacks a comparison with other sparsity techniques like pruning or quantization.\n4. The author should consider rearranging Table 5 or highlighting the optimal results, as the current version is somewhat challenging to compare the results of different methods\n\n[1] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1529–1538, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)