Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models
Authors: Qiong Wu
Zhaoxi Ke
Yiyi Zhou
Xiaoshuai Sun
Rongrong Ji
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal large language model
dynamic routing
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recently, mixture of experts (MoE) has become a popular paradigm for achieving the trade-off between modal capacity and efficiency of multimodal large language models (MLLMs). Different from previous efforts, we are dedicated to exploring the dynamic experts in existing MLLMs and showing that a standard MLLM can also be a mixture of experts. However, achieving this target is still notoriously challenging. The well-trained MLLMs are more accustomed to the fixed pathway and a drastic change in its inference manner also greatly impedes its performance. To address these issues, we propose a novel dynamic expert routing method for existing MLLMs, termed Routing Experts (RoE), which can achieve example-dependent optimal path routing without obvious structure tweaks. Meanwhile, a new structure sparsity regularization is also introduced to force the well-trained MLLMs to learn more short-cut pathways. In addition, we also address the alignment of the training and inference of MLLMs in terms of network routing. To validate RoE, we apply it to a set of existing MLLMs, including LLaVA-1.5, LLaVA-HR and VILA, and conduct extensive experiments on a bunch of VL benchmarks. The experiment results not only show the effectiveness of our RoE in improving MLLMs' efficiency, but also yield obvious advantages over MoE-LLaVA in both performance and speed, e.g.,  an average performance gain of 3.3% on 5 benchmarks while being 1.61 times faster. Our code is anonymously released at https://github.com/DoubtedSteam/RoE

Published as a conference paper at ICLR 2025
ROUTING EXPERTS: LEARNING TO ROUTE DYNAMIC EXPERTS IN EXISTING MULTI-MODAL LARGE LAN-
GUAGE MODELS
Qiong Wu Zhaoxi ke Yiyi Xiaoshuai Sun Rongrong Ji
Key Laboratory of Multimedia Trusted Perception and Efficient Computing,
Ministry of Education of China, Xiamen University, 361005, P.R. China.
2 Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China.
{qiong, kezhaoxi 8 xmu wmu.edu.com (zhouyiyi, xssun, rrji}0xmi edu en,
ABSTRACT
Recently, mixture of experts (MoE) has become popular paradigm for achieving the trade-off between modal capacity and efficiency of multimodal large language models (MLLMs). Different from previous efforts, are dedicated exploring the dynamic experts in existing MLLMs and showing that standard MLLM can also be mixture of experts. However, achieving this target is still notoriously challenging. The well-trained MILMs are more accustomed to the fixed pathway and drastic change in its inference manner also greatly impedes its performance. To address these issues, we propose a novel dynamic expert routing method for existing MI.LMs, termed Routing Experts (RoE), which can achieve example- dependent optimal path routing without obvious structure tweaks. Meanwhile, a new structure sparsity regularization is also introduced to force the well-trained MLLMs to learn more short-cut pathways. In addition, we also address the align- ment of the training and inference of MLLMs terms network routing. To validate RoE, apply it to set of existing MLLMs, including LLaVA-1155 LLaVA-HR and VILA, and conduct extensive experiments on a bunch of VL benchmarks. The experiment results only show the effectiveness of our RoE in improving MILMs* efficiency, but also yield obvious advantages over MoE- LLaVA both performance and speed, e.g., an average performance gain of 3.3% on 5 benchmarks while being 1.61 times faster. Our code is anonymously released at htttts: com/Dobbtedtteann
INTRODUCTION
Recently, the great success of large language models (LLMs) Radford al. (2018); Zhang et al.
(2022); ; Bai et al. (2023a); Touvron et al. (2023) attracts an influx of interest in extending them to
more modalities, e.g., vision and language (VL) Jiang et (2020) Luo al. (202bb); Tong et
2024) Zhou et al. 2019). Despite great progress. multi-modal large language models (MLLMs) Li
al. (2023b); Dai et al. (2023); Rose al. (2023); Wang et al. (2024); Koh et al. (2024) also suffer
from excessive computation due to the introduction of more modality tokens. For instance, LLaVA
Liu et al. (2023b) requires 6.15 times more computation than unimodal inference on ScienceQA
Lu et al. (2022). Inspired by the progress of LLMs Radford et al.) (0118;; Zhang et al.
2022
Touvron et al. (2023), recent efforts Bai et al. 2023a); Ainslie et al. (2023 Jiang et al.
2024);
Raposo al. (024) also devoted exploring
MLLMs with Mixture Mexture-of-Expee Experts (MoE)
structure, thereby archiving a good trade-off between model capacity and inference efficiency.
Different from these pioneers Gou al. (2023); Shen et al. (2023); Lin et al. (2024). focus exploring the dynamic experts in MILMs that already exist and show that a well-trained common MILM can also be mixture of experts. The motivation is akin MoE, that is, LLMs MLLMs need enough parameter capacity to meet scaling law Kaplan et al. (2020), but is evident that the entire model is often redundant for specific tasks, especially the easy ones Wu al. (2024). For
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
Image:
Image:
Qestion:
24
24
Q Question:
W is the farmer doing the image?
Is the dog walting? Aswwer the question using single word pirase.
30
1.2
0.8
04
00:00
0.
68
L1- Distaancc
L1-Distance
Figure 1: The visualization of the 11 11-distances between the input and output features of each layer of LLaVA-7B Liu et al. (2023b). A lower d1-distance] indicates that this layer has less impact on the feature update of this example, which also suggests that it is not that important during inference. For two examples, the contributions of different layers are also different.
instance, the advanced MILMS like LLaVA-1.5 Liu al. (023a) exhibit much stronger generaliza- tion capability than previous
(VL) models Li et al. (2019); ; Kim et al. (2021) ; Dou
al. (2022); Gao al. (2023), but still on par with the bespoke ones Lu et al. (2019); Kim et al.
2021
with much smaller parameter sizes on the benchmarks like VQAv2 Goyal et al. (2017).
However, in terms of methodology, we are keen to explore the dynamic and sparse structures of MILMS that already exist, rather than building a new sparse model like previous MoE methods Shen et al. (2023); Lin et al. (2024). We observe that the activations MLLMs' different layers for the examples are distinct. As shown in Fig. some layers barely contribute to the transformation and reasoning of a given example. This finding implies that the inherent knowledge of common MLLMs is likely to be distributed as in MoE models Eigen et al. (2013). indicating the feasibility of routing the expert subnetworks in an already existing MLLM.
However, achieving this target is still intractable. In particular, aim to adaptively skip the less important layers of MILMS for each example, thereby achieving better efficiency, as shown in Fig. Although intuitive, this attempt at MLLMs still suffers from several challenges. The first one is the feature gap that occurs in dynamic inference. Unlike previous dynamic models [in et al. (2024); Jiang al. (2024); Sun al. (2024); Shen al. (2024); Luo al. (2024b), which mostly trained from scratch and well accommodate dynamic inference, this layer-wihe wise skipping will make MLLMs encounter drastic change in feature space during inference, greatly limiting its performance upper-bound. Meanwhile, how to make MILMS choose a short-cut pathway is also difficult. Since MI.LMMs are already end-to-end well trained, they usually prefer not skip under the default training objectives. In addition, existing MLLMss Dai et al. (2023); [ [ et al. (2023a) Lin ctt al. (2023); Liu et al. (2023b); Zhou et al. (2024) often organize multiple examples as a multi-turn conversation for efficient training, which however contradicts the dynamic routing of each example. Overall, these ingredients greatly hinder the achievement of dynamic routing in existing MLLMs.
To address these issues, we propose an innovative dynamic routing method for MLLMs, termed Routing Experts (RoE). RoE regards each layer of MLLMs as an independent expert, and its ob- jective is find out and connect the important In practice, RoE uses path routers to decide whether to skip layers. To compensate the issue of the feature gap, we introduce the adapter-based connections to replace the less important layers, which are easy to deploy and can well serve feature transformations Wu et al. (2024). To optimize RoE, we also propose a novel sparsity regularization to encourage the learning of sparse and diverse pathway routing. Combined with this objective, the simple yet effective routing tokens are further proposed facilitate the optimization of RoE in multi-turn conversations, addressing the issue of misalign- ment between training and inference. With these innovative designs, RoE shows that standard and well-trained MI.LM can also be a mixture of experts.
as an optimal routing path for each example.
To validate RoE, we apply it to a set of advanced MILMs, including LLaVA- 1.5 Liu et al. (2023a), LLaVA-HR Luo et al. (2024c) and VILA Lin et al. (2023), on 10 competitive VL benchmarks, including VQA2.0 Goyal et al. (001)), GQA Hudson & Manning (2019), TextVQA Singh et al. 2019), POPE Li et al. (2023c). MME Fu et al. (2023), and MM- Vet Yu et al. (2023). The experi- mental results show that our RoE can greatly speed up the inference of common MLLMs, while still maintaining their competitive performance on various benchmarks. For instance, our RoE improves the inference speed of VA-155 by 21.3% without performance drops most benchmarks. Com-
2

---

Published as a conference paper at ICLR 2025
pared with the previous MoE-based method, e.g., MoB-LLaVA Lin et al. (2024). RoE not only has better performance on all benchmarks, but also exhibits faster inference speed, egg., 6.77 v.s. 4.95 examples per second for traditional VL benchmarks average.
Overall, our contributions are three-fold:
We present a novel attempt of dynamic routing in existing MILMMs termed Routing Experts (RoE), which aims to transform existing MLLMs into a mixture of experts without obvious structure tweaks.
To achieve effective and efficient RoE for the well-trained MLLMs, we introduce adapter- based skip connection to alleviate the feature gap problem and novel sparsity regulariza- tion to help MLLMs learn dynamic and sparse inference. Besides, the routing token design also aligns the training and inference of RoE-MLLMs.
On ten highly-co benchmarks, RoE can significantly improve the efficiency of three advanced MLLMs while retaining similar or even better performance.
2 RELATED WORK
2.1 MIXTURE-OP-EXPERT
Mixture-of- Mixture-of-Experts Experts (MoF) Figen et al. (2013): Jain et al. (2024); [Zhao et al. (2024) is dynamic and sparse paradigm that can achieve good trade-off between model capability and efficiency. Its main property is that MoE models can dynamically select the most appropriate experts from several can- didates for different inputs, thereby improving model efficiency. In terms of methodology, existing MoE models can be categorized into the soft and the hard ones, respectively. In soft MoEj Lepikhin et al. (220)); Fedus et al. (2022); Lin et al. (2024). the model output is weighted aggregation of the experts with high confidence. For example, MoB-LLLVA Lin et al. (2024) combines outputs from multiple/ feedforward networks (FFNs) enhance the model capabilities. Mistral-MoF] Diang al.) (2024) uses the outputs of top-two experts for different examples. Although effective, soft MoE is often hard to achieve real speed acceleration as expected, since the inference of all experts needs to be computed. In contrast, hard MoE models Kudugunta et al. (2021 Bao et al. (2022); Zhu al. (2022); Satar et al.] (2022): [Wagg et al. (2022b); Shen et al. (2023): Li et (2023d); Ma et al. [2023 ; Long et al. (2023) dynamically activate the experts, introducing less additional calculation overhead. For instance, VLMo Bao et al. (2022) and VL-MoE (Shen et al. (2023) activate the expert with the highest confidence. LaCE et al. (2023d) activates experts according to the predefined token given by the input. Although MoE can select appropriate experts to deal with different inputs, it still uses the same complexity for tasks of different difficulties. In the latest developments, some methods Ainslie et al. 2003): Dotzel et al. (2024); Jaiswal et al. (2024); |Raposo et al. (2024); Luo et al. (2024a) introduce experts with different computational overhead to improve the efficiency- For instance, CoLTS Ainslic et al. (2023) proposes a heavy and light option for each module in transformer layer. DLO'Zhen et (0024) expands the Transformer layers vertically and selectively activates some ones from them. However, these MoE models often need to re-design the network structure and train the model from scratch, lacking the effective use of existing MLLMs. Recently, some works introduce dynamic inference. SkipBERT Wang et al. (2022a) improves shallow layers' efficiency in BERT by combining words through n-grams. SmartBERT layer-skipping and early-exit using the [cls] token. MoD Raposo et al. (024)) focuses tokens to take |
al. (2023) combines
shortcuts according to a certain ratio in each layer. Orthogonal to these works, we focus on exploring the inherent expert structure in MLLMs that already exist.
2.2 MULTI-MODAL LARGE LANGUAGE MODEL
Driven by the success of large language models (LLMs) Radford et al. (2018): Touvron et al. (2023): Chen et al. (002c)); Bai et al. (2023)); Jiang et al. (2024); Zhang et al. (2024). the research of
multimodal large language models (MLL MMs) Alayrac et al. (2022) Zhu et al. 2023); Bai et al.
2023b); Chen et al. 22023a b); Li et al. (023b) Liu et al. (2023b c:: Peng et al. 2023): Luo et al.
2023a 2024c); Zhou et al. (2024); Zhu et al. (2024) also gains increasing attention recently. The main paradigm of MLLMs is to directly connect the visual encoder and LLM with an additional network. For instance, BLIP-2 2 Li et al. (202hb) introduces QFormer to bridge the gap between
3

---

Published as a conference paper at ICLR 2025
FFN
FFN
Adapter
FFN
MHA
Adapter
FEN
Adapter
FFN
Adapter
FFN
MHA
MHA
MHA
MHA
FFN
FEN
Adapter
FFN
Reaning
Reuning Tokens
MHA
MHA
Image
Routing
Image Tokeus
Routing
4
Tokens
Tokens
Rouring
0, Takena
Tokens
(a) Mixture of Experts
Routing Expert
(c) The alignment Training and Testing with Routing Tokens
Figure 2: Illustration of the proposed Routing Experts (RoE). Existing MoE models (a) often build new sparse structure with multiple FFNs as experts, and each pathway takes the same computation for all examples. Our RoF (b) aims to explore the expert pathways within the model itself via adapter-based skip connections, realizing dynamic computation for different examples. (c) Routing tokens used to decide layer-wise path selection, i.e., the adapter-basee based skip connection or the default Transformer layer. It also serves to align the training and testing of MLLMs.
vision and language modalities, integrating visual tokens into LLMs.. Similarly, MINI-GPT4 Zhu eet al. (2023) uses projection layer map visual features into the semantic space of the L.M. LLaVA Liu et al. (2023b) shares the same paradigm with MINI-GPT, and also proposes carefully designed training strategy. In terms of network design, MLLMs often use stack of Transformer decoding layers Bai et al. (2023a); Touvron et al. (2023) for multi-modal inference following LLMs. However, with the introduction of visual tokens, the already high computation of this dense structure is further exacerbated. To address this issue, recent MILMs like MoE-LLaVa Lin et al. (2024) resort to sparse and dynamic design of MILMs. However, as mentioned, computation of multiple paralleled experts still limits the efficiency improvement. Different from these efforts, we aim to explore the dynamic inference of MLLMs to improve efficiency while retaining performance.
PRELIMINARY
In this section, first recap the principle of Mixture of Experts (MoE) for MLLMs. As shown in Fig. (aa), existing MoE-MLLMM like MoE--LLVAA Lin et al. (2024) often build multiple FFNs as the experts of each layer. During inference, only this case, an MoE layer in MLLM G(.) can be defined by
several experts activated in each layer. In
K
x = = + Σ Gx)(x,) Rey (x:),
(I)
1=1
where G-y(-) denotes the j-th expert in the i-th layer, and (-) are routing weights predicted by the router Rt X; Rnx represents the inputs for i-th layer, where n and d denote the length and dimension. In practice, MoE models only activate subset of experts according to the input features. Despite effectiveness, each inference still has the same computation for all tasks, which is also redundant for some examples.
Like LLMssAinslic et al. (2023); Raposo et al. (2024), existing MLLMs are also obviously redundant in many cases. As aforementioned, not all layers equally contribute to the final prediction. In this case, we focus on exploring the dynamic experts in MILMSs that already exist.
Concretely, we regard each layer of an MLLM as expert, and skip the less important ones to form routing path G(rr)::
G'= G1 G2 Gn,
where G' G'(x) is the activated subnetwork, and (G1, G2, G.., G,) layers chosen by the router. Its number is smaller than the default length n.
However, the absence of some layers in Eq inevitably impedes the feature transformation during inference, especially for the well-trained MLLMs. This issue also makes MLLMs prefer not to skip layers during the training of dynamic routing.
4

---

Published as a conference paper at ICLR 2025
4 ROUTING EXPERTS
4.1 METHOD
In this paper, propose innovative Routing Experts (RoE) scheme for the dynamic routing in the existing MI.LMs. The objective RoB is
argmin [(G(T, 7[(0'))) |0%,
(3)
where 0' C 0 is subset of MILMM, and O0| represents the activated parameters.
As discussed above, a direct skipping is prone to hindering feature transformation, i.e. the feature gap between layers. In this case, we introduce adapter-based skip connection for MLLMs, and the dynamic expert pathway G(æ))' is obtained by
=²= M1 M2 o.... o Mn:
G. (x)u > (2(x,)1,
(4)
where
M. =
A,, Rx,)) R((x))1-
Here, M, is the expert of the i-th layer, A; is lightweight adapter[Sung al. (0022). R, is binary routing function to decide whether the i-th layer need to be skipped:
R(x;) Softmax(r; W,),
(5)
where r; E R R1
xi is router token and will be introduced Sec 4.3
In terms of the adpater A,(:), its low-rank network be defined by
x' = ReI.U(xWa)) W..
(6)
where Wa E paxe and W. E Rexd are two weight matrices, and <<<<> d.
Compared with Eq Eq introduces the use of adapter-based skip connection. Although the adapter involves some computation, but it is much more lightweight than standard MLLM layer. More importantly, has been proven be capable of feature adaption for large models terms of
parameter efficient tuning Sung et al. (2022); Wu et al. 2024).
4.2 STRUCTURE SPARSITY REGULARIZATION
Although RoE can cope with the issue of feature gap, the MLLM is still likely to use the entire network to infer examples during and after training, as discussed above.
In this case, we introduce a structure sparsity regularization to facilitate the training of RoE:
C. = mt Σ R, (x.),,),),
where n denotes the number of layers, and predefined rate of skipped layers. This regularization term can force the model to meet the target skip rate during training.
However, 7 does not consider the varying difficulties of examples. Intuitively, the more difficult the example the more complex inference is required. Thus, we weight the regularization based on the tuning loss of each example. RoE uses the prediction loss value as an indicator difficulty, and combines with the overall optimization objective, defined by
-= Li + -C:
where L, is the default objective, and | demotes the loss value for the example. And is hyper-parameter to balance the performance and efficiency. With Eq. 8 the routing for the simpler examples will be optimized more by the sparsity regularization.
Although Eq. g is well designed, its actual effectiveness is still greatly limited by the training scheme of existing MLLMs. To explain, existing MLLMs Liu et al. (2023a); [in et al. (2023); Luo et al. 2024c) often combine multiple VL examples in one multi-turn conversation as a single example, thereby speeding up training. Thus, this type of training examples will share a common routing strategy for parallel computation under the default setting. However, each conversation in the training example should encouraged to learn an unique routing path to maximize the benefits of our sparsity regularization. To address this issue, further introduce the design of routing token.
5

---

Published as a conference paper at ICLR 2025
4.3 ROUTING TOKEN AND THE ALIGNMENT OF TRAINING AND TESTING
As discussed above, recent MLLMs like LLavA Liu et al. (023a) often combine multiple VL examples one multi-turn conversation. During training, the answers of all examples are predicted and optimized in parallel, which poses a practical issue for the training of dynamic routing, ie., how train and optimize RoE for all examples at the same time?
To overcome this problem, we introduce the design of routing token for RoE-MLLMs. In practice, insert the routing tokens r; into the input sequence of each example:
X; [r I, (
9 A
(Q, A (8)) is the question-answer pair, where where I Ro represent the visual tokens.
Q.!h (k) E Rm!" xd and A, (M) E Rn(" xd are the question and answer tokens. The inserted routing to- kens are learnable vectors that can aggregate information from the corresponding question Q And
(0) is the router token for the image. Then, the routing for j-th question-ana answer pair predicted by
(x,), = Softmax(-Ir) (1r ( w,,j > 0,
(10)
where [ denotes concatenation, W, R2ax22 is a weight matrix, and is the temperature that decreases as training progresses. For the image sequence, its routing weights are computed by
1 QT
R,(x;) (0) Softmax
Σ
1r 1 Jw,,,
(11)
where q is the number of questions.
This design allows each question to engage in the prediction of its specific expert pathway while maintaining training efficiency. In this case, we can align the training and inference of MLLMs for dynamic routing.
4.4 THE TRAINING SCHEME RoE
In this paper, we also carefully design the training scheme of RoE consisting of three main stages,
Stage 1: Adapter warmup. We first optimize the adapter-based skip connections, thereby making them be capable of feature transformation. In particular, we will randomly select the default layers and some adapter connections as the expert network according to a predefined sparsity target. To reduce the difficulty of optimization, we will freeze the entire MLLMs and only update the adapters.
Stage 2: Router warmup. When the adapter connections are well trained, we begin optimize the routers for path routing. At this stage, the MLLM is still frozen, and both adapters and routers are trained. Meanwhile, the structure sparsity regularization of RoE begins to be used during training.
Stage 3:
Instruction tuning. Lastly, we updated the entire RoE and MLLM for the instruction tuning, and the training objectives include the sparsity regularization and the default ones.
Although RoE involves three training stages, the actual expenditure is cost-effective. Above all, RoE does not require the expensive VL alignment pertaining. Meanwhile, the learnable parameter spaces of the adapters and routers are small, so they can be quickly optimized by few SFT steps. Similarly, the final instruction tuning can also converge quickly since the MLLM is already well- trained. For instance, under the setting of LLaVA Liu et al. (2023a), the full training process of RoE only takes about half the number of default SFT tuning steps.
5 EXPERIMENT
DATASETS AND METRICS
The benchmarks used in this paper are five common vision-language and five recently proposed MLLM benchmarks. The common VL benchmarks include VQAv2 et al. (2017). GQA Hud-
son & Manning 2019) ScienceQA Lue (022). VizWiz Gurari al. (0018) and TextVQA Singh

---

Published as a conference paper at ICLR 2025
Table I: Results of RoE with different skip rates on three MLLMs. "Acc.", "Speed" and "Skip"
denote accuracy, samples per second and the actual skip rate, respectively.
Method
SQA!
GQA
MMB
SEED
Average
Acc.
Speed
Skip
Acc.
Speed
Skip
Acc.
Speed
Skip
Acc.
Speed
Skip
Acc.
Speed
Skip
LLaVA Liu at 2023a
66688 7.55 0.00% 62.0
6,99
0.00%
|64.3 8.37
0.0018
|88.6
8.33
0.00%
62.9
7.81
0.00/
RoF--LLaVAAox
68.4 7.65
10.26%
61,4
7.07
4.59%
64.3
9.62
20.55%
58.2
8.41
9.04%
63,5
8.19
7.77%
RoF. L...a...
68.7
9.15
20.55%%
61.3
7.52
7.86%
64.6
9.88
23.64%
57.8
9.85
24.52%
63.1
9.10
19.15%
RoF-LLaVAgox
68.4
9.67
23.03%
61.4
7.65
8.81%
64.8
10.14
28.94%
58.2
10.43
30.43%
63.1
9.47
22.80%
VII.A[Linet
668.2 8:27
0.00%
62.3
0.00%%
668.9
0.00%
8..36
8.36
0.00%
66.1
0.00%
69.5
8.39
9.19%
62.2
8.01
4.83%
66..6
8.63
10.59%
61.3
8.50
11.41%
65,2
8.38
11.94%
RoE-VILAgos
68.4
10.49
23.93%
61.1
8.20
12.02%
67.8
10.37
19.57%
61.2
9.85
22. 24.34%
64.6
9.73
19.45%
VILAgss
69.4
10.67
25.12%
60.3
8.21
13.41%
66.8
11.66
27.56%
60.2
10.73
27.661
64.2
10.32
23.44%
LLaVA-HR et 2024g]
65..1
4.82
0.00%
64.2
0.00%
644.9
4.76
0.00%
64.22
3.74
0.00%
64.6
0.00%
67.4
4.96
7.96%
62.5
5.01
7.65%
64.6
4.82
6.96%
62.2
3,86
8.43%
64.2
4.66
7.68%:
RoE-LLaVA-HRior
56.1
4.97
12.77%
60.8
5.09
11.07%
52.9
4.89
10.63%
58.8
3.92
13.62%
57.2
4.72
12.02%
Table 2: Ablation study of RoE. "Acc.", "Speed" and "Skip" denote accuracy, samples per second and actual skip rate, respectively. Here, "+Regzo% refers to the use of the sparse regularization.
Method
SQA!
GQA
MMB
SEED
Average
Acc.
Speed
Skip
Acc.
Speed
Skip
Acc.
Speed
Skip
Acc.
Speed
Skip
Acc.
Speed
Skip
ILaVA
66.8
7.55
0.00%
62.0
6.99
0.00%
64.3
8.37
0.00%
|88.6
8.33
0.00%
662.9
7.81
0.00%
Router
69.0 7.37
3.23%
61.2
6.29
0.01%
65.5
7.64
0.03%
58.4
7.67
1.13%
63,5
7.24
1.10%
Regav/
64.3
8.63
15.48%
59.6
7.57
7.00%
63.8
9.32
17.89%
56.6
9.18
18.49%
61.1
8.59
14.72%
Adapter
68.7
9.15
20.55%
61.3
7.52
7.86%
64.6
9.88
23.64%
57.8
9.85
24.52%
63.1
9.10
19.15%
Table 3: The training costs of RoE on three MLLMs. "Adapter" and "Router" denote the adapter warmup and router warmup of RoE. We use the GPU hour of A800 to measure the training time (Time). The values of "Data" are the number of examples for training. The base MLLMs only involve pretraining and SFT tuning ("Finetune").
Pretraining
Adapter
Router
Finetune
Total
Method
Time
Data
Time
Data
Time
Data
Time
Data
Time
Data
VILA
6326.5
50M
0.0
0
0.0
0
120.9
1M
6447.4 93.6
51M
ROE-VILA
0.0
0
25.3
100k
18.7
67k
49.6
166k
333k
LLaVA
55.4
558k
0.0
0.0
0
82.6
665k
138.0
1.2M
RoE-LLaVA
0.0
0
25.3
100k
18.7
67k
49.6
166k
93.6
333k
LLaVA-HR
37.2
558k
0.0
0
0.0
0
128.4
665k
165.6
1.2M
RoE-LLaVA-HR
0.0
39.1
100k
29.0
67k
76.7
166k
144.8
333k
et al. (2019). During testing, we use the data splits organized in the instruction format of LLa Liu et al. (2023a). And report the accuracy of these datasets. The MLL.Mspeccffs benchmarks are POPE Li et al. (2023c), MME Fu et al. (202)), MMB Liu et al. (023d). SEED Li et al. (002a)) and MM- Vet Yu et al. (2023). Compared common VL evaluations, these benchmarks aim to evaluate MLLMs from various aspects, such as fine-grained reasoning and visual hallucination.
5.2 IMPLEMENTATION DETAILS
We apply RoE to three popular MLLMs, namely LLaVA-1.5 Liu et al. (0223)). LLaVA-HR Luo et al. (2024c) and VILA Lin al. (003),, and term the new models as RoE-LLLVA, RoE-LLaVA--HR and RoE- VILA, respectively. For all MILMs, the hidden dimension of RoE's adapter connections is set to 1,024. The randomly sample 15%, 10% and 25% of the 665k instruction data of LLa Liu et al. (2023a) for our three-stage training, respectively. During training, MLLMs are optimized with a learning rate of 2 x 10-6 while the routers and adapter connections are updated with a learning rate of 4 x 10" The training epoch is set to 1, and early stop is applied. The rest settings are kept the same with the original Ms..Ms. More details can refer code project.
a is set to 0.5 to control the impact of sparsity regularization. We
7

---

Published as a conference paper at ICLR 2025
Table 4: Comparison with existing MLLMS on MLLM Benchmarks. "Param.", "Res.", "Acc." and "Speed" denote parameter scale, input image resolution, accuracy and sample per second, respectively. The best and second best results are marked in bold and underline, respectively.
Method
LLM
Param Res.
Acc. Speed
POPE
MME Score Speed
Acc. Speod
MMB
SEED Acc. Speed
Score Speed
Met
Dense MLLMs
Qwen Bai
Qwen-7B
33.22
7,40
56.3
2,42
Qwen-Vi VL VL-Chat Chat Bai of
2023b
Qwen-7B
9.6B
448
1487.5
3.96
60.6
7.55
58.2
2.59
ILaVA Tau
Vicuna-7IB
7.2B
85.9
1510.7
8.37
58.6
8.33
VILA Lin et 2033
Vicuna-7B
7.2B
336
85,5
9.21
1533.0
8.64
68.9
8.51
61.1
8.36
34.9
0.48
LLAVA-HR cr 2024g
Vicuna-7B
7.4B
1024
85.9
4.70
1554.9
4.77
64.9
4.48
64.2
3.46
31.2
0.76
Sparse MLLMs
MoE-LLLaVA-I663:
et 2024
Stable1. L6B
2.9B
336
85.7
7.65
1318.2
8.06
660.2
9.90
26.9
0.43
M.F.L..VA.2..BBx
Phi-2.7B
5.3B
336
86.3
5.95
1423,0
5.83
65.2
5.27
34.3
0.25
RoF-LLaVA
Vicuna-7B 7.3B
336
86.1
9.38
1522.7
9.03
64.3
9.62
58.2
8.41
31.9
0.42
RoE-VILA
Vicuna 7B
7.3B
336
86.8
9.25
1446.0
8.95
67.6
8.63
61.3
8.50
36.7
0.43
RoF-LLaVA-HR
Vicuna-7B 7.5B
1024
88.1
4.75
1558.2
4.82
64.6
4.82
62.2
3.86
30.0
0.68
Table 5: Comparison with existing MLLMs on 5 traditional VL benchmarks. "Param.", "Res.". "Ace." " and "Speed" denote parameter seale, input image resolution, accuracy and sample per second, respectively. The best and second best results are marked in bold and underline, respectively.
Method
Param,
VQA Acc. Speed
Acc. Speed
GQA
VizWiz Ace. Speed Ace. Speed Ace. Speed
SQA!
VQA
Ace. Speed
Average
Dense MLLLMy
Owen-VL. Bai 20236
Qwen-7B
9.6B
5,23
159.3
3.48
3,92
66..1
6.97
3.77
6608
4.67
Qwen-V1 Vien-Vi-Ciarraa Card al 20236
Qwen-7B
9.68
448
78.2
5.30
57.5
3.63
38.9
3.22
68.2
6.10
61.5
5.21
60.9
4.69
LLaVA Liu al, 20256
Vicuna
7.2B
336
78.5
6.97
62.0
6.99
50.0
6.44
66.8
7.55
58.2
584
63.1
6.76
VILA [Lin al
Vicuna-7B
7,2B
8.01
8.03
5.75
8.27
5.70
2.15
LLaV at 2024g
Vicuna-7B
7.48
1024
81.9
4.42
64.2
4.55
48.7
4.06
65.1
4.71
67.1
3.81
65.4
4.31
Sporie MLLMy
MoB-LLaVA-LL6BB Lin ct
2024
StableL-M11668
2,9B
336
76.7
7.79
160.3
7.43
36,2
6.27
62.6
8.09
50.1
4.48
57.2
6.81
LLLVA-2.78B x Lib ct
MIZA
Phi-2.7B
5.3B
336
77.6
6.01
61.4
5,23
43.9
3.95
68.5
5.80
51.4
3.76
60.6
4.95
RoF-I.LVV
Vicuna-7B
7.38
336
7.02
61
7.07
52.5
6.52
68.4
7.65
5.59
63.8
6.77
RoE-VILA
Vicuna-78 7B
7.38
336
78.8
8.25
62.2
8.01
53.7
6.28
69.5
8.39
59.3
5.75
64.7
7.34
RoE-LLaVA-HR
Vicuna- 7B
7.5B
1024
80.9
4.79
62.5
5.01
47.6
4.12
67.4
4.96
64.6
4.02
64.6
4.58
5.3
EXPERIMENTAL RESULTS
5.3.1
QUANTITATIVE ANALYSIS
Results of RoE on different MLLMs. In Table. 1 we first present the results of RoE applying VL. Liu al. (2023a), LLaVA-HR Luo et al. (202cc) and VILA Lin al. (2023) with different skip rates. From this table, we can observe that RoE can help existing MLLMs to reduce a large number of redundant parameters and computations. For instance, RoF. VILA30%% skips 23.44% of the parameters and speed up inference by about 24.5% on average. Notably, the performance of RoE- VILA30% only drops by 1.38%. We can also observe that the impact of skip rate distinct for different MLLMs. Specifically, for RoE-LLaVA, the performance does not drop obviously as the skip rate increases, i.e. -0.2% on average with an actual skip rate of 22.80%, while LLa VA-HR is more sensitive to network skipping. For example, the increase of skip rate from 7.68% to 12.02% results in about -4.34% performance drop on average. Nevertheless, RoE can further improve the compactness of I.I V-RR through its dynamic routing. Another observation from these results is that the RoF-MLLLM always has a higher skip rate on multiple-choice questions than the open-ended ones, egg. 23.03% on SQA vs. 8.81% GQA on RoE-LLa VAga%, suggesting its actual skip rate is related to the task difficulty. Overall, these results well validate the effectiveness of our RoE and also confirm the redundancy of existing MLLMs.
A Ablation Study, In Tab 2] we conduct a set of experiments to ablate the designs of RoE. In this table, "+Router" means that we directly equip LLaVA-1.5 with path routers as described in Eq We can first observe that this direct attempt is hard to achieve the expected structure sparsity on LLaVA. As shown Tab 2 its average skip rate is only 1.1%, and it is even as low as 0.1% on the difficult tasks like GQA. This result suggests that the model barely chooses to skip layers during inference, which well confirms our assumption about the challenges of routing the well-trained MLLMs. In practice, the additional computation of routers even leads to the latency of +7.29%. Based on "+Router", "+Reg20%" " further applies the structure sparsity regularization to optimize the routers. With this regularization term, routers are better trained to evaluate and skip the less important layers of MLLMs. For instance, RoE-LLaVA can skip up to 10 layers on MMB, greatly improving the inference speed by +21. 11.9.. Nevertheless, we still observe obvious performance
8

---

Published as a conference paper at ICLR 2025
User: Who he?
MoE-LLaVA-227Bwx
LLaVA: The the image is Vincent Van Gogh, famous artist known for his unique style and contributions to the world of
RoE-LLaVA (ours): The man in the image is Vincent Van Gogh, famous artist known for his distinctive style and contributions to the world of art.
The man in the imago is famous painter, Vincent Gogh.
Rob:
32,52
(a)
LLAVA
29.24
token's
User: Where is there?
MoE-LLaVA-2277::
LLaVA: This a truin station, with a train on the tracks and building nearby.
RoE-LLaVA (vurs):
RoF
33.54
This is a scene at a train station, with tain the tracks and a building nearby.
This a train station, with train the tracks and people walking around.
Mol:
1892
tnkenis
Input Image:
User: Can you explain this meme! RoE-LLaVV: The imuge features a pan
User: How many fried chicken!
Sometimes just look pictures of carth from spaion and marvel how besutiful all
RoE-LLaVA: There are ten pieces of fried chicken in the image.
filled with fried chicken pieces, shaped to resemble chicken pieces are spread across the pan. with pieces placed closer the edges and others more towards the center. The overall presentation is creative and unique, combines the familiar concept of a map with the delicious taste of fried chicken.
of the world. The
16
(c)
User: Is there any fried chicken? Please answer Yes or No.
RoE-LLaVA: Yes.
16
24
35
Input Question: How many people there? Please answer the question with single number.
RoF-ILaaA::
RoF-LLLVV: 10
16
Figure 3: Examples of our RoE on LLaVA. Example (a) and (b) show the comparison between RoE- LLaVA and LLaVA and MoE-LLaVA. Our RoE-LLaVA can answer the questions as accurately and in detail as LLaVA, while being faster, ie., more tokens per second (tokens/st). Example (c) shows the predictions of RoF-LLaVA for the same image and different questions. RoE can adjust the choice and depth of expert pathways according the question, i.e., the bar charts (the yellow ones denote the skipped layers). Example (d) shows the predictions for the same question but different images. RoE can also route different optimal expert pathways according different visual content.
degradation after layer skipping, e.g., -2.0% on SEED. As we discussed in Sec. 4.1 directly skipping layers typically leads to dramatic changes in feature space. To compensate for this, the lightweight adapter-based skip connections are then ablated, i.e., "+Adapter" in Tab 2 It can be seen that this simple design can greatly improve the average performance by up +2.0%. Overall, these ablation results well confirm our motivation for dynamic routing in the well-trained MLLMs, and also validate the designs of RoE.
The training costs of RoE. In Tab we report the training costs of RoE on three base MLLMs. From this table, we can observe that the implementation of RoE is much cheaper than building a new MLLM. For instance, training VILA from scratch takes about 6447.4 7.4 GPU hours, but its extension VILA-RoE is only about 93.6 GPU hours since the pre-training not required. In addition, the optimization of RoF is also very efficient. We can see that for all three base MILMs, our RoE not only requires fewer SFT examples to train, but also has a shorter total training time than the can conclude that RoE is also
SFT tuning of these models. Overall, from these statistics, training- training-efficient method for the improvement of existing MLLMs.
Comparison with existing MLLMs. In Tab. 4 and Tab. 5 we compare the performance and ef- ficiency of RoE-MLLMs with more MLLMs. In Tab. we can first observe the comprehensive advantages of RoE-MLLMs over the compared sparse MLLMs on four MLLM-specific bench- marks. In particular, the computation of MoE-LLaVA series is more expensive than our RoE- VILA although its parameter scale is smaller. For instance, compared with MoE-LLLaVA-1.6x- RoE-

---

Published as a conference paper at ICLR 2025
VILA improves the scores by 9.8% on MM-Vei Vet. And VILA also improves the inference speed by 1.11 times on MME. Similar advantages of RoE can also be witnessed in PoPE. Compared to MoE-LLaVA-2.785 x4, RoE-VILA not only achieves +0.5% performance gains but also speeds up the inference by 55.5%. When compared to the dense MI.LMs, the henefits of RoF-MILLMs are still obvious. For instance, RoF-LLaVA-HH improves the score by +3.3 MME, and RoF- achieves +1.8 performance gains on MM-Vet, while still keeping a faster inference speed. Tab. 5 gives the performance comparison on common VL tasks. Compared other sparse MLLMs, RoE- MLLMs still achieve better results on all benchmarks with superior efficiency. For instance, RoE- LOBE-LLaVA-22 7B 4 by +2.7 on VQA, while being +16.8% faster. Compared to dense MLLMs, the proposed RoE-MLLMs also show distinct advantages in efficiency, which can speed up the inference by 2.65%-6.26%% In terms of performance, RoB-MLLMM even outperform the original dense MIL.Ms on several benchmarks, e.g. +1.6 of RoF-LLaVA against L.L. VA on ScienceQA. Overall, these results further confirm the great effectiveness and efficiency of our RoE.
LLaVA outperforms
5.3.2 QUALITATIVE ANALYSIS
To gain insight into the proposed RoE, we visualize its predictions and skipped layers in Fig. 3 In Fig. 3 (a)-(b), we first compare its predictions with LLaVA and MoE-LLaVA-2.7Bxx x4. We can observe that the implement of RoE barely impedes the expression compared with the default LL As shown in Fig 3 (a), RoE-LLaVA VA can give accurate and detailed answers for the question like the default VA..
contrast, the response of the other sparse model MoF-ILLaV briefer. Besides, it can be also seen that RoF-LLaVA has a much faster inference speed than the compared models for the same question. For two examples, RoF-LLLaV can speed up inference by up to 13.9% and 77.3% than LLaVA and MoE-LLaVA, respectively. In Fig. 3 (c)-(d), we examine the behaviors of RoE in terms of the image and the question, respectively. In Fig. 3 (c), RoE-LLa is required to answer different questions for the same image. From these examples, fewer layers when answering the first question, which requires a detailed answer. In contrast, for the other two questions of which answer results suggest that the actual skip rate of RoE is related to the answer content, and the more complex the answer, the fewer layers the RoE skips. In Fig. (d) shows the response of RoE-LLaVA under the same question but different images. For the first image. of which scene is simpler, RoE can skip more layers to answer the number of people. Instead, ROE requires more reasoning steps to infer the answer of the second example, where the classroom scene is more complex. In this case, the behavior of RoE is also related to the complexity of visual content. Overall, from these examples, we can conclude that RoE is example-dependent and can route the optimal expert paths for different examples according to their difficulties, which well confirms our motivation.
see that RoE skips
simpler, RoE can skip more layers during inference. These
6 CONCLUSION
In this paper, we focus on exploring the dynamic experts in existing MLLMs, and propose a novel and effective modeling scheme called Routing Expert (RoE). To address the challenges for the well- trained MLLMs, RoE introduces the adapter-based skip connection to mitigate the issue of feature gap and a novel sparsity regularization to encourage MLLMs to learn sparse inference. Meanwhile, the routing token design is also proposed to address the training and testing misalignment of existing MLLMs. Extensive experiments on MLLMs and 10 benchmarks demonstrate that our RoE method can greatly improve the efficiency of existing MLLMs without obvious structure modification and with low training cost, while maintaining close or even better performance.
7 ACKNOWLEDGE
This work was supported by the National Science Fund for Distinguished Young Scholars (No. (No.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. U21A20472, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No.2022106001) and the Fun- damental Research Funds for the Central Universities (Xiamen University: No. 20720240053).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Joshua Ainslic, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp. Yi Tay, al. Colt5: Faster long-range transformers with conditional computation. Computing Research Repository (CoRR), 2023.
Jean- Ban--Bapttteeeee Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. Computing Research Repository (CoRR), 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. Computing Research Repository (CoRR), 2023b.
Hangho Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Sub- hojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with ffittue-of-moddalty--opoo 32912, 2022.
Advances in Neural Information Processing Systems, 35:32897-
Li Bo, Zhang Peiyuan, Zhang Kaichen, Pu Fanyi, Du Xinrun, Dong Yuhao, Liu Haotian, Zhang Yuanhan, Zhang Ge, Li Chunyuan, and Liu Ziwei. Lmms-eval: Accelerating the development of large multimoal models, March 2024. URL https::/ com/
EvolvingIMMs Lab/ lmms Lmms-eval
Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Shikrah Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306.1512 15195, 2023a.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision lan- guage models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- -ingus-inn tasks. arXiv preprint arXiv:2312.14238, 2023c.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose models with instruction tuning. Computing Research Repository (CoRR), 2023.
purpose vision-language
Jordan Dotzel, Yash Akhauri, Ahmed S AbouElhamayed, Carly Jiang, Mohamed Abdelfattah, and Zhiru Zhang. Radial networks: Dynamic layer routing for high-performance large language mod- els. Computing Research Repository (CoRR), 2024.
Zi-Yi Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang. Lu Yuan, Nanyun Peng, al. An empirical study of training end-to-end vision- and-language transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18166-18176. 2022.
David Eigen, Mare Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. Computing Research Repository (CoRR), 2013.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, pp. 5232- 5270, 2022.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evalua- tion benchmark for multimodal large language models. Computing Research Repository (CoRR), 2023.
11

---

Published as a conference paper at ICLR 2025
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Hama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint 3xXv::330./50100 2023.
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit- Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning. Computing Research Repository (CoRR), 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Boren Hu, Yun Zhu, Jiacheng Li, and Siliang Tang. Smartbert: A promotion of dynamic early exiting mechanism for accelerating bert inference. arXiv preprint arXiv:2303. 002.0006066.... 2023.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Yash Jain, Harkirat Behl, Zsolt Kira, and Vibhav Vineet. Damex: Dataset-aware mixture-of-cxperts for visual understanding of nixture-of-datasett Advances in Neural Information Processing Sys- tems, 36, 2024.
Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Shiwei Liu, Tianlong Chen, and Aditya Akella, Fin- skipllm: A hidden gem for autoregressive decoding with adaptive feed forward skipping. Com- puting Research Repository (CoRR), 2024.
Albert Q Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. Computing Research Repository (CoRR), 2024.
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Leilee-MMller, and Xinlei Chen. In defense of grid features for visual question answering. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10267-10276, 2020.
Zhang Kaichen, Li Bo, Zhang Peiyuan, Pu Fanyi, Adrian Cahyono Joshua, Hu Kairui, Liu Shuai, Zhang Yuanhan, Yang Jingkang, Li Chunyuan. and Liu Ziwei. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URL http::::///xxxi..rr///bab///
12772
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without con- volution or region supervision. In International Conference on Machine Learning (ICML), pp. 5583-5594, 2021.
Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Thang Luong, and Orhan Firat. Beyond distillation: Task-level mfxture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3577-3599, 2021.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2025
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Benchmarking multimodal llms with generative comprehension. Computing Research Reposi- tory (CoRR), 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-iman pre-training with frozen image encoders and large language models. In ICML, 2023b.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: (tt:::9988.0005 03557, 2019.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. Computing Research Repository (CoRR), 2023c.
Yunshui Li, Binyuan Hui, ZhiChao Yin, Min Yang, Fei Huang, and Yongbin Li. Pace: Unified multi- modal dialogue pre-training with progressive and compositional experts. Computing Research Repository (CoRR), 2023d.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large vision-languasl models. Computing Research Repository (CoRR), 2024.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07553. 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, arXiv preprint arXiv:2310.03744 2023a.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, pp. 34892 34916, 16, 2023b.
Shilong Liu, Hao Cheng. Haotian Liu, Hao Zhang, Feng Li. Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. Llava-plus: Learning to use tools for creating multimodal agents, 2023c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? Computing Research Repository (CoRR), 2023d.
Zijun Long, George Killick, Richard McCreadie, and Gerardo Aragon Camarasa. Multiway- adapater: Adapting large-scale multi-modal models for scalable image-text retrieval. Computing Research Repository (CoRR), 2023.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. arXiv preprint arXiv:
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022.
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in neural information processing systems (NeurIPS), 2023a.
Gen Luo. Yiyi Zhou, Jiamu Sun, Xiaoshuai Sun, and Rongrong Ji. survivor in the era of large- scale pretraining: An empirical study of one-stage referring expression comprehension. IEEE Transactions on Multimedia, 2023b.
Gen Luo, Yiyi Zhou, Minglang Huang, Tianhe Ren, Xiaoshuai Sun, and Rongrong Ji. Moil: Mo- mentum imitation learning for efficient vision- languae adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
13

---

Published as a conference paper at ICLR 2025
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yongjian Wu, Yue Gao, and Rongrong Ji. Towards language- guided visual recognition via dynamic convolutions. International Journal of Computer Vision, 132(1):1-19, 2024b.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: of adaptation for multimodal large language models. arXiv preprint arXiv:2403.03000, 2024c.
Guangyuan Ma, Xing Wu, Peng Wang, and Songlin Hu. Cot-mote: Exploring contextual masked auto-encoder pre-training with mixture--ffeeetual--xprtt for passage retrieval. Computing Re- search Repository (CoRR), 2023.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-trate training. 2018.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based lan- guage models. Computing Research Repository (CoRR). 2024.
Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang. Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings. arXiv preprint mrXiv:2305.02311 2023.
Burak Satar, Hongyuan Zhu, Hanwang Zhang, and Joo Hwee Lim. Rome: Role-aware mixture-of-
expert transformer for text-to-video retrieval. Computing Research Repository (CoRR), 2022.
Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multi- modal experts for generalist multimodal large language models. arXiv preprint arXiv:2407.12700 2024.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. Computing Research Repository (CoRR), 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Manxi Sun, Wei Liu, Jian Luan, Pengzhi Gao, and Bin Wang. Mixture of diverse size experts. arXiv preprint arXiv:2409. 10, 2024.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efffcient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227-5237, 2022.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings multimodal llms. arXiv preprint arXiv:240..06209, 2024.
Hugo Touvron, Thibaut Lavril Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. Computing Reseurch Repository (CoRR), 2023.
Haixin Wang, Xinlong Yang, Jianlong Chang, Dian Jin, Jinan Sun, Shikun Zhang, Xiao Luo, and Qi Tian. Parameter-effictne tuning of large-scale multimodal foundation model. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
Jue Wang, Ke Chen, Gang Chen, Lidan Shou, and Julian McAuley. Skipbert: Efficient inference with shallow layer skipping. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7287-7301, 2022a.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-lanee saiiunla tasks. Computing Research Repository (CoRR), 2022b.
Qiong Wu, Wei Yu, Yiyi Zhou, Shubin Huang, Xiaoshuai Sun, and Rongrong Ji. Parameter and computation efficient transfer learning for vision-language pre-trained models. Advances in Neu- ral Information Processing Systems, 36, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. Com- puling Research Repository (CoRR), 2023.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang. and Wei Lu. Tinyllama: An open-source small language model. Computing Research Repository (CoRR), 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. Compuring Research Repository (CoRR), 2022.
Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, Kuai Li, Chen Chen, Tao Yang, and Maosong Sun. Ca-lora: Adapting existing lora for compressed Iims to enable efficient multi-tasking on personal devices. In First Conference on Language Modeling, 2024.
Tan Zhen, Dong Daize, Zhao Xinyu, Peng Jie, Cheng Yu, and Chen Tianlong. Dlo: Dynamic layer operation for efficient vertical scaling of llms. arXiv preprint arXiv:2407. /0040077//00,,.0 2024.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. Computing Research Repository (CoRR), 2024.
Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Deyu Meng, Yue Gao, and Chunhua Shen. Plenty is plague: Fine- ginengraieed learning for visual question answering. IEEE transactions on pattern analysis and machine intelligence, 44(2):697-709, 2019.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models. Computing Research Repository (CoRR), 2023.
Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang. Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-moe Learning sparse generalist models with conditional moes. Advances in Neural Information Processing Systems, 35:2664-2678, 2022.
Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. Computing Research Repository (CoRR), 2024.
15


Reviews:
[{"review_id": "Qk7ZgzT9dA", "reviewer": "Reviewer_PWB7", "summary": "This paper proposes RoE, which skips layers in an existing MLLM achieve efficiency and effectiveness. The router for managing the layer skipping is expected to skip layers that have redundancy. The skipped layer is substitute by an adapter for mitigating feature gap.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow\n\n2. The problem solved is interesting and meaningful\n\n3. The proposed method seems to be interesting and effective", "weaknesses": "1. Even though the paper is motivated through MoE, the method is more focusing on layer skipping, which is a generally well-studied field for LLM. There should be a subsection in the related work talking about this field. Moreover, these two papers [1,2] seem to be very relevant and should be compared or discussed. The current version makes it hard to judge the novelty or contribution.\n\n[1] Raposo, David, et al. \"Mixture-of-Depths: Dynamically allocating compute in transformer-based language models.\" arXiv preprint arXiv:2404.02258 (2024).\n[2] Tan, Zhen, et al. \"DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs.\" arXiv preprint arXiv:2407.11030 (2024).\n\n2. The major motivation lie in the feature redundancy of layers in the MLLM, as shown in Fig 1. Can the author plot similar figures for the learned RoE model, to show that the redundancy is mitigated?\n\n3. There seem to be no direct supervision signal for calculating the feature similarity and guiding the learning of the router. How to make sure the skipped layers are indeed redundant? Also, can the paper show the training loss to indicate that the convergence of the method?\n\n4. Even though the paper focuses on VLLMs, the major design seems can also be applied to LLMs. It would be interesting to see how this will impact LLMs.\n\n5. Since the redundancy is highly correlated to the hardship of the input instance, how to decide the sparsity before the training? If it's a hyper-parameter for tuning across datasets / tasks, then this might heavily impact the applicability of the proposed method for unseen tasks. Can the authors provide some insights on how to choose the sparsity? Also, the current tasks are more focusing on easier tasks like VQA. Is the method still effective / necessary for newer or harder tasks and benchmarks like grounding or segmentation?\n\n6. Since the efficiency is the major target, can the author provide comparison of actual averaged FLOPs in the experiments, to explicitly show the effectiveness and importance of the proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaxfK28cl4", "reviewer": "Reviewer_iEwZ", "summary": "This paper introduces a method of introducing sparsity in multimodal LLMs by skipping some transformer layers. Importantly the skipped layers have a low-rank adaptor applied, and the exact set of layers skipped or not depends on the input due to a learned routing function. The work is in essence combining ideas of model pruning with MoEs and applying it to multimodal LLMs. The authors introduce several techniques to effectively train this model, such as warming up the routers and adaptors, and enforcing sparsity in the router. They show their model can maintain very close to SOTA accuracy across a variety of multimodal LLMs while increasing throughput by 10-20%.", "questions": "Suggestion for improvement is to compare against another technique for model pruning or distillation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is clearly written, and the main ideas are simply presented. The work shows good speedups of up to 20% without much degradation of model accuracy.", "weaknesses": "The paper focuses mostly on the connection of their work to MoEs, but not as much on the connection to existing model pruning / layer removal efforts. Also while the paper compares accuracy & speed-up compared to the baseline models, they don't compare to baseline pruning or distillation techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FRH4y3lml1", "reviewer": "Reviewer_akxJ", "summary": "This paper explores how to make a MLLM (multimodal large language model) more efficient. They find that different layers of the MLLM contribute differently to each sample; therefore, the paper proposes adaptively skipping over less important layers. They do this by replacing full layers with adapters, and learning a routing function at each layer that makes a binary decision to use the adapter or the full layer. This effectively converts an existing MLLM into a \"mixture of experts\". They find that this approach, called RoE, results in significant speedups at inference time while suffering only a slight degradation in accuracy.", "questions": "1. Can you provide more visualization on the accuracy/speed tradeoff of each model?\n2. Can you provide additional ablations, such as showing what happens when a subset of the layers is deterministically set to use the adapter.\n3. Can you provide more examples that are routed to adapters more often versus routed to the existing layer more often? Do these correspond with easier/harder samples?\n4. Clarity suggestions.\n5. Would welcome more discussion/results on improving and quantifying the tradeoffs made in RoE and its costs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: simple way of taking an existing model and making it more efficient/into a MOE by swapping out existing layers with adapters and training a router. \n\nQuality: results show fairly consistent speedups in MLLMs. RoE also outperforms existing MOEs in accuracy on downstream tasks.", "weaknesses": "Quality: \n- Only one scenario (SQA in table 5) has RoE being strictly better than other models in both accuracy and speed. All other settings exhibit a tradeoff, and it is not unclear how good/bad this tradeoff is. It would be nice if the paper could visualize the pareto frontier. There are also some cases where RoE is slower than dense MLLM counterparts.\n- Some additional ablations would be helpful, such as adapter+no router+no reg; that is, just using the adapter at each layer. \n- More analysis would be better, i.e., what is being routed between the adapter and the existing layer (Figure 3d touches on this) \n\nClarity: \n- Notation like $\\{G_1, G_2, \\dots, G_n\\}$ are layers chosen by the router. Its number is smaller than the default length $n$ is ambiguous. Should use separate $n$'s. \n- Equation (3) is not properly explained - what is $I$? What is $T$?\n- What are the training objectives for Stage 1 and Stage 2 of RoE?\n\nSignificance: there are many tradeoffs in the proposed method. Therefore, it is unclear how much people would use this in practice (i.e., do people want to spend ~93.6 GPU hours to make an existing model faster but worse---and not clear by how much)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jwkVhfRVzz", "reviewer": "Reviewer_2K7T", "summary": "This paper introduces Routing Experts (RoE), a novel approach to transform existing multimodal LLMs into mixture-of-experts models without significant architectural changes. The key innovation is treating each layer of pre-trained MLLMs as a potential expert that can be dynamically routed or skipped based on input complexity.", "questions": "- While the study demonstrates thorough experiments across different LLaMA-based MLLMs, the generalizability to non-LLaMA architectures (e.g., Qwen) remains unexplored. Testing RoE on diverse language model backbones would better validate its broader applicability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and good in coherence.\n- The authors present innovative architectural designs that effectively integrate Mixture-of-Depths into Multimodal Large Language Models\n- The empirical validation is comprehensive, encompassing diverse model architectures and benchmark datasets.\n- The proposed approach is computationally efficient, requiring minimal fine-tuning overhead for implementation.", "weaknesses": "**Major**\n- From Table 1, RoE-LLaVA-HR shows a large drop in performance. While the authors note that \"LLaVA-HR is more sensitive to network skipping ... Nevertheless, RoE can further improve the compactness.\" They should explain why this happens and whether the improved compactness is worth the performance loss.\n- From Table 2, comparing RoE to *Router* that entirely skips model layers may not be fair enough. The study needs separate tests for each part of RoE (adapter, regularization, and router token) to show how each contributes.\n- The sparsity ratio in Table 4 and 5 is not clearly stated, and the inference speed improvements are not very impressive. This raises questions about how well RoE can handle more complex tasks and higher sparsity levels.\n\n**Minor**\n- Formatting: Too much empty space under figures and between sections.\n- Inconsistent Terms: \"L1-Distance\" is written differently in Figure 1 and its caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)