Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models
Authors: Yimu Wang
Mozhgan Nasr Azadani
Sean Sedwards
Krzysztof Czarnecki
Venue: NeurIPS 2025 poster ()
Keywords: Vision Language Model
Knowledge Distillation
Primary area: applications
TL;DR: HAWAII distills knowledge from multiple visual experts into a single vision encoder using teacher-specific and general-knowledge LoRA adapters with routers, achieving strong performance with low computational cost.

Abstract: Improving the visual understanding ability of vision-language models (VLMs) is crucial for enhancing their performance across various tasks. While using multiple pretrained visual experts has shown great promise, it often incurs significant computational costs during training and inference. To address this challenge, we propose HAWAII, a novel framework that distills knowledge from multiple visual experts into a single vision encoder, enabling it to inherit the complementary strengths of several experts with minimal computational overhead. To mitigate conflicts among different teachers and switch between different teacher-specific knowledge, instead of using a fixed set of adapters for multiple teachers, we propose to use teacher-specific Low-Rank Adaptation (LoRA) adapters with a corresponding router. Each adapter is aligned with a specific teacher, avoiding noisy guidance during distillation. To enable efficient knowledge distillation, we propose fine-grained and coarse-grained distillation. At the fine-grained level, token importance scores are employed to emphasize the most informative tokens from each teacher adaptively. At the coarse-grained level, we summarize the knowledge from multiple teachers and transfer it to the student using a set of general-knowledge LoRA adapters with a router. Extensive experiments on various vision-language tasks demonstrate the superiority of HAWAII, compared to the popular open-source VLMs.

Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models
Yimu Wang, Mozhgan Nasr Azadani, Sean Sedwards, Krzysztof Czarnecki
University of Waterloo, Canada
Cyimu. , mnasraza, www.sseeaaaa...rreeeeeeeeeeeeeee sedwards, k2ccarne) Buatteroo..o
Abstract
Improving the visual understanding ability of vision-language models (VLMs) is crucial for enhancing their performance across various tasks. While using multiple pretrained visual experts has shown great promise, it often incurs significant compu- tational costs during training and inference. To address this challenge, we propose HAWAII, a novel framework that distills knowledge from multiple visual experts into single vision encoder, enabling it to inherit the complementary strengths of several experts with minimal computational overhead. To mitigate conflicts among different teachers and switch between different teacher-specific knowledge, instead of using fixed set of adapters for multiple teachers, we propose to use teacher- specific Low-Rank Adaptation (LoRA) adapters with a corresponding router. Fach adapter aligned with specific teacher, avoiding noisy guidance during distil- lation. To enable efficient knowledge distillation, we propose fine-grained and coarse-grained distillation. At the fine-grained level, token importance scores are employed to emphasize the most informative tokens from each teacher adaptively. At the coarse-grained level, we summarize the knowledge from multiple teachers LoRA adapters
and transfer it to the student using set of with a router. Extensive experiments on various vision-language tasks demonstrate the superiority of HAWAII compared to popular open-source VLMs. The code is available at http://ggthh...ommmmmmaaagsws///
1 Introduction
Vision-language models (VLMs) [ 2 enable machines to perform complex reasoning over multi- modal inputs by combining the powerful language reasoning capabilities of pretrained large language models (LL
3 4 5 with the rich perceptual understanding offered by vision foundation mod- These two components are connected through alignment modules, such Q-Formers [9] or MLP projections [10]. which map visual tokens into a representation space compatible with LLMs. At the heart of this pipeline, the vision encoder plays central role, its ability extract semantically rich visual features directly impacts the generation and reasoning capabilities of the VLM.
els 6 7 3
Recent studies have shown that incorporating multiple vision experts improves performance by large margin [T]]
14 [3]. Nevertheless, these gains in effectiveness often come at the cost of 19 20 : multi-expert setups require computing visual tokens from all vision experts during both training and inference, making them expensive and less practical for deployment, especially in latency-sensitive
13
efficiency [16] 17
18
resource-constrained settings [21] 22 23 As result, there is growing interest in approaches that can retain the benefits of multiple vision experts while avoiding their substantial inference-time costs.
Knowledge distillation (KD) (24), as a general framework for transferring knowledge from a larger
model (teacher) to smaller model (student), has been widely used in various domains [25] 26 27 28
As a pioneer study of KD in VLMs, MoVE-KD 22 distills knowledge from multiple visual experts
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

into single vision encoder using a. fixed ser of Low-Rank, Adaptation (LoRA) adapters [30] for all teachers, enhancing visual understanding while only adding a small set of trainable parameters. However, learning from multiple teachers is challenging [3 32 as the training data, model architec- ture, and training objectives of each teacher could be different. It can lead to noisy and redundant knowledge transfer, which can hinder the learning process with suboptimal performance [33]
To this end, we propose novel hierarchical visual knowledge transfer method for efficient VLMs, namely HAWAII It is designed to distill knowledge from multiple visual experts, i.e., SAM [6]. ConvNext [34, EVA 8 and Pix2Struct [35), into single vision encoder, specifically, CLIP's vision encoder, enabling it to inherit the complementary strengths of these experts with minimal computational overhead. HAWAIl consists of a novel mixture-of -itturreof-LoR--aaee LoRA-adapter (MoLA) module and
hierarchical knowledge distillation (HKD) mechanism that enables the student encoder to distill knowledge caanse-grained and fine-grained levels.
Fine-grained distillation. As each teacher's knowledge is different, due to the heterogeneity of training data, architecture, and optimization methods, in MoLA, teacher-specifie LoRA adapters are employed to avoid conflicts between teachers' knowledge. Each adapter is aligned with its teacher separately, allowing the student encoder to learn from diverse teachers while mitigating noisy distillation. Moreover, to emphasize the informative tokens generated by each teacher, at the fine-grained level, HKD utilizes a new token importance scoring method, which assigns weights to tokens according to the similarity to the text instructions and visual features.
Coarse-grained distillation. To obtain the collective consensus among visual teachers, HKD summarizes the knowledge from multiple teachers using a projector. Then, MoL.A incorporates a set of ceneral-knowlede LoRA adapters and router to align the student with the collective consensus for a global alignment.
In summary, the main contributions of this work are:
We propose HAWAII, novel framework that distills knowledge from multiple pretrained visual experts into single vision encoder, improving the visual understanding ability of VLMs without incurring substantial computational overhead.
The proposed MoLA module consists of teacher-specific LoRA adapters and general- knowledge LoRA adapters that enable the student encoder to learn from diverse teachers separately (fine-grained and globally ccoarse-grained avoiding noisy and redundant knowl- edge transfer.
HKD distills knowledge from multiple teachers gearse-grained and fine-grained levels. At the fine-grained level, HKD utilizes teacher-specife LoRA adapters and token impor- tance scoring to select and learn from the most informative tokens from each teacher, as indicated by the visual and text tokens. At the coarse- -rarneeg level, HKD summarizes the the student encoder globally using
knowledge from multiple teachers and transfers general-knowledge LoRA adapters.
Extensive experiments on various vision- Lasion-language tasks 33 37 38 39
45) show that HAWAII achieves better performance on all the benchmark datasets compared to the baseline model (L.LaV--1.5 (46)). In particular, the performance on VizWiz, SQA, and MMBench is improved by 7.8%, 5.5%, and 4.0%, respectively.
2 HAWAII
In this section, we introduce the HAWAJI framework, which learns from multiple powerful visual teachers for better visual perception ability. HAWAII inherits the complementary strengths of several experts without incurring substantial computational overhead. First, we introduce the architecture of HAWAII. Second, present the details of MoLA, which consists of set of teacher-specific and general-knowledge LoRA adapters in Section 2.2 Last, we provide the details of our hierarchical knowledge distillation method, which contains the coarse- and fine-grained distillation in Section 2.3
2.1 Architecture
The overall architecture of HAWAII is presented in the upper part of Figure | It follows the general design (vision sppert-projector-LLEE of existing MLLMs (2] 10 47) The vision expert is trained

---

Visual Projector f
LLM
Response
Vision Encoder
Add & Norm
FFN
MoLA
Visual Feat.
Image
I
fu
Norm
Self-Attn
KD Loss
Logi
Instructions
General Router
Sum.
y
Vision Encoder fu
Feett.
General Adp. 1
T. Adp.
Adp.
Malaen State
General Adp. 2 ?
Concat MLP
af
x]
Lig
Teacher Adp.
Teacher
Feat.
Ii
KD Loss
S1
Teacher 0g
Teacher 2
KD Loss
S2 Feat.
1
Feat.
Teacher Router
Lig
Token Importance
(a) MoLA
(b) Coarse-Grained Distillation
(c) Fine-Grained Distillation
Figure The overall architecture of HAWAII We use two teachers for simplicity. (a) MOLA (Section 2.2p consists of teacher-specifii LoRA adapters (Teacher Adp.) and
LoRA adapters (General Adp.) with two routers controlling the activation of adapters. (b) Coarse- grained distillation (Section 2.3.1 first summarizes the knowledge from multiple teachers and then transfers the student encoder globally. "TI Feat.", "T2 Feat,", and "Sum. T. Feat." represents the visual features 1?" generated by different teachers and the summarized teacher features L.g. (c) In the fine-grained distillation (Section 2.3.2), teacher-specific LoRA adapters (T. Adp.) and token importance scoring (Figure 2 are employed to select and learn from the most informative tokens.
to distill knowledge from multiple pretrained vision experts and produces visual tokens used for visual comprehension. The projector maps the visual tokens the LLM input space, and the LLM generates the cestructiio-oollowinn response.
The vision encoder fe(-) takes the input image and generates a set of visual tokens 1 RmXD where m is the number of visual tokens and D is the dimension of each token. To boost the performance, instead of using multiple vision encoders [19] 47 48). which would be computationally expensive, only one student vision encoder is employed. And, trained to distill knowledge from multiple pretrained vision experts 6 7) We introduce the mixture-of of--ooff--oaaalleeee (MoLA, Section 2.2h module that enables the student encoder to learn from diverse teachers fine-grained (Section] 2.3.2p and coarse gaasee-rained (Section 2.3.Ip manner.
visual projector fp(-) is applied to project the generated visual token 1 to the LLM input space.
An LLM (.LM(-) then takes the mapped visual tokens fT() and the textual instruction tokens T as input to generate the instruction-follon foolloiinn-fllowiing response Y {y: (veL) as
p(
II I(lulf((r)))T v, 2-1
(I)
where L is the length of the response and ycc is the previous tokens of yi-
2.2
Mixture of LoRA Adapters
Directly fine-tuning the student encoder is challenging, as often leads to overfitting on the limited fine-tuning data and catastrophic forgetting 22 49). To avoid this, we propose mixture-of-Le -o-RAA-- adapter (MoLA) module consisting of teacher-specific LoRA adapters and general ceneral-knowledge LoRA adapters (30 to enable the student encoder to learn from diverse teachers without forgetting. MoLA illustrated in Figure (a).
Teacher-specifie LoRA adapters. Learning from multiple teachers is challenging [3 32 33 as
each teacher 66] 8 might have different training data, model architecture, and training objectives. Directly transferring diverse teachers' knowledge to the student could lead to noisy distillation and No performance drop. To avoid this, introduce set of teacher-specific LoRA adapters {a? J==1
3

---

where N₂ is the number of teachers. Each adapter is designed to align with one teacher only, which avoids the conflicts between multiple teachers (see Section 2.3.2p. Those adapters are applied each feedforward layer of the student encoder (-)
General-knowledge LoRA adapters. For learning the collective consensus from teachers and the LoRA adapters {af Ji=1
N₂ that are applied to each feedforward layer of the student encoder fo (-). where Ng is the number of general-knowledge LoRA adapters. The details of this general (global) knowledge transfer are provided in Section 2.3.1
training data, we introduce a set of
We adopt the general (sparse) design of mixture-of-eeperts (MoE) [50] 5T to select the LoRA adapters based on the hidden inputs of each layer. Specifically, we employ two sparse routers, i.e, (.) and f,)()), to select the teacher-specific LoRA adapters and anneral-knowlede. LoRA adapters, respectively. Formally, for each feedforward layer of the student encoder, the MoE output (.) is computed as
(h) (()) + of +1)++af + +af (b).
with i = ARGMAX(F, (h)) and j (H)
where h is the hidden input of the current layer and F(-) is the current layer. We denote the visual tokens generated by the student encoder with MoLA as 7
2.3 Hierarchical Knowledge Distillation
To integrate diverse teachers' knowledge into single student encoder, we propose hierarchical knowledge Distillation (HKD) mechanism that transfers knowledge two levels of granularity, i.e. coarse-grained and fine-grained levels. Specifically, for coarse-graines distillation (Section 2.3.1). we summarize the knowledge from multiple teachers (collective consensus) and transfer to the student encoder globally. For fine- fine-grained distillation (Section 2.3.2) teacher-specifie LoRA adapters are employed to align with each teacher separately for a precise noise transfer. Moreover, to attend to the most informative tokens during knowledge transfer, introduce token importance scoring method (Figure 2) based on the similarity among teachers' visual tokens and the input instructions.
2.3.1 Coarse-Grained Distillation (CGKD)
To globally distill the knowledge from multiple teachers to the student encoder, we propose a coarse- grained distillation (CGKD) mechanism that first summarizes the knowledge from multiple teachers and then transfers it to the student encoder.
To obtain the collective consensus, i.e., summarized teacher feature, each teacher's visual features are first unshuffled 22 47 52) to have the same length [2] as the student's 's visual features I E Rmxx Then, those visual tokens channel-wise concatenated and the summarized feature I is obtained by applying two-layer MLP (-) as
T fey (CONCAT (1F,13 (I., RmDD
(3)
where 1,T is the unshuffled visual tokens from the i-th teacher, and CONCAT(-) is the channel-wisely concatenation operation.
Next, we apply the coarse-grained distillation loss L to transfer the collective consensus by min- imizing the mean square error loss (MSE) between the summarized features I and the student encoder output 1 as
Leg = MSE(IS, (1
2.3.2 Fine-Grained Distillation (FGKD)
Using LoRA adapters [30 to transfer knowledge from one teacher student has proven to be successful. However, transferring knowledge from multiple teachers to a single student is challenging. especially when using a fixed set LoRA adapters (2 for all the teachers. The reason is that the noisy and redundant teachers' knowledge can hinder the learning process and lead to suboptimal
performance [3 33), due to the conflicts among teachers, which arises from the heterogeneity of training data, architectures, and the training algorithms.

---

To address this challenge, propose the fine-gres fine-grained distillation (FGKD) that exploits teacher- specific LoRA adapters and token importance scoring. Each teacher-specific LoRA adapter is designed to align with one teacher only, allowing the student to learn from each teacher separately. Token importance scoring is used select and attend the most informative tokens from each teacher during knowledge transfer, reducing the noise and redundancy.
Teacher-specific LoRA adapters. We expect each teacher-specific LoRA adapter learn the knowledge from one teacher only, such that the knowledge transfer is more effective and less noisy, We denote the output of the student encoder with only the i-th saccer--ppecfic LoRA adapter of being activated for each layer as 1,. Specifically, each feedforward layer of the student encoder, apply the LoRA adapter of (-) as F(h) + of (h), where F()) is the current layer and h is the input to the layer. In that case, 1 only needs to align with the i-th teacher's visual feature I?", making the knowledge transfer procedure smooth and precise.
Token importance scoring. The key to knowledge distil- lation is to transfer the most important information [24] As previous studies show that not all tokens are equally in- formative [88 [19 20 22), identify the most informative tokens, we introduce new similarity-
CONCAT (F.T) ury
81 R 1xm
importance score that considers teachers' visual tokens and the in- put instructions T, allowing us to prioritize tokens that are more relevant the task context. Specifically, for the i-th teacher, we compute the token importance score 8t E R1xm as
Scores
Teacher's Feat.
Similanty
The calculation of token im- portance score s,. To focus on the most informative tokens, consider the sim- ilarity among the teacher's 's features and the input instructions T.
Figure 2:
CONCAT (17 (IT)
8: MEAN
SOFTMAX
VD
(5)
where I.. E RR and 7 are the visual tokens and input instructions projected by a learnable two-layer MLP to have the same dimension the student features. D is the dimension of the visual tokens.
Now, with the token importance scores {ss)se)m), the fine-grained distillation loss Lig calculated as
N₂
Ci
5:
C-MSE(1,,
(6)
N
2.4 Training Objectives
The overall training objective of HAWAII is to minimize the loss consisting of the text generation loss
Lgen 153, the coarse- grained distillation loss Leg (Equation (4), the fine-grained distillation loss
Cig (Equation ), and the MoE balance loss Lmb [5] 154 155 This is given
Lagen (L Lug) + A2
(7)
where A and x2 are the syper-parametent to balance the losses. We set A1 = 0.5 and X2 = 0.05 for all our experiments.
3 Experiments
3.1 Experimental setup
Implementation details. We use Vicuna-v1.5-7B 3 as the LLM and use CLIP I for the vision encoder, with the teachers of CLIP (as CLIP is updated) being ConvNeXt 34). Pix2Struct [35]. SAM [6], and EVA-02 [8]. The base version of HAWAII uses CLIP, ConvNeXt, and EVA-02 as the vision teachers, while for HAWAII f, we further add Pix2Struct as the teacher. To understand how different teachers contribute to the performance, we also conduct experiments with CLIP, ConvNeXt, EVA-02, and SAM as the teachers, denoted HAWAII F. The visual projector is 2-layer MLP with the GELU activation function [56]. For MoLA, we use three (or four) teacher-specific LoRA adapters and three
LoRA adapters for each FFN layer of the student encoder.
5

---

Methods
VQA
VizWiz
GQA
SQA
POPE
MME
MMBeneh
MMMU
A12D
SecdBeach
BLIP-2 9 (Vicunna- 1313)
42.5
61.0
85.3
1293.8
IDEFICS-9B [57 (LLaMA-7B)
25,9
35.5
38.4
48.2
Vwen-VL. 58 (Qwen-7B)
63.8
35.2
59.3
67.1
38.2
56.3
mPI.UGG-Ow12 (L-.MA-7R)
68.7
64.5
Vicunna 15-77 7B
InstruciBLIP 60
49.2
60.5
36.0
30.6
Vidco-LLaVA 61
51.8
48.1
60.3
66.4
84.4
60.9
MoVE-KD 20
58.3
52.3
63.2
69.4
86.9
1524.5
66.3
LLaVA-1.5 1.5 46 (Baseline)
58.2
50.0
62.0
66.8
85,9
1510.7
64.3
34.7
55.5
66.1
62.8
70.5
87,3
1540.2
66.9
36.6
56.2
67.5
1..4
1
1.0.4
1.1
0.0.
1 15.7
7.6
1.9
0.7
1.4
HAWAn' (HAWAII + Pix2Struct)
58.6
54.2
63.6
69.5
86.8
1533.6
67.1
36.0
56.2
67.2
HAWAD (HAWAII SAM)
63.3
87.5
66.7
55.8
Table 1: Performance comparisons of HAWAII and the baseline VLMs using Vicunna-1 1.5mma (if not specified). HAWAJI utilize CLIP, ConvNeXt, and EVA-02 as the teachers (the same setting with MoVE-KD), HAWAII' further adds Pix2Struct as the teacher, and HAWAII uses CLIP, Cenv EVA-02, and SAM the teachers. The best results in bold and the second best results underlined.
Each adapter a LoRA block [30 with rank of 32. The routers are sparse and 2-layer MLPs with the GELU activation function. Each router selects only the LoRA adapter with the highest probability. Models are run on eight NVIDIA A6000 GPUs with 48GB of memory.
Training stages. We follow the standard paradigm of LLaVA-1.5 [46]. The training of HAWAII consists of two stages, i.e., pretraining and finettuning The pretraining stage is to align the vision encoder with the LLM. During this stage, only the vision projector, LoRA adapters, and the routers are trained. The supervised fine-tuning stage instruction-followwe response. In this stage, the whole model is trained.
align the vision encoder with the LLM and the
Training datasets. HAWAII the same training data ALaVA-v1. [46]. Specifically, in the pretraining stage, we use 558K image-text pairs, while in the supervised fine-tuning stage, we use 665K instructionnfollooinn image-text data boost the performance.
Benchmarks and baselines. We evaluate HAWAII on several image understanding tasks 36 42 43 44 45). Details are deferred to the Appendix. We compare HAWAII with and a VLM with knowledge
38 39
40
41
several baseline methods, including general VLMs 57 58 59 60 61] distillation 229]
3.2 Main Results
The results are shown in Table [ Compared to the baseline method (LLaVA-1.5). HAWAII achieves sigmificant improvements on most benchmarks, demonstrating its effectiveness. Results also demon- strate that compared to the existing knowledge distillation method [29] that uses the same teachers as HAWAII, HAWAII achieves better performance on most benchmarks, demonstrating the effectiveness of the proposed MoLA module and HKD mechanism.
3.3 Ablation Studies
In this part, we conduct ablation studies to analyze the effectiveness of the proposed components in HAWAIL
Ablation on FGKD, CGKD, and MOLA. The results are shown in Table 2 When all components included, HAWAII achieves the best performance most tasks (highlighted row), with an average of 63.7% across all tasks. The baseline model (LLaVA-1.5)) with only FGKD (w/o token scoring) and teacher-specific LoRA adapters achieves 63.2% on average. Further adding the token importance scoring mechanism improves the performance to 63.5%. However, we also observe that the performance GQA is slightly decreased, which might be due the fact that GQA requires more general knowledge rather than specific knowledge from vision teachers. Adding CGKD and general-knowledgee LoRA adapters further improves the performance 63.7% average.
Number of visual teachers. To understand how different teachers provide complementary knowledge for visual understanding, conduct experiments with different teachers, shown in Table The

---

Methods
VQA TMr
VizWiz
GQA
SQA
POPE
MME
MMBeach
MMMU
A12D
SeedBench'
Avg.
LLaVA-1.5
58.2
50.0
62.0
66.8
85.9
1510.7
64.3
34.7
55.5
66.1
61,9
FGKD (w/ot token scocing)
59.0
52.5
63.1
70.1
86.6
1592.1
66.8
36.7
54.6
66.3
63.2
tokea scoring
59.1
52.5
62.8
70.2
87.4
1541.7
67.3
35.9
56.1
67.0
63.5
CGKD
53.9
62.8
70.5
15402
66.9
56.2
67.5
63.7
DoRA
58.4
53.2
61.8
69.3
87.7
1558.5
66.9
35.2
53.5
67.8
634
Table 2: Ablation study on various vision-language tasks HAWAIL. We normalize the results of MME to compute the average results. FGKD and CGKD denote fine-grained distillation with teacher- specific LoRA adapters and coarse-grained distillation with general-knowledde knowledge LoRA adapters. W. DoRA represents the variant trained with DoRA for comparison.
#
VQA Text
GQA
SQA
POPE
MME
MMMU
A12D
SeedBench'
1
58.7
62.6
70.1
84.5
1516.2
37.0
55.5
67.4
3
58.7
62.8
70.5
87.3
1540.2
36.6
56.2
67.5
5
58.6
62.8
70.4
85.2
1530.2
36.4
55.0
66.9
Table 3: Performance of HAWAII with different numbers of general-knowless adapters.
VQA Text
GQA
SQA
POPE
MME
MMMU
A12D
SeedBench'
LLaVA-1.5-13B
61.3
63.3
71.6
85.9
1531.3
35.5
59.3
68.2
MoVE-KD-133
59.7
64.2
73,2
85.7
1568.1 1
HAWAII-13B
61.7
64.7
75.0
86.6
1568.7
35.7
60.0
68.5
Table 4: Performance comparison using the Micunna-1.5-133
VQA Text
GQA
SQA
POPE
MME
MMMU
A12D
64.9
64.2
70.1
86.5
1519.0
35.8
64.9
MOVE-KD (LLaVA-Next-7B)
63.7
64.5
70.7
86.7
1537.2
HAWAII (LLtVA-Next-7B)
65.5
65.2
72.0
87.8
1551.3
37.4
65.6
Table 5: Perofmance of HAWAII on LLaVA-Next-7B.
basic version of HAWAII uses CLIP, ConvNeXt, and EVA-02 the teachers. Further adding Pix2Struct as the teacher improves the performance on VizWiz, GQA, and MMBench, compared to HAWAII. However, maybe due the redundancy of knowledge, the performances
VQA Text SQA, and SeedBench are slightly decreased. We further test the performance of HAWAII with CLIP, ConvNeXt, EVA-02, and SAM as the teachers, denoted as HAWAII in Table 1 Results show that HAWAJI i improves performance on VQA Text VizWiz, GQA, SQA, POPE, MMMU, and SeedBench', compared to HAWAII, as SAM might bring strong fine-graines descriptive visual understanding ability to the model. However, we also observe that the performance on MME decreases with adding more teachers, which might be due to the fact that MME requires more general common sense knowledge for reasoning rather than specific knowledge from vision teachers.
Number of general-knowledge adapters. The number of teacher-specifie LoRA adapters is depen- dent on the number of visual teachers, whereas the number of general-knowledge LoRA adapters hyperparameter: To understand the optimal number of general-knowledee adapters, present an ablation in Table The results show that increasing the number of adapters to three improves performance on most benchmarks, while five adapters can lead slight degradation, indicating that excessive redundancy may introduce overfitting.
Generalizing to larger base models. To test the efficiency of our proposed method, we conducted experiments with Vicunna- 1.B. The results in Table f show that HAWAII achieves significant improvements. Specifically, HAWAII improves the performance SQA from 71.6 75.0. However, we also notice that with larger base models, the performance on POPE decreases as compared to that with 7B model.
The impact of the base method. To understand how our proposed knowledge distillation generalizes across different base models, we conducted experiments with LLaVA-Next-7B 62). The results in
7

---

Alan W Walter, and Council H
U: What direction is Chile in Unuguay?
U: The volure ehids cakollaed
is Walter's
U: Where is the motorcycle from?
U: What is the number of blue jersey froat?
name?
Hawaii: (West).
Hawnii: B (Cylinder).
Hawaii:
Hawaii: New york.
Hawaii: 11.
other unall diis dus. the what color?
U: What type of activity
U: Which mood does this image convey?
U: Which matches the picture?
U: What is the nature of the relations of these animals?
happening this image?
Hawaii: B (Cyan).
Hawai:: (Sightseeing).
Hawaii: C (Happy).
Hawaii: (radial symmetry). Hawaii: (Mutualism).
Figure 3: HAWAII is able to perform vision-language understanding tasks, such emotion under- standing, OCR, spatial reasoning, attribute reasoning, and relation reasoning. The examples are from the following benchmarks: VQA Text (37], MMBench [42, and SeedBench 45
Table 5 show that HAWAII achieves significant improvements on most benchmarks, compared to the baselines.
The impact of different LoRA methods. We use LoRA our design because of generalizability. To understand how different LoRA adapters impact the performance, we conducted experiments with DoRA (63) replacing LoRA. The results are shown in Table DoRA, which is more advanced than LoRA, is less generalizable than LoRA, as evidenced by the performance degradation on some benchmarks.
3.4 Qualitative Results
Visualization of inference examples. We perform qual- itative evaluation highlight the diverse reasoning capa- bilities of our model across range of challenging visual understanding tasks
written yellow sticker on the fridge?
MoVE-KD: No Smoking.
45). As illustrated in Fig- ure 3. HAWAII demonstrates strong attribute reasoning, accurately identifying fine-grained visual characteristies such color, texture, and shape. For tasks involving OCR and mathematical content, the model effectively reads and interprets text in images. Beyond factual perception, HAWAIl is capable of higher-level understanding, such as inferring image emotion and reasoning about contextual relationships and spatial arrangements. For instance, it can assess emotional tone from facial expressions and body language, and discern nature-related dependencies. These examples showcase the model's comprehensive visual-language understanding, grounded in both low-level perception and abstract reasoning.
Hawaii: Waming.
U: Which the following organisms primary consumer food web?
MoVE-KD: Black crappic.
Hawaii: Copapod.
Figure Comparison between HAWAII and MoVE-KD 29 on OCR and visual- semantic reasoning capabilities.
highlights HAWAII 's stronger visual- semantic reasoning, as accurately interprets ecological relationships in complex diagrams and effectively minimizes text ballucinations in OCR tasks.
Moreover, a comparison with MoVE-KD 29 (Figure
Visualization of token importance scores by different teachers and the instructions. Different teachers and instructions typically attend to different regions of the image, providing diverse visual cues that are important for the model to develop a comprehensive understanding. To understand how the token importance scores distribute, we visualize similarity scores between different teachers and the instructions Figure As shown, the teachers and the instruction exhibit distinct preferences. Text instructions usually focus the center objects in the images, which usually indicated by the
8

---

MARTHN
Pix2Struct
ConvNext
CLIP
RID BRAVO
Text Pix2Struct
Text ConvNext
Text CLIP
KRESS
Figure 5: Visualization of the similarity score used in calculating importance score (Section 2.3.2 using HAWAIl.
questions. For visual teachers, CLIP usually attends to the center objects, while ConvNext tends to care more about the common objects in an image (for example, people in the top right image). In contrast CLIP and ConvNext, Pix2Struct focuses the small signs and texts in the image, which useful for OCR-related tasks.
Related work
Multi-expert knowledge. the context vision-language learning, multi-expert knowledge typically refers to the use of multiple pretrained visual models, each specialized in a particular domain or task, to provide richer and more diverse visual understanding. One common strategy for incorporating such knowledge is through auxiliary supervision multitask learning 164 65], where expert models trained on tasks such as segmentation, object detection, or depth estimation provide additional learning signals during training. These experts are typically integrated via auxiliary losses or parallel task-specific heads (66). allowing the model to benefit from complementary visual perspectives. While this approach has shown effectiveness, it often requires task-specific annotations and careful balancing of multiple objectives, which can complicate training and limit scalability.
Another strategy for incorporating multi-expert knowledge into vision components of VLMs involves using multiple visual encoders to extract diverse representations, which are then fused to form unified visual understanding. These methods 13
typically focus on efficiently integrating visual tokens generated by a mixture of pretrained visual experts. By drawing on the complementary strengths of these encoders, such approaches aim enhance the model's visual perception capabilities. However, they often introduce substantial computational overhead due to the large number visual tokens, particularly in approaches that concatenate token sequences [1] 13 68]. In contrast, HAWAII adopts different strategy by using multiple vision encoders as teachers to distill their knowledge into a single student encoder, enabling it to inherit their complementary strengths while maintaining efficiency.
47
48
67
Knowledge distillation. Knowledge distillation (KD) 24 is a process where smaller, more efficient model called the student learns from the output logits or feature representations of a larger, pretrained model known as the teacher. In the context of vision-language learning, KD has been explored in several directions. Some approaches 699 70 focus on distilling large vision-language models into smaller ones. This line of work aims to compress the knowledge of powerful multimodal models into more compact and efficient versions that can still perform effectively on vision- language tasks. In contrast to our work, these methods prioritize reducing the overall model size. Instead, our approach focuses enhancing the visual capabilities of the vision encoder within VLM by distilling knowledge from multiple expert teachers without necessarily reducing the VLM itself. Another common use of KD is to train efficient vision foundation models 7 by distilling smaller vision

---

backbones from larger teacher(s) in standalone setting, separate from the VLM training pipeline. For example, Intern 300- [7] is distilled from ViT-6B using feature distillation with a cosine similarity loss applied between the hidden states of the final transformer layers. Similarly, RADIO 72 trains a vision model from scratch by merging multiple backbone models into a unified architecture through multi-teacher distillation. It employs feature-level distillation using cosine distance loss, with equal weighting applied to the outputs of each teacher. While effective, these approaches are highly computationally intensive and require massive datasets and substantial compute resources. In contrast to these standalone approaches, our work focuses on optimizing the student vision encoder within the training loop of a vision- vision-language model, allowing it to benefit directly from multimodal supervision and alignment during training.
The work closest to ours is MoVE-KD KD 29 which distills knowledge from multiple visual experts into a single vision encoder using a weighted distillation loss with a fixed set of LoRA adapters (30] The weights are shared between different teachers based on the attention weights from CLIIP []. which introduces a bias toward CLIP. In contrast, HAWAII introduce teacher-specific LoRA adapters which are aligned with each teacher separately, allowing the student encoder to learn from diverse teachers while avoiding noisy distillation. Moreover, the token importance scoring in HAWAII is based on each teacher's visual features and the input instructions, which helps to select the most informative tokens from each teacher without introducing bias toward any specific teacher.
Conclusion, Limitations, and Societal Impacts
Conclusion. We introduced HAWAII, a novel framework that distills knowledge from multiple pretrained visual experts into a single vision encoder. HAWAII consists of novel mixture-of-LoRA- adapter (MoLA) module and a new hierarchical knowledge distillation (HKD) mechanism. MoLA consists of teacher-specific LoRA adapters and general-knowledge LoRA adapters that enable the student encoder to learn from diverse teachers while learning general knowledge from the training data. HKD distills knowledge from multiple teachers at noarse-grained and fine-graines levels. The searse-grainee distillation summarizes the knowledge from multiple teachers and transfers it to the student encoder globally. The fine-graines distillation utilizes ceacher-specffic LoRA adapters and token importance scoring to select the most informative tokens from each teacher for distillation. Extensive experiments various vision- language tasks demonstrate the superiority of HAWAII over existing methods with minimal computational overhead.
Limitations. Due to the limitation of computational resources, we only used five pretrained vision experts in our experiments. Also, we only evaluated HAWAII using the Vicuna-v1.5-77 K as the LLM. In the future, it would be interesting to explore the performance of HAWAII with more pretrained vision experts and different LLMs. We only distill knowledge from the visual experts to the vision encoder, while the knowledge distillation from a bigger LLM to a smaller LLM is not considered. We believe that further improvements can be achieved by distilling knowledge from a bigger L.M smaller LLM.
Societal impacts and safeguards. The proposed HAWAII framework is designed to enhance the performance of VLMs. Thus, it inherits the same societal impacts as existing VLMs. The use of HAWAII and VLMs in general may raise concerns related to bias, misinformation, and privacy, However, we have taken steps to mitigate these risks by carefully curating the training data and implementing safeguards to ensure responsible use.
Acknowledgement
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC)-CSE Research Community project entitled "An End-to-End Approach to Safe and Secure AI Systems" and NSERC's Postdoctoral Fellowship. Researchers funded through the NSERC-CSE Research Communities Grants do not represent the Communications Security Establishment Canada or the Government of Canada. Any research, opinions, or positions they produce as part of this initiative do not represent the official views of the Government of Canada.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[2] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei La, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks, January [cs] version: 3.
2024. arXiv:2312.14238
[3] Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa. Imsys. org (accessed 14 April 2023), 2,3);6, 2023.
[4] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023. arXiv:2302.1321 13977 [es].
[5] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[6] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything, April 2023. arXiv:2304.02643 [cs].
[7] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07//99 2023.
[8] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. In 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19358-19369, Vancouver, BC, Canada, June 2023. IEEE.
[9] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman image pre- wre-train with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19744 PMLR, 2023.
[10] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. November 2023.
[11] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Kegin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprinl urXiv:2311.07577 2023.
[12] Peter Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-l: A fully open, vision-centric exploration of multimodal llss. Advances Neural Information Processing Systems, 77:887310-8335, 2024.
[13] Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. In European Conference on Computer Vision, pages 113-132. Springer, 2024.
[14] Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. arXiv preprint arXiv:2407.12709 2024.
11

---

[15] Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS. Yishen Ji, Shiyi Lan, Hao Zhang Yilin Zhao, Subhashree Radhakrishnan, al. Eagle Building post-training data strategies from scratch for frontier vision-language models. arXiv preprint arXiv:2501. (.tiv::2501.488111 14818, 2025.
[16]
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang, An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Lision-Language Models. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Giil Varol, editors, Computer Vision ECCV 2024, volume 15139, pages 19-35, Cham, 2024. Springer Nature Switzerland. Series Title: Lecture Notes in Computer Science.
[17] Bo Tong, Bokai Lai, Yiyi Zhou, Gen Luo, Yunhang Shen, Ke Li, Xiaoshuai Sun, and Rongrong Ji. FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression, December 2024. arXiv:2412.0433 [cs].
[18] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA- Adaptive Token Reduction for Efficient I. Large Multimodal Models, May 2024. arXiv:2403.1520 153888 [cs].
[19] Yimu Wang, Mozhgan Nasr Azadani, Sean Sedwards, and Krzysztof Czarnecki. Leo-mini: An efficient multimodal large language model using conditional token reduction and mixture of multi-modal experts. arXiv preprint arXiv:2504.044 046533 2025.
[20] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. VLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token, January 2025. arXiv:2501.0388 [cs].
[21] Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, and Oleg Sinavski. LingoQA: Visual Question Answering for Autonomous Driving. In Aleš Leonardis. Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024. pages 252,,69, Cham, 2024. Springer Nature Switzerland. TLDR: This work introduces LingoQA, novel dataset and benchmark for visual question answering in autonomous driving and proposes a truthfulness classifier, called I. Lingo-Judge, that achieves 0.95 Spearman correlation coefficient to human evaluations, surpassing existing techniques like METEOR, BLEU, CIDEr, and GPT-4.
[22]
Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang. James M. Rehg. and Chao Zheng. MAPLM: A Real- Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding. In 2024 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 21819-21888 830, Seattle, WA, USA, June 2024. IEEE. TLDR: A new vision-language benchmark that can be used to finetune traffic and HD map domain-specific foundation models, and annotate and leverage large-scale, broad-coverage traffic and map data extracted from huge HD map annotations proposed.
[23]
Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochender- fer, Chiho Choi, and Behzad Dariush. Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7498-7507, Waikoloa, HI, USA, January 2024. IEFF. TLDR: novel dataset, Rank2Tell 1, a multi-modal ego-centric dataset for Ranking the importance level and Telling the reason for the importance is introduced, which provides dense annotations of various semantic, spatial, temporal, and relational attributes of various important objects in complex traffic scenarios.
[24] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1503.02531, 2015.
[25] Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R. Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov. Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving, September 2023. arXiv:2309. 14491 [cs].
[26] Yi Xie, Yihong Lin, Wenjie Cai, Xuemiao Xu, Huaidong Zhang, Yong Du, and Shengfeng He. D3still: Decoupled Differential Distillation for Asymmetric Image Retrieval. In 2024 IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 11181-17190 Seattle, WA, USA, June 2024. IEEE
12

---

[27] Yi-Ting Hsiao, Siavash Khodadadeh, Kevin Duarte, Wei-An Lin, Hui Qu, Mingi Kwon, and Ratheesh Kalarot. Plug-and-Plas and-Play Diffusion Distillation. In 2024 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13743-13752, Seattle, WA, USA, June 2024. IEEE.
[28] Jing Xu, Jiazheng Li, and Jingzhao Zhang. Scalable Model Merging with Progressive Layer- wise Distillation, February 2025. MrXiv:2502.12706 [cs].
[29] Jiajun Cao, Yuan Zhang, Tao Huang, Ming Lu, Qizhe Zhang, Ruichuan An, Ningning MA, and Shanghang Zhang. Move-kd: Knowledge distillation for vims with mixture of visual encoders. arXiv preprint 25Xvv2501.00700 009, 2025.
[30] Edward J. Hu, Yclong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, Octoher 2021. arXiv:2106.09663 [cs].
[31] Sihui Luo, Xinchao Wang, Gongfan Fang. Yao Hu, Dapeng Tao, and Mingli Song. Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning. In Proceedings of the Twenty-Eighu International Joint Conference on Artificial Intelligence, pages 3087- 3093, Macao, China, August 2019. International Joint Conferences Artificial Intelligence Organization.
[32] Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 3503-3512, Seoul, Korea (South), October 2019, IEEE.
[33] Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Amalgamating Knowledge towards Comprehensive Classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3068-3075, July 2019.
[34] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976- 11986, 2022.
[35] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisensch- los, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding. In Proceedings of the 40th International Conference on Machine Learning, pages 18893-18911 PMLR, July 2023. ISSN: 2640-3498.
[36] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. VizWiz Grand Challenge: Answering Visual Questions from Blind People. In 2018 IEEE/CVF Conference on Computer Vision and Paltern Recognition, pages 1608-3611 Salt Lake City, UT, USA, June 2018. IEEE.
[37] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8309-8318, 2019.
[38] Drew A. Hudson and Christopher D. Manning. GQA: A New Dataset for World Visual Reasoning and Compositional Question Answering. In 2019 IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR). pages 6693-6702 Long Beach, CA, USA, June 2019. IEFE.
[39] Pan Lu. Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th conference on neural information processing systems (NeurIPS), 2022.
[40] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji- Wen. Evaluating Ob- ject Hallucination in Large Vision-Language Models. In Houda Bouamor, Juan Pino, and
13

---

Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing. pages 292-305, Singapore. December 2023. Association for Computational Linguistics.
[41] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, March 2024. arXiv:2306.133 133394 [es].
[42] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is Your Multi-modal Model an All-Around Player? In Computer Vision - ECCV 2024, pages 216-233. 2024.
[43] Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A Massive Multi multi-Discipliee Multimodal Understanding and Reasoning Benchmark for Expert AGI. In 2024 IEEE/CVF Conference on Computer Vision and Puttern Seattle, WA, USA, June 2024. IEFE.
Recognition (CVPR), pages
[44] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A Diagram is Worth Dozen Images. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016, pages 235-251, Cham, 2016. Springer International Publishing.
[45] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. SEED-Bench: Benchmarking Multimodal Large Language Models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13299-13308, 2024.
[46] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. arXiv, October 2023. arXiv:23 1rXiv:2310.03744 [cs].
[47] Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang. Hongxu Yin, Karan Sapra, Yaser Yacoob, et al. Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. In The Thirteenth International Conference Learning Representations, 2025.
[48] Mozhgan Nasr Azadani, James Riddell, Sean Sedwards, and Krzysztof Czarnecki. Leo: Boosting mixture of vision encoders for multimodal large language models. arXiv preprint NKKv:25001.06866 2025.
[49] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning, July 2022. 12Xiv:2203.12119 [cs].
[50] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of LoRA Experts. October 2023.
[51] Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng. Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards Ultimate Expert Special- ization in Mixture-of- Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguisties (Volume I: Long Papers), pages 1280-1297, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
Language Models. In Lun-Wei Ku, Andre Martins, and Vivek
[52] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874-1883, June 2016. ISSN: 1063-6919.
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony
14

---

Hartshomn, Saghar Hosseini. Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, llyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. arXiv:2307.099 .09288 [cs].
[54] Tianlin Liu, Mathieu Blondel, Carlos Riquelme Ruiz, and Joan Puigcerver, Routers in Vision Mixture of Experts: An Empirical Study. Transactions on Machine Learning Research, February 2024.
[55] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMOE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1932-1945, Bangkok, Thailand, August 2024, Association for Computational Linguistics.
[56] Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs), June 2023. arXiv:1606.08415 [cs].
[57] Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image- Text Documents. November 2023.
[58] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen- A Versatile Languuee Model for Understanding, Localization, Text Reading, and Beyond, October 2023. arXiv:2308. 12966 TLDR: The Qwen- VL series is introduced, a set of large-scale vision-language models designed to perceive and understand both text and images that outperforms existing Large Vision Language Models (LVL.Ms).
[59] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Zhang, and Fei Huang. mPL.UGOWI2:: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. pages 13040-13051, 2024.
[60] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBI.iP Towards General-purpose Vision-Language Models with Instruction Tuning. November 2023.
[61] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971 5971-5984, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[62] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024.
[63] Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiana Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-Decomposed Low-Rank Adaptation. In Proceedings of the 41st International Conference Machine Learning, pages 32100-32121. PMLR, July 2024. ISSN: 2640-3498.
[64] Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: A vision-language model with multi-task experts. Trans. Mach. Learn, Res., pages 1835-8856, 2024.
15

---

[65] Yuanchen Wu, Junlong Du, Ke Yan, Shouhong Ding, and Xiaoqiang Li. Tove: Efficient vision- language learning via knowledge transfer from vision experts. arXiv preprint arXiv:2504.00691, 2025.
[66] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. European Conference on Computer Vision, pages 348-367. Springer, 2022.
[67] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403./8814, 2024.
[68] Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, et al. Mousi: Poly-visual-exper vision-language models. arXiv preprint arXiv:2401.1722,, 2024.
[69] Yuxuan Cai, Jiangning Zhang. Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang. and Xiang Bai. Llava-kd: A framework of distilling multimodal large language models. arXiv preprint arXiv:2410. 16236, 2024.
[70] Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Lei Zhang, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, et al. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprint arXiv:2408. 15881, 2024.
[71] Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-intermvl: a flexible-transfer pocket multi- modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1-17, 2024.
[72] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomera- tive vision foundation model reduce domains into one. Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 12490-12500, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. We focus distilling visual perception knowledge from multiple pretrained visual experts into a single vision encoder, enabling it to inherit the complementary strengths of these experts while maintaining efficiency.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
This paper has no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided all the information needed to reproduce the main experi- mental results of the paper. The code and models will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The data is publicly available. The code and models will be released upon acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
aublic/guudees/Codeebbbmsss cosssiionoolyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuiiaeeeaec
Saaeee//CoSSSSbisssS.S.. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details necessary to understand the results in Section D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the limited computational resources, we do not report error bars in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources needed to repro- duce the experiments in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips.. http::pp....pubbbiichcccccttttie
Answer: [Yes]
Justification: We confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive socictal impacts and negative socictal impacts of the work in Section
Guidelines:
20

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have discussed it in Section
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have properly cited the assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code and models will be released upon acceptance with sufficient docu- mentation.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
23

---

A Experiments
A.1 Benchmark Datasets
We evaluate our method on the following benchmark datasets: MME 41 MMBench
42 VizWiz
Seed-
Bench 451. GQA [8). SQA (3). MMMU 43
POPE 40
A12D 44
36 and
TextVQA (37.
MME [11]. The MME benchmark is designed to rigorously evaluate a model's perceptual and cognitive abilities through 14 subtasks. It employs carefully constructed instruction-answer pairs and concise instructions to minimize data leakage and ensure fair evaluation. This setup provides a robust measure of model's ' performance across various tasks.
MMBench [42]. MMBench offers hierarchical evaluation framework, categorizing model capabil- ities into three levels. The first level (L-1) focuses on perception and reasoning. The second level (L-2) expands this to six sub-abilities, while the third level (L-3) further refines these into 20 specific dimensions. This structured approach allows for nuanced and comprehensive assessment of model's 's multifaceted abilities.
Seed-Bench 45
SEED-Bench consists of 19K multiple-choice questions with accurate human annotations, covering 12 evaluation dimensions including both the spatial and temporal understanding.
GQA (38]. GQA is structured around three core components: scene graphs, questions, and images. It includes not only the images themselves but also detailed spatial features and object-level attributes. The questions are crafted to assess a model's ability to comprehend visual scenes and perform reasoning tasks based on the image content.
ScienceQA (39]. ScienceQA spans a wide array of domains, including natural, language, and social sciences. Questions are hierarchically categorized into 26 topics, 127 categories, and 379 skills, providing diverse and comprehensive testbed for evaluating multimodal understanding. multi-step reasoning, and interpretability.
MMMU [43]. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such charts, diagrams, maps, tables, music sheets, and chemical structures.
POPE [4). POPE is tailored to assess object ballucination in models. It presents a series of binary questions about the presence objects in images, using accuracy, recall, precision, and score as metrics. This approach offers precise evaluation of hallucination levels under different sampling strategies.
A12D 44]. A12D is dataset of over 5000 grade school science diagrams with over 150000 rich annotations, their ground truth syntactic parses, and more than 15000 corresponding multiple choice questions.
VizWiz [36]. VizWiz consists of over 31,000 visual questions originating from blind people who each took picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question.
TextVQA [37]. TextVQA emphasizes the integration textual information within images. evaluates model's proficiency in reading and reasoning about text embedded in visual content, requiring both visual and textual comprehension to answer questions accurately.
A.2 Comparison with MILMS with Multiple Vision Encoders
To better understand how HAWAII compares with the existing MLLMs with multiple vision en- coders [13
68), we present the comparison in Table Results show that HAWAII achieves competitive or significant improvements on most benchmarks, demonstrating the effectiveness of HAWAII. However, we also notice performance degradation on some benchmarks, such as POPE, GQA, and SeedBench.
24

---

VizWiz
GQA
SQA
POPE
MME
A12D
MMMU
SeedBench
Eagle-XS 47
54,4
64,9
69.8
88.8
1528
36.3
73.9
(CLIP DINO Pix2Struut)
MouSi
LLyoutLMn33DDNNo24CCLPPP
63.6
69.0
86.5
67.5
Brave 13
54.2
52.7
87.6
(CLIP-)5(CCIIP)
50.0
62.0
66.8
85.9
1510.7
55.5
34.7
66.1
MoVE-KD
52.3
63.2
69.4
86.9
1524.5
HAWAII
53.9
70.5
87.3
1540.2
56.2
36.6
67.5
HAWAII (HAWAII Pix2Struct)
54.2
63.6
69.5
86.8
1533.6
56.2
36.0
67.2
HAWAIE (HAWAII SAM)
54.3
63.3
70.8
87.5
1528.6
55.8
36.9
67.9
A
4.11
1.3
1
1.1.3
12.2
0.6
4.6.0
Table 6: Comparison with MLLMs with multiple vision encoders.
04
80
80
03
of Bc500Bencch
Chaice Caparsa TecsOA
Cholcer Exparts Vutila
0.4
0.0
2.50
Figure 6: Visualization of the routing choice using HAWAII-v1.00 .0. Best viewed in color.
A.3
Ablation Study
Routing between specific teachers' knowledge. To further understand how HAWAII switches between different teachers' knowledge, we visualize the routing results F6gure It is obvious that HAWAII selects different expert's knowledge across different benchmark datasets and different layers. notable observation for most of the cases, HAWAII does not choose CLIP for understanding visual contents. We observe that for MME, VizWiz, and SEEDBench, the model has similar selection preference, while for MMMU, model mainly choose CLIP and ConvNext.
25


Reviews:
[{"review_id": "i9uJAyYKmM", "reviewer": "Reviewer_7ZEx", "summary": "This  paper introduces HAWAII, a hierarchical visual knowledge distillation framework that transfer knowledge from multiple pre-trained vision foundation models such as ConvNeXt, EVA, Pix2Struct and SAM, into a single vision encoder to improve efficiency. HAWAII utilizes teacher-specific LoRA adapters and a mixture-of-LoRA-adapters module to align multiple teachers and student. In the while, a hierarchical knowledge distillation mechanism is proposed to combine global and local information to optimize knowledge transfer. Extensive experiments have demonstrated its superior performance across vision-language tasks compared to baselines.", "questions": "1. In Table 1, when adding teachers like Pix2Struct, performance on some tasks (e.g., VQA$^{Text}$) slightly decreases due to knowledge redundancy, but the paper does not quantify the optimal number of teachers to avoid such trade-offs.\n\n2. What is the optimal number of LoRA adapters for balancing performance and computation?\n\n3. How does HAWAII perform with domain-specific visual experts (e.g., medical or autonomous driving models)? Further, can the hierarchical distillation mechanism be applied to multi-modal knowledge beyond vision?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths \n- A good ides is proposed in this paper, i.e., hierarchical distillation: fine-grained (token-level) and coarse-grained (global consensus) strategies ensure comprehensive knowledge transfer.\n- Significantly improves on benchmarks like VizWiz (+7.8) and SQA (+5.5) compared to LLaVA-1.5 baseline.\n\n\n## Weakesnnes\n- While teacher-specific LoRA adapters aim to avoid conflicts, the paper does not explicitly analyze how adapters for different teachers interact during training, leaving uncertainty about potential interference effects.\n\n- The performance of HAWAII on transfer tasks (e.g., adapting from synthetic data to natural images) is unaddressed, limiting understanding of its generalization beyond training distributions.\n\n- The proposed HAWAII focuses solely on visual expert distillation and does not explore integrating knowledge from non-visual modalities (e.g., audio or text).", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments and valuable feedback. \nBelow, please find our specific responses (R) to weaknesses (W) and questions (Q).\n\n---\n\n> W1: How do adapters for different teachers interact during training.\n\nR: Thank you for the question. During training, we activate one LoRA expert at a time to get the specific student feature $I^{S}_{i}, \\forall i \\in [N_t]$ for fine-grained distillation (Sec. 2.3.2). \nAlso, for calculating coarse-grained distillation loss and text generation loss, only one teacher-specific LoRA adapter and one general-knowledge adapter are activated based on the routers (Eq. (2)). \nOverall, each teacher-specific LoRA adapter will be updated to align with its corresponding teacher independently in fine-grained distillation (Sec. 2.3.2), while coarse-grained distillation (Sec. 2.3.1) will update one teacher-specific LoRA adapter selected by the router. \nThat means they will not interfere with each other during training.\n\n---\n\n> W2: Generalization beyond training distributions.\n\nR: Thank you for the suggestion. Following previous works [1,2], in our experiments, we use LLaVA-1.5 pretraining and SFT data for training the model, which means we report zero-shot performance on most of the benchmark datasets. \nMeanwhile, we are also interested in having more evaluations on synthetic data. Please let us know if there are any relevant, valuable benchmarks we could test our models on.\n\n---\n\n> W3 and Q3: No exploration on non-visual distillation.\n\nR: Thank you for the comment. As mentioned in the limitations of our paper, we acknowledge that further improvements might be achieved by LLM distillation. \nIn this paper, we focus on visual distillation and propose novel coarse-grained and fine-grained distillation to better transfer knowledge from multiple visual experts into one unified visual encoder. \nWe will leave the combination of LLM and visual distillation as future work.\n\n---\n\n> Q1: Optimal number of visual teachers.\n\nR: In Table 1 of our paper and the following table, we did observe that under different settings of the teachers, the model performs differently on diverse benchmark datasets, which might be due to the diverse knowledge brought by teachers. \nThus, instead of finding an optimal number or set of visual teachers for all the diverse tasks, we suggest that users select the teachers that best suit the downstream tasks for better performance.\n\n|                               | VizWiz   | GQA      | SQA      | POPE     | MME        | AI2D     | SeedBench |\n|-------------------------------|----------|----------|----------|----------|------------|----------|-----------|\n| LLaVA-1.5 (CLIP as the visual encoder)              | 50       | 62       | 66.8     | 85.9     | 1510.7     | 55.5     | 66.1      |\n| HAWAII (Ours, CLIP as the visual encoder)                 | 53.9     | 62.8     | 70.5     | 87.3     | **1540.2** | **56.2**     | 67.5      |\n| HAWAII† (HAWAII + Pix2Struct as teachers) | 54.2     | 63.6     | 69.5     | 86.8     | 1533.6     | **56.2**     | 67.2      |\n| HAWAII‡ (HAWAII + SAM as teachers)        | **54.3** | **63.3** | **70.8** | **87.5** | 1528.6     | 55.8     | **67.9**  |\n\n---\n\n> Q2: Optimal number of LoRA adapters.\n\nR: The number of teacher-specific LoRA adapters is dependent on the number of visual teachers. \nAnd the number of general-knowledge LoRA adapters is determined by the users. \nWe have added an ablation on the number of general-knowledge LoRA adapters, as shown below\n\n| # of General-Knowledge adapters | VQA Text | GQA      | SQA      | POPE     | MME        | MMMU     | AI2D     | SeedBench |\n|--------------------------------|----------|----------|----------|----------|------------|----------|----------|-----------|\n| 1                              | **58.7** | 62.6     | 70.1     | 84.5     | 1516.2     | **37.0** | 55.5     | 67.4      |\n| 3 (Main Configuration)         | **58.7** | **62.8** | **70.5** | **87.3** | **1540.2** | 36.6     | **56.2** | **67.5**  |\n| 5                              | 58.6     | **62.8** | 70.4     | 85.2     | 1530.2     | 36.4     | 55.0       | 66.9      |\n\nIn our paper, we use 3 general-knowledge experts as the main setting for HAWAII. \nResults show that, the setting of 3 general-knowledge LoRA adapters outperforms other settings with 1 and 5 adapters on most of the benchmark datasets. \nWe also observe that on MMMU, increasing the number of general-knowledge LoRA adapters decreases performance, while on GQA, using 3 or 5 adapters works the best.\n\n---\n\n> Q3: Include domain-specific visual experts.\n\nR: Thank you for the suggestions. In our paper, following MoVE-KD [2] and Eagle [3], we use CLIP, EVA, and ConvNext as the basic settings of teachers, with Pix2Struct and SAM as additional teachers. \nThey serve different purposes: CLIP (general-purpose), ConvNeXt (image classification), Pix2Struct (OCR/document understanding), EVA-02 (multi-task generalist), and SAM (segmentation). \nThese teachers differ in architecture, training data, and task specialization, providing a broad and complementary set of capabilities. And this diversity is intended to balance generality with scalability. Such a diverse set of visual teachers brings comprehensive knowledge from different domains, benefiting the model. \nMoreover, we would like to explore some other choices for the visual experts, such as medical or driving models. \nPlease let us know if there are any specific models you would suggest.\n\n---\n\nReference:\n\n[1] Visual Instruction Tuning. NeurIPS 2023.\n\n[2] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025.\n\n[3] Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders. ICLR 2025."}, {"review_id": "9BfXHzrWek", "reviewer": "Reviewer_A5aZ", "summary": "This paper introduces a new framework to improve the efficiency and performance of VLMs, by distilling knowledge from multiple visual experts into a single vision encoder. The key innovation is the Mixture-of-LoRA-Adapters (MOLA) module, which includes teacher-specific and general-knowledge LoRA adapters. These adapters are selectively activated using sparse routers, allowing the model to learn from diverse teachers without noisy or conflicting guidance. HAWAII performs hierarchical knowledge distillation at both coarse and fine granularity. In fine-grained distillation, the model uses teacher-specific adapters and a novel token importance scoring mechanism to focus on the most informative visual tokens from each teacher. In coarse-grained distillation, it combines features from all teachers into a summarized representation, which is then transferred to the student encoder via general LoRA adapters. Extensive experiments show that HAWAII outperforms previous methods, including MoVE-KD, on various VLM benchmarks like VizWiz, SQA, and MMBench.", "questions": "1. How are the teacher models selected? \nThe paper lacks a clear explanation of the rationale behind selecting CLIP, ConvNeXt, and EVA-02 as the teacher models. It is unclear whether these models were chosen due to their complementary strengths, diverse training objectives, or empirical performance.\n2. Is knowledge distillation applied during both pretraining and fine-tuning stages?\nWhile the training pipeline is described, the paper does not clearly state whether the hierarchical knowledge distillation is performed in both the pretraining and SFT stages, or only in one.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written, logically structured, and easy to follow. The authors provide a detailed explanation of their method and offer comprehensive illustrations to aid understanding.\n2. The paper addresses a meaningful and practical problem in the vision-language domain—how to efficiently integrate knowledge from multiple visual experts into a single model. This is relevant to the community, especially for resource-constrained settings.\n\nWeaknesses:\n1. Limited Novelty. While the paper proposes using teacher-specific LoRA adapters and hierarchical knowledge distillation, these techniques are already widely used in other domains.\n2. Insufficient Experiments. The experimental validation lacks critical comparisons. In Table 1, there is no baseline that uses multiple visual encoders at inference time (e.g., model ensembles or token concatenation), which makes it difficult to quantify the effectiveness of the distillation process. Moreover, the ablation studies are limited. The authors should include comparisons showing how different vision experts (e.g., CLIP, SAM, Pix2Struct) impact the final student model’s performance when used individually or in combination.\n3. Outdated Baselines. Many of the compared models (e.g., BLIP-2, LLaVA-1.5) are early-stage VLMs. The paper does not compare against more recent and stronger models like Qwen-VL 2.5 and InternVL 2.5. Given that the method aims to enhance vision encoders via multi-teacher distillation, it would be important to test whether the proposed method can still improve state-of-the-art VLMs when more advanced vision backbones are used.", "was_revised": true, "final_justification": "The author's rebuttal has addressed my concerns, and I will raise the rating.", "rebuttal": "We appreciate your thoughtful comments and valuable feedback. \nBelow, please find our specific responses (R) to weaknesses (W) and questions (Q).\n\n---\n\n> W1: Novelty.\n\nR: Thank you for the comments. \nIn this paper, we propose HAWAII that distills knowledge from multiple visual experts into a single vision encoder. \nTo mitigate conflicts among different teachers and switch between different teacher-specific knowledge, we introduce teacher-specific LoRA adapters, each corresponding to one specific visual teacher. \nWe also include token importance scores to emphasize the most informative tokens from each teacher adaptively. \nMoreover, we also employ a set of general-knowledge LoRA adapters to learn the common knowledge among different teachers and training data. \n\nWe would like to clarify that although the term hierarchical knowledge distillation has been used in other domains (e.g., [3, 4]), the underlying techniques differ significantly. \nTheir methods focus on aligning features extracted from the hierarchical architecture of one teacher, such as features from the shallow layers to the deep layers. \nOn the other side, our proposed method, HAWAII, treats hierarchical knowledge distillation as fine-grained (distilling knowledge from one teacher only for solving conflicts between teachers) and coarse-grained knowledge distillation (distilling knowledge from the summarized knowledge of multiple teachers), which is totally different from those methods in other domains.\n\nTo the best of our knowledge, we are the first to use two sets of LoRA adapters to distill knowledge from multiple visual experts into one visual encoder. We might miss some references in relevant areas. Please let us know if there are any works that have a similar design of two sets of adapters for distilling knowledge. \n\n---\n\n> W2: No baseline on models with multiple visual encoders. \n\nR: \nWhile we appreciate the importance of such baselines with multiple experts [1], our method fundamentally differs in design: we distill knowledge to a single unified vision encoder, rather than maintaining multiple experts during both training and inference. \nDirectly comparing these two paradigms would therefore not be fair, as the former with multiple experts typically involves higher computational cost and a larger number of parameters. We have collected the latency of LLaVA-1.5-7B, Eagle-X4-Plus (with four visual experts), and our HAWAII-7B as shown below. We use Pytorch Profiler to measure the FLOPs and CUDA time using the teaser figure from the llava paper and “What is shown in this image?” as input. We use one A6000 GPU with 48 GB GPU memory to inference the models.\n|                  | FLOPs (T) | CUDA Time (s) |\n|------------------|-----------|---------------|\n| LLaVA-1.5-7B     | 8.9       | 1.2           |\n| Eagle-X4-Plus    | 29.4      | 2.1           |\n| HAWAII-7B (ours) | 9.3       | 1.3           |\n\nIt shows that, our HAWAII, which only uses one vision encoder, has smaller FLOPs and CUDA times compared to the Eagle model, which contains multiple (4-5) visual experts. \nWe will explicitly highlight this distinction in the paper and include a discussion in the final version.\n\n---\n\n> W2: No ablations on how different vision experts affect performance. \n\nR: Thank you for the comment. We have presented an ablation study on different settings of visual teachers in Table 1 of our paper. It shows that having more visual teachers improves performance, and due to the heterogeneous knowledge in different visual teachers, the performance boost in different benchmark datasets varies. \nFor performance using different visual experts [1], we have included the following table for your reference.\n|                               | VizWiz   | GQA      | SQA      | POPE     | MME        | AI2D     | SeedBench |\n|-------------------------------|----------|----------|----------|----------|------------|----------|-----------|\n| LLaVA-1.5 (CLIP as the visual encoder)              | 50       | 62       | 66.8     | 85.9     | 1510.7     | 55.5     | 66.1      |\n| LLaVA-1.5* (SAM as the visual encoder)              | 49       | 57.3     | 66.8     | 84.3     | 1216       | **69.2** | 56.9      |\n| HAWAII (Ours, CLIP as the visual encoder)                 | 53.9     | 62.8     | 70.5     | 87.3     | **1540.2** | 56.2     | 67.5      |\n| HAWAII† (HAWAII + Pix2Struct as teachers) | 54.2     | 63.6     | 69.5     | 86.8     | 1533.6     | 56.2     | 67.2      |\n| HAWAII‡ (HAWAII + SAM as teachers)        | **54.3** | **63.3** | **70.8** | **87.5** | 1528.6     | 55.8     | **67.9**  |\n\nOur HAWAII uses CLIP as the student encoder with CLIP, ConvNext, and EVA being the teachers. Our HAWAII achieves best performance on all the benchmark datasets except AI2D. *represent results from [1]. \nWe will include a more detailed analysis in the final version.\n\n---\n> W3: Apply HAWAII to more methods.\n\nR: Thank you for the suggestion. Following MoVE-KD [2], which serves as the most relevant and established knowledge distillation baseline for our model, we use LLaVA-1.5 as our base model to ensure a fair comparison. Now, we are employing our methods to LLaVA-Next. The experimental results will be updated in this thread in the following days. \nMoreover, we are also planning to use Qwen-2.5-VL as an additional base model. The results will be added to the final version of our paper.\n\n---\n\n> Q1: How are the teacher models selected?\n\nR: Thank you for the question. We select CLIP, ConvNeXt, and EVA-02 as our main design, following MoVE-KD [2], which serves as the most relevant and established knowledge distillation baseline for our model to ensure a fair comparison. We also include Pix2Struct and SAM for analyzing the impact of different visual experts, as shown in Table 1. \n\nMoreover, they serve different purposes: CLIP (general-purpose), ConvNeXt (image classification), Pix2Struct (OCR/document understanding), EVA-02 (multi-task generalist), and SAM (segmentation). These teachers differ in architecture, training data, and task specialization, providing a broad and complementary set of capabilities. And this diversity is intended to balance generality with scalability. Such a diverse set of visual teachers brings comprehensive knowledge from different domains, benefiting the model. We will clarify this in our paper.\n\n---\n\n> Q2: Is knowledge distillation applied during both pretraining and fine-tuning stages?\n\nR: Yes. It is applied in both pretraining and supervised fine-tuning stages. \nWe will clarify this in the paper. \n\n---\n\nReference:\n\n[1] Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders. ICLR 2025.\n\n[2] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025.\n\n[3] SignKD: Multi-modal Hierarchical Knowledge Distillation for Continuous Sign Language Recognition.\n\n[4] Hierarchical Self-supervised Augmented Knowledge Distillation. arXiv:2107.13715."}, {"review_id": "6fmWAGsc3P", "reviewer": "Reviewer_p3zt", "summary": "HAWAII is a framework that distills multiple pretrained visual experts into a single vision encoder, capturing their complementary strengths without incurring high computational costs. Instead of relying on a fixed set of adapters for each teacher, HAWAII assigns a dedicated LoRA adapter to each expert and uses a router to switch between them, preventing noisy or conflicting guidance during distillation.", "questions": "Refer to Weaknesses", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghts:\n\n- Multi-Teacher Knowledge Distillation: HAWAII effectively distills knowledge from multiple pretrained visual experts into a single student encoder, ensuring comprehensive learning from diverse sources.\n\n- Mixture-of-LoRA Adapter (MOLA): The novel MOLA module incorporates both teacher-specific and general-knowledge LoRA adapters, enabling the student encoder to absorb specialized expertise while retaining broad, task-agnostic representations.\n\n- Hierarchical Knowledge Distillation (HKD): HKD operates at two levels—coarse-grained distillation aggregates global knowledge from all teachers, while fine-grained distillation selects the most informative tokens from each teacher—resulting in richer, more targeted guidance.\n\nWeaknesses:\n\nI feel it is great work, but there are some four concerns. First one is that the authors only use LLaVA-1.5 as baseline to apply your method, but this paper could be improved by using multiple recent released models such as Qwen2/2.5-VL and InternVL2.5/3 or the models you are using for evaluation baselines. The second is that there are none of challening benchmarks like MM-Vet, MM-Vet-v2, and MMMU-Pro. Third is that I am wondering why the authors dealt with only LoRA adapter although there are various adapters based on DoRA, VeRA, OFT, LoKr, LoHa, etc. The last thing is that the possibility for LLM-distillation, because the authors only deal with distillation of visual properties but it may be naturally wondering that why didn't the author consider LLM-part distillation.", "was_revised": true, "final_justification": "I support this paper's acceptance because the authors did a really great in having a rebuttal.", "rebuttal": "We appreciate your thoughtful comments and valuable feedback. \nBelow, please find our specific responses (R) to weaknesses (W) and questions (Q).\n\n---\n\n> W1: Apply HAWAII to more methods.\n\nR: Thank you for the suggestion. Following MoVE-KD [1], which serves as the most relevant and established knowledge distillation baseline for our model, we use LLaVA-1.5 as our base model to ensure a fair comparison. Now, we are employing our methods to LLaVA-Next. The experimental results will be updated in this thread in the following days. \nMoreover, we are also planning to use Qwen-2.5-VL as an additional base model. The results will be added to the final version of our paper.\n\n\n---\n\n> W2: Evaluation on challenging benchmarks like MM-Vet, MM-Vet-v2, and MMMU-Pro.\n\nR: Thank you for the suggestion. We have added evaluation on MM-Vet, MMVet-V2, and MMMU-Pro using lmms-eval. It shows that our model HAWAII outperforms LLaVA-1.5-7B on MMMU-Pro, MMVet, and MMVet-2.\n|              | MMMU_pro_standard | MMMU_pro_vision | MMVet | MMVet-2 |\n|--------------|-------------------|-----------------|-------|---------|\n| LLaVA-1.5-7B | 20.3              | 12.1            | 31.1  | 28.3    |\n| HAWAII       | 21.5              | 12.6            | 33.0  | 29.4    |\n\n\nWe will test all the models in our paper on these three datasets and the results will be added to the final version. Please let us know if there are any other valuable benchmark datasets. We are more than happy to test our model and report the results.\n\n\n---\n\n> W3: Why not other LoRA adapters?\n\nR: Thank you for the question. We use LoRA in our design because of its generalizability.\nWe have conducted an experiment using DoRA as shown below.\n|                  | VQA Text | GQA      | SQA      | POPE     | MME        | MMMU     | AI2D     | SeedBench |\n|------------------|----------|----------|----------|----------|------------|----------|----------|-----------|\n| LLaVA-1.5-7B     | 58.2     | 62       | 66.8     | 85.9     | 1510.7     | 34.7     | 55.5     | 66.1      |\n| MoVE-KD-7B       | 58.3     | **63.2** | 69.3     | 86.9     | 1524.5     | 35.3     | 55.6     | 66.5      |\n| HAWAII-7B (LoRA) | **58.7** | 62.8     | **70.5** | 87.3 | 1540.2 | **36.6** | **56.2** | 67.5  |\n| HAWAII-7B (DoRA) | 58.4     | 61.8     | 69.3     | **87.7**     | **1558.5**     | 35.2     | 55.2     | **67.8**      |\n\nThe results show that, DoRA has comparable performance on most of the benchmark datasets compared with LoRA. It outperforms LoRA on POPE, MME, and AI2D with worse performance on other benchmarks. \nWe will add the experiments with more LoRA adapters for testing the scalability of our methods in the final version.\n\n\n---\n\n> W4: Exploration on LLM distillation.\n\nR: Thank you for the comment. As mentioned in the limitations of our paper, we acknowledge that further improvements might be achieved by LLM distillation. \nIn this paper, we focus on visual distillation and propose novel coarse-grained and fine-grained distillation for better transferring knowledge from multiple visual experts into one unified visual encoder. \nWe will leave the combination of LLM and visual distillation as future work.\n\n---\nReference: \n\n[1] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025."}, {"review_id": "kLwhWv9EKQ", "reviewer": "Reviewer_XygR", "summary": "The article introduced a novel framework that distills knowledge from multiple visual experts into a single vision encoder, enabling it to inherit the complementary strengths of several experts with minimal computational overhead.", "questions": "1. I noticed that you have chosen a rank of 16 for your LoRA adapters.  Have you experimented with higher ranks, such as 32, to see if there are any additional performance improvements? Because the MoVE-KD uses the rank of 32.\n2. How long does training take?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：HAWAII proposes a hierarchical knowledge distillation (HKD) mechanism that includes both coarse-grained and fine-grained distillation. This allows for a more nuanced transfer of knowledge from multiple teachers to a single student encoder. In contrast, MoVE-KD primarily focuses on a single level of distillation, which may not capture the full spectrum of knowledge as effectively.\nHAWAII introduces the MOLA module, which consists of teacher-specific LoRA adapters and general-knowledge LoRA adapters. This design enables the student encoder to learn from diverse teachers separately (fine-grained) and globally (coarse-grained), avoiding noisy and redundant knowledge transfer.\n\nWeakness：The experiments in the \"HAWAII\" paper are primarily conducted on the 7B LLaVA-1.5 model. There is a lack of validation on models of different sizes, both larger and smaller than 7B. This limits the generalizability of the findings.  The effectiveness of the HAWAII framework in enhancing visual understanding might vary across different model architectures. Testing on a wider range of models, such as LLaVA-NeXT, would provide a more comprehensive evaluation of the framework's robustness and applicability. (I think this is a very core experiment and cannot be simply summarized in Limitations.)", "was_revised": true, "final_justification": "The authors have provided additional responses and clarifications to all four of my concerns, which effectively address my questions and further validate the proposed method.\nAs a result, I have increased my score to 5 (accept). The key improvements include:\n﻿\n* Providing empirical validation with the larger models, which demonstrates the scalability and robustness of the proposed approach in more complex settings.\n﻿\n* Extending the evaluation to a broader range of models, such as LLaVA-Next, thereby supporting the generalizability of the method.\n﻿\n* Offering detailed clarification on the choice and impact of hyperparameters.\n﻿\n* Supplementing information regarding the training time and computational resources involved.\n\nThese thoughtful additions significantly strengthen the paper.", "rebuttal": "We appreciate your thoughtful comments and valuable feedback. \nBelow, please find our specific responses (R) to weaknesses (W) and questions (Q).\n\n---\n\n> W1: Only test on 7B model.\n\nR: Thank you for the comment. We have added the experimental results on the LLaVA-1.5-13B model as shown below.\n|               | VQA Text | GQA      | SQA    | POPE     | MME        | MMMU | AI2D | SeedBench |\n|---------------|----------|----------|--------|----------|------------|------|------|-----------|\n| LLaVA-1.5-13B | 61.3     | 63.3     | 71.6   | 85.9     | 1531.3     | 35.5 | 59.3 | 68.2      |\n| MoVE-KD-13B   | 59.7     | 64.2     | 73.2   | 85.7     | 1568.1     | -    | -    | -         |\n| **HAWAII-13B (Ours)**    | **61.7** | **64.7** | **75.0** | **86.6** | **1568.7** | **35.7** | **60**   | **68.5**      |\n\nOverall, HAWAII-13B (with CLIP, EVA, and ConvNext as teachers) achieves best performance on all the benchmark datasets, compared to LLaVA-1.5-13B and MoVE-KD-13B [1], which are the two most relevant baselines in our methods. We will add experimental results using smaller models in our final version.\n\n---\n\n> W2: Testing on a wider range of models.\n\nR: Thank you for the suggestion. Following MoVE-KD [1], which serves as the most relevant and established knowledge distillation baseline for our model, we use LLaVA-1.5 as our base model to ensure a fair comparison. Now, we are employing our methods to LLaVA-Next. The experimental results will be updated in this thread in the following days. \nMoreover, we are also planning to use Qwen-2.5-VL as an additional base model. The results will be added to the final version of our paper.\n\n---\n\n> Q1: Why rank 16 for LoRA?\n\nR: We use 16 for training efficiency. Due to time limitation, we could not complete the experiments with the LoRA rank of 32. But the corresponding results will be added to the final version for a fairer comparison to MoVE-KD [1]. \n\n---\n\n> Q2: How long does training take?\n\nR: We used 8 A6000 with 48G GPU memory for each. It takes 20 hours for pretraining and 48 hours for supervised fine-tuning on the 7B model. We also tested on 8 A100 with 80G GPU memory for each. It is much faster, with 9 hours for pretraining and 20 hours for fine-tuning. We will add these training details in the final version.\n\n---\n\n[1] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025."}]

Author rebuttal:

Meta-review: The paper presents a practical and well-executed approach to multi-expert visual distillation, featuring clear architectural choices and hierarchical objectives that yield consistent gains across benchmarks and model sizes, while maintaining efficiency relative to multi-encoder systems. Novelty is moderate, and some evaluations are pending (additional base models and LoRA rank sweep); however, the rebuttal added substantial evidence, addressed key methodological questions, and provided efficiency comparisons.
Decision: Accept (poster)