CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs
Authors: Jiwan Kim
Kibum Kim
Sangwoo Seo
Chanyoung Park
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Model
Knowledege Distillation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as a solution to their high computational complexity, making them more practical for real-world applications. In this regard, the knowledge distillation (KD) approach has emerged as a promising alternative, which transfers the rich visual and linguistic knowledge from a larger model (teacher) to a smaller model (student). However, we observe that existing KD methods struggle to effectively distill the teacher MLLM's rich visual perception abilities to the student, a challenge that has been largely overlooked in previous studies. Through a systematic analysis, we identify visual attention misalignment between student and teacher as the main cause of this issue. Based on this insight, we propose CompoDistill, a novel KD framework that explicitly aligns the student's visual attention with that of the teacher to enhance the student's visual perception abilities. Our extensive experiments show that CompoDistill significantly improves performance on compositional reasoning tasks that require visual perception abilities while maintaining strong performance on visual question answering tasks, as done in existing studies. Furthermore, CompoDistill demonstrates effectiveness with a more advanced backbone, highlighting its generalizability.

Published as a conference paper at ICLR 2026
COMPODISTILL: ATTENTION DISTILLATION FOR COMPOSITIONAL REASONING IN MULTIMODAL LLMS
Jiwan Kim
Kibum Kim
Sangwoo Seo
Chanyoung Park
1 KAIST
ABSTRACT
Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as solution their high computational complexity, making them more practical for real-world applications. In this regard, the knowledge dis- tillation (KD) approach has emerged the rich visual and linguistic knowledge from larger model (teacher) to a smaller model (student). However, we observe that existing KD methods struggle to ef- fectively distill the teacher MLLM's ss rich visual perception abilities to the student, a challenge that has been largely overlooked in previous studies. Through a sys- identify visual attention misalignment between student and
promising alternative, which transfers
tematic analysis, teacher as the main cause of this issue. Based on this insight, we propose Com- poDistill, a novel KD framework that explicitly aligns the student's ' visual atten- tion with that of the teacher to enhance the student's visual perception abilities. Our extensive experiments show that CompoDistill significantly improves per- formance compositional reasoning tasks that require visual perception abilities while maintaining strong performance on visual question answering tasks, as done in existing studies. Furthermore, CompoDistill demonstrates effectiveness with a more advanced backhone, highlighting its generalizability.
0.74
"A the table." "
0.72
No Improycmemnt monnnddd CR
Student (LLaVA-KD-28)
Teacher (LLaVA-4B)
0.70
Conpurret
M
Compa *
0.66
0.64
LLW
0.62
Madee
LLWVRDI
VQA Performance !
Performance
(a) Comparison methods
(b) attention misalignment
Figure 1: (a) Comparison of the teacher (LLaVA-4B), SFT (LLaVA-2B), and various 2B KD meth- ods distilled from the same teacher. (b) Visual attention misalignment between the student VA- KD-2B) and teacher (LLaVA--4)) where the student attends to irrelevant image regions for the text query A woman is the table, in contrast to the teacher. For more examples including our proposed method, CompoDistill, please refer to Appendix A
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Liu et al., 2023 Lin et al. 2024b) have demon- strated superior performance various vision-language tasks over vision- Langualle models (Rad-] ford et al. 2021 Zhai et al.] 2023), marking a significant step forward in multimodal learning. Yet, their advancement has largely been driven by scaling laws, which has resulted in ever-larger and more computationally demanding architectures [OpenAl 2023 Owen et al., (025).. The substantial computational and memory costs associated with such massive models pose considerable challenges for practical deployment. This has, in turn, intensified interest in developing efficient MLLMs that can maintain strong multimodal capabilities while mitigating resource requirements.
"Corresponding author

---

Published as a conference paper at ICLR 2026
To this end, knowledge distillation (KD) CCai et al. 2024 Shu et al. 2024 Xu al. 2024 has
emerged as a promising approach, transferring rich knowledge from a larger leacher model to a smaller student model. These KD-based methods have demonstrated effectiveness in visual question answering (VQA) tasks (e.g., GQA (Hudson & Manning, 2019)) which require visual recognition ability, outperforming the standard supervised fine-tuning (SFT mode in Figure (a)).
Despite the progress of KD methods in visual recognition ability, a crucial question remains unex- plored: Is visual perception ability equally well distilled? Visual recognition refers to tasks like object classification, where the goal is to identify objects in images based on their features. On the other hand, Visual perception involves more complex abilities, such as understanding relationships among objects and accurately capturing their attributes capabilities essential for real-world mul- timodal applications To answer the question. in Figure 1 [a), we compare the state-of-the-art KD methods (Shu et al. 2024 Cai et al., 2024 Xu et al. 2024) on VQA and compositional reasoning (CR) datasets, where VQA serves as a benchmark for evaluating visual recognition ability and CR is designed assess visual perception ability[] We observe that, although existing KD methods show significant performance improvements on the VQA, their performance on the CR is on par with the SFT model, which is unexpected. Based on this observation, we argue that existing KD methods struggle to effectively distill visual perception ability from the teacher model.
Beyond a simple performance comparison, we investigate the failure of the existing KD methods in distilling visual perception ability and analyze the attention maps of both the student (i.e., LLaVA- KD-2B (Cai et al. (024g) and the teacher (i.e., LLaVA-4B (Liu et al.] 2023)) models. Specifically, by visualizing their focus areas for given text, we aim to identify differences in attention, which we believe are closely related to perception ability. As shown in Figure [bb,, we observe that the student model struggles to focus on the relevant regions, while the teacher model captures them ef- fectively, revealing a clear mismatch in the attention distributions between the teacher and the student models. The discrepancy in attention distributions, which we term visual attention misalignment, reveals that although KD methods aim to transfer knowledge from the teacher to the student through knowledge distillation, the student fails inherit the teacher's 's powerful visual attention mechanism, thereby limiting the effective distillation of visual perception ability.
In this regard, we hypothesize that the limited performance improvement of KD methods in CR tasks arises from the visual attention misalignment between the teacher and student models. To empirically validate our hypothesis, we conduct controlled experiments examining the relationship between attention behavior and visual task performance in Section 3 Our results demonstrate that this misalignment is directly responsible for the unexpectedly low performance of the existing KD methods in CR tasks, and this study is the first to establish that addressing visual attention misalign- ment is the key to enhancing the student model's visual perception ability.
Motivated by these findings, we propose Compl novel KD framework aimed at effectively distilling the teacher's rich visual perception abilities to the student model by addressing visual attention misalignment. We introduce the Visual ATtention alignment (VAT) module to explicit align the visual attention of the student model with that of the teacher model, using a simple yet effective group matching strategy, in which each student layer is matched with a group of teacher layers in a one-to-many manner, to handle the difference in the number of LLM layers between the teacher and student. However, the teacher's 's visual attention is optimized for its own vision-language space, making a simple transfer via the VAT module ineffeetive within the student's distinct and incompatible space. This mismatch creates conflict between the student's feature space and the imposed attention mechanism, ultimately restricting the model's inherent perceptual abilities. To this end, we propose the Teacher Adapter Fetch (TAF) module to bridge this feature space gap and enable synergy with the VAT module. Building on this, introduce meticulously designed three- stage training strategy that leverages both modules to comprehensively distill visual perception.
Through extensive experiments on multiple CR datasets, we demonstrate that CompoDistill sig- nificantly outperforms existing KD methods, demonstrating the effectiveness of CompoDistill in enhancing the student model's visual perception ability. Meanwhile, DotpoDistill maintains com-
'Sharing the same architecture and size as the student models in KD. this model is trained only with visual instruction uning Liu et al. (2023) via supervised fine-tuning. We henceforth denote it as the SFT model.
RRgardiing a detailed comparison of the differences between the two abilities, please refer to Appendix B
VVQ datasets include GQA (Hudson & Manning 2019). TextVQA (Singh al. 2019 and MME (Fu
al.
2024). and CR datasets include SugarCrepe (Hsich al. 2023). SADE (Ma et al. 2023). BiVLC (Mi-
randa et al. 2024). and Winoground (Thrush et al. 2022). Results Figure [[a) correspond to the average
performance across the datasets.
2

---

Published as a conference paper at ICLR 2026
petitive performance VQA datasets, preserving its visual recognition abilities. Furthermore, demonstrate the effectiveness of CompoDistill with a more advanced backbone, highlighting its generalizability.
We summarize our contributions as follows: (1) We identify, for the first time, that existing KD methods in MLLMs fail to distill visual perception ability from the teacher and provide a detailed attention-based analysis, offering insights on how to enhance this ability. (2) We propose Com- poDistill, novel KD framework that transfers both visual recognition and perception abilities through the Visual ATtention alignment and the Teacher Adapter Fetch Module. (3) We achieve significant improvements in CR tasks by effectively distilling the visual perception ability, while maintaining competitive performance in VQA tasks by preserving the visual recognition ability.
2 PRELIMINARY
Multimodal Large Language Models (MLLMs). Following the LLaVA (Liu et 2023) design, MLLMs consist of three main components: a pre-trained LLM (., a vision encoder ()),, and an adapter P. (.). Given a text query Q and an image E IR "x*xxx x3, where H and W denote the image height and width, the vision encoder extracts patch-level features Zp = (J) E RNN. Xd,, with NN being the number of visual patches and dp the hidden dimension of V The adapter projects them into the language space as x, Pas (zp)) E R.x. where d is the hidden dimension of LM Hereafter, we refer to X. as the visual tokens. Meanwhile, the query Q is tokenized into embeddings
X, Ndd where N, is the number of text tokens. The combined sequence [X,, xx] E IR (N++NNN)
is processed by the LM to compute the probability of the target answer as:
p(YLK) )== ply: Xe, Yes),
(1)
where y-i is the sequence of the answer tokens up to the i-th token and K is the answer length. During training, the cross-entropy loss derived from Equation is minimized, denoted as LLM.
WHY Is THE VISUAL PERCEPTION ABILITY NOT DISTILLED PROPERLY?
In Section D we noted that existing KD methods, despite improvements in terms of VQA, show unexpected results on compositional reasoning (CR) tasks, which we attribute to visual attention misalignment. This section provides an in-depth analysis elucidate the underlying causes of this issue. Our analysis first identifies the key factor for distilling visual ability through an attention-based analysis (Section B.I). then demonstrates this factor's direct impact on performance (Section 3.2). Finally, we validate that alleviating the visual attention misalignment associated with this key factor facilitates more effective transfer of visual perception (Section 3.3).
3.1 IDENTIFYING THE KEY FACTOR: TEACHER-SS STUDENT ATTENTION SIMILARITY OVER VISUAL TOKENS IN THE VISUAL UNDERSTANDING LAYER
In this section, we aim to identify critical factor that can serve as criterion for successfully distilling visual abilities (i.e., visual recognition and perception). To achieve this, we examine the layer-wise functions of the LLM within MLLMs, building on insights from prior works (Chen et al.] 2025a
Yoon et al. 2025). This is followed by dreettion-bssed analysis to pinpoint the key factor.
Layer-wiss Functionality in MLLMs. Following previous studies (Chen et al., 2025a Yoon et al.) 2025). we adopt a layer wise view of MLLM processing: Early layers align heterogeneous modali- ties with the LLM's text space, Intermediate layers integrate these signals for fine-grained semantic understanding. and Later layers generate the final response. Our focus is on the intermediate layer which we term the visual understanding layers, since these layers play a critical role in forming foundational visual reasoning and comprehension (Chen et al. (2255)) which are closely tied to the visual abilities (recognition and perception) central to study.
Attention Analysis for VQA and CR Tasks. Building on our focus on the visual understanding
layers, we design an experiment to investigate if teacher-student attention alignment can explain the difference in the degree of performance improvement between VQA and CR tasks, as observed in Figure [ [). To this end, we compare the attention distribution of the teacher model (i.e., LLaVA- 4B (Liu et al. 2023) with that of two distinct models: student model distilled directly from the
*Following (No et al. 2025 intermediate layers refer the 30-70% range of the total LLM layers.
3

---

Published as a conference paper at ICLR 2026
teacher (i.e., LLaVA- (Cai et al. 2024)) and SFT model (i.e., LLaVA-2B) that is architec- turally identical to the student model but trained without distillation.
As shown in Figure 2(a), we measure the attention similarity to quantify the aligmment between given model (student SFT) and the teacher. This is computed as the cosine similarity of their attention distributions, where the answer token y, is the query (Q) and the visual tokens
(xv) or text tokens (x+) are the key (K). Our primary analysis centers on the attention distribu- tions over visual tokens during answer generation, as these are critical for the visual abilities be- ing studied. For a more comprehensive comparison, we also investigate attention distribution over text tokens. We analyze the attention similarity of the student and SFT models the teacher. Specifically, we investigate how these similarities in attention behavior are re- lated with each model's performance in VQA and CR task, which brings us closer identifying the key factor. To guide this investigation, we formulate two key research questions:
de Mesiss Seneedion www.edeinnnnnniaee cerebirty
Q1) Where do the performance gains of KD on VQA (visual recognition) tasks originate? For VQA, where the student model demonstrates clear per- formance improvements, Figure (b) shows that, within the visual under- standing layers, the teacher-student at- tention similarity over visual tokens is significantly higher than that between the teacher-SFT. However, no such im- provement is observed for text tokens (Figure d)). This finding brings us identify the clue of the key factor: if the teacher-student vi- sual attention similarity in the visual understanding layers is high, distillation is effectively achieved, resulting in performance improvement of the student.
(23 Minualy /Tent attestion
(4)
Figure 2: (a) Attention of the answer token over visual tokens and text tokens during its generation. The transparency indicates the degree of attention the answer token gives to these tokens. (b- e) Layer-wise Teacher-Student and Teacher-SFT atlention sim- ilarities over visual tokens ((b), (c)) and text tokens ((d). (e)). GQA is used for VQA ((b), (d)), and SugarCrepe is used for CR ((c). (e)). Results on other datasets are shown Appendix C
Q2) Why does KD fail to deliver comparable improvements on CR (visual perception) tasks? We extend our analysis to CR tasks, where the student model performs on par with the SFT model. As shown in Figure c) and (e), we observe that the teacher-student attention similarity is comparable to the teacher-SIT attention similarity in the visual understanding layers, even over the visual to- kens, which contrasts with our observation on VQA. That is, the visual understanding layers, the student model shows no better alignment with the teacher model in terms of the attention over vi- sual tokens than the SFT model, despite such alignment being crucial for success in the downstream performance as shown in the case of VQA. Drawing on the findings from Q1 and Q2, we argue that the inability of existing KD methods to effectively distill visual perception ability stems from the moderate level of stucher-student attention similarity in the visual understanding layers.
3.2 RELATIONSHIP BETWEEN TEACHER-STUDENT ATTENTION SIMILARITY AND DOWNSTREAM PERFORMANCE
While the stadhet-student attention similarity over visual to- kens appears to be key factor in determining whether visual abilities have been distilled, its direct relationship with down-
Annwer Token Probabiitty
0.7
0.65
0.6
0.35
0.40
stream performance remains unclear. In other words, it is un- certain whether the higher teacher-student attention similar- ity (green line), relative to the teacher-SPF SFT attention similarity (yellow line) in the visual understanding layers (Figure 2b)), explains the superior performance of the student model on the VQA dataset. To this end, we quantify the relationship between the teacher-student visual attention similarity and the answer token probability given in Equation on the VQA dataset (Figure 3). where visual recognition ability has been successfully distilled from the teacher model. More precisely, we randomly sampled 5,000 instances from the GQA test set, grouped them according to the student-teacher similarity over vi- sual tokens during the answer generation step, and then measured the average probability assigned to the ground-truth answers. The results reveal a clear positive trend: higher attention similarity is
0.3
0.27
0.2
0.4
0.5
0.6
0.7
0.8
Visual Attention Simitarity
Figure 3: Relationship between atten- tion similarity and performance the GQA (VQA) dataset.
4

---

Published as a conference paper at ICLR 2026
consistently associated with higher answer probabilities, providing direct evidence that aligning the student's attention with the teacher's s attention is a key factor driving better performance on VQA.
3.3 A SIMPLE SOLUTION: REPLACING STUDENT'S VISUAL ATTENTION WITH TEACHER
Is Teacher Attention Really Beneficial? A natural ques- tion arises: Does increasing the leacher-student atlention similarity over visual tokens -a key factor for VQA per- formance-ales enhunce performance on CR tasks? To test this, we perform a direct intervention during inference on CR. Before generating the answer, we substitute the stu- dent's original attention over visual tokens with the average of the teacher's and student's ' attention, thereby increasing the student's ss attention similarity to the teacher (Figure 4 This yields modest but consistent performance gains in the Sug- arCrepe dataset (Swap. Replace and Add are three types of CR sub-tasks in the dataset). Although small, these improve- ments confirm that the student benefits from incorporating the teacher's s attention, reinforcing our hypothesis that attention alignment plays a crucial role in effective knowledge transfer.
0.50
0.94 0.37
0.85
0:11
3
0075
purferrd 1 0.45
0.70
0.42
0.80
LLAWWDD
0.50
Swap
Replace
Add
Evaluation Type
Figure 4: Change in performance the student attention substituted with the teacher attention on the SugarCrepe (CR) dataset.
(5
In summary, our analysis identifies visual attention misalignment the key barrier effectively distilling the teacher's perception abilities to the student, highlighting the crucial need to accurately transfer the teacher's s visual attention.
4 METHODOLOGY: COMPODISTILL
Motivated by the findings in Section 3 our framework, called CompoDistill, is designed to effec- tively transfer the teacher's 's visual perception abilities by addressing the visual attention misalign- ment. We first introduce our two core components: the Visual ATtention alignment (VAT) module (Section[4.1 which aligns the student's 's visual attention mechanism with that of the teacher, and the Teacher Adapter Fetch (TAF) module (Section 4.2) which ensures the student processes visual space consistently with the teacher. These modules are then integrated into a meticulously designed three-stage training strategy (Section 4.3) The overall framework is shown in Figure 5
4.1 VISUAL ATTENTION ALIGNMENT (VAT) MODULE
Attention Distillation Loss. To transfer the teacher's 's attention mechanism to the student, we uti- lize the attention matrix of the Transformer (Vaswani et al. (017) within the LLM layer, which represents the importance of each token relative to other tokens. The attention matrix for the input tokens (i.e., Xa and xx) is computed as: A softmax (QK" /Va) E (NN N.), where
Q (N) and K E NN), are the query and key matrices derived from the input tokens, respectively. From the overall attention matrix A, as discussed in Section 3 our goal is to distill the attention over visual tokens from the teacher to the student in the visual understanding lay- ers, leading to extract sub-matrix À A that focuses visual attention. Specifically, for each visual understanding layer I, this sub-matrix includes only the columns (keys) of A corresponding to the visual tokens, resulting in A, = As[;; Nol E [NN++ N Based on the visual attention- related sub-matrices of the teacher (A)) and student (Åf), we compute the cosine distance between them for the attention distillation loss as sim AA., Ař..). where 4 and le denote the index of the visual understanding layers of the teacher and student, respectively.
Group Layer Matching, However, since the teacher has more layers than the student, directly matching a student layer index I, with a teacher layer index 4 is challenging. A naive solution to map each to one by uniformly sampling teacher layers according to the ratio of their depths. For example, if the student has 5 layers and the teacher has 10 layers, then teacher layers (1,3,5, 5, 7, 9} are selected and aligned with the 5 student layers. However, this approach is subop- timal, as it overlooks the richer perception abilities distributed across the teacher's s layers and does
We initially attempted to completely replace the student's attention with that of the teacher. However, this rather led slight performance degradation compared with vanilla LLaVA-KD, which we attribute to mismatch between the student's 's feature space and the teacher's 's attention.
5

---

Published as a conference paper at ICLR 2026
not ensure accurate alignmen Instead, we propose simple yet effective Group Layer Matching strategy, where each I, is aligned with a group of {4:) in a one-to-many manner, enabling the student to capture broader teacher knowledge while roughly preserving the layer order.
Formally, let the sequence of student's visual understanding layer indices be Lg (':.-.... -....-..1
. and that of the teacher be LT = (4/ ,..,......... .,,,),,,4")) where the j-th layer in the visual understanding layers, and & and m denote the total number of visual understanding layers for the student and teacher, respectively, with & < m. For each student layer 12. we define corresponding group of teacher layers, Gj. consisting of . consecutive teacher layers, formed using sliding window approach. For example, the group of teachers 11 assigned for the first student layer l2 could be {4, l2 .?, J. while the group G2 for the second student layer l would be {iZ, ,?....,,? 2 2 and on. So, the total number of groups is the same as the number of student's target layers, k.
To distill the visual attention of a teacher group Gj into stu- dent layer Lg, we formulate the objective by minimizing the co- sine distance between the attention matrix of student layer, A and the averaged attention matrices of its corresponding teacher group, Aj. as follows:
ALM
CxL
Dacoder
LUN Layoe
LUN Decoder Layel
LLM Decoder Layer
LADL 1 A Σ sim (A) A
Σ Аj. KEGy
Poceder Layer
LIN Descder Layer
ITN
FFN
CADD
Attention
Attention
Furthermore, beyond the naive one-to-one matching approach, our proposed group matching strategy is superior to the more advanced adaptive matching method (Lee et al. 2025b) in terms
Layer Nomr
Layer Norr
Layo
6:0
of effectiveness, as demonstrated in Section 5.3
LUN Decederi syer
Toloe ber
Adapler
4.2 TEACHER ADAPTER FETCH (TAF) MODULE
TSturzerrr
Adapter
Adapter
The adapter, responsible for projecting the visual space into the language space of the LLM, is crucial for generating the visual tokens with which the attention mechanism processes. In this regard, given that the teacher's attention mechanism transferred via VAT is tightly coupled with the output of its own adapter, vision-language space mismatch occurs when this attention is imposed on a student with a different, incompatible space, hin- dering effective knowledge transfer. To address this, we intro- duce the Teacher Adapter Fetch (TAF) module, which directly leverages the teacher's frozen, pretrained adapter (PC and adds a lightweight trainable MLP (Px w. ) only for dimensional align- ment. This ensures the student processes the visual input through the same lens as the teacher, making the attention transfer effective. Formally, the student's visual token expressed as follows:
Vidoon
Vitton
Teacher MLLM
Muttimodal Imputs
mon no rilh hand..
Figure 5: Overview of CompoDis- till. It consists of the VAT module and the TAF module.
x, = PS, (Pú (2p)) E Nox.""
where d, is the hidden dimension of the student LLM. This approach allows for the student to bridge the gap between the teacher-imposed attention mechanism and its own visual feature space.
Discussion. It is important to note that attention distillation itself is not novel concept, similar approaches have been explored in various domains (Wang et al., 2020 Sajedi et al. 2023 Zhou et al. 2025), including diffusion models, dataset distillation, and language models. However, its application to MLLMs particularly challenging due to mismatched visual feature spaces between teacher and student. To this end, we introduce the TAF module as a simple yet effective solution that makes attention distillation practical in this setting. Lastly, our work is not intended to propose a fundamentally new distillation method; rather, its primary contribution lies in identifying a critical factor that can serve as a criterion for successfully distilling visual abilities in MLLMM,, i.e. visual attention misalignment, as explained in Section 3
IIn Table
we show that this approach is less effective than the proposed Group Layer Matching strategy. TTo ensure the use of all teacher layers Tor, we define in closed form + 1.

---

Published as a conference paper at ICLR 2026
4.3 THREE- STAGE DISTILLATION FRAMEWORK
Our knowledge distillation framework consists of three stages, built upon two foundational train- ing objectives. The first objective is language modeling autoregressive loss Lr.M. defined in Section [ Moreover, following previous studies (Shu et al.] 2024; (Cai et al.] (202)). we employ a Kullback-Leibler (KL) divergence loss (L to align the student's predictive distribution (p,)
with the teacher's (px). encouraging the student to mimic the teacher's final output at the logit level:
1 K
Xi, X1, log Xe..
LKI.
(4)
where yn is the n-th vocabulary token and N is the total vocabulary size.
Stage 1: Distilled Pre-Training (DPT). The first stage aims to align the visual feature space with the language space. To this end, instead of initializing adapter from scratch, we construct it using our Teacher Adapter Fetch module, which reuses the teacher's s pretrained adapter, which is kept frozen during training. While the vision encoder and LLM remain frozen, the student adapter p. is optimized with the language modeling loss (LLM) and the KL-divergence loss (CKL).
Stage 2: Distilled Fine-Tuning (DFT). In the second stage, aim to enhance the student's 's visual perception by aligning its visual attention mechanism over visual tokens with that of the teacher via the Visual ATtention Alignment module. During this stage, both the student LLM and the student adapter Ps are fine-tuned. The overall objective is defined LIM + LKL + CADI.
Stage 3: Supervised Fine- Fine-Tuning (SFT). Finally, motivated by prior work (Lee et al., 2025b), fine-tune only the student using the autoregressive language modeling loss (LLM)) with both the student LLM and the student adapter Ps v., trained in the same manner as in Stage 2. This final stage consolidates the rich knowledge transferred from the teacher during Stages and 2 into the student's own parameters and further strengths its instruction-oolowinn capability.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS.
Evaluation Benchmarks. For evaluation, we use two categories of vision-language tasks. Gen- eral VQA: We evaluate visual recognition abilities using well-established benchmarks including VQAv2 (Goyal et al. 2017). GQA (Hudson & Manning 2019). and VizWiz (Bigham et al. 2010)
for question answering. TextVQA (Singh et al. (219)) for scene text comprehension, and MME (Fu
et 2024) for comprehensive multimodal evaluation. Compositional Reasoning: To evaluate vi- sual perception abilities, we use several challenging benchmarks: SugarCrepe (Hsich et al. (023),
SADE (Ma et al., 2023), BiVLC (Miranda et al. 2024), and Winoground (Thrush et al.) (022). We
accuracy as the evaluation metric. Refer to Appendix D for more details regarding the basclines.
Implementation Details. Both the student and teacher employ SigLIP (Zhai et al.) (023) vision en- coder and the Qwen 1.5 (Yang et al. (2024) LLM series, with the student having 1.8B parameters and the teacher having 4B parameters. Regarding the details of the training datasets and hyperparameter settings, please refer to Appendix
5.2 QUANTITATIVE RESULTS ON VQA AND CR TASKS
In Table [ we compare Dospo with a diverse range of MLLMS of different sizes on VQA and CR tasks, which evaluate visual recognition and perception abilities, respectively. We have the following key observations: 1) Existing KD methods outperform the SFT model (I.La on VQA (visual recognition) tasks, but not on CR (visual perception) tasks, where their performance remains comparable to that of standard 2B models. This indicates that KD methods struggle to dis- till visual perceptual abilities. 2) CompoDistill overcomes this limitation by significantly improving CR performance to a level competitive with 4B models- an achievement not attained by previ- ous KD methods-whhh simultaneously maintaining strong, state-of-the of-te-o--hh-arr performance on VQA. This demonstrates the ability of CompoDistill to enhance visual perception while maintaining vi- sual recognition. 3) CompoDistill achieves these results with high data efficiency, relying on just
"To ensure fair comparisons, all KD models were distilled from the same teacher model (i.e., LLaVA-4B) and share the same I.M backhone (i.e., Qwen 1.5).
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison CompoDistill with KID-based methods and other MILMSs VQA and CR. Samples Training data samples. Reproduced results using the official source code. The best and second second-based models marked in bold and underlined, respectively for models sizes under 2B.
Size
Method
LLM
4 Samplex
Visual Question Answering
Composiional Reuscning
VQAA22
VizWiz GOJ
TexVQA
MME |Ang SSugueerege
SADE BIVLC Winoground [Ag
LLaVA-78
Qweel 5-7B
79.8
60.
68.6
87.3
78.5
58.4
LLaVA-7IB
Qwen2.5-70
1.2 M
81.4
50.6
63.8
51.6
77.5
67.6
93.1
82.9
68.3
08.2
78,
CogVLLM-7B
Vicara-78
78.3
64.5
57.2
82.8
77.4
Deopssek-VV VL-78
DLLM-7B
2000
86.2
78.8
LLaVA-4B' (Teacher)
Qwenl.5-10
56.7
75.5
38
80.4
57.1
65.9
63.5
Imp-38
Phu2-2788 7B
1,5 1.5M
K1.2
54.1
63.5
59
72.3
66.2
7M.
61.1
59.5
38.3
59.3
Bunny-3 3B
Phi2-2 7B
2.6 M
79.8
43.8
62.5
56.
76.4
63
75.8
67.2
59.7
53.1
64.0
Mok--LLaVe VA--33
Phu2-2.718
2.6 M
79.9
43.7
02.6
S7.0
11.6
63.0
30.3
70.4
64.4
54.1
67.3
MobileVI
Mobilal aMA-2 78
S7
72.0
67.9
72.
43.0
40.3
MiniCPM-
MiniCPM 2.4B
520 M
90.0
70.6
05.3
55.3
70.3
MiniCiemini-22 28
Gemma-28
2.7
60.7
56.
67.0
67.0
62.0
50.5
59.4
Vt.pseek-VV--I... 1BB
DIIM-1.15 1B
2000 M
36.8
59.3
58.4
65.3
36.6
38.6
39.9
60.8
44.0
MobileVLM-1133 LLM11
MobileLLaMMA 1.43
1.2.M
39.3
52.1
65.1
09.8
62.0
53.8
43.4
57.7
Imp-2R
Qweel 5-1 .B
1.5% M
79.2
39.2
61.9
54.5
652
60.0
71.0
59.6
58.5
$1.1
40.
Runny-28
Oweel 5-1
2.6 M
76.6
34.2
59.6
51.2
63.0
57.7
68.6
66.5
$1.8
283
60.3
Mott-1LaaA
Oweal
2.2 M
76.2
32.6
61.5
28.0
64.6
56.6
80.8
023
G2.
50.4
G39
LLaVA-28 ESFT godel)
Oweal.5-1
1.2 M
75.6
31.2
$7.9
506
61.2
54.9
73.1
63.9
33.3
47.2
-LaVA--D--BB
@wen 3B
M
78.8
44.8
62.3
534
68.9
61.6
75.3
63.6
57.9
49.3
61.3
L.IAVAD--2
Qwenl 5-1 .B
1.2 M
75.8
35.2
58.7
294
63.8
56.6
76.9
61.5
54.1
48.4
60.2
LLaWA-MeDD-28
Qweel 5-1.88
5.0.
76.3
43.0
58.8
52.7
63.4
58.9
76.9
63.8
00.3
49.4
62.6
CompoDisill-28
Qweel 5-1.88
1.2M
78.8
32.
62.2
56.4
70.1
61.9
82.9
69.4
63.3
51.2
66.7
1. 1.2M training samples. This is a sharp contrast to other models requiring much larger datasets (e.g., LLa VA-MoD with 5M, MiniCPM-V with 570M), proving the effectiveness of CompoDistill toward a truly efficient MLLM.
Table 2: Ablation for VAT and TAF modules.
Row
VAT module
TAF module
II
Visual Question Answering
Compositioal Reasoning
|| GQA
TeaVQQA
MME
Avg
Sugarerope
SADE BIVLC
Wineground
Avg
(b)
54.3
54.6
64.8
57.9
81.0
66.4
62.1
50.5
65.0
(c)
60.8
56.5
66.5
61.3
78.0
67.0
62.4
48.1
63.8
(d)
62.2
56.4
70.1
62.9
82.9
69.4
63.3
51.1
66.7
5.3 ABLATION STUDIES
Effect of Core Components. In Table 2 we conduct ablation studies to understand the effect of each component (i.e., VAT and TAF modules). Note that the variant without either component (row (a)) follows the proposed three-stage framework (Section 4.3), using only the language modeling loss (Section [ and logit-based KD loss (Equation 4). Effeett of VAT module: We observe that the
VAT module significantly improves the performance especially on CR tasks (row (a) vs. (b) and row (c) vs. (d)), demonstrating the effectiveness of enhancing visual perception abilities. This confirms that explicit visual attention alignment is crucial for distilling visual perception ability, as discussed in Section g Effect of TAF module: We observe that equipping the TAF module improves the performance on VQA and CR tasks (row (a) vs. (c) and row (b) vs. (d)), indicating that bridging the student's s feature space with the imposed attention mechanism is crucial for effective knowledge
transfer. Finally, the fledgef model (row (d)) achieves the best performance on both VQA and CR tasks, demonstrating the benefit of the VAT and TAF modules.
Table 3: Detailed ablation for VAT module. The blue line performs the same CompoDistill.
(a) Attention loss type
(b) Target layers
(c) Layer matching strategy
Attention Loss
VQA Avg
CR Avg
Target Layers
VQA Avg
CR Avg
Match strategy
VQA Avg
CR Avg
61.3
63.8
Early 30%)
61.2
63.7
Simple Adaptive
61.5
65.6
60.3
65.2
Later (70% m-)
61.7
64.6
62.0
65.7
KL. Div.
60.7
65.5
All
62.4
66.6
Group
62.9
66.7
Cos. Sim.
62.9
66.7
Intermediate
62.9
66.7
Fine-grained Analysis on VAT module. We perform fine-grained ablation studies study the im- pact of specific design choices- namely, the attention loss type, the target matching layer, and the layer matching
the VAT module. 1) We first analyze the impact of the attention loss function (Table Sap. While any form of attention loss improves CR over the baseline, using Cosine Similarity (Cos. Sim.) significantly outperforms both MSE and KL Divergence. This result suggests that it more erucial for the student to learn the relative importance among visual patches
8

---

Published as a conference paper at ICLR 2026
rather than forcing an exact match of their absolute attention scores. 2) Next, we investigate which layers to target for distillation (Table 3b). Performing distillation on the intermediate layers (30- 70%) yields the highest performance. This finding confirms our analysis that visual understanding layers are crucial for visual-semantic integration and highlights the effectiveness of distilling specif- ically from these layers, a strategy consistent with prior research (Kaduri et al., 2024). 3) Lastly, we evaluate different layer matching strategies (Table Be). Our proposed Group matching achieves the best performance compared to both Simple matching (which uniformly samples teacher layers) and Adaptive matching (which finds optimal pairs based on layer distance). This suggests that grouping layers provides a more stable and effective signal for transferring the teacher's s complex attention behaviors, especially when student and teacher architectures differ in depth P
Further Benefit on Other Complex Task. Beyond compo- sitional reasoning, we explore a further benefit of enhancing visual perception for complex tasks. Specifically, we expect that CompoDistill can help mitigate the relational hallucina- tions, thanks to its ability to accurately understand object re- lationships, as discussed in Section T To test this, we evalu- ate CispoDistill on the R-Bench (Wu et al. 2024) and Reef- knot (Zheng al. 2023) benchmarks, both of which focus evaluating relational hallucinations. As shown in Table 4 CotpoDistill significantly outperforms other KD methods and achieves per- formance nearly on par with the teacher. The results highlight that enhancing the visual perception ability can be beneficial to not only compositional reasoning tasks but also other complex tasks that require precise understanding of relationships among objects.
Table 4: Comparison for relational hallucination using Fl score.
Model
ReBench 1
Reefknet 1
Teacher (LLaVA-dB)
79.1
67.9
Student (LLaVA-26)
74.3
61.3
LLaVA-KD-22 KD-2B
76.5
60.3
LLaVA-MoD-28
76.2
63.4
CompoDisUll-28
78.6
66.7
SCALABILITY EXPERIMENTS
Richer Data Improves Distillation Performance. We ana- lyze the effect of data scale in Table Performance improves significantly when moving from SFT distillation, and in- creases further when the training dataset size is doubled [Tù This highlights that both the quality and quantity of training data are crucial for effective knowledge transfer, with CR showing particular sensitivity to data scaling,
Table 5: Data Scaling Experiment.
Student
e Sumple
VQA Ang
CR A
Quann (SFT) Qweul Qwenl. 188
1.2M
56.6
60:7
24
83.3
69,9
Larger Teachers Produce Stronger Students. Next, we exam- ine the influence of teacher model size in Table 6 Our experi- ments show that students consistently benefit from larger teach- ers, regardless of the student's own size. For instance, a 1.8B student distilled from a 7B teacher outperforms one distilled from 4B teacher. This demonstrates that higher-capacity teacher is crucial for transferring stronger reasoning abilities, providing more effective source of knowledge for the student.
Table 6: Experiments with different size of teacher/student.
Ave
Qwent. 10111-1-110 12B (STT)
26.6
60.7
Qwen1-5-1.. 5-133
Qpeent 5-40
0wes2.5.7B
629
66.7
Qwen1. 0.58 (SFT)
$2.0
28.4
Qwmi
Owenl
1
Oweet 5-48
Our Distillation Method Generalizes Across Backbones. Finally, we test our framework's generalizability by replac- ing the Qwen backbone with the MobileLLaMA family (ML- LaMA) in Table 7 Our distillation method remains effec- tive even with this entirely different architecture. This result contirms the robustness and generalizability of our approach, demonstrating that its principles are not tied to a specific model family but are broadly applicable.
Table 7: Experiments with different LLM backbone.
Student
Teacher
VQA Avg
CR.Av Avg
MIT A.L.aMM--177 MLLaMa L.70
43.7
MLLaMa 30
53.1
48.9
7 RELATED WORKS
Multimodal Large Language Models. Visual instruction tuning (Liu et al.] 2023) has propelled
MLLMs to strong performance on diverse benchmarks (Chen et al. 2024 Yang et al. [2044]] OpenAl
2023
yet persistent weaknesses remain in fine-grained visual tasks (Tong et al. 2024 Qi et al.
2025
Early efforts to address these limitations focused on scaling up vision encoders (Lu et al.
2024
Kar et al. 2024) or designing more expressive projectors (Cha et al. 2024 Liu et al.] 2024).
More recently, the focus has shifted to the internal information flow, with research identifying critical
FFor a detailed explanation of the layer matching strategics, see Appendix
For details on the training data, Appendix G
9

---

Published as a conference paper at ICLR 2026
failures such misaligned attention (Jiang et al., 2025; Neo et al. 2025
Darcet et al. 2024) and
the dilution of visual features in intermediate layers (Kaduri et al. 2024
Yoon et al. 2025] Chen
it al. 2025b). In contrast to these approaches, our analysis focuses on the visual attention dynamics between student and teacher models trained via direct knowledge distillation.
Knowledge Distillation. Knowledge distillation (KD) compresses a large teacher model into a smaller, efficient student (Hinton et al. 22155) a technique widely applied to LLMs via logit match- 2020).
the multimodal domain, methods have adapted this for visual grounding (Cal et al. [2224 Feng et al. 2025) or used modular strategies to overcome architec- tural limits (Shu et al. 2024). The scope has since expanded to distillation across model families (Lee) ct al. 2025b a) and a deeper focus on internal mechanics. For instance, recent alignment- ooriented methods like VIRAL (Yoon et al., 2025) regularize intermediate representations to preserve visual semantics, moving beyond simple logit-level supervision. While these studies primarily propose new distillation techniques, our work takes different approach. We instead provide a detailed analysis to identify the specific bottlenecks that hinder the effectiveness of knowledge distillation in MLLMMs.
ing (Sun et al. 2019 Jiao et al.
Compositional Reasoning Benchmarks. Compositional reasoning, the ability to understand the in- terplay of objects and their relations, remains significant hurdle for vision- language models. Initial benchmarks like Winoground (Thrush et al. (022) first exposed these weaknesses in early models. Building on this, a new generation of more robust benchmarks has emerged to address evaluation biases. These include SugarCrepe (Hsieh et al. 2023). which uses LLM -LL--genrrtted hard negatives; SADE (Ma et al. 2023). which specifically diagnoses and mitigates the language bias found in gen- erative models through debiased test set that neutralizes syntactic shortcuts; and BiVLC (Miranda al. 2024), which focuses on bidirectional retrieval with synthetic negatives.
CONCLUSION
In this work, we aim to enhance the visual perception abilities of Knowledge (iniilation(DD-bballl Multimodal Large Language Models (MLLMs), which has been largely overlooked by the previous KD studies. To this end, we conduct a systematic analysis and identify visual attention misalignment as key factor hindering the effective distillation of visual perception from teacher to student. Build- ing on this analysis, we propose CompoDistill, novel KD framework that incorporates Visual ATtention alignment (VAT) module to explicitly address this misalignment. Furthermore, we intro- duce the Teacher Adapter Fetch (TAF) module to ensure that teacher-impan -mpoeed attention mechanism is compatible with the student's s feature space, making synergy with VÁT module. Through exten- sive experiments on VQA and CR benchmarks, we demonstrate that CompoDistill significantly enhances visual perception abilities while preserving strong visual recognition abilities, as achieved in existing KD works. Regarding the limitation and future work, please refer to Appendix PP)
We believe that this work provides novel perspective the student's 's visual attention misalignment and makes a contribution to the pursuit of efficient MILLMs, especially in KD-based research, by establishing the first dedicated direction toward enhancing visual perception abilities.
ACKNOWLEDGEMENT
This work was supported by the Institute of Information & Communications Technology Planning & (yaluation(ITTP) grant funded by the Korea governmen((MSIT) (RS-2025-02304966, AI Star (ellowship(KAIST) as well as another (IITP) grant funded by the Korea govermmen((MIIT) (RS- 2022-11220077, Reasoning, and Inference from Heterogeneous Data)
ETHICS STATEMENT
Our research contributes the development of more efficient and accessible Multimodal Large Language Models (MLLMs). By enabling the creation of smaller, yet highly capable student models, work helps lower the significant computational barriers to deploying advanced AL, promoting its broader adoption. compliance with the ICLR Code of Ethics, and to the best of our knowledge, we have not faced any ethical issues during this research. Additionally, all datasets and baselines used in our experiments are freely accessible to the public.
REPRODUCIBILITY STATEMENT
To ensure reproducibility of experiment results throughout the paper, we describe the details of ex- perimental setting and training details 55 and Appendix respectively. Furthermore, we provide source code in httpps//giihuh..oo/////////////s (ttp:////
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL htttss//arxiv.oom org/ abs 2502. 13923
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time an- swers to visual questions. In Proceedings of the 23nd annual ACM symposium User interface software and technology, pp. 333-342, 2010.
Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, Zhucun Xue, Yong Liu, and Xiang Bai. Llava-kd: A framework of distilling multimodal large language models. arXiv preprint 241i::2400.16666, 2024.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Nicoletta Calzolari, Chu-Ren Huang. Hansaem Kim, James Puste- jovsky. Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.). Proceedings of the 29th International Conference on Computational Linguistics, pp. 151 1-1520, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https: httpa://aalaatthhoooocoo
org 2022. wwwp://////......./oooooo cwwinco 13107
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungscok Roh. Honeybee: -oocaallt--ehanneed projector for multimodal 1lm, 2024, URL http:pps//aaxi...ggr/bb///////// 067.3111..777.2
Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, and Junxian He. Bring reason to vision: Understanding perception and reasoning through model merging. arXiv preprint arXiv:2505.05464 2025a.
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for vims? an atten- tion mechanism perspective focus areas, 2025b. Utthttts:://an /ttp///axvvvv//////////.. org/ abs/2bas00 01773
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic disual-linguiise tasks, 2024. URL httpps:/ //axxiv wwwprwvww....o..e... www.//gg////.o.ooooo/o../o...t// / 2312. 2.1.2.1422. 14238
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, and Chunhua Shen. Mobilevim v2: Faster and stronger baseline for vision language model, 2024. htttps://aaxiv.orrrrs//t
037603766
Christopher Clark and Matt Gardner. Simple and effective multi parti-par reading comprehension, 2017.
177.110733
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2024. URL Rolnttps::/ /tattps://axxae http:///xxxxxxxxxx.aaaaa wwtp:oo////0/000000 .6188
Qianhan Feng, Wenshuo Li, Tong Lin, and Xinghao Chen. Align-kd: Distilling cross-modal align- ment knowledge for mobile vision-language large model enhancement. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4178 4188, 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL
2336 13394
Yash Goyal. Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017. 2017.
11

---

Published as a conference paper at ICLR 2026
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.1 15530,
1530, 2024.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015. URL htRthhts:://axii>> org. abs/1503, 02531
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural informa- tion processing systems, 36:31096-33 31116, 2023.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang. Chongyi Wang, Yuan Yao. Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL hhttps://arxi>> org/ abs/ E4C4.06395
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700- 6709, 2019.
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucina- tions via attention lens, 2025, URL https: /ttp:i..//aaxxvv..oov org abs / 22112 16724
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang. and Qun Liu. Tinybert: Distilling bert for natural language understanding, 2020. URL https://arxix:
10351
Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? a deve into the vision of vision language models, 2024. URL wtt.hhttp//// htttps://axxv..org//ee..
2411. 17491
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- 08163
izations via question answering, 2018.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Scong Jae Hwang. Your large vision-language model only needs a few attention heads for visual grounding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9339-9350, 2025.
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-languag. models, 2024. URL https:
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images, 2016. /tll/htttta///axii.ccor wwtp:////orraaa//00003
07396
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongycon Nam. Jinyoung Park, Jinycong Yim, Won- scok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understand- ing transformer, 2022. URL httttp:://aaattet
org abs/ 2111 15664
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016. URL /
http://aaxivv.vo www.wiiwww...aaaaaaaaoaaaaaaaaae /ttp:///wwww...//////////22002//
Byung-Kward Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang, and Yueh-Hup Hua Wu. Gen- recal: Generation after recalibration from large to small vasion-language models. arXiv preprint arXiv:2506. 15681, 2025a.
Byung-Kwe Kwan Lee, Ryo Hachiuma, Yu- Yu-Chiang Frank Wang, Yong Man Ro, and Yuch-Hua Wu. Vlsi: Verbalized eyves--o-inntractinn from large to small vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29545-29557, 2025b.
12

---

Published as a conference paper at ICLR 2026
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning, et al. Moe-llava: Mixture of experts for large usion-languaae models. arXiv preprint 15Xv:2201..5947, 2024a.
Lin, Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEÉE/CVF conference on computer vision and pattern recognition, pp. 26689-26699, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL https://arxis: wwt. 2230 08485
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning, 2024. URL https:://rxiv..rr
abs 2ans/233 /211 0. 03744
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
11X10:1711.05101, 2017.
Haoyu Lu. Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vi: Towards real-world vision-language understanding, 2024. URL https:
//axx/v.ore/abs/2203..
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. URL httpht:s:// http://xxxi...rr/aa//////00/
09513
Teli Ma, Rong Li, and Junwei Liang. An examination of the compositionality of large generative vision-language models. arXiv preprint arXiv:2308. 10509, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. URL https://arxiv.
hxg/abs/1906. 00067
Ahmed Masry. Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL https:
//arxiv.org/ wtw.vvv.orr//b///233.3.. 10244
Imanol Miranda, Ander Salaberria, Eneko Agirre, and Gorka Azkune. Bivle: Extending vision- language compositionality evaluation with text-to-image retrieval. Advances in Neural Informa- tion Processing Systems, 37:101880-101904 2024.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 947-952, 2019. doi: 10.1 20.1009/DDAR.2019.0000
Clement Neo. Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpret- ing visual information processing in vision- vision-language models, 2025. URL https: /ttpss//arxxv. htg//ab/22410.. 07149
OpenAl. Gpt-4v(ision) technical work and authors, 2023. URL 4v/ Accessed: 2025-09-18.
com/
cont contributions
aware- Jianing Qi, Jiawei Liu, Hao Tang. and Zhigang Zhu. Beyond semantics: Rediscoveerng aware
ness in vision- languaee models. 2025. URL htttt:tt////aattt http://orggbbbbb.../a./s.
Qwen, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https::/aaxxvv org/ abs /00os/2411 2412. 15115
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PmLR, 2021.
13

---

Published as a conference paper at ICLR 2026
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17097-17107, 2023.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mot- taghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022. URL
https: http//axivvv./ org/ abs 2206. 01718
Zhenwei Shao, Zhou Yu, Jun Yu, Xuecheng Ouyang, Lihao Zheng, Zhenbiao Gai, Mingyang Wang, Zhenzhong Kuang, and Jiajun Ding. Imp: Highly capable large multimodal models for mobile devices. IEEE Transactions on Multimedia, 2025.
Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Lei Zhang, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, et al. Llava-mod: Making llava tiny via moe knowledge distilla- tion. arXiv preprint ar Arxiv:2408. 15881, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression, 2019.
wttp:/ogggaabbb/oooee 09355
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5238-52885 5288-224,770 2022.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms, 2024. URL https://arxiv.
org/abs/22 24011 06209
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong. Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2024. URL atttps://arxiv.ozz
abs/ 0bbs/3111 2311.03075
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task task-agmostic compression of pre-trained transformers, 2020. URL
Stttps://erxxt www.//ggg//////o..///.///eeeee/e 11022...955 10957
Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in large vision-language models, 2024. URL
https://arxiss
wlwp/////////// 6406. 1110066..66.. 16449
Shilin Xu, Xiangtai Li, Haobo Yuan, Lu Qi. Yunhai Tong, and Ming--suar Yang. Llavadi: What matters for multimodal large language models distillation. arXiv preprint arXiv:2407.194 19409, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma. Jianxin Yang. Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan, Yang Yao, Yichang Zhang. Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://aras org/abs......o.ooooo 2407 10671
14

---

Published as a conference paper at ICLR 2026
Heeji Yoon, Jaewoo Jung, Junwan Kim, Hyungyu Choi, Heeseong Shin, Sangbeom Lim, Honggyu An, Chaehyun Kim, Jisang Han, Donghyun Kim, Chanho Eom, Sunghwan Hong, and Seungry- ong Kim. Visual representation aligmment for multimodal large language models, 2025. URL
07979
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions, 2016. URL htttp::/aaxxiii...ooom org/ http://aaa./.e...... 00272
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 11975-11986, 2023.
Tianyang Zhao, Kunwar Yashraj Singh, Srikar Appalaraju, Peng Tang, Vijay Mahadevan, Man- matha, and Ying Nian Wu. No head left behind-multi-head alignment distillation for transform- In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7514-7524
2024.
Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large lan- guage models, 2025. htLhstps::/ arxiv. org/ abs/ 2408 09429
Yang Zhou, Xu Gao, Zichong Chen, and Hui Huang. Attention distillation: A unified approach to visual characteristics transfer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18270-18288 2025.
15

---

Published as a conference paper at ICLR 2026
Supplementary Material
CommoDDstill:: Attention Distillation for Compositional Reasoning in Multimodal LLMs
Additional Visualizations on Áttention Misalignment
18
B Differences between Visual Recognition and Perception Abilities
18
Visual Recognition
19
B.2 Visual Perception
20
Additional Experiments on Attention Similarity
20
D Baseline Methods
20
Additional Implementation Details
21
E.1 Training Strategy and Syper-parametery
21
Explanation of the Baselines for Layer Matching Strategy
22
Explanation for the Extended Training Data
22
H
Additional Experiments Results
23
Detailed Experiment Results on VAT module
23
H.1.1
Detailed Experiment Results on Attention Loss Type
23
H.1.2
Detailed Experiment Results Attention Target Layers
23
H.I.3
Detailed Experiment Results on Layers Matching Strategy
24
H.2
Detailed Experiment Results on Relational Hallucination
24
H.3
(Additional) Detailed Experiments Results on Training Strategy.
24
Detailed Experiments Results on Scalability
25
H.4.1
Detailed Experiments Results Data Scaling
25
H.4.2
Detailed Experiments Results on Student and Teacher LLM Size
25
H.4.3
Detailed Experiments Results on different LLM Backbones
26
Generalization Other Architecture
26
Discussion about Training Overhead
27
Additional Experiment on Áttention Mixing
28
Additional Experiment on Teacher Adapter Fetch Module
28
M Additional Experiment Visual ATtension Alignment Module
29
16

---

Published as a conference paper at ICLR 2026
N Discussion on Possibility about Teacher Overfitting
29
Additional Experiment on Visual Grounding
30
Limitation and Future work
31
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL VISUALIZATIONS ON ATTENTION MISALIGNMENT
In this section, we provide additional qualitative examples of the attention misalignment between the student and the teacher model, concept introduced in Section I To facilitate comprehension, we have highlighted crucial textual components in red, specifically phrases that require relational un- derstanding or accurate attribute recognition. For a comprehensive comparison, we present results for additional student model (LLaVADI-2B) and our method (CompoDistill), alongside the base- line LLaVA-KD-28 and LLa VA-4B (Teacher). Notably, the visualizations show that CompoDistill produces attention map remarkably similar to the teacher's. s. correctly focusing on the key visual areas relevant to the text. These comparisons are illustrated Figure 6
and Figure
Original
"A man taking a photo with right hand."
Teacher (LLaVA-4B)
Student
(LLaVA-KD)
Student (LLaVADI)
Student (CompoDistill)
Original
knife
Teacher (LLaVA-4B)
Student (LLaVA-KD)
Student (LLaVADI)
Student
(CompoDistill)
Visual attention misalignment
Figure 6: Examples of attention misalignment.
B DIFFERENCES BETWEEN VISUAL RECOGNITION AND PERCEPTION ABILITIES
This section provides detailed explanation of the distinction between visual recognition and visual perception abilities, which are evaluated using Visual Question Answering (VQA) and Composi-
18

---

Published as a conference paper at ICLR 2026
Original
"Two men ride bikes next to the cars."
Teacher (LLaVA-4B)
Student (LLaVA-KD)
Student (LLaVADI)
Student (CompoDistill)
Original
woman is holding bicycle."
Teacher (LLaVA-4B)
Student (LLaVA-KD)
Student (LLaVADI)
Student (CompoDistill)
Visual attention misallgnment
Figure 7: Examples of attention misalignment.
tional Reasoning (CR) datasets, respectively. Also in Figure we illustrated an example of VQA and CR question.
B. VISUAL RECOGNITION
Visual recognition is the foundational ability of model to identify and categorize objects, scenes, and their basic attributes within an image. It fundamentally answers the question, such as "What this image?: This process relies on matching learned visual patterns-such as textures, shapes, and colors to specific labels or concepts. For example, when model identifies a four-legged furry animal "dog". it performing visual recognition. This ability is analogous to building vocab- ulary of the visual world.
Most standard VQA datasets (e.g.. VQAv2 (Goyal et al. 2017). Vizwiz (Bigham et al. 2010), and GQA (Hudson & Manning 2019)) are primarily designed evaluate this recognition capability. The questions in these datasets are typically direct and fact-based, probing for the presence, count, simple properties of objects. They often be answered correctly if the model successfully
19

---

Published as a conference paper at ICLR 2026
Q. What the color bicycle?
Q. Which caption best describes the image?
1. A woman is riding bike.
2. woman is holding bike.
A. Red
(1)
(a) VQA Question
(b) Compositional Reasoning Question
Figure 8: An example of standard VQA question, which requires simple object identification, versus a Compositional Reasoning (CR) question, which requires accurately distinguishing between two detailed and confusable multiple choices.
recognizes the key objects and their most salient features, without needing to understand complex inter-object relationships.
B.2 VISUAL PERCEPTION
Visual perception more advanced cognitive ability that goes beyond simple identification. involves interpreting and understanding the relationships between objects, their precise at- tributes, their spatial arrangement, and the overall context of the scene. If recognition is about what, perception about how and why such as how objects are arranged, how they interact, and why the scene is composed in particular way. This requires the model to not just list the contents of an image, but build coherent, structured understanding of it.
Compositional Reasoning (CR) datasets are specifically designed to evaluate this perceptual ability. The questions structured to be challenging for models that rely solely simple keyword match- ing or recognition. To answer correctly, a model must accurately bind specific attributes to their corresponding objects and correctly interpret the spatial or semantic relationships between them. These questions often test model's robustness where the correct objects and attributes present but not the configuration described by the question.
C ADDITIONAL EXPERIMENTS ON ATTENTION SIMILARITY
we extend our attention similarity analysis two additional benchmark datasets: VQAv2 (Goyal et al. (017) for VQA and Winoground (Thrush ctt al. 2022) for CR. replicating the experimental setup used for GQA (Hudson & Manning 2019) and SugarCrepe (Hsieh al. 2023).
To further validate our findings in Section 3.1
The results, visualized in Figure 9 demonstrate a consistent trend with the conclusions drawn in visual tokens in perfor-
main analysis. The analysis reaffirms the critical role of attention mance improvement. Specifically, on the Winoground (Thrush et al. 2022) (CR) task, where the student model's performance gain over the SFT baseline was marginal, we again observe no signifi- cant increase in teacher-student attention similarity. This result corroborates our main argument that higher attention similarity over visual tokens in the visual understanding layers is a key factor for the effective distillation of visual perception.
D BASELINE METHODS
We include several baselines from two main groups. The first group consists of Knowledge Distil- aution-based methods, including LLaVADD-2B (Xu et al. 2024), LLaVA-KD-PD (Cai et al. 2024). and LLaVA-MoD-2B (Shu et al. 2024). To ensure fair comparisons, all KD models were dis-
20

---

Published as a conference paper at ICLR 2026
1.00
Performance Studeet: SFT (LLaWA-KD-28) (LLaVA-20)
SimiTeachet SFT) SamiTeacher Studend)
Performance mance
Sin(Weccher SFT) sincwaclhes student)
(LLaVA-KD-28) (LLavi 281
0.70
0.60
0.50
0.40
0.20
0.10
0.00
(b) Visual attention similarity on VQA
(c) Visual attention similarity on CR
1.00
SimcTescheer SFT1
Performance Student: GFT: (LLaVA (LLeVA--3))
Samileacher 4F1)
Student: 78.8 SFT:
SimiTeaccerr
SuniThacherr Stualent)
[LLaVA-KD 281 LaVi-15)
0.80
0.60
0.40
2
10
12
14
16
18
20
22
2
6
10
12
14
16
18
20
22
(d) Text attention similarity on VQA
(e) Text attention similarity on CR
Figure 9: Layerwise attention similarity of visual tokens and text tokens between student/SFT mod- els and the teacher model.
tilled from the teacher model (i.e., LLaVA-4B) and share the same LLM backbone (i.e.,
Qwen 1.5). The second group comprises a broad range of General MLLMS with parameter sizes in the range of 1.3B-3B, comparable to that of the compared KD methods, for direct performance comparison. This includes models such as Imp-2B (Shao et al. 2025), Bunny-2B (He et al. 2024). MoE-LLaVA-2B (Lin et 2024a), LLaVA-22 2B (Liu et al. 2024), and Deepseck-VL, VL-1.BB (Lu et al. 2024), MobileVLM--L...
(Chu al. 2024). well as larger models (4B-7B) like MinCCPMVV.
2.4B (Hu et al.) 2024). CogVLM-7B (Wang et al. 2024) and Qwen2.5-Vi LLL7BB (Bai et al. 2025 to
provide state-of-the of-the-rrt context.
E ADDITIONAL IMPLEMENTATION DETAILS
E. TRAINING STRATEGY AND HYPER-PARAMETERS
Our model is initialized with a SigLIP-B/14@* 384 vision encoder and Qwenl.5-1.88 LLM. For adapter,
3-layer MLP. Across all training stages, trained for one epoch using Adam W optimizer (Loshchilov & Hutter 2017). The AdamW optimizer with 31 0.9 and 3r = 0.98, cosine decay learning rate scheduler, weight decay of 0.0, and warm-up ratio of 0.03. We process images at resolution of 384x384, with input sequence lengths of 729 for the vision encoder and 2048 for the LLM. All training is performed with Float 16 numerical precision and Zero2 model parallelism with 8 NVIDIA L40S 48GB GPUs. For the DPT stage, use the LLaVA-1. VA-55-558K (Liu et al. 2024) dataset with a batch size of 256 and learning rate of le-3. For the DFT and SFT stages, we use the batch size of 128 and learning rate of le-4.
consistent setup: Each stage is
(Liu et al. 2024) dataset with
Our training is divided into three distinct stages, each lasting for one epoch:
Distilled Pre- Pre-Training (DPT): In this initial stage, our primary goal is align the visual representa- tions with the language embedding space. To achieve this, we freeze both the vision encoder and the LLM, and exclusively optimize the parameters of the adapter. This stage uses global batch size of 256.
21

---

Published as a conference paper at ICLR 2026
Distilled Fine-Tuming (DFT): The scope of training is expanded in the second stage to distill knowl- edge more deeply into the language model. The vision encoder remains frozen, but we now co- optimize both the LLM and the adapter. For this stage, the global batch size is reduced to 128.
Supervised Fine-Tuning (SFT): In the final stage, the model, initialized from the DFT checkpoint, is further refined. The training configuration mirrors the DFT stage: the vision encoder is frozen, while the LLM and adapter are co-optimized. This stage also uses global batch size of 128 to complete the training process.
The detailed training hyper- papee-parameters are show in Table
Table 8: Training hyper-paramelers of each stage.
Configuration
Distilled Pre-Training
Distilled Fine-Tuning Supervised Fine- Fine-Tuning
LLM
Vision Encoder
X
Adapter
LLM init.
Qwen1.5-1.88
Qwen 1,..8B SigLIP-B/14@3M)
From DFT
Vision Encoder init.
Image resolution
384 x 384 729
Vision Encoder Sequence length
LLM sequence length
2048
Optimizer
Adam
Optimizer syper-parameter
01 0.9, (g 0.98 2e-4
Learning rate
Learning rate scheduler
Cosine decay 0.0
Weight decay
Training epuch
Warm-up up ratio
0.03
Global batch size
256
128
128
Numerical precision
F
Model parallelism
Zero2
F EXPLANATION OF THE BASELINES FOR LAYER MATCHING STRATEGY
In this section, we provide a detailed explanation of the different layer matching strategies evaluated in our main ablation study (T)ble Layer matching is critical component of attention distillation, as it defines the correspondence between teacher and student layers for knowledge transfer. While our proposed method, Group matching, proved to be the most effective, we also explored two alternative bascline strategies to thoroughly investigate the design space. We will describe each of these in turn:
First, Simple matching, straightforward approach that uniformly samples teacher layers to match the number of student layers. Second, Adaptive matching. Expanding on (Lee et al. 2025b), this method first computes a matrix of Kullback-Leibler (KL) distances between every student target layer and every teacher target layer. Each student layer is then greedily paired with the teacher layer that has the minimum KI. distance, under a consecutive constraint. This constraint ensures that a given student layer can only select teacher layer that comes after the one selected by the previous student layer, thereby maintaining the sequential integrity of the model's architecture.
G EXPLANATION FOR THE EXTENDED TRAINING DATA
This section provides detailed information the training data used for the data scaling component of our main scalability experiments. Our baseline training utilizes a base dataset of approximately 1.2M samples, comprising the LLaVA-1.5-55888 (Liu et al. 2024) for Distilled Pre- Pre-Training (DPT) and LLa VA-mixx665K (Liu et al. 2024 for the Distilled Fine- Tuning (DFT) and Supervised Fine- Tuning (SFT) stages.
To scale the data in Sec 6 we incorporate an additional 1.2M samples, originally curated for the
Dense-to-Se Sp-rss Distillation (D2S) in LLaVA-MoD (Shu et al. 2024). As detailed in Table 9 this extended dataset is a diverse mixture covering wide range of tasks, including General QA, Ground- ing, Science, Chart& & Document understanding, OCR, and Knowledge-basee reasoning.
22

---

Published as a conference paper at ICLR 2026
Table 9: Training dataset of scaling data experiment.
Stage
Task
Dataset
Distilled Pre-Training (DPT)
Captioning
LLaVA-L.5-558K
[iuu al.
2024
Distilled Fine-Tuning (DFT)
Conversation
I.I 4V5-
Jiu et al.
2024
Supervised Fine-Tuning (SFT)
Conversation
LLaVA-mix-665K
LLu et al.
2024
Captioning Conversation
ShareGPTaV-100K
TextCaps
LLaVA-mix-665K (Liu et al. 2024
GQA (Hudson & Manning 2019). V VQAv2 Goyal et al. 2017]]
General QA
OKVOA (Marino er at. 2019
Data Scaling Experiment
Grounding
VG (Krishma al. 201h Re
Yu et 2016
Science
A1217 (Kebbhavi eral. 2016,
(all Oci
ScienceOAA
2022
DVQA (Kafle et al. 2018), ChartQA (Masry 2022
Chart
DocQA CCark & Gärdner 2017
OCR
OCRVQA (inisird e a SynthDoG-EN (Kim et al.
2022
Knowledge
A-OKVQA (Schwenk or ara 2022
GeoQA+ (Cao & Xiao
2022
H ADDITIONAL EXPERIMENTS RESULTS
H.1 DETAILED EXPERIMENT RESULTS ON VAT MODULE
H.1.1 DETAILED EXPERIMENT RESULTS ON ATTENTION Loss TYPE
Table 10: Ablation for various attention loss type.
Atteation Loss Type
Attentoon Matching
Aneentinn Layers
||
Question Answering
Compositiona Reasoning
VQA
Avg
Sugarrope
BiVLC
Winegrund
Adlation
weo Attention Lowe
60.8
56.5
66.5
61.3
78.0
67.0
62.4
48.1
6388
MSE KL. Div.
Group Group Group
insermediat
59.9
55.3
65.7
60.3
79.9
68.3
62.8
49.7
66.2
IntermoSSatee
60.7
53.6
65.9
60.7
801
67.9
63.5
50.5
65.5
Cos, Sim.
insennediate
62:2
36.4
70.1
62.
82.9
69.0
63.3
51.1
06.7
We provide detailed analysis of the impact of different attention loss functions, as summarized in Table T0] Our goal was to determine the most effective way to quantif'y the difference between student and teacher attention maps for distillation. The results show that while all tested loss func- tions improve performance on Compositional Reasoning (CR) tasks over the baseline, their effects on Visual Question Answering (VQA) are mixed. Notably, both Mean Squared Error (MSE) and KL Divergence (KL. Div.) slightly degrade VQA performance.
Cosine Similarity (Cos. Sim.) proved to be the most effective loss function, improving performance across both the VQA and CR domains. We believe this is because it is more crucial for the student model to learn the relative importance of different visual patches, rather than replicating the exact absolute values (MSE) or the probability distribution (KL. Div.) of the teacher's 's attention scores.
H.1.2 DETAILED EXPERIMENT RESULTS ON ATTENTION TARGET LAYERS
Table 11: Ablation for attention target layers.
Attention Loss Type
Attention Masching
Attention Layers
Visual Question Answering
Compositonal Reasoning
|| GQA
TesVVQA
MME Avg
Sugarcrepe
SADE BIVLC
Winoground Arg
AMiztion on targer loyers for attention distilattonn
Cos. Sim.
Group Group Group
eurly
61.7
Cos. Sim.
all
02.
83.1
Cos. Sim.
Group
Intermediate
56.1
62.9 82.9
63.3
51.1
66.7
We investigate which layers are the most effective targets for attention distillation, with detailed results in Table II] We compared four strategies: distilling from early layers (~30%), later layers (70%~), all layers, and our primary approach of targeting the intermediate layers (visual under- standing layers).
The results clearly indicate that targeting the visual understanding layers yields the best overall per- formance on both VQA and CR tasks. While distilling from all layers provides strong and sometimes comparable results, this approach is less computationally efficient. The more focused intermediate
23

---

Published as a conference paper at ICLR 2026
strategy ultimately achieves better trade-off, slightly outperforming the all-layer approach in over- all scores without the associated computational overhead. Conversely, targeting only the early or later layers leads to a noticeable drop in performance.
This confirms our analysis that the intermediate layers, which function as the core visual under- standing layers, are the most critical for -isual-semantic integration. Distilling specifically from this block provides the most potent and effective signal for knowledge transfer, a finding consistent with prior research (Kaduri al. 2024 Neo et al. 2025).
H.1.3 DETAILED EXPERIMENT RESULTS ON LAYERS MATCHING STRATEGY
Table 12: Ablation for attention matching strategy.
Attension Loss Type
Attentoon Matching
Amention Layers
II
Question Answering
Compositionl Reasoning
GQA
TextVQA
Avg
Sugurcrepe
BIVLC
Winoground
Avg
Ablutioe lavora
attoaatoon distilation
Cos. Sim.
Simple Adaptive Crropp
internediatte
60.2
56.3
68.0
61.5
8166
67.1
63.
50.7
65.6
Cas. Sim
insermedaat
61.4
55.9
03.6
620
82.
68.3
628
49.3
66.7
Cos, Sim
Intermistate
62.2
56.4
70
629
829
694
63.
51.1
667
This part details our ablation on different strategies for matching student and teacher layers, with full results in Table 12 and detailed explanation about each strategy is in Section [F] We compared our proposed Group matching against two strong baselines: Simple matching (uniform sampling of teacher layers) and Adaptive matching (pairing based on feature distance). The data shows that while both Simple and Adaptive strategies improve performance over no-distillation baseline, our Group matching consistently achieves the best results across all benchmarks. Notably, our method demonstrates a clear performance advantage over the next best method, Adaptive matching. Further- more, our approach more computationally efficient, as it avoid the complex calculations required to dynamically match layers inherent to adaptive strategies. This suggests that grouping layers pro- vides more stable and robust signal for the student. By averaging the behavior of block of teacher layers, our method likely smooths out layer-speciff idiosyncrasies and transfers a more generalized attention strategy.
H.2 DETAILED EXPERIMENT RESULTS ON RELATIONAL HALLUCINATION
Table 13: Relational Hallucination Experiment.
R-Bench
Reefknot
Model
Precision
Recall
FI Score
Perception
Cognitive
Total
Teacher (LLaVA-4B)
66.4
97.8
79.1
45.3
78.0
67.9
Studuct (LLaVA-4B)
60.5
96.3
74.3
40.1
70.8
61.3
L.aVA-KD
63.5
96.2
76.5
41.8
68.8
60.3
LLaVA MoD 2B
62.5
97.6
76.2
42.0
73.1
63.4
CompoDistill-22
65.7
97.6
78.6
43.2
77.3
67.9
As mentioned in the main text, we conducted experiments on relational hallucination benchmarks to demonstrate further benefit of our proposed method. Table 13 presents the detailed quantita- tive results of this evaluation on the R-Bench (Wu et al., 2024) and Reefknot (Zheng et al., 2025)
benchmarks.
The results substantiate our claim, showing that CompoDistill significantly outperforms other knowledge distillation methods on the R-Bench benchmark and nearly closes the performance gap to the teacher model. More strikingly, on par with the teacher model, showcasing its ability to handle complex relational challenges. These findings provide strong evidence that enhancing visual perception via CompoDistill effectively mit- igates relational hallucinations and boosts performance on complex visual reasoning tasks.
the Reefknot benchmark, our method achieves performance
H.3 (ADDITIONAL)) DETAILED EXPERIMENTS RESULTS ON TRAINING STRATEGY.
In this section, we provide additional experimental results analyze the effectiveness of our pro- posed threc-stage training strategy (DPT-DFT-SFT), To demonstrate its efficacy. we investigate the specific impact of each individual stage, with detailed results presented in Table 14
24

---

Published as a conference paper at ICLR 2026
First, the Distilled Pre- Pre-Trainig (DPT) stage shows significant impact, particularly Visual Ques- tion Answering (VQA) tasks. As seen by comparing configurations with and without DPT (e.g.. (1) vs. (3)), replacing standard pre-training with our distilled approach consistently yields substantial improvements in VQA scores. This highlights DPT's role building strong visual knowledge from the teacher.
Next, the Distilled Feature-Tuning (DFT) stage is crucial for enhancing Compositional Reasoning (CR) capabilities. The inclusion of DFT leads the most significant gains CR performance across all benchmarks. We attribute this improvement to the effective transfer of the teacher's ' fine-grained attention patterns through our attention distillation process, which is vital for understanding complex object relationships.
Finally, the Supervised Fine-Tuming (SFT) stage is indispensable. The results from the configuration without SFT (4) show catastrophic failure on VQA tasks, as the model completely fails to follow task instructions. This demonstrates that SFT is an absolutely critical final step for aligning the model's distilled knowledge with the specific formats and demands of downstream tasks.
Table 14: Ablation for training recipe and teacher adapter fetch module. Faiill to follow instructions (i.e., answer only single word).
Training Recipe
Tencher Adapter
Visual Question Answering
Compositional Reasoning
GQA
TextVQq
MME
Avg
Sugarerepe
SADE
BIVLC
Winoground
Avg
Ablation on different training recipe
(I) PT SET
57.9
50.6
61.2
56.6
73.3
63.9
$8.5
47.2
60.7
(2) PT + DET
59.5
54.6
64.8
59.6
79.
67.3
61.8
50.3
64.6
(3) DPT + SFT
60.4
56.3
64.4
60.4
72.6
61.6
58.1
49.9
60.6
18
28.0
DPT SFT
56
65.1
60.6
67.9
48.9
65.3
(*) DPT + SET
62.2
5644
70.1
62.9
82.9
69.4
63.3
$1.1
66.7
H.4 DETAILED EXPERIMENTS RESULTS ON SCALABILITY
H.4.1 DETAILED EXPERIMENTS RESULTS DATA SCALING
This section provides a detailed analysis of our experiments scaling data quality and data quantity, with full results presented in Table 15 For a clear point of comparison, we also include the perfor- mance of baseline model trained only with Supervised Fine- Fine-Tuning (SFT) on the same initial data volume.
Our findings clearly demonstrate the two-fold benefit of our approach. First, by using our distillation method, the model's performance significantly improves over the SFT baseline, even with the same amount of data. This confirms that distillation provides a higher-quality training signal. Second, when we use more of this high-quality data (doubling the training samples), the model's perfor- mance is further enhanced. These results validate that our distillation approach offers a substantial performance gain, which is then amplified by increasing the quantity of the training data. A detailed explanation of the training dataset configuration is provided in Appendix G
Table 15: Data scaling experiment.
Student
Training Samples
||
Visual Question Answering
Compositional Reasoning
GQA
Tex:VQA
Avg
Sugarcrope
BiVLC
Winoground
Avg
soaling experiment
61.2
Owenl Owenl.5-BB
12
Owent 1-188
Owenl.5-1B
2/
68.9
H.4.2 DETAILED EXPERIMENTS RESULTS ON STUDENT AND TEACHER LLM SIZE
This section provides a more detailed analysis of the relationship between teacher and student model sizes in our distillation framework, with full results presented Table I6
We first examine a 1.8B parameter student model. As the table shows, distilling from a larger 7B teacher yields stronger student model compared to distilling from 4B teacher, with improved performance on both VQA and CR tasks. It worth noting that we selected Qwen2.5-7B model as the larger teacher. This decision was made because preliminary evaluations showed that the perfor- mance of the Qwen 557B model not significantly higher than that of the Qwen 5-4B version;
25

---

Published as a conference paper at ICLR 2026
using the more capable Qwen2.5 architecture ensured more significant and meaningful gap in teacher capacity for this experiment.
This trend is further validated with smaller, 0.5B parameter student. The results show a clear progression: the 0.5B student first shows dramatic improvement distilled from a 1.8B teacher. Performance increases again, and quite substantially, when the same 0.5B student is distilled from an even larger 4B teacher. This demonstrates that even smaller student models can effectively absorb the enhanced capabilities of higher-capacity teachers.
its SFT-only baseline when
In summary, these experiments consistently show that a larger, more capable teacher model is a critical factor in producing stronger student, regardless of the student's size. A higher-capacity teacher provides a richer and more effective source knowledge, successfully transferring more powerful reasoning abilities through our distillation process.
Table 16: Student and teacher LLM Size experiment.
Student LLM
Teacher LLM
# Training Sumples
Visual Question Answering
||
Compositional Reasoming
GQA
TeatQQA
MME
Avg
|| Sugarcrope
SADE
BiVLC
Wineground
Avg
Large teacher experiment
Owen1 5-1.8B (SFT)
1,2 M
$7.9
50.6
61.2
56.6
73.3
63.9
58.5
17.2
60.7
Qwenl
Owanl 5-48
1.2 M
62.2
36.4
70.
62.9
82.9
69.4
63.3
51.1
66.7
Owen 1.581.8
Owen2.5.78
1.2 M
62.9
57.1
69.8
63.4
84.1
70.7
63.8
32.5
67.8
Owent 5.0.8 (SFT)
M
$4.9
45.5
$8.5
$2.0
52.9
51,4
39.5
49.8
48.4
Owenl.5.0..5
Owen Owenl.5.1. 1.881.8
M
57.3
48.9
58.1
54.7
57.3
52.8
44.0
30.4
$1.1
Owenl.5-0..3
Owenl.5-4B
M
60.1
50.3
59.5
56.6
59.5
59.0
49.1
30.5
54.5
H.4.3 DETAILED EXPERIMENTS RESULTS ON DIFFERENT LLM BACKBONES
To verify the generalizability of our distillation framework, we conducted an experiment using an entirely different architectural backbone, the Mobilel.LaMA (Chu et al. 2024) family. This section provides a detailed analysis of these results, which are presented the Table
Specifically, we applied our distillation method to a 1.4B parameter MobileLLaMMA as the student model, using 2.7B parameter MobileLLaMA model the teacher. We then compared its per- formance to a baseline 1.4B student trained with only Supervised Fine-Tuning (SFT). The results clearly show that our distillation method remains highly effective on this new architecture. The dis- tilled student model significantly outperforms the SFT-only baseline across both Visual Question Answering (VQA) and Compositional Reasoning (CR) benchmarks.
This successful application demonstrates the robustness and architectural independence of our ap- proach. It confirms that core principles our distillation method are not specifically tailored to the Qwen (Qwen et al. 2025) model family but can be broadly applied to improve the performance of different MLLM backbones, validating the generalizability of our findings.
Table 17: Different LLM backbones experiment.
Studest LLM
Tescher LLM
Training Samples
Visual Question Anewering
Compositioal Reawming
GQA
TestVQA
MME
Avg
Sugarcrope
SADE
BIVLC
Winagoonnd
Avg
Differel buckhone exporiment
MobileLLaMA-LL
52.2
37.5
MobileLLMMA-1.e MobilaL 2M
I GENERALIZATIIN TO OTHER ARCHITECTURE
To rigorously validate the robustness and generalizability the proposed CompoDistill framework beyond the LLaVA architecture, we extended our evaluation to the Qwen3-Vi VL family.
Experimental Setup. The Qwen3-VL. architecture differs fundamentally from the LLaVA design used our main experiments. It incorporates distinct training schemes and advanced architectural components, such as Native Resolution ViT (Na and DeepStack. We conducted experiments across varying model scales, ranging from 2B to 8B parameters. to assess scalability, For training efficiency in these ablation studies, we utilized a 50% subset of the Cambrian737K dataset, with the input resolution parameter max pixel va lues set to 32,768.
26

---

Published as a conference paper at ICLR 2026
Table 18: Different architecture experiment.
Student LLM
Teacher LLM
Training Method
Visual Question Answering
Compositional Reasoning
GQA
TextVQA
MME
Avg
Sugurcrepe
SADE
BIVLCC
Avg
Qwen3-VL-2B Owen3-VL-4B
SFT SFT
$7.2
50.9
70.4
59.5
85.5
79.3
64.9
76.5
59.7
56.7
80.0
65.5
91.6
84,4
67.2
81.1
Owen3-VVL-SB
SFT
60.0
58.2
80.9
66.4
93.0
85.5
68.1
82.2
Qwen3-Vi.220
Owen3-V1 VB.-BB
Naive KD
58.9
55.3
72.1
62.1
86.2
79.6
65.8
77.2
Owen3 2B
Owen3 4B
CompoDistill
99.4
58.9
73.2
63.8
91.0
81.4
67.1
79.8
Qwen3 VL
Qwen3 VL-SB
CompoDistill
59.5
59.2
73.7
64.1
91.5
81.5
67.4
80.1
Results and Analysis. Table 18 presents the performance comparison Visual Question An- swering (VQA) and Critical Reasoning (CR) tasks. Despite the structural differences the vision encoder and language model, CompoDistill consistently demonstrates strong performance improve- ments. This confirms that the visual attention misalignment bottleneck not specific to LLaVV but is broader phenomenon in MLLMs, and that our multi-stage distillation strategy generalizes effectively to diverse architectures and larger model scales.
DISCUSSION ABOUT TRAINING OVERHEAD
In this section, we provide a comprehensive analysis of the computational efficiency of our pro- posed framework, CompoDistill, compared to existing state-of-the-art knowledge distillation (KD) methods.
Quantitative Comparison. Table 19 summarizes the training overhead of method against other baselines, LLa and LLa VA-MoD. Our framework requires significantly fewer training hours compared to the baselines while maintaining comparable number of training stages. specif- ically, the total training time of CotpoDistill is approximately 170 hours (consisting of DPT: 28h, DFT: 88h, and SFT: 54h), which substantially faster than LLaVA-KD (~3200 hours) and LLaVA- MoD (~960 hours).
Table 19: Comparison of training overhead between our method and existing baselines. Our method achieves the lowest total training time with streamlined loss components.
Method
Training Stages
Loss Components
Total Training Time (hrs)
LLaVA-KI KD
3 stages
4 losses losses losses
~30
LLaVA-MoD
4 stages
9960
CompoDistill
3 stages
~~0
Computational Complexity Analysis. The efficiency of our method stems from the design of the distillation objectives. LLLVA-KD introduces significant computational bottleneck by calculating an expensive correlation matrix for all visual tokens between the student and teacher, resulting in complexity of ONN? d). where N, is the number of visual tokens and d the hidden dimen- sion. Similarly, the extended training time of LLaVA-MoD is attributed to the complex routing and computation within its Mixture of Experts (MoE) architecture. In contrast, our VAT attention loss CompoDistill operates with a much lower time complexity of O(N, x L), where L denotes the number of layers. This linear complexity with respect to visual tokens allows for highly efficient training without sacrificing performance.
Resource-Constraine Settings. To further address practicality under resource-constrainge en- vironments, the memory overhead of multi-module framework can be effectively mitigated. Since the teacher model is static during distillation, its outputs (logits and attention maps) can be pre-computed and stored offline. This strategy decouples the teacher's 's memory requirement from the student's training loop, allowing our method be deployable on GPUs with limited VRAM capacity.
27

---

Published as a conference paper at ICLR 2026
K ADDITIONAL EXPERIMENT ON ATTENTION MIXING
In this section, we investigate the causal relationship between the similarity of visual attention to the teacher model and the performance vision-centric tasks. While main paper (Figure 4 in Sec(.33 suggests this correlation, we conducted additional ablation studies to empirically verify that the performance gain primarily originates from the distilled teacher's visual attention.
Experimental Setup. We extended the experimental setup from See 3.3 by introducing three vari- ations of attention mixing strategies during inference. Keeping all other conditions identical, we compared the following settings:
LLaVA-KD: The baseline method.
+ Teacher Visual Attention (Ours): Incorporating visual attention from the distilled teacher (Qwen1.5-4B based).
+ Teacher Text Attention: Using the teacher's text attention instead of visual attention.
+ Other Visual Attention: Using visual attention from a larger, external MLLM (Qwen1.5-7B)) to test if simply using stronger model suffices.
+
Random: Injecting random attention maps as a control.
+ Teacher Attention: Using both the teacher's visual and text attention.
Quantitative Results. Table 20 presents the accuracy scores across three compositional reasoning tasks: Swap, Replace, and Add.
Table 20: Different architecture experiment.
Method
Swap 912
Replace 3,846
Add
# Data Num
2,754
(a)
VL--A
0.6218
0.8023
0.8435
(b)
Teacher Visual Attention (Ours)
0.6419
0.8140
0.8747
(C)
Teacher Text Attention
0.6157
0.8045
0.8535
(d) + Other Visual Attention (Qwenl .5-7B)
(S-BB
0.6354
0.8059
0.8567
(c) Random
0.2094
0.1708
0.1959
Teacher Attention (Visual Text)
0.6360
0.8210
0.8531
Analysis. The results provide clear empirical support for our framework design. First, utilizing the teacher's visual attention (b) consistently outperforms the baseline (a) across all datasets, confirming that visual attention is the key driving factor. This is further corroborated by the comparison between (b) and (c), which reveals that unlike visual attention, text attention does not yield similar improve- ments for vision-centric tasks. Second, critical observation arises from the comparison between (b) and (d), demonstrating that alignment more critical than mere model power. Although the external model (Qwenl.5-7B)
1-5-BB is significantly more powerful than teacher (Qwen1 1.54B), simply mixing its attention fails to achieve optimal performance. This indicates that attention alignment effective only when the student and teacher feature spaces are intrinsically aligned.
In conclusion, these experiments justify the design of VAT and TAF modules. confirming that precise alignment with the teacher's 's visual attention a requisite for improving compositional rea- soning capabilities.
L ADDITIONAL EXPERIMENT ON TEACHER ADAPTER FETCH MODULE
In this section, we address the potential concern that reusing the frozen teacher adapter (Teacher Adapter Fetch, TAF) might constrain the student's flexibility lock it into the teacher's visual- linguistic idiosyncrasies. We compare our approach against training a student-side adapter from scratch to clarify the trade-offs between efficiency and adaptability.
On the Lock-in Effect.. We first clarify that the primary goal of our framework to faithfully distill the teacher's 's rich visual capabilities into an efficient student model. Since both models share the same SFT and KD data, mimicking the teacher's generation behavior is deliberate design
28

---

Published as a conference paper at ICLR 2026
choice to maximize capability transfer rather than limitation. As supported by prior distillation studies, accurate alignment with the teacher often enhances the student's generalization rather than restricting it.
Comparative Experiment: Student-Side Adapter Training. To empirically validate the effec- tiveness of TAF, we conducted comparative experiment against
approach.
Setup: Instead of reusing the teacher's frozen adapter, we initialized new vision adapter on the student side and trained it align with the teacher's representation space. This process required an additional training stage using the LLaVA-1.3-5555 dataset before proceeding to the DPT, DFT, and SFT stages.
Overhead: We note that this explicit alignment strategy introduced significant computational overhead, requiring approximately 40 additional training hours.
Results and Analysis. Table 21 compares the performance of the Baseline (SFT), the Student- Side-Match method, and our proposed CompoDistill (TAF). As shown in the results, our TAF mod- ule consistently outperforms the Student-Side-Me MatdeMMacch approach. This demonstrates that reusing the teacher's adapter is not only more computationally efficient but also provides a more effective mech- anism for aligning the student's visual space with the teacher's, without compromising adaptability to the target tasks.
Table 21: Comparison of training strategies with computational overhead.
Student MIL.M
Method
Teacher
Extra Time
GOA
TexVQQ
VQA
MME
Avg
Wineground
CR SugurCrepe
BVVLC
Avg
S-1.8B
SFT
57.9
50.6
61.2
54.9
47.2
73.3
58.5
59.7
Owen1.S.1.8
Studear- Side-Match Match (Ours)
Qweni-5.4B @wen1 1-5-8
+00 +0h
56.8
52.3
68.5
59.2
50.6
78.2
60.8
63.2
Qwen1 18
56.4
70.1
61.9
82.9
63.8
M ADDITIONAL EXPERIMENT ON VISUAL ATTENSION ALIGNMENT MODULE
In this section, we investigate the impact of updating attention parameters during the distillation process. Specifically, address the hypothesis that freezing attention blocks in the middle layers while only updating Feed-Forward Networks might help mitigate potential over-mimicry or improve generalization.
Experimental Validation:
To empirically verify the necessity of attention tuning, we conducted a comparative experiment with two settings:
Freezing vs. Tuning.
Freeze: Freezing the parameters of the attention blocks in the middle layers and updating only the FFNs.
Tuning (Ours): Updating both attention blocks and FFNs as proposed in our VAT module.
Results and Analysis. Table presents the performance comparison VQA and Critical Rea- soning (CR) tasks. As shown the results, freezing the attention blocks leads to consistent perfor- mance degradation across all metrics compared to our standard setting. We interpret this as evidence that the self-attention mechanism is critical for learning the visual reasoning process. Restricting the update of attention parameters acts as bottleneck, preventing the student from effectively learning the spatial relationships and where to look required for complex visual tasks. This confirms that adapting the attention mechanism essential for successfully distilling the teacher's visual capabil- ities.
N DISCUSSION ON POSSIBILITY ABOUT TEACHER OVERFITTING
29

---

Published as a conference paper at ICLR 2026
Table 22: Impact of Attention Block tuning in the Visual Attention Alignment (VAT) module.
Student
Teacher
Attention Block
Strategy
GQA
TextVQA
VQA
MME
Avg
Winoground
CR SugarCrepe
BiVLC 63.3
Avg.
Owen1 5-1.8B
Qwen1.5-4 Qwen 5-4B
Freeze Tuning (Ours)
61.6
55.9
64.6
60.7
49.4
80.4
Qwenl 5-1 8B
62.2
56.4
70.1
61.9
51.2
82.9
65.8
In this section, address the concern regarding the risk of the student model strictly overfitting to the teacher's visual focus patterns, which might dampen exploration or degrade robustness. We provide both empirical evidence and a discussion on our loss function design to demonstrate that CompoDistill encourages intrinsic visual reasoning rather than brittle mimicry.
Empirical Analysis of Focus Patterns. To verify whether the student merely mimics the teacher acquires intrinsic ability to discern where to look, analyzed the relationship between at- tention alignment and task performance using the SugarCrepe dataset. We stratified the test samples based on the visual attention similarity between the teacher and student at visual understanding layers.
Table 23 summarizes the performance and data distribution across different similarity intervals. We observe two key trends:
Absence of Over-fitting: The majority of samples (55.5%) are concentrated in the moderate sim- ilarity range of [0.6, 0.7), rather than clustering in the highest similarity intervals (e.g., [0.9, 1.0). This distribution indicates that the student does not simply mimic the teacher's 's attention map but learns generalized representation.
Robust Intrinsic Reasoning: Notably, CompoDistill maintains strong performance (0.76) even in intervals where attention similarity is relatively low (0..5, 0.6)). This implies that the student has acquired robust visual reasoning capabilities, cnabling it to derive correct answers even when its visual focus partially deviates from that of the teacher.
Table 23: Analysis performance and sample distribution relative to Attention Similarity.
Similarity Range
[0.5, 0.6)
[0.6, 0.7)
[0.7, 0.8)
[0.8, 0.9)
[0.9, 1.0]
Performance Data distribution 5.5%
0.76
0.87 55.5%
0.82 15.0%
0.79 11.6%
0.82
12.4%
Design Choice of VAT Loss Function. Furthermore, our specific choice of Cosine Similarity for the Visual Attention Aligument (VAT) loss is a deliberate design decision to mitigate overfitting. In our preliminary ablation studies, we observed that stricter alignment objectives, such Mean Squared Error (MSE) or KL Divergence, led to performance degradation in VQA tasks.
We attribute this to the nature of the loss functions: while MSE forces the student to match the ab- solute magnitude of the teacher's values (encouraging point-to-poing mimicry), Cosine Similarity aligns the direction of the attention vectors. This approach teaches the student the relative impor- lance of visual tokens without enforcing rigid adherence to the teacher's is exact distribution, thereby fostering the learning of underlying reasoning patterns while preserving robustness.
O ADDITIONAL EXPERIMENT ON VISUAL GROUNDING
In this section, we empirically validate the impact of the Visual Attention Alignment (VAT) mod- ule on localization (grounding) capabilities. Given that CompoDistill demonstrates superior perfor- mance in fine-grained visual-centric tasks (e.g., Compositional Reasoning and Hallucination mit- igation), we hypothesized that the improved visual attention alignment would naturally extend to precise visual grounding.
Experimental Setup. We evaluated the models on the RefCOCO benchmark. To ensure a fair comparison. all models, including the baselines (SFT and LLaVA-KD), utilized Qwenl.5-1.88 as the backbone L.M.. Performance is measured using the Accuracy @0.5 (Acc@0.5) metric.
30

---

Published as a conference paper at ICLR 2026
presents the quantitative results. Our proposed method, Com- poDistill, significantly outperforms both the SFT baseline and the LLaVA-KD method. Specifically, CompoDistill achieves an accuracy of 57.8%, representing substantial improvement over LLaVA- KD (45.4%) This empirical evidence confirms that the fine-grained visual attention alignment pro- vided by VAT not only improves reasoning capabilities but also translates effectively to precise object localization.
Results and Analysis. Table 24
Table 24: Performance comparison on the visual grounding task (ReiCOCO-val),
Method
Backbone LLM
RefCOCO-val (Acc@0.5)
SFT
Owen1 15-.8B Qwenl. 5-1.8B
16.2
LLaVA-KD
45.4
CompoDistill (Ours)
Qwenl.5-1..0 55-.BB
57.8
P LIMITATION AND FUTURE WORK
A potential limitation of CompoDistill lies in its inability to capture the distinct information carried by each head within multi-hean attention (Zhao al. 2024 Kang et 2025 as it distills only the average of the teacher's s visual attention across all heads. This simplification may lead to information loss by overlooking the diverse roles played by individual heads. Moreover, CompoDistill assumes that the teacher and student MILMs belong required for logit-based distillation (Equation 4in the main paper). This constraint poses challenge when attempting to distill visual knowledge from teachers belonging to different model families.
same LLM series, as consistency vocabulary is
As future work, we aim to incorporate the head-specific characteristics of the teacher's s visual at- tention into the distillation process, enabling the student capture more fine- ane-grained and nuanced visual cues.
31


Reviews:
[{"review_id": "LdWKTVcT1j", "reviewer": "Reviewer_4Sdb", "summary": "The paper introduces CompoDistill, a knowledge-distillation framework for MLLMs aimed at transferring visual perception (compositional reasoning) rather than only visual recognition. The key diagnosis is visual attention misalignment between teacher and student in the visual understanding layers; the method (i) explicitly aligns attention over visual tokens via a Visual ATtention alignment (VAT) loss with a group layer matching scheme, and (ii) bridges teacher–student vision-language space mismatch via a Teacher Adapter Fetch (TAF) that reuses the teacher’s adapter with a light MLP mapper. A three-stage training (distilled pretraining → distilled finetuning with VAT → SFT) is used. Experiments across VQA and compositional reasoning benchmarks show gains on CR while keeping VQA strong; ablations support VAT/TAF design choices.", "questions": "1. What happens if you freeze only the attention blocks in mid layers during VAT but allow FFNs to adapt. Does that help avoid over-mimicry?\n\n2. Any evidence that VAT improves localization (pointing/grounding) accuracy measured directly?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Simple, orthogonal mechanisms. VAT aligns visual-token attention using cosine distance with a one-to-many group layer matching that’s more effective than simple or adaptive matching. TAF pragmatically resolves feature-space mismatch by reusing the teacher’s frozen adapter with a light MLP, making attention transfer workable in practice.\n\n2. Consistent CR gains without sacrificing VQA. Across SugarCrepe, SADE, BiVLC, and Winoground, CompoDistill outperforms KD and maintains competitive VQA, showing perception improves while recognition holds steady. The intermediate-layer focus and cosine loss are empirically validated as strong choices. \n\n3. Breadth and robustness signals. Relational hallucination benchmarks (R-Bench, Reefknot) show additional improvements, hinting at better grounding. Results with a stronger backbone indicate the approach is not tied to a single encoder/LLM stack.", "weaknesses": "**1.Causal evidence is still light.**\n\nThe analyses show correlation between attention similarity and accuracy and a mild improvement from attention mixing, but they stop short of stronger causal tests (e.g., counterfactual training where only non-visual tokens are aligned). For example, on Figure 2c it shows the difference between student or SFT model with teacher is small on CR; similar trivial difference in Figure 4 is also observed. The author should have done more either empirical or theoretical analysis justifying the analysis. Tightening the causal chain would reduce the risk that VAT is a proxy for other regularization effects. \n\n**2. Adapter reuse may constrain flexibility.**\n\nTAF fetches a frozen teacher adapter and adds a small projector, this could lock the student into the teacher’s visual-linguistic idiosyncrasies and hinder adaptation to novel modalities or encoders. A comparison against training a student-side adapter to match the teacher space (without reuse) would clarify trade-offs.\n\n**3.Risk of overfitting to teacher’s focus patterns.**\n\nFor hard CR that requires shifting attention relative to the teacher (e.g., ambiguous or adversarial scenes), strict alignment might dampen student exploration and degrade robustness. Evaluating under label noise, adversarial cropping, or counterfactual edits would reveal whether VAT encourages brittle mimicry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MrJf0VcMC", "reviewer": "Reviewer_BxQL", "summary": "In the draft, the authors proposed CompoDistill, a new Knowledge Distillation (KD) framework to better transfer the teacher model's visual perception ability to student model. Multiple experiments show that the proposed KD framework could help improve performance on compositional reasoning tasks while maintaining performance on VQA tasks.", "questions": "Please refer to the paper weakness section for more details and provide more justification accordingly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors conducted systematic analysis and identified visual attention misalignment is the key issue that prevent the distillation of the teacher MLLM's visual perception abilities to student model. Both the analysis and findings make sense and are technically sound to me.\n2. The two proposed components of Visual ATtention alignment (VAT) and Teacher Adapter Fetch (TAF) are both technically sound to me as well. VAT better align the visual attention and TAF better align the following process. \n3. The authors conducted extensive experiments to justify the proposed frameworks.\n4. Writing is good and easy to follow.", "weaknesses": "1. The current approach seems limited to same model family. How to extend the distillation to cross modal family?\n2. The proposed three stage training is very complicated compared to the standard single stage approach. More justification might be beneficial to make sure the gain outweigh the cost.\n3. For Table 2. it seems that VAT module is hurting on TextVQA tasks (56.4 vs. 56.5). Any idea what might be the issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TFsV9DizTl", "reviewer": "Reviewer_euXb", "summary": "This work addresses a gap in the effectiveness of knowledge distillation techniques in distilling visual perception techniques from teacher MLLMs to student models. An analysis is conducted to identify the reason for this issue, which is attributed to visual attention misalignment between the teacher & student models. A new knowledge distillation approach is proposed called CompoDistill which aims to address this issue by aligning visual attention patterns across the student and teacher models. Experiments are conducted which show that the proposed method offers improvements for compositional reasoning tasks.", "questions": "1. From the ablation results (Table 3) it seems that VAT is more critical for compositional reasoning while TAF is more important for VQA. Do you have any insights into why this is the case?\n2. Have you evaluated the effectiveness of CompoDistill for distillation from larger models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This work identifies and addresses an important gap with existing knowledge distillation techniques which has been ignored in previous studies.\n2. The analysis of attention pattern misalignment provides good conceptual motivation and support for the proposed CompoDistill method.\n3. The experimental results cover a good range of VQA and compositional reasoning datasets. \n4. Ablation studies provide interesting insights into the importance of different CompoDistill components (VAT & TAF).", "weaknesses": "1. Discussion of related work is relegated to the appendix, which seems inappropriate for a full-length main conference paper.\n2. The main experiments (Table 1) provide results for CompoDistill using only a single student/teacher model pair based on Qwen 1.5 (1.8B parameters for the student and 4B for the teacher). This makes it difficult to assess how well the method will generalize to different model architectures and larger model sizes. Limited aggregated results are provided for MobileLLaMA in Table 7, but it seems that the teacher model used here was also quite small. \n3. CompoDistill does not seem to offer significant improvements over other baselines on VQA. I understand that compositional reasoning requires more finegrained perceptual abilities, but shouldn't greater visual attention alignment between the student and teacher also lead to some benefits for VQA as well?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fw2CBgdvqd", "reviewer": "Reviewer_RxKW", "summary": "This paper identifies visual attention misalignment as the main bottleneck in distilling visual perception ability in MLLMs through extensive attention-based analyses and large-scale experiments. The authors propose a knowledge distillation framework combining Visual Attention Alignment and Teacher Adapter Fetch based on the findings, which achieves significant improvements in compositional reasoning performance while maintaining strong visual recognition ability, and demonstrates robust generalizability and state-of-the-art performance across multiple benchmarks and model architectures.", "questions": "1. How much additional training cost does your multi-module, three-stage approach incur compared to existing KD methods?\n2. Have authors validated the conclusions and method on architectures other than LLaVA?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper conducts extensive and systematic analysis prior to method design, deeply investigating the root cause of poor visual perception distillation through detailed teacher–student attention comparisons across tasks and layers, and by quantifying the relationship between attention similarity and performance.\n\n2. It proposes an innovative and well-structured knowledge distillation framework that combines Visual Attention Alignment and Teacher Adapter Fetch to address both attention alignment and vision–language feature space mismatch issues, with a Group Layer Matching strategy that preserves richer teacher perception knowledge compared to traditional one-to-one matching.\n\n3. The work presents extensive experiments across multiple compositional reasoning benchmarks, showing significant performance gains while maintaining strong visual recognition ability on VQA datasets, and demonstrating robust generalizability and state-of-the-art results across different backbone architectures.", "weaknesses": "1. The analysis in this paper is primarily conducted on the LLaVA, and the conclusions have not been systematically validated on other types of MLLMs with different training data compositions, architectural designs, and especially those employing ViT-based vision encoders. This raises uncertainty about whether the identified “visual attention misalignment” bottleneck and the effectiveness of the proposed CompoDistill framework generalize well to broader scenarios.\n\n2. The proposed KD framework incorporates multiple modules and a multi-stage training strategy, which may introduce higher training overhead compared to existing KD methods. However, the paper does not provide a quantitative comparison of this additional cost nor discuss its applicability under resource-constrained settings, limiting a full assessment of the method’s practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)