Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Authors: Zichen Wen
Shaobo Wang
Yufa Zhou
Junyuan Zhang
Qintong Zhang
Yifeng Gao
Zhaorun Chen
Bin Wang
Weijia Li
Conghui He
Linfeng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Efficiency
Multi-model Large Language Model
Token Compression
Primary area: applications
TL;DR: We propose a progressive consistency distillation framework that enhances the efficiency of MLLMs by significantly reducing computational cost while preserving strong performance.

Abstract: Visual tokens consume substantial computational resources in multi-modal large models (MLLMs), significantly compromising their efficiency. Recent works have attempted to improve efficiency by compressing visual tokens during training, either through modifications to model components or by introducing additional parameters. However, they often overlook the increased learning difficulty caused by such compression, as the model’s parameter space struggles to quickly adapt to the substantial perturbations in the feature space induced by token compression. In this work, we propose to develop Efficient MLLMs via Progressive Consistency Distillation (EPIC), a progressive learning framework. Specifically, by decomposing the feature space perturbations introduced by token compression along the token-wise and layer-wise dimensions, we introduce token consistency distillation and layer consistency distillation, respectively, aiming to reduce the training difficulty by leveraging guidance from a teacher model and following a progressive learning trajectory. Extensive experiments demonstrate the superior effectiveness, robustness, and generalization capabilities of our proposed framework.

Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Zichen Wen 1.2
Shaobo Wang'
Yufa Zhou
Junyuan Zhang* Qintong Zhang'
Yifeng Gao
Zhaorun Chen Bin Wang" Weijia Li
Conghui H Linfeng Zhang
EPIC Lab, Shanghai Jiao Tong University SShangaii Laboratory
DDuee University "The University of Hong Kong
'Peking University University of Chicago "Snn Yat-sen University
heconghui@pjlab wrg.g................oommm cn, zhanglinfen@@sjtuu edu. cn
o Code: http://gthhbbboom////cteee//////
Abstract
Visual tokens consume substantial computational resources in multi-modal large models (MLLMs), significantly compromising their efficiency. Recent works have attempted to improve efficiency by compressing visual tokens during training, either through modifications to model components or by introducing additional parameters. However, they often overlook the increased learning difficulty caused by such compression, the model's 's parameter space struggles to quickly adapt the substantial perturbations in the feature space induced by token compression. this work, we propose to develop Efficient MLLMs via ProgressIve Consistency Distillation (EPIC), progressive learning framework. Specifically, by decompos- ing the feature space perturbations introduced by token compression along the token-wise and layer-wise dimensions, we introduce token consistency distillation and layer consistency distillation, respectively, aiming to reduce the training dif- ficulty by leveraging guidance from a teacher model and following progressive learning trajectory. Extensive experiments demonstrate the superior effectiveness, robustness, and generalization capabilities of our proposed framework.
Introduction
equip large language models (LLMs) 54 23 with the ability to understand visual information, exhibiting remarkable capa- bilities across diverse range of multi-modal tasks, including image captioning, visual question answering (VQA), video understanding [61], and multi-modal reasoning [62].
Multi-modal large language models (MLLMs) 32
81
39 42
However, unlike LLMs
66 59], which only need to process small number of information- presents
dense text tokens the introduction of substantial visual tokens in MILMS
significant computational challenges. This issue becomes particularly pronounced when handling high-resolution images (35] or multi-frame videos (53].
To address this issue, a natural approach is to reduce visual tokens
Recent advances have introduced various token compression techniques eliminate vision tokens in training approach [78 633. Most of them adopt either a token importance-basse strategy 77 24] or a token redundancy-basee strategy [57] [22 Although the aforementioned non-parametric methods avoid additional training costs, they inevitably incur significant performance degradation. To achieve
67
"Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

* Parmerric Optimun
To Easy drup frum 0% to
0% Token Compresion
20% token compression
Congresion
point
Optiman
Eu 40%
deop from 20%
compresion
Local optima crapt Hard to drop from this to dif, tuken
Parm
Opimmm ostoon
compression optinnal point
(a) Loss Landscape of Compression Ratio = 0%
(b) Loss Landscape of Compression Ratio = 20%
(c) Loss Landscape of Compression Ratio 40%
(d) Loss Landscape of Compression Ratio 60%
Figure 1: Progressive Consistency Distillation vs. Direct Training. Each subplot shows the loss landscape under the corresponding token compression ratio, with the optimum indicated. Our method reaches the objective via progressive learning trajectories, while direct training remains challenging.
a better performance--fficiennyy tradc-off, recent training-awars token compression methods have attracted significant attention [ 14. Beyond early approaches [22 that employed Q-former, MQT- LLaVA 26 proposes more flexible Q-former capable of dynamically encoding visual information into variable-length visual tokens.
proposes TokenPacker, coarse-to to-rse- visual projector that progressively refines coarse tokens with visual information. In addition to refining the model architecture and upgrading model components, VoCo-LLaMA [70] and VL-VMniini [76] build upon observations of the transfer of visual information within language models, attempting to transfer visual information into a small set of VoCo tokens via attention modification, or into textual tokens with transformer- basee pre-fusion module.
B3
Although these methods improve model efficiency while largely preserving performance, their improvements primarily come from architectural enhancements or newly introduced modules, often overlooking the learning challenges posed by token compression during training. As illustrated in Figure when token compression is applied during training, the distribution of the compressed token sequence inevitably differs from that of the full token set. This discrepancy can be regarded as a perturbation in the feature space, which shifts the model's optimal point in the parameter space. The higher the compression ratio, the greater the perturbation introduced, and consequently, the further the optimal point drifts. The goal of training-aware token compression is to guide the model to progressively adapt from the original optimum (under full tokens) to a new optimum corresponding to the compressed token distribution. Figure (d) shows that directly training a model with compressed tokens often leads to suboptimal solutions, as the optimization process can easily get trapped in poor local minima, making it difficult to reach the desired optimum under heavy compression.
In this work, we propose a progressive consistency distil- lation learning framework EPIC tailored for token com- pression, where a single MILM simultaneously acts as both teacher and student through weight sharing. From a token-wise perspective, we introduce Token Consis- tency Distillation (TCD). At the early stages of training, both the teacher and student models adopt a very low token compression ratio, indicating relatively casy learning task without significant optimal point drifts (as shown in Fig. (b)). As training progresses, the com- pression becomes increasingly aggressive, forming a progressive learning trajectory (see Fig. (b) to (d)). Al- though the final optimal point under heavy compression remains far from the initial optimum, each intermediate optimum along the trajectory is relatively close to its pre- decessor, making each transition more manageable and casier to optimize. Moreover, the teacher consistently uses slightly lower compression ratio than the student, introducing a compression ratio gap between them. We argue that when the gap too large, the student may struggle to benefit effectively from the teacher's ' guid- ance 20 Therefore, the compression ratio gap is also designed to follow a progressive learning strategy, gradually increasing over time to ease the learning process. From a layer-wise perspective, we introduce Layer Consistency Distillation (LCDD.. Based on observations from prior work (7], the significance of visual tokens diminishes notably in deeper layers, suggesting that compressing tokens
TCD (Ouns), 64 visual MMMM
65.0
LLaVA-s.. S, 526 viua!
MMBench Accuracy (%)
9.T FLOPs
80.0
57.5
56.0
LLaVA-11.5 5
VoCo LLaMA
TakenPackeer
MOT-LLawA
50.0
QT-LLaVA
47.5
TCD (Ours)
LCD (Ours)
10
100
578
# #isual Tokens
Figure 2: MMBench accuracy vs. number of visual tokens for various methods. TCD (Ours) and LCD (Ours) achieve competi- tive accuracy with far fewer tokens, lower FLOPs, and smaller KV cache compared to LLaVA-v1.5, highlighting its efficiency.

---

at these layers has minimal impact on the model's 's feature space and output. Therefore, in LCD, the token compression is progressively shifted from deeper to shallower layers as training progresses, implicitly following an easy-to-hard learning paradigm. Meanwhile, compression gap between the teacher and the student is maintained to encourage effective guidance from the teacher.
Our primary contribution lies in proposing EPIC, a progressive consistency distillation learning framework, which demonstrates compatibility with diverse token compression techniques. Within this framework, we introduce Token Consistency Distillation and Layer Consistency Distillation from token-wiss and layer-wise dimensions, respectively. This approach enables the training of robust and highly generalizable models through a progressive learning strategy without requiring modifications to the model architecture. Compared prior approaches, comprehensive experiments validate the superior effectiveness, robustness, and generalization capabilities of our proposed framework.
2 Related Works
Multi-modal Large Language Models. Multi-modal Large Language Models (MLLMs), pio-
have successfully showcased promising results wide variety of vision-language perception and reasoning tasks. Existing MILMMs typically employ a pre-trained vision encoder (e.g., CLIP
neered by [4
81 15
71
g
4
and SPgLIP 73 551 to extract visual features, which are then pro-
jected into the LLM's input space via visual projector (e.g., MLP and Q-former (32)), enabling the model to process both visual embeddings and user instructions for multimodal understanding and response generation. Recent studies 775 74) have highlighted the limitations of MLLMs in fine-grained visual perception tasks. To this end, more advanced MLLMS have attempted to increase the number of encoded visual tokens by employing dynamic resolutions [1] I0 40, 31] or arbitrary resolutions [58] 4 to process nigh-resolution images, thereby enhancing their performance in visual understanding tasks. However, due to the quadratic complexity of the attention mechanism [36], the resulting longer token sequences pose significant challenges to both inference speed and memory
usage.
Visual Token Compression. Visual tokens typically outnumber text tokens by orders of magnitude
while containing greater spatial redundancy than information-dense text 48 Recent work has
explored both training-free [24
77 80 43 [77 and faiiing--wwrre [5]
1
60
compression
approaches, with the latter showing superior performance potential despite requiring additional
training (63. Training-free methods generally follow two paradigms, importance-bane based strategies
like Fast' 7 and VLL (78) that use attention scores, and redundancy- approaches such as DART 64 and G-Prune (28) that assess token similarity. Early training-aware work focused on paramcter-free strategies, including LLaVA-PruMerrs ' attention-bann based merging [50) and VoCo- LLaMA's compressed VoCo tokens (70]. Subsequent research explored architectural modifications through lightweight component replacements [16
19 34. More advanced approaches include MQT- LLaVA's dynamic Q-former for variable- token encoding (26 and TokenPacker's ' coarse-to- iterative condensation [33] Recent methods like VA-Minn [76] achieve near-lossless token compression through extra auxiliary modules, though they often overlook the training challenges introduced by feature space perturbations during token compression.
3 Methodology
3.1 Preliminary
Multi-modal Large Language Models (MLLMs). An MLLM typically consists of three modules: visual encoder VE, modality projector (e.g., MLP), and a language model LM. Given image I, the visual encoder extracts patch-level features Z. E R. N Nxd,, projected into visual tokens e E R." N Xdn;
ey MLP(VE(Z)),
(I)
where N is the number of image patches, d, is the encoder output dimension, and d, is the LM hidden size. Meanwhile, a text prompt P is tokenized and embedded into a sequence of text tokens C, Lxdn, The visual and text tokens then concatenated to form the full input sequence:
e ex] E
Positional embeddings are added to x to encode spatial and sequential structure. The language model then autoregressively generates output tokens y/ E (with vocabulary V) one token time:
y LM(X, vc.), for i 0,
(3)
3

---

(a) Vanilla Distillation
(i) Token Consistency Distillation
Compression Ratio
-. KL loss
p stu
plea
r U P
R
Student
Two Different
4
Student
Models
Teacher
max(0, - A
Teacher
smaller mode!
bigger model
Training Time
mmmm
(ii) Layer Consistency Distillation
(b) Progressive Consistency Distillation (Ours)
Compression From Deep Layers to Shallow Layers over Trinning Time
KL loss
p
stu
plea
Layer 1
Layer 2
Layer 3
Layer N-1
Layer N
Shared Weights
Student
Teacher
Student retains fewer tokens
Teacher retaiiss more tokens
x
...
x
t=T
t=1
Figure 3: An overview of Progressive Consistency Distillation. (i) Token Consistency Distillation
progressively increases token compression ratio over time. (ii) Layer Consistency Distillation shifts token compression from deep to shallow layers, promoting layer-wise consistency during training.
where M<v {30. U1, 11-1) denotes the previously generated tokens.
Progressive Consistency Distillation Learning. Training MLLMs under aggressive token com- pression introduces significant feature space perturbations that hinder convergence. To address this, we propose Progressive Consistency Distillation Learning (PCDL), which gradually increases compression difficulty and facilitates the model's progressive convergence to the final objective (see Fig. D. Our Token Consistency Distillation (TCD) progressively increases the token compres- sion ratio over training steps from token-wise perspective, while Layer Consistency Distillation (LCD) initially applies token compression in deeper layers (minimal impact 7 76) before gradually shifting shallower layers, performing progressive learning in layer-wise manner. Both employ Consistency Distillation with Shared Weights, where teachers with slightly lower compression ratios (e.g., 5% less) provide manageable guidance [20), progressively transitioning to stronger teachers (e.g., 10% lower ratio) for staged mentorship.
3.2 Theoretical Intuition: 1D Prototype for Progressive Consistency Distillation Learning
Scalar center path. To provide intuition for Progressive Consistency Distillation Learning (PCDL), we introduce a one-dimensional prototype where model predictions are scalar values 0 E R, and each target is given by compression-dependenl center G, IR for compression ratio E [0, mmxx]. We assume that the mapping e [0, mmax] + R satisfies:
(S1) Monotonicity: is differentiable with e 2 0 for all r.
(S2) Lipschitz slope: There exists 0 such that |4/1 7 and |c., Cr2l 7/1 r2/ for all r1, T2-
(S3) Convexity: e is convex, i.c., (r(r) 2 O.
Two quadratic objectives. We compare two learning objectives:
Lair (r, 0) = (0 e2
Croog 0) = (0 - c2) (x-)2,))
with constants 0 < A < 1, 1 A < Tmax The second term acts as regularizer pulls the prediction toward a slightly less compressed teacher target, mimicking the KL-based distillation loss in EPIC
Analogy to EPIC. In EPIC, the teacher operates at a slightly lower compression ratio (r- A) and provides smoother targets via KL consistency. The scalar prototype captures this: Ldie corresponds to direct supervision from the current target c,, while Lpoo encodes progressive distillation by incorporating past (easier) target cr-A- As in EPIC, the regularizer improves training stability by reducing sensitivity to abrupt changes in the input space induced by token compression.
Exact minimizers. Both objectives are strongly convex in f with closed-fome solutions:
gprog + Acr
gdit
Cr,
1+>

---

Schedule and path length. Let 0 = < <n < < PT = Tmax denote any compression schedule. The total variation of a path {.+} is defined as:
T TV( Σ 2++1 - rel-
1=0
We show below that the progressive path yields strictly shorter total variation, reflecting the smoother optimization trajectory induced by PCDL.. We delay the full proof to Appendix [B]
Theorem 1 (Scalar path gain, Proof in Appendix B). Under assumptions (SI) the total variation
of the progressive path is strictly smaller:
TV((O)))) 3 Tmaxx:
1 + An
(V(P))))))) I
1+X (V < TV (V(())))
c(r) - c(r A)
where
sup
A] c(r + A) c(r) E {
rE|A, Tiax
3.3 Token Consistency Distillation
We consider single MLLM fo that plays both teacher and student roles by sharing parameters 0. Let I denote the input image and P denote the language prompt. Let C(I, r, E) be a token compression operator that, at layer l, compresses the visual tokens extracted from I with ratio E [0, 1]. i.e.,
retaining only a fraction 1 of visual tokens. Lett t {0. ... T} index training steps.
At iteration 4, we sample a student compression ratio from a gradually shifting uniform distribution: rt ~ U (Remin,..
))
(4)
where R linearly increases from m (e.g., 5%) to R Riux (e.g., 90%) with training steps, and R grows more slowly from 0% to most 50% training progresses. This creates an easy-to-hard curriculum for the student model by gradually narrowing and shifting the sampling range towards more aggressive compression.
We then define the teacher model's compression ratio as r!a max(0, r:th - -.),
(5)
where the compression gap A, also increases gradually from Ag = 6min to Ar = 6max (e.g., up to 30%), ensuring that the teacher consistently sees slightly less compressed inputs than the student.
Concretely, we form two forward passes through the shared model fo:
h tea fo tea P);P); h fo(C
(6)
where both token compressions occur at the fixed Transformer layer L.
We incorporate the Token Consistency Distillation (TCD) loss as an auxiliary objective alongside the main supervised fine-tuning (SFT) loss. Specifically, the overall training objective is formulated as:
Lusa (0) (1 A) (0) + X Lfc
(7)
where Lsrr(P) (f) denotes the autoregressive cross-entropy loss over language outputs, and is balancing coefficient for the distillation loss, empirically set to 0.7.
The TCD loss is defined the Kullback-Leibler (KL) divergence
29
25 between the output logits
distributions of the teacher and student:
LTCD EI.P. [KL (p || 1 [t
where pi = Softmax(h //* and n = Softmax (7* are the temperature-scaled output logits distributions from the teacher and student, respectively, and is a temperature hyperparameter.
By progressively increasing both the student's compression range [Rsty Resu and the teacher-studed gap A,, implement progressive learning in token-wise manner. At early training stages, the student experiences mild compression and benefits from a closely aligned teacher, while later stages, endures stronger compression with increasingly distinct teacher guidance.
Any plug and play token compressor (e.g., FastV 77. DART 64)
compression operator here.
5

---

3.4 Layer Consistency Distillation
Prior work 7 76 shows visual tokens receive negligible attention in deeper layers while shallow layers play more critical role for visual modality. This motivates our Layer Consistency Distillation (LCD) strategy: performing compression in deeper layers first minimizes output perturbation and feature space distortion, then progressively moves to shallower layers.
Let be the total number of Transformer layers in the language model. Define a normalized training progress: 3r
7. We then select a single compression layer for both teacher and student:
4 Round(L B, emin)), 4, miins miin 1,...,L
where min denotes the shallowest compression layer. Thus at t = O we compress at the deepest layer to L, and by T the shallowest layer fT min, realizing layer-wise progressive learning.
At training iteration 6, we sample the student model'ss token compression ratio from [Tminy mmax]
[[(mmm; l'max);
(10)
Typically, Tmin is set to 0.2, while Tmax is configured as 0.9 and define the teacher model's compres- sion ratio by subtracting the compression ratio gap Ag:
1 max(0: (0, r:th - At).
(11)
We then perform two forward passes through the shared model Se at layer ly:
h tea fo (C P); h fo nu ())
(12)
The Layer Consistency Distillation (LCD) loss is defined analogously to TCD, as the KL divergence between the teacher and student output logits distributions:
(O) = E, (p [
p Softmax(h** p Softmax(h$*/1) (13)
Finally, we integrate both distillation terms into the overall training objective:
(rocal()) = (1 - A) (SFT(!) + X (O)
(14)
4 Experiments
4.1 Experimental Setting
Implementation Details. We implement EPIC based on LLaVA 41
40 without introducing any
modifications the model architecture. Specifically, we adopt CLIP VIT-L/I4 [4] as vision encoder, utilizing its officially pretrained projector, and Vicuna-v1.5 [12] as our LLM. algname only requires performing the second stage training, which involves visual instruction tuning on the LLaVA-665K instruction fine-tuning dataset. To demonstrate the effectiveness and generalizability of method, we incorporate three representative token compression techniques (DART (64, FastV (7], and Random token pruning) into EPIC for training. More implementation details about our proposed method and baselines are provided in Appendix D
We evaluate our model across 10 representative visual understanding benchmarks. Further details about benchmarks can be found in the Appendix D33
Evaluation Benchmarks.
Baselines. As shown in Table I we compare our method with various visual token compression
techniques, including QT-LLaVA VA, MQT-LLaVA (26). LLaMA- VID [34), VoCo-LLaMA (70). Token- Packer [33], and VA-Mini [76). We also list other MLLM's results for comparison, including
BLIP-2 [2] InstructBLIP [15]. IDEFICS 33), Qwen- WW B. Qwen-VL-Chan VL-Chat 33, SPHINX (38).
mPLUG-Ow12 (6). and vanilla LLaVA 41
Please refer to Appendix D.4 for more details.
4.2 Experimental Results on Benchmarks
Table I presents experimental results on 10 representative visual benchmarks. Notably, our method and MQT-LLaVA are among the few approaches enabling flexible control over token compression (36 ~ 256 tokens) with single trained model, adapting efficiently to varying resource constraints.

---

Table 1: Performance 10 visual understanding benchmarks. "Res." is resolution, and '#Vision Tokens' is the number of vision tokens. Both training and inference employ DART as the token compression strategy for our methods. Parentheses in (%) column show diffs vs. LLaVA-v1.5.
Methods
L.M
Res.
AVision Tokens
vQ 6Q4
VirWar
SQs!
VQA"
POPE
MME
MMB
MMB-
CN
OCR Bench
Arg. (%)
BL.P-22
k2
Vicuns- 13B
224
32
65.0
41.0
19.6
61.0
42.5
85.3
InstractB1.1P
16
Vicuns788
234
32
66.3
40.2
34.5
60.5
50.1
63.9
1500
36.0
2.59
Instrsc(BLL.P
Vicun-
33.4
$4.1
1530
36.9
252
IDRFICS-9R
1aMA-77
274
50.9
35.5
25.9
75.3
1027
48.2
244
IDFFICS-00R
30
1LMMA-688
234
64
60.0
45.2
36.0
30.9
1076
54.5
20.1
277
Qwen-V] VL. X
Qwen-7B
448
256
99.3
35.2
67.1
63.8
1708
38.2
133
Qwen-V1.Clom
Qwen-7B
448
156
SPHINX
114MA-138
234
78.1
39.9
69.3
51.6
50.7
66.9
SPMINX-2X 18
11aMA-138
762
2890
80.7
63.1
44.9
70.6
61.2
$7.2
65.9
@PLLIG-OW12
11aMA-78
448
1034
79.4
$6.1
$4.5
68.7 7
$4.3
64.5
Video-LLaVA
Vicun=7BB
$3.1
1542
MaVA-v155
Vicuns-BB
336
576
72.2
61.9
52.5
68.3 3
SK.
$5.9
1783
64.1
55.8
319
61.4
I.MMs
with Jewer vision tokens
AveragePPoolig
Vicuna70
336
64
63.0
55.5
18.4
68.6
52.6
79.2
1579
59.6
69.5
2.58
55.9 (:55)
Vicun-78
356
$1.4
75.4
1402
46.4
MQT--1aaA
20
Vi
336
62.0
57.7
53.6
69.2
28.6
83.9
1777
60.5
244
66
MOT-LLaVA
20
Vicuna:BB 7B
336
64
65.6
58.7
56.3
68.1 4
32.5
83.1
1810
61.3
53.7
260
56.8 (4.0)
MQT-LIaVA
25
Vicun-7B
336
178
66.2
99.8
$4.6
69.3
35.7
84.3
1773
62.1
53.6
766
57.6 (3.9)
MOTT-LLaVA
192
66.9
55.1
1784
62.0
263
57.7 (377)
MOT-LLaVA
Vicunz-B
336
256
68.3
60.1
36.6
69.0
37.1
$4.6
1740
61.7
53.0
273
57.8 (3.0)
QU-LLaYA
Vicuns78
336
256
60.3
$1.5
68.1
36.9
$4.1
1771
62.1
53.9
265
LLMMA-VID
Vicuna
336
55.5
56.2
68.8
89.0
83.1
VoCo-LLaMA
Vicuns7B
80.8
1594
56.4
TokenPacker
Vicuns-B
336
144
71.3
62.0
56.6
70.5
43.8
56.2
1716
63.9
303
59 9 (115)
TokenPPacher
3
Vicuna 7B
336
36
58.6
50.2
83.7
62.8
LLaVA-Miii
1
Vicun=7B
336
144
58.1
56.3
14.8
25.3
26.0
82.3
1325
24.8
132
MLaVA-Maal
70
Vicuna7BB
336
10.4
21.4
25.1
52.3
1324
23.8
Oury
LL TCD
Vicuna78 78
336
236
72.7
61.4
$6.]
69.8
$7.0
85.8
1807
66.1
54.8
310
(+03)
LLaVA-11. ICD
Vicuna
336
192
71.6
56,0
70.0
56.9
55.3
1813
65.8
304
61.4
L
69.7
1961
65.6
299
61.3 (4.1)
LL
Vicun=-BB
336
64
66. I
$7.1
35.1
71.1
$4.8
79.2
1809
64.2
53.0
286
59 (2.0)
LLAVA-1.++CD ICD
Vicuna 7B
336
36
62.1
$4.9
55.2
71.3
53.6
75.8
1747
62.4
51.5
262
57.5 (339)
LCC
Vicun=76
336
72.6
31.4
312
(103)
1LVAA-1...
Vicuns-B
336
192
71.3
61.5
57.6
70.0
56.7
85.3
1830
64.4
55.9
316
62 (+00)
LLAVA-15+
Vicuna: 7B
336
128
69.2
60.6
$7.9
69.8
$6.3
54.2
1832
61.1
55.3
306
61.3 (OLD)
LLVAA-15+ LCD
Vicun=7B 7B
336
64
66.0
58.3
57.8
69.7
54.3
81.2
1794
62.1
52.9
280
59, 4 (:200
1aVA-115+
Vicun-78
336
When retaining 128 tokens, our framework achieves performance comparable to vanilla LLaVA- v1,5-7B, while surpassing it with 192+ visual tokens, strongly suggesting significant redundancy in visual tokens. Compared to other training-aware methods involving model modifications (e.g., MQT-LLaVA, TokenPacker), our approach maintains superior average performance, particularly excelling on MME, MMBench, and VQA V2. This indicates that effective training strategies are as crucial as architectural modifications for token compression. The results also demonstrate our method's robustness across compression ratios: with just 64 tokens, performance degrades by merely 2% versus vanilla LLaVA, while maintaining minimal (< 1%) variation at 128 ~ 256 visual tokens.
4.3 Efficiency
Table 2: Inference efficiency analysis of EPIC, A denotes the reduction ratio. All experiments are on
POPE (8, 910 samples) using an A100 GPU. Token compression is fixed at the 2nd layer.
Method
Visual Tokens
KV cache (MB)
CUDA Time (s) I
FLOPs (T)
Value
A
Value
A
Value
A
LLaVA-v1.557B
576
367.2
1103.5
9.3
EPIC + FastV
64
40.9
88.9%
749,1
32.1%
1.5
83,9%
EPIC- DART [64
64
40.9
88.9%
744.3
32.6%
1.5
83.9%
EPIC- Random
64
40.9
88.9%
697.3
36.8%
1.5
83.9%
We discuss the efficiency of EPIC, including training-time efficiency and inference-time efficiency. As shown in Table Appendix [D2] in contrast to other training aware token compression method, especially those that alter the model architecture, our approach only requires supervised fine-tuning, completing training in approximately 12 hours on 8 A A100 GPUs. Most token compression methods that modify the model architecture require two or even three training stages. Furthermore, the replacement addition of new model components necessitates more training iterations to properly
7

---

Table 3: Ablation study Token Consistency Distillation. "w/o Distillation Loss" disables teacher supervision by zeroing LTCD. "w/o Progressive Compression Ratio" uses a fixed 88.9% compression ratio. DART is employed as the token compression strategy during training.
Method
VQA V2
GQA
VizWiz
SQA'
VQA
POPE
MMB-CN OCRBench Avg.
Retain
(1 77.8%)
TCD (Ours)
69.7
59.9
54.9
70.8
56.6
84.5
1861
65.6
54.3
299
61.3
w/o Distillation Loss
67.2
60.4
53.2
68.6
57.3
83.2
1745
63.8
53.5
289
59,8
w/o Progressive Compression Ratio
67.1
58.9
49.7
70.0
54,1
84.3
1788
63.8
51.5
277
59.1 (-2.2)
Retain 6d Tokens
(J. 88.9%)
TCD (Ours)
66.1
57.1
55.1
71.1
54.8
79.2
1809
64.2
53.0
286
59.4
wlo Distillation Loss
65,7
57.9
53.6
69.8
54,9
78.3
1671
61.6
52.1
272
58.1 (-1.3)
w/o Progressive Compression Ratio
64.3
57.9
50.3
70.3
52.0
82.7
1774
62.8
52.5
258
58.2 (-1.))
adapt these parameters, resulting in significantly greater computational expenditure (e.g., 30 ~ 48 hours on A100 GPUs). This substantially increases the overall training cost. Focus on the inference- time efficiency, which in our framework is primarily influenced by the token compression strategies employed during the inference process. Table presents the KV cache memory usage, CUDA time, and FLOPs obtained when applying three different token compression methods during inference for models trained using Token Consistency Distillation. The FLOPs are calculated using calflops [69], while the KV cache memory usage is estimated with the help of LLM-Viewer [72). can observed that when retaining 64 tokens, all methods achieve improvements reducing KV cache memory, FI.OPs, and latency. In particular, Random token compression, which incurs no additional computational overhead, achieves an actual speedup of nearly 1.6x.
Analyses
In this section, we conduct a thorough investigation aimed at addressing the following key questions: (1) How much does the weight-sharing teacher guidance contribute to performance? (2) What occurs without progressive learning in token/layer-wiss dimensions? (3) How generalizable is EPIC across various token compression strategies? (4) Is extreme token compression (1 or 2 tokens) necessary?
5.1 Ablation Studies
Table 4: Ablation study on Layer Consistency Distillation. "w/o Progressive Compression Layer": fixed at the 2nd layer. DART employed as the token compression strategy during training.
Method
VQA V2
GQA VizWiz SQA TQAT
POPE MME MMB MMB-CN OCRBench Avg. (%)
Retain 128 Tokens (1 77.8%)
LCD (Ours)
69.2
60.6
57.9
69.8
56.3
84.2
1832
64.1
55.3
306
61.3
Distiliation
62.9
55.4
60.5
Progressive Compression Layer
68.7
59.3
54.3
70.6
56.2
82.2
1776
63.1
54.9
298
60.3 (-1.0)
Renain 64 Tokens
88.9%)
LCD (Ours)
66.0
58.3
57.8
69.7
54.3
81.2
1794
62.1
52.9
280
59.4
wlo Distillation Loss
64.4
58.2
55.9
69.6
54.8
80.9
1735
61.3
52.7
275
58.7 (-0.7)
wio Progressive Compreasion Layer
63.5
56.5
54.7
71.5
54.2
75.6
1734
61.9
51.7
260
57.8 (-1.5)
To validate the guiding role of the weight-sharing teacher model (RQ1) and the importance of the progressive learning strategy in training-awan token compression (RQ2), we conducted ablation studies on both components. Specifically, for token consistency distillation (TCD) and layer consis- tency distillation (LCD), in addition to eliminating the distillation loss remove the teacher model's influence, we also fix the compression ratio (e.g., 88.9%) and compression layer (e.g., the 2nd layer) to eliminate the progressive learning strategy in both the token-wiss and layer-wise dimensions. This essentially degenerates the process into imposing a significant perturbation in the feature space, and the model is trained to adapt in the parameter space, like direct training. As demonstrated in Tables the experimental results indicate that without teacher guidance, both TCD and LCD exhibit significant performance degradation across multiple benchmarks on average, with particularly pronounced declines vision-centric benchmarks such MME and MMBench.

---

5.2
How Well Does Proposed Framework Generalize across Different Methods?
Performance (%)
57.5
Performance (%)
65.0
00.0
Performance (%)
65.0
56.0
Performance
70.0
60.0
55.0
60.0
05.0
50.0
60.0
50.0
47.5
90.0
FastV
Random
FastV
Random
Methods
Methods
Methods
Methods
(a) GQA
(h) MMBench
(c) POPR
(d) VQA V2
Figure 4: Following LLaVA-v1.5's architecture and data, we apply DART for token consistency dis- tillation. "w/o train" denotes vanilla LLaVA. At inference, all methods use 88.9% token compression.
Beyond the generalization across different compression ratios observed in our comparative experi- ments of Sec. 4.2 we further investigate the adaptability of our proposed framework to diverse token compression strategies (RQ3). Specifically, as outlined in Sec. [4. we integrate three plug-and-plas play token compression strategies into our framework during training. We then conduct cross-strategy evaluations to assess how models trained with one specific strategy (e.g., FastV, DART) perform when applied with alternative compression methods during inference. As shown in Figure token consis- tency distillation consistently improves model performance across all benchmarks and compression methods. Notably, even when trained solely with DART-based compression, the model generalizes well to FastV and Random compression, yielding consistent performance gains. Furthermore, after training with proposed framework, the performance gap between token compression strategies is significantly reduced. Notably, previously underperforming strategies exhibit more substantial improvements than their stronger counterparts. For additional experiments and discussions the generalization capability of our method, please refer to Appendix
5.3 Is Extreme Token Compression Necessary?
1200
80
rndoetion from 576 128 yessal nobens
Sharp FL DPP
ICU (Qurs)
Latency (eccond)
1000
11.00
compremsioe
Pruper
(Cum!
POPE Scare)
70
60
Area
TOPE (F1 (-Soore)
700
High ROI Area
ykli
60
FLO redictio slows down, balon 55 yaral bokens
000
speedin
Estretic sakee
Low
Low Rora
compessiio hags
Rorston
spesdup bencfia.
TCD FualV
LGD + Fasti
700
TCD DART
LGP DART
575 250 125
64
576 258 125
36 15
100
#Visual Tokens
Visuul Tokens
#Vssual
Viiuual
Figure 5: All experiments use the model trained following LLaVA-v1.5. FLOPSs and latency are measured on the POPE. Visual token and latency experiments repeated three times for reliability.
We observe that many token compression methods pursue aggressive compression (e.g., one or two tokens). Table shows that extreme token compression still leads to notable performance degradation for many methods. While this consistently reduces KV cache memory, it raises the question: does such extreme compression always translate faster inference (RQ4)? To answer this, conducted detailed analysis and experiments. Figure shows the relationship between FLOPs and the number of retained visual tokens. When reducing tokens from the full set (576 tokens) to 128, FLOPs drop ignificantlyffrom 9.3T to around 2T. However, under more extreme compression, FLOPs reduction becomes noticeably smaller. A similar or even more pronounced trend is observed between token count and actual latency. In some cases, retaining 64 tokens yields better performance than fewer tokens (e.g., 36 18). possible hypothesis is that overly fragmented feature slices increase memory access time. Moreover, reducing tokens to 64 largely preserves vanilla model performance. We refer to this range as the High Return-on-Investin Investment (High ROI) Area. Further reduction beyond 64 offers only marginal latency gains but sharply degrades performance- this is the Low ROI Area. Model efficiency depends on whether it is computation or memory-bound. With heavily reduced tokens, GPU compute is underutilized and latency is dominated by memory access, making the

---

system memory- where further reduction brings little speedup. Overall, we argue that extreme compression unnecessary; instead, focus should be on balancing latency and performance.
6 Conclusion
In this paper, propose EPIC, learning framework that enhances the efficiency of multi-modal large language models (MLLMs) via Progressive Consistency Distillation. It integrates with existing token compression strategies without modifying the model architecture. achieving efficiency in both training and inference. Experimental results demonstrate that MLLMs trained with our framework achieve comparable average performance to the vanilla model using only 128 visual tokens. Notably, on 4 out of 10 visual understanding benchmarks, our approach even outperforms the vanilla model despite using significantly fewer visual tokens. Furthermore, extensive experiments validate the effectiveness of EPIC, highlighting its robustness across token compression ratios and generalization across strategies. Meanwhile, analysis reveals that while token compression offers significant benefits, excessive compression may lead to poor latency-performance trade-off, underscoring the importance of balancing compression levels for optimal performance.
Acknowledgements
This research was supported by the Shanghai Science and Technology Program (Grant No. 25ZR 1402278) and Shanghai Artificial Intelligence Laboratory. Besides, thank Huawei As- cend Cloud Ecological Development Project for the support of Ascend 910 processors.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprini arXiv:2309. 2023.
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen- VL: A frontier large vision-language model with versatile abilitics. arXiv preprint arXiv:2308. 12966, 2023.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[5] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal Ilm. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 13817-13827, 2024.
[6] Junjie Chen, Xuyang Liu, Zichen Wen, Yiyu Wang, Siteng Huang, and Honggang Chen. Variation-aware vision token dropping for faster large vision-language models. arXiv preprint arXiv:2509.01555, 2025.
[7] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vasion-language models. In European Conference on Computer Vision, pages 19-35. Springer, 2024.
[8] Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al. Mj-bench: Is your multimodal reward model really good judge for text-to-image generation? arXiv preprint arXiv:2407.04842. 2024.
10

---

[9] Zhaorun Chen, Francesco Pinto, Minzhou Pan, and Bo Li. Safewatch: An efficient following video guardrail model with transparent explanations. In The Thirteenth International Conference on Learning Representations.
[10] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 2Xiv:2412.05277 2024.
[11] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.
[12] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An opensource chatbot impressing gpt-4 with 90% chatgpt quality. ArXiv preprint, 2023.
[13] Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang. Yiming Hu, Fei Wei, Xinyu Zhang. Bo Zhang, Xiaolin Wei, et al. Mobilevim: fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint 2XXx:2331216888 1(2):3, 2023.
[14] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024.
[15] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[16] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[17] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) 2023.
[18] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 16344-66399, 2022.
[19] Xiaoyi Dong, Pan Zhang. Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcon xcomposer2-4khe A pionecring large vision-language model handling resolutions from 336 pixels to 4k hd. Advances in Neural Information Processing Systems, 37:42566 42592, 2024.
[20] Tayebeh Fani and Farid Ghaemi. Implications of vygotsky's zone of proximal development (zpd) in teacher education: Zptd and self-scaffolding. Procedia-Social and Behavioral Sciences, 29:1549-1535 2011.
[21] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 2023.
[22] Yash Goyal, Tejas Khot, Douglas Summers-Sun Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[23] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

---

[24] Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression. arXiv preprint arXiv:2410.0855. 2024.
[25] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 10
[26] Wenbo Hu, Zi-Yi Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Ma- tryoshka query transformer for large lasoon- models. Advances Neural Information Processing Systems, 77:50068-50018 2024.
[27] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[28] Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, and Yiyi Zhou. What kind of visual tokens do we need? training-free visual token pruning for multi-modal large language models from the perspective of graph. arXiv preprint arXiv:2501.02266. 2025.
[29] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
[30] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelies: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36:71683-71702, 2023.
[31] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 2.Xx::2008.03266 2024.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[33] Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal 11m. arXiv preprint arXiv:2407.02392 2024.
[34] Yanwei Li, Chengyao Wang, and Jiaya Jia, Llama-vid: An image is worth tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[35] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. 2.X.::2403118844 2024.
[36] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proc. of EMNLP, 2023.
[37] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012. 2023.
[38] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint wrXiv:2331.005777 2023.
[39] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tningg In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306. 2024.
[40] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
12

---

[41] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 34892-34916. Curran Associates, Inc., 2023.
[42] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 2024.
[43] Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, and Honggang Chen. Compression with global guidance: Towards training-free high-resolution milms acceleration. arXiv preprint arXiv:2501.05179, 2025.
[44] Xuyang Liu, Zichen Wen, Shaobo Wang. Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, et al. Shifting ai efficiency from model-centric to data-centric compression. arXiv preprint arXiv:2505.19// 2025.
[45] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui Hc, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307, 002888.......... 2023.
[46] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024.
[47] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NewrIPS, 2022.
[48] David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[50] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
[51] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
[52] Xudong Tan, Peng Ye, Chongjun Tu, Jianjian Cao. Yaoxin Yang, Lin Zhang, Dongzhan Zhou, and Tao Chen. Tokencarve: Information cooommatoo-preservvngg visual token compression in multimodal large language models. arXiv preprint
200.. 2025.
[53] Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey. arXiv preprint arXiv:23/2.17433
2023.
[54] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 00288, 2023.
[55] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual languaee encoders with improved semantic understanding. localization, and dense features. arXiv preprint arXiv: /4786, 2025.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
13

---

[57] Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Quétu, and Enzo Tartaglione. Folder: Accelerating multi-modal large language models with enhanced performance. arXiv preprini arXiv:2501.02430) 2025.
[58] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Gc, et al. Qwen2-vl: Enhancing vision- langugee model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[59] Shaobo Wang, Xiangqi Jin, Ziming Wang, Jize Wang. Jiajun Zhang, Kaixin Li, Zichen Wen, Zhong Li, Conghui He, Xuming Hu, et al. Data whisperer: Efficient data selection for task- specific Ilm fine-tuning via few-shot in-context learning. arXiv preprint arXiv:2505.12213 2025.
[60] Shaobo Wang, Jiaming Wang, Jiajun Zhang, Cong Wang, Yue Min, Zichen Wen, Fei Huang, Huiqiang Jiang, Junyang Lin, Dayiheng Liu, and Linfeng Zhang. Winning the pruning gamble: unified approach to joint sample and token pruning for efficient supervised fine-tuning. arXiv preprint M.Xiv:250992387. 2025.
[61] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng. Jilan Xu, Zun Wang, al. Intervvide2:: Scaling video foundation models for multimodal video understanding. Arxiv e-prints, pages arXiv-2403, 2024.
[62] Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint arXiv:2401.0688 2024.
[63] Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem? arXiv preprint arXiv:2502. 1101, 2025.
[64] Zichen Wen, Yifeng Gao, Shaobo Wang. Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502. / 1494, 2025.
[65] Minhao Xiong, Zichen Wen, Zhuangcheng Gu, Xuyang Liu, Rui Zhang, Hengrui Kang, Jiabing Yang, Junyuan Zhang, Weijia Li, Conghui He, et al. Prune2drive: A plug-and-play framework for accelerating vision-language models in autonomous driving. arXiv preprint arXiv:2508. 13305, 2025.
[66] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.151/1 15115, 2024.
[67] Yantai Yang. Yuhao Wang, Zichen Wen, Luo Zhongwei, Chang Zou, Zhipeng Zhang, Chuan Wen, and Linfeng Zhang. Efficientvla: Training-fre freee acceleration and compression for vision- language-action models. arXiv preprint arXiv:2506. 101000 2025.
[68] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing
large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040-13051, 2024.
[69] Xiaoju Ye. calftops: flops and params calculate tool for neural networks in pytorch framework. 2023.
[70] Xubing Ye, Yukang Gan, Xiaoke Huang. Yixiao Ge, and Yansong Tang. Voco-Ilama: Towards vision compression with large language models. arXiv preprint arXiv:2406. 12275, 2024.
[71] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint 2rx.1:2306.1335. 2023.
14

---

[72] Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Llm inference unveiled: Survey and roofline model insights, 2024.
[73] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE/CVF international conference on computer vision, pages 19955-19986, 2023.
[74] Jiarui Zhang, Jinyi Hu, Mahyar Khayathhoei, Filip Iliieski, and Maosong Sun. Exploring perceptual limitation of multimodal large language models. arXiv preprint arXiv:2402.07384 2024.
[75] Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Visual cropping improves zero-shot question answering of multimodal large language models. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
[76] Shaolei Zhang. Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.0388 2025.
[77] Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zenghui Ding, Xianjun Yang, and Yining Sun. Beyond training: Dynamic token merging for zero-shot video understanding. arXiv preprint arXiv:2411. 14401, 2024.
[78] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsification for efficient vasion-language model inference. arXiv preprint arXiv:2410.04111, 2024.
[79] Zeliang Zhang, Phu Pham, Wentian Zhao, Kun Wan, Yu-Jhe Li, Jianing Zhou, Daniel Miranda, Ajinkya Kale, and Chenliang Xu. Treat visual tokens as text? but your milm only needs fewer efforts to see. arXiv preprint arXiv:24/0.061669 2024.
[80] Shiyu Zhao, Zhenting Wang. Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. Accelerating multimodel large language models by searching optimal vision token reduction. arXiv preprint arXiv:2412.00556 2024.
[81] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 2tth:2304.1052.. 2023.
15

---

Appendix
A More Generalizability Experiments
16
Detailed Theoretical Analysis
17
Integrated Progressive Consistency Distillation
18
Experimental Setup
19
D.I Token Compression Techniques During Training
19
D.2 Training Details
19
D.3 Benchmarks
20
Overview of the Basel lines
20
D44.1 General MLLMS
20
D.4.2
Training- aware Token Compression Methods
21
Limitations and Future Works
21
Broader Impacts
21
A
More Generalizability Experiments
Performance (%)
298
Performance (%)
Mio Soin Trah
Performance (%)
60.0
Perfo mannee
$8.0
50.0
FartV
Random
DART
FacV/
Random
DART
Fast/
Random
DART
FastV
Random
DART
Methods
Methods
Methods
Methods
(a) GQA
(b) MMBench
(c) POPE
(d) VQA V2
Figure 6: Following model architecture and training data of ---.v..----7B we apply the Random token compression for token consistency distillation during training. Here, "w/o train" denotes the vanilla LLaVA. During inference, all methods achieve token compression ratio of 88.9%.
66.0
Performance (%)
Performance (%()
70.0
Performance (%6
Iran
Performance
52.5
00.0
80.0
38.0
$0.0
47.5
50
Random
Methods
Methods
Methods
Methods
GQA
MMBench
(c) POPE
VQA
Figure 7: Following model architecture and training data of LLaVA-v1.5-7B, apply the FastV for token consistency distillation during training. Here, "w/o train" denotes the vanilla LLaVA. During inference, all methods achieve a token compression ratio of 88.9%.
In addition to the cross-method generalization validation of our proposed framework conducted performed more comprehensive and meticulous experiments to further verify its
in Sec. 5.2
16

---

effectiveness. Specifically, as illustrated in Figures 6 and 7 we employ Random Token Compression and FastV, respectively, to perform token consistency distillation on the multi-modal large language model. Experimental results demonstrate that regardless of the token compression strategy used during training with our proposed method, the trained model consistently achieves improved inference performance across various token compression strategies. This strongly demonstrates the gener- alizability and effectiveness of our approach, indicating that models trained under our framework genuinely adapt the pattern of missing visual tokens without being constrained to fixed set of preserved tokens.
Detailed Theoretical Analysis
Proof of Theorem
7
1. Direct path. Since o c(ri), the total variation is
T-1
TV(10 Σ - c(r:)). -(dr))))-rr)))
By monotonicity (S1) and Lipschitz continuity (S2), each term satisfies
c(re+1) - c(ra)= S 77(1+ - -r),
so summing yields:
T-1 (re41 - rz) = 7T
TV( S Σ
2. Progressive path. From the closed-form minimizer,
c(r:)) + Xe(re A)
gprog
1+
we have:
(e(r(+1) - o(ri)) + Mc(r(+1 - A) cerr - A))
Oprog Oprog
1+>
Define A. c(r++1) -((r)) 2 0 and å, cre+t1 A) dre A) 2 0. Then:
A XÂ, 1 AKz
gprog
Aprog
A. where K4 A
1+1
1+1
Bounding Kr. Define the worst-case slope ratio:
c(r) crr - A) A] c(r + A) - c(r)
sup
AE Fmax
Each Ky compares the lagged slope to the current slope. the schedule appropriately spaced so that rt [A, Tmax A), then have:
c((+1 - A) - c(ri A)
c(r+t1
c(r(+1) dri)
Convexity (S3) ensures that c' dr(r) is non-decreasing,
c(r) c(r A) s c(r A) c(r)
K 15
is not affine (i.e,, (r) > 0 on set of positive measure), the inequality strict for some r, and thus < 1.
4. Total variation bound. Summing steps gives:
T 1+ A Σ
1+ As. TV((O)))),
TV(OO)
Σ
Oproz
(prog
A.
1+λ
1+A
with strict inequality when 1, completing the proof.
17

---

Table 5: Performance 10 visual understanding benchmarks. 'Res.' is resolution, and '#Vision Tokens' is the number of vision tokens fed to the LLM backbone. Both training and inference of Integrated Progressive Consistency Distillation (ICD) employ DART as the token compression strategy for our methods. Parentheses in Avg. column show diffs vs. LLaVA-v1.5.
Methods
L.M
Res.
AVision Tokens
VQ4 6Q4
VieWir SQA'
VQA"
POPR
MME
MMB
MMB-
OCR
Arg.
BIIP--2
Vicun--138
234
65.0
19.6
42.5
55.3
Instruct55L.P
16
Vicuna-BB
224
66.3
49.2
34,5
60.5
50.1
83.9
1.000
36.0
259
Instrue181.1P
(E)
Vicuns-135 13B
224
12
64.2
49.5
33.4
63.1
$4.1
1530
36.9
17.4
252
IDEFICS-9R
10
1LMMA-78
234
64
50.9
38.4
35,5
25.9
75,3
1027
48.2
245
IDEFICS-S08
LLMMA-65B
1076
51.5
277
Qwen-VI.
Qwen-7B
448
256
99.3
35.2
67.1
63.8
1708
18.2
133
Qwen-7B
448
256
$7.5
38.9
68.2
61.5
1891
60.5
267
SPHINX
[3]
LLMMA-13B
224
289
78.1
62.6
39,9
69.3
51.6
S0.7
66.9
SPHINX-2
LLMMA-13B
762
2500
mPLUG-Onw2
LLaMA-7B
648
1024
79,4
56.1
50.5
68.7
$6.3
64.5
Video-LLLVA 3
Vicuna-7B 7B
224
236
65.9
60.3
48.]
66.1
$1.8
83.1
1542
60.5
49.3
161
55.7
LLVA-v15
Vicuns7B
336
576
72.2
61.9
$2.5
68.3
5K.1
85.9
1785
64.1
55.8
319
61.4
LMMs
with fewer vision tokens
AveragePPoling
Vicun=7B
336
64
63.0
55.5
48.4
65.6
$2.6
79.2
1579
59.6
49.5
2.58
55.9 (5.5)
MQT-L.LaVA
20
Vicuns78
336
2
4
49.6
50.0
66
14.8
75.4
1402
48.9
40.5
169
(154
MOT-LLaVA
2b
Vicuna
336
36
62.0
51.7
53.6
69.2
28.6
82.9
1777
60.5
51.6
244
55.4
MOTTLLLaAA
Vicun=7B
MQT--1aaA
20
Vicuns7B
336
178
66.2
90.8
34.6
35.7
84.3
1773
62,1
366
57,6 (339)
MOT-LLaVA
20
Vicuna 7B
336
192
66.9
59.9
$4.6
69.1 1
35.8
85.1
1784
62.0
53.9
2633
57.7 (377)
MQT-LLaVA
Vicun=7B
336
256
68.3
60.1
54.6
69.0
37.11
$4.6
1740
61.7
53.0
273
57,8 (3.0)
OT-LLaYA
Vicuna-7U
51.5
36,9
$4.1
1771
62.1
265
LLaMA-VID
8E
Vicuna
336
55.5
36.2
65.8
69.0
$3.1
VoCo-LLaMA
NO
Vicuna7B
336
1
55.6
$4.6
68.4
31.7
90.8
1594
56.4
46.2
69
TokenPadker
Vicuna-78
336
144
71.3
62.0
36.6
70.5
43.8
56.2
1716
63.9
53.4
303
59.9
TokenPacher
53.7
Mavi--Mii
Vicun=-B
336
144
58.1
56.3
14,8
25.3
26.0
$2.3
1325
24.8
132
LLaVA-Mial
20
Vicuna: 7B
336
64
56.6
10.4
27.4
28.1
$2.3
1324
23.8
145
Ours
LLaVA-v1.5
Vicuns-78
72.2
54.7
55.7
55.7
312
61.8 (+04)
LLaVA-v15 ICD
Vicuna
336
192
71.9
61.4
54.8
70.3
57.6
$4.8
1783
64.8
55.9
316
61.7 (103)
LLVAA-1 ICD
Vicuna- 7B
336
128
69.6
60.3
55.2
70.1
57.1
83.5
1770
63.9
55.2
298
60.8 (0.0)
LLAAAs15 ICD
Vicuna718 78
336
66.5
55.3
70.4
79.3
1720
63.8
MLVA
Vicuna:
62.1
54.4
70.9
52.8
74.0
1635
61.2
246
56.5 (449)
C Integrated Progressive Consistency Distillation
As outlined in Sec. [ our Progressive Consistency Distillation framework incorporates two progres- sive learning mechanisms: Token Consistency Distillation (TCD) in boken-wise manner and Layer Consistency Distillation (LCD) in a layer-wise manner. Experimental results demonstrate that both approaches achieve strong performance across all key metrics: model accuracy, efficiency, robustness, and generalization capability. To further investigate this direction, we explore the integration of progressive learning from both token-wise and layer-wis, perspectives. critical question arises: Can this integrated approach continue to maintain or even enhance the model's performance? ? Building upon these two approaches, we design Integrated Progressive Consistency Distillation (ICD). Unlike the original Token Consistency Distillation (TCD), where token-wise progression is governed by global training progress, our method enables layer-wise iterative application of TCD, effectively integrating both token-wise and layer-wise dimensions. Specifically, during training, the token com- pression layer progressively shifts from deeper shallower layers. Within each layer, the compression ratio follows the TCD schedule, sampling from small to large values. Upon transitioning to the next layer, the ratio resets to its initial value, and the process repeats. As shown in Table the Integrated Progressive Consistency Distillation approach achieves comparable performance on representative visual benchmarks. Notably, it even surpasses the vanilla LLaVA in average performance while retaining only 192 visual tokens (66.7% J). Furthermore, when compared to other training-aware token compression approaches (such as MQT-LI and PakenPacker). ICD demonstrates superior performance under identical retained visual tokens.
18

---

D Experimental Setup
D.I Token Compression Techniques During Training
We adopt three representative token compression methods in our proposed framework: FastV, which follows an importance-bane -based strategy; DART, which leverages redundancy-base pruning; and random token pruning, the simplest form of token compression.
DART 64 is a training-free ffrenngg-feeeeee and plug-and-play method that prunes visual tokens based on to- ken duplication while maintaining compatibility with efficient attention mechanisms like Flash Attention [8 17
FastV 7 is aallg-andd-lay token compression technique that builds the observation that visual tokens tend to have diminishing contributions to model outputs in deeper layers. By utilizing attention scores assess token importance, it prunes less critical visual tokens at earlier layers of the language model.
Random is token compression method that requires no additional signals or computation, and is primarily used to validate the effectiveness and generalizability of our proposed approach.
Table 6: Method Comparison Across Different Stages
Method
Stage 1
Stage 2
Stage 3
Training Time J (h)
QT-LLaVA MQT-LLaVA
30h x A100s
34h x A6000s
LLaMA-VID /
v
48h x 8 A100s
LLaVA-Mini EPIC (Ours)
26h x 8 A100s
12.2h x 8 A100s
D.2 Training Details
For fair comparison, our framework not only adheres to the same model architecture and instruction- tuning data as vanilla LLaVA but also maintains identical hyperparameter settings. Moreover, since we have not modified the model architecture, we can directly use pre-trained projector-unlic MQT-LLaVA, QT-LLaVV, and L.La VA-Minn,, which require mandatory Stage 1 pre-training. The training process, conducted on A100 GPUs, takes approximately faithfully reproduced the entire LLaVA training process following the official LLaVA-v1.5 training guidelines, All other token compression baselines are either trained following the settings provided in the original papers evaluated using publicly available model checkpoints. For the ablation study in Section the experiment without the distillation loss was trained using the same LLa VA-655K SFT data, with all other training parameters and procedures kept identical to those of EPIC. For the variants without a progressive compression ratio (Table or progressive compression layer (Table 4. fixed the second layer the compression layer. Overall, our ablation studies were conducted under a strictly fair and consistent experimental setup.
hours. Furthermore, we
Table 7: Detailed hyperparameter settings.
Settings
Stage
Batch size
128
Learning rate
2c-5
Learning schedule
Cosine decay
Warmup ratio
0,03
Weight decay
0
Epoch
1
Optimizer
AdamW W
DeepSpeed stage
3
Max token
2048
19

---

D.3 Benchmarks
.ME 21
a comprehensive benchmark for evaluating the performance of MLLMs in multi- modal tasks. It measures models' capabilities across two key areas: perception and cognition, using 14 specially designed subtasks that test interpretative and analytical skills.
MMBench <5] employs dual approach: it provides extensive dataset that broadens the range and variety of evaluation questions, and introduces the innovative CircularEval strategy, which uses ChatGPT convert free-form predictions into structured choices. MMBencch-CN is the Chinese version of the benchmark.
ScienceQA 47 is a multi-modal benchmark aimed assessing and diagnosing Al systems' multi-hop reasoning and interpretability in the science domain. It includes a dataset of around 21K multiple-choice questions across various scientific topics, complete with detailed answer annotations, related lectures, and explanations.
GQA [27] is dataset designed for advanced visual reasoning in real-world scenarios, using scene graph-based structures generate 22 million diverse, features novel set of evaluation metrics focused on consistency, grounding, and plausibility, setting high standard for lisoonlanguaage task assessment.
paatticallyy-roormmmmdd questions.
POPE 36 is an evaluation method for examining object hallucination in MILMs. It transforms the evaluation into a binary classification task, asking MLLMs simple Yes-or-No questions to identify hallucinated objects. POPE employs various object sampling strategies reveal model tendencies towards hallucination.
VQA V2 [22] evaluates the model's visual perception capabilities through open-ended questions. consists of 265,016 images, covering a wide variety of real-world scenes and objects, providing rich visual contexts for the questions. For each question, there are 10 ground truth answers provided by human annotators, which allows for comprehensive evaluation of the performance of different models answering the questions accurately.
TextVQA [5T] focuses on the comprehensive integration of diverse text information within images. It meticulously evaluates the model's text understanding and reasoning abilities through a series of visual question-answeriin
tasks with rich textual information. Models need to not only understand the visual content of the images but also be able to read and reason about the text within the images answer the questions accurately.
OCRBench [46] is comprehensive benchmark for evaluating the OCR capabilities of multi-modal language models across five key tasks: text recognition, scene text-centric and document-oricnn -rieente VQA, key information extraction, and handwritten mathematical expression recognition.
D.4
Overview of the Baselines
D.4.1
General MLLMs
is a vision-language pretraining framework that efficiently combines frozen image encoders with large language models (LLMs). It adopts two-stage training strategy leveraging lightweight Querying Transformer to bridge the vision-language modality gap, enabling compute- efficient, zero-shot image-to-text generation aligned with natural language instructions.
BLIP-2
32
InstructBLIP [5] builds on BLIP-2 by introducing instruction tuning and an instruction-aware Query Transformer. This enhances the model's ability to extract features for a wide range of vision-language tasks. It achieved fate-of-thheart zero-shot performance across 13 benchmarks and demonstrated strong results on fine-tuned tasks such as ScienceQA.
IDEFICS [30] is an apen-access vision- language model based on the Flamingo architecture. Available in both base and instruction -uned variants (9B and 80B parameters), IDEFICS is trained entirely on publicly available data and models, promoting transparency and accessibility.
Qwen- VL & Qwen VL-Chat [3] extend the Qwen-LM 2 foundation with visual encoder and specialized input-output interface. Through three-stage training pipeline and a rich multilingual, multimodal corpus, the models achieve strong capabilities in grounding and optical character recognition (OCR) tasks.
SPHINX (38 a multimodal large language model that applies joint mixing across model weights, tuning objectives, visual embeddings, and image scales. By unfreezing the LLM during pretraining
20

---

and integrating diverse instructional and visual signals, SPHINX demonstrates strong performance in fine-grained vision-language understanding and reasoning tasks.
mPLUG-Ow12 [68] features modular architecture with language decoder interface for unified modality coordination. It employs shared cross-modal modules alongside modality-adapten components enhance feature retention and generalization in both unimodal and multimodal settings.
Video-LLaVA extends multimodal language modeling to unified video and image understand- ing. By aligning visual modalities into shared language feature space prior to projection, the model enables effective joint training across visual domains, achieving state-of-the- art results on diverse video and image benchmarks without requiring paired video-image data.
improves upon the original LLaVA [4] model through enhanced visual instruc- tion tuning. Utilizing CLIP- ViT-LL3366x visual backbone and MLP-based projection, it achieves strong performance with high data efficiency. Trained on only 1.2 million publicly available images, it excels at academic-oriented VQA tasks using straightforward prompting strategies.
LLaVA-v1.5 (39
D.4.2
Training-aware Token Compression Methods
[58] merges every four adjacent visual patches into a single token, effectively reducing the number of visual tokens.
Average Pooling, inspired by the token merging strategy in Qwen2-VL V
QT-LLaVA replaces the original modality projector in LLaVA with Q-former [32] that uses learnable queries to project the input token sequence into a shorter sequence. In this work, we fix retain 256 visual tokens.
MQT-LLaVA [26] proposes flexible Query Transformer that enables encoding images into variable number of visual tokens (up to predefined maximum), allowing dynamic adaptation to different tasks and computational budgets.
LLaMA-VID compresses both the instruction and the image into single token each, resulting in just two tokens per image. This design enables efficient understanding of longer video sequences.
VoCo-LLaMA 70 utilizes language models to compress all vision tokens, significantly improving computational efficiency while maintaining multimodal understanding.
TokenPacker 3 introduces visual projector that adopts coarse-to- coonse-t-fineeeee strategy to reduce the number of visual tokens by up to 80%, substantially decreasing the computational cost.
LLaVA-Mini 76 employs a modality pre-fusion module, constructed with Transformer blocks, to integrate visual information into the text tokens in advance. This approach reduces the number of visual tokens fed into the language model, thereby lowering computational cost.
E Limitations and Future Works
As discussed in the Appendix D.2 our proposed progressive consistency distillation framework was only applied during the visual instruction tuning phase when training the multi-modal large language models (MLLMs). While this approach has already demonstrated remarkable effectiveness in terms of both model performance and training efficiency, several promising research directions remain unexplored. For instance, how might our method perform applied during the model's pretraining stage? Specifically, implementing our framework during projector pretraining, rather than initializing it with publicly available projector weights, could potentially yield greater performance improvements. We hypothesize that this would allow the modality projector's parameter space to adapt token compression-induced feature space perturbations prior fine-tuning, thereby further facilitating subsequent supervised fine-tuning. In future work, we will systematically investigate extending our framework to all training stages (including pretraining and supervised finetuning), with rigorous analysis of both the performance gains and impacts training- time efficiency.
F Broader Impacts
In this work, we propose framework to develop efficient multi-modal large language models (MLLMs) through Progressive Consistency Distillation, which can be seamlessly integrated with various token compression strategies without modifying the original model architecture. Our approach
21

---

demonstrates strong effectiveness, robustness, and generalizability. On one hand, this contributes to improving the efficiency of MLLMs, thereby facilitating their deployment and practical applications at societal level, especially for resource-constraing edge devices. On the other hand, after training under our framework with comen-compressed inputs, the resulting model parameters shift from the optimum of the vanilla model (trained on full inputs) to a new optimum suited for compressed inputs. However, we do not further align these models with human preferences after token compression-ban training. While the models perform well across range of multi-modal tasks, they may carry potential risks of adversarial vulnerabilities or undesirable outputs.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contributions and scope of this paper are outlined the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix we outline the potential limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
22

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In the main paper and Appendix, we provide the full set of assumptions and complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details of all model versions, dataset versions, hyperparameters, and experimental settings in Sec. 4.I and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
23

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: In accordance with the requirements of the supporting organization, the code will be released after the review process is completed. We are committed to providing sufficient instructions to ensure reproducibility at that time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttpccccppppppp....e.eeee.eo.cpt aeeiii/gguies//eee /ooeeeS/emmomdmmiiicca ssiinni asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and evaluation details have been provided in Sec. and Appendix [
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
24

---

Answer: [Yes]
Justification: Some experiments, including latency statistics, were conducted multiple times, and the corresponding error bars or bands are reported in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include descriptions of the computational resources used Sec. 4.3 and Appedix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///www.....cco/aa///cccccaa
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
25

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe broader impacts of our work in Appendix F
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets (e.g., code, data, models) used the paper have been properly credited, and the corresponding papers have been cited.
26

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
27

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
28


Reviews:
[{"review_id": "nq8fWtLQ4H", "reviewer": "Reviewer_mKBF", "summary": "This paper tackles the problem of efficient compression of visual tokens in MLLMs. The authors propose a new progressive training-aware token compression that enables various \"scales\" of compression: token level or layer level. The main principle could be considered as a curriculum learning strategy with easy learning tasks with a low token compression ratio at the beginning of the training and more difficult ones after. Their approach is based on a teacher-student architecture with a progressive consistency distillation. The paper contains a theoretical motivation and a formalization of the proposed approach. An experimental validation is provided on 10 visual understanding benchmarks. Experimental results show the interest of the approach. It also includes ablation studies.", "questions": "+ The authors propose to use a linear increase of the compression ratio in the EPIC framework. The choice is relevant, but alternative progressive schemes could also be proposed. For instance, recent approaches have been proposed for learning a curriculum temperature of KD distillation that could be relevant. How the proposed approach could be enriched with other curriculum strategies?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "** Strengths**\n\n+ The progressive distillation principle and associated framework are relevant and interesting, with an experimental validation that supports the main claim of the paper.\n+ The multi-scale distillation (token and layer-wise) is also a nice idea.\n+ The proposed framework enables the easy integration of  plug-and-play token compression strategies \n+ The paper is well written with a clear formalization of the proposed approach\n+ The attempt to provide a theoretical intuition of the progressive consistency distillation is a positive point of the paper.\n+ Experimental part supports the claims of the paper.\n\n**Weaknesses**\n\n+ Some cognitive motivations for the proposed approach are given without any justifications. For instance, it is not clear to me how the D. Marr works enable us to argue that LLMs only need a small amount of information-dense text tokens. \n+While the motivation is to improve the efficiency of compression of visual tokens in MLLMs, it's not clear to me to what extent the proposed approach is specific to this type of model. Couldn't it be generalized to purely visual models or tokens of other modalities? It's not clear to me to what extent the proposed approach is specific to this type of model. Couldn't it be generalized to purely visual models or tokens of other modalities?", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mKBF,\n\nThank you for your insightful and constructive feedback. We sincerely appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.\n\n> **W1: Some cognitive motivations for the proposed approach are given without any justifications. For instance, it is not clear to me how the D. Marr works enable us to argue that LLMs only need a small amount of information-dense text tokens.**\n\n**RW1:** We sincerely apologize for any confusion this may have caused. In fact, our reference to D. Marr’s work[1] aims to highlight the higher information density of text tokens compared to visual tokens. This distinction is clearly emphasized in his research. Unlike LLMs, MLLMs must process not only text tokens but also a significantly larger number of visual tokens derived from images or videos. Therefore, we believe it is reasonable to argue that LLMs only require a small number of information-dense text tokens in most scenarios. We will clarify this connection between D.Marr’s work and our motivation in the revised version.\n\n> **W2: While the motivation is to improve the efficiency of compression of visual tokens in MLLMs, it's not clear to me to what extent the proposed approach is specific to this type of model. Couldn't it be generalized to purely visual models or tokens of other modalities?**\n\n**RW2:** Thank you for this insightful question. We focus on MLLMs because visual token redundancy is especially critical in high-resolution and long-video scenarios, directly affecting performance and user experience. That said, our compression framework is not limited to MLLMs. It can be extended to pure vision models, such as ViTs, by applying token reduction within the visual encoder under our training paradigm. Beyond vision, our approach could extend to other modalities. For example, CoT tokens in reasoning models often exhibit redundancy, and applying targeted compression strategies within a compatible training framework could enhance efficiency in language-centric models as well. \nWe conducted experimental validation by integrating the token compression operation into the purely visual model—ViT—within the EPIC framework, updating the ViT weights, while keeping all other experimental settings identical to those in Table 1. The token pruning strategy for training and evaluation is DART.\nAs shown in the table, applying our method within the visual model (ViT) still achieves strong performance, attaining 99.5% of the average score of the vanilla model. This demonstrates the broad applicability and effectiveness of our approach.\n\n|                         Model                         |  MME  | POPE  |  GQA  | TextVQA |  SQA  | MMbench_en | MMbench_cn | OCRBench | VizWiz | VQA_v2 | Average |\n| :---------------------------------------------------: | :---: | :---: | :---: | :-----: | :---: | :--------: | :--------: | :------: | :----: | :----: | :-----: |\n|                     LLaVA v1.5 7B                     | 1785  | 85.9  | 61.9  |  58.1   | 68.3  |    64.1    |    55.8    |   319    |  52.5  |  72.2  |  61.4   |\n| + TCD (Pruning Ratio 0.778)     | 1861  | 84.5  | 59.9  |  56.6   | 70.8  |    65.6    |    54.3    |   299    |  54.9  |  69.7  |  61.3   |\n| + TCD in ViT (Pruning Ratio 0.778) | 1823  | 83.5  | 60.9  |  57.2   | 68.6  |    64.3    |    55.4    |   312    |  53.6  |  70.7  |  61.1   |\n\n\nWe believe these are promising directions and plan to explore them in future work.\n\n\n> **Q1: The authors propose to use a linear increase of the compression ratio in the EPIC framework. The choice is relevant, but alternative progressive schemes could also be proposed. For instance, recent approaches have been proposed for learning a curriculum temperature of KD distillation that could be relevant. How the proposed approach could be enriched with other curriculum strategies?**\n\n**RQ1:** Thank you for the insightful suggestion. In our current work, we adopted a linear schedule due to its simplicity and demonstrated its effectiveness across multiple benchmarks. Our progressive learning strategy is specifically designed in conjunction with token compression, aiming to improve the efficiency of MLLMs by dynamically adjusting either the compression ratio or the compression layer throughout training. We also agree that richer curriculum strategies could further enhance training. The curriculum temperature technique you mentioned is a general curriculum learning approach used in KD. Importantly, it is complementary to our framework and can be seamlessly integrated to further guide student learning.  \\\nWe therefore followed [2] to incorporate curriculum temperature into EPIC, conducting experiments under the same setup as main table 1 (token pruning strategy: DART). As shown in the results below, the addition of curriculum temperature leads to performance improvements on certain benchmarks, yet the overall average performance exhibits only minor fluctuations compared to TCD. This suggests that simply stacking progressive learning strategies may not be sufficient to unlock their full potential—more sophisticated integration may be required. Alternatively, our method might already be approaching the performance upper bound under token compression, where curriculum temperature does exert an effect, but its benefits are not clearly reflected in the metrics.\n\n|                                  Model                                  |  MME  | POPE  |  GQA  | TextVQA |  SQA  | MMbench_en | MMbench_cn | OCRBench | VizWiz | VQA_v2 | Average |\n| :---------------------------------------------------------------------: | :---: | :---: | :---: | :-----: | :---: | :--------: | :--------: | :------: | :----: | :----: | :-----: |\n|                              LLaVA v1.5 7B                              | 1785  | 85.9  | 61.9  |  58.1   | 68.3  |    64.1    |    55.8    |   319    |  52.5  |  72.2  |  61.4   |\n|             + TCD (Pruning Ratio 0.778)              | 1861  | 84.5  | 59.9  |  56.6   | 70.8  |    65.6    |    54.3    |   299    |  54.9  |  69.7  |  61.3   |\n| + TCD&Curriculum Temperature (Pruning Ratio 0.778) | 1835  | 85.2  | 60.3  |  56.4   | 70.9  |    65.5    |    54.8    |   301    |  54.2  |  70.1  |  61.3   |\n\nWe will include a discussion of this potential integration in the revised version and consider it a valuable direction for future work.\n\n**Reference**\n\n[1] Marr, David. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. \\\n[2] Li, Zheng, et al. \"Curriculum temperature for knowledge distillation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 2. 2023.\n\nBest Regards,\n\nThe Authors"}, {"review_id": "82oQTpEbtl", "reviewer": "Reviewer_aYx1", "summary": "The paper introduces EPIC (Progressive Consistency Distillation), a training framework that makes multi-modal LLMs (MLLMs) more efficient by progressively compressing vision tokens. EPIC decomposes the perturbation caused by token pruning along two axes: (1) Token Consistency Distillation (TCD) gradually increases the compression ratio during finetuning, with a weight-sharing “teacher” that always keeps ≈ 5–10 pp fewer tokens than the “student”. (2) Layer Consistency Distillation (LCD) begins compressing only in the deepest Transformer layer and then shifts the compression layer toward shallower layers as training progresses.", "questions": "What is the wall-clock and A100-hour overhead of the extra teacher forward pass versus direct finetuning? A cost-accuracy plot would clarify practicality.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Clear algorithm description and theoretical intuition.\n- Comprehensive benchmark (10 vision–language tasks) and ablations on teacher loss & schedules.\n- Shows inference-time gains (KV cache ↓ 89 %, FLOPs ↓ 84 %, latency 1.6×).\n- Demonstrates that moderate token reduction (64-128) is a better speed/accuracy trade-off than extreme compression.\n\nWeaknesses\n\n- All experiments use a single 7 B LLaVA backbone. It can be beneficial to add larger MLLMs. \n- Efficiency measured on POPE only; no wall-clock comparison against state-of-the-art token-reduction baselines (TokenPacker, MQT-LLaVA) with identical hardware.", "was_revised": true, "final_justification": "The author addressed my concerns. I am happy to maintain my score.", "rebuttal": "Dear Reviewer aYx1,\n\nThank you for your insightful feedback. We appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.\n\n> **W1: Add larger MLLMs.**\n\n**RW1:** Thank you for great suggestion. Evaluating across different parameter scales can further validate the effectiveness of EPIC. We have evaluated our method across MLLMs with varying architectures and sizes. All experiments are trained on the same LLaVA-665K SFT dataset and use the same experimental configuration as in Table 1. As shown in the table below, our method demonstrates strong performance across different parameter scales and diverse model architectures. **Even when 77.8% of visual tokens are pruned, our approach retains over 95% of the average performance of the vanilla model, and with a 66.7% pruning ratio, it achieves performance on par with the vanilla model. Under the same pruning ratio, our method achieves up to a 4.9% higher average performance compared to direct SFT.** We will include these results in the revised version.\n\n|                   Model                   |  MME  | POPE  |  GQA  | TextVQA |  SQA  | MMbench_en | OCRBench | VizWiz | VQA_v2 |    Average    |\n| :---------------------------------------: | :---: | :---: | :---: | :-----: | :---: | :--------: | :------: | :----: | :----: | :-----------: |\n|               Qwen2-VL-0.5B               | 1398  | 84.6  | 55.8  |  45.2   | 60.1  |    42.6    |   246    |  35.9  |  60.7  |     51.0      |\n| + Direct SFT (Pruning Ratio 0.667) | 1272 | 80.2 | 54.3 | 42.6 | 58.7 | 41.1 | 208 | 32.5 | 56.3 | 48.0 (94.1%) |\n| + TCD (Pruning Ratio 0.667) | 1379  | 83.1  | 56.1  |  44.7   | 61.3  |    42.0    |   231    |  34.8  |  59.8  | 50.5 (99.0%)  |\n| + TCD (Pruning Ratio 0.778) | 1375  | 80.0  | 55.2  |  41.1   | 61.2  |    41.6    |   207    |  34.1  |  56.7  | 48.9 (95.9%)  |\n||\n|               Qwen2.5-VL-3B               | 1738  | 86.6  | 62.5  |  58.4   | 74.2  |    66.8    |   315    |  48.7  |  72.0  |     62.5     |\n| + Direct SFT (Pruning Ratio 0.667) | 1702 | 82.3 | 60.1 | 55.8 | 73.1 | 64.8 | 300 | 48.1 | 69.3 | 60.5 (96.8%) |\n| + TCD (Pruning Ratio 0.667) | 1732  | 85.1  | 62.0  |  57.1   | 74.1  |    66.8    |   301    |  48.3  |  71.3  | 61.85 (99.0%) |\n| + TCD (Pruning Ratio 0.778) | 1711  | 83.7  | 61.4  |  55.6   | 73.5  |    65.3    |   287    |  47.6  |  70.1  | 60.78 (96.9%) |\n||\n|                Qwen2-VL-7B                | 2212  | 85.9  | 67.3  |  80.1   | 82.1  |    79.2    |   664    |  53.6  |  77.9  |     74.6      |\n| + Direct SFT (Pruning Ratio 0.667) | 2110 | 82.8 | 66.1 | 77.3 | 80.1 | 78.5 | 649 | 52.0 | 75.9 | 72.6 (97.3%) |\n| + TCD (Pruning Ratio 0.667) | 2199  | 84.2  | 67.3  |  78.9   | 82.3  |    80.1    |   657    |  53.7  |  78.0  | 74.3 (99.6%)  |\n| + TCD (Pruning Ratio 0.778) | 2157  | 83.8  | 65.6  |  78.1   | 81.6  |    78.7    |   647    |  52.3  |  76.5  | 73.1 (98.0%)  |\n||\n|              LLaVA-v1.5-13B               | 1850  | 86.1  | 62.3  |  61.5   | 72.3  |    67.2    |   332    |  57.5  |  74.7  |     52.8      |\n| + Direct SFT (Pruning Ratio 0.667) | 1778 | 80.9 | 60.1 | 58.2 | 71.9 | 65.2 | 306 | 56.4 | 73.1 | 50.9 (96.4%) |\n| + TCD (Pruning Ratio 0.667) | 1835  | 84.9  | 62.4  |  60.5   | 72.7  |    67.6    |   328    |  58.5  |  74.4  | 52.7 (99.8%)  |\n| + TCD (Pruning Ratio 0.778) | 1800  | 83.4  | 61.6  |  59.1   | 72.8  |    67.3    |   300    |  58.4  |  71.2  | 51.6 (97.7%)  |\n||\n|               LLaVA-Next-7B               | 1783  | 87.1  | 63.6  |  67.5   | 68.7  |    67.6    |   540    |  63.1  |  74.7  |     67.8      |\n| + Direct SFT (Pruning Ratio 0.667) | 1715 | 84.2 | 61.4 | 66.0 | 67.3 | 65.3 | 518 | 61.1 | 73.2 | 65.7 (96.9%) |\n| + TCD (Pruning Ratio 0.667) | 1809  | 85.9  | 63.1  |  66.9   | 68.7  |    68.7    |   537    |  62.7  |  73.4  | 67.5 (99.6%)  |\n| + TCD (Pruning Ratio 0.778) | 1740  | 85.4  | 62.2  |  65.1   | 68.3  |    67.9    |   523    |  60.9  |  74.4  | 66.5 (98.1%)  |\n\n> **W2: More wall-clock comparison**\n\n**RW2:** Thank you for the valuable suggestion. We have added inference efficiency evaluations across more benchmarks and conducted fair CUDA-time comparisons with state-of-the-art baselines, all under identical single A100 GPU hardware settings. As shown in table below, our three token compression strategies achieve inference latency comparable to TokenPacker and MQT-LLaVA when retaining **64 visual tokens**. In some cases (e.g., Random), our methods even outperform them in speed due to the absence of additional computation overhead. \\\nFurthermore, we would like to emphasize that the central contribution of our work lies in the EPIC progressive learning framework, whose another key advantage is in training-time efficiency. As shown in Table 5 of the submission, our method achieves faster training compared to all baselines. The inference-time speedup primarily depends on the integrated token compression strategy, and our framework is modular and compatible with a wide range of such methods. In summary, our approach achieves competitive inference efficiency while significantly improving the training pipeline. We will include these results in the revised version.\n\n|       Benchmarks        | MME (2374 samples) | textvqa (5000 samples) | POPE (8910 samples) | MMB (4377 samples) |\n| :---------------------: | :----------------: | :--------------------: | :-----------------: | :----------------: |\n|  Vanilla LLaVA-v1.5-7B  |       300.7s       |         977.6s         |       1103.5s       |       772.6s       |\n|    DART     |       202.1s       |         712.8s         |       744.3s        |       436.3s       |\n|    FastV    |       211.0s       |         713.5s         |       749.1s        |       436.9s       |\n|   Random    |       189.7s       |         705.8s         |       697.3s        |       420.7s       |\n||\n|  MQT-LLaVA  |       207.8s       |         720.1s         |       740.6s        |       430.4s       |\n| TokenPacker |       199.1s       |         710.7s         |       700.9s        |       422.6s       |\n\n> **Q1: A cost-accuracy plot on direct SFT and ours method.**\n\n**RQ1:** Thank you for your suggestion. Due to policy limits, we present the data for the cost-accuracy plot in the table below. We trained both LLaVA-v1.5-7B with direct SFT and our method—using the same LLaVA-665K—on a setup with 8×A100 GPUs, saving checkpoints at multiple training steps and record corresponding time. The results show that EPIC incurs a moderate and acceptable increase in training time. **Remarkably, the model trained with EPIC achieves performance on par with, or even surpassing, the vanilla model, despite removing 66.7% of the visual tokens during inference.** This outcome is indeed encouraging, demonstrating that high multimodal representation efficiency can be achieved without sacrificing accuracy. We will present the above data in a cost-accuracy plot in the revised version. Thank you once again.\n\n|                 Method                  | Training Steps | Wall-clock Time |  MME   | POPE  |  GQA  | TextVQA |  SQA  | MMbench_en | MMbench_cn | OCRBench | VizWiz | SEEDBench-2 | VQA-v2 | Average Score |\n| :-------------------------------------: | :------------: | :-------------: | :----: | :---: | :---: | :-----: | :---: | :--------: | :--------: | :------: | :----: | :---------: | :----: | :-----------: |\n| LLaVA v1.5 7B |      1000      |     1:55:36     | 1770 | 86.0  | 57.0  |  53.1   | 67.8  |    57.7    |    50.2    |  305   |  57.0  |    44.2     |  67.4  |     57.7     |\n| + Direct SFT (Pruning Ratio 0.667) | 1000 | 1:55:36 | 1734 | 84.1 | 55.7 | 50.2 | 66.2 | 55.6 | 48.6 | 289 | 55.8 | 43.4 | 65.9 | 56.0 |\n| + TCD (Pruning Ratio 0.667)    |      1000      |     2:22:05     | 1754 | 85.4  | 56.7  |  53.2   | 67.5  |    57.0    |    49.4    |  306   |  57.1  |    43.3     |  66.7  |     57.2     |\n||\n| LLaVA v1.5 7B |      2000      |     3:54:47     | 1744 | 84.9  | 59.4  |  55.3   | 68.5  |    58.4    |    49.1    |  313   |  44.2  |    45.9     |  69.8  |     57.2     |\n| + Direct SFT (Pruning Ratio 0.667) | 2000 | 3:54:47 | 1712 | 81.0 | 57.9 | 53.1 | 66.9 | 57.2 | 47.8 | 301 | 43.1 | 43.9 | 67.3 | 55.4 |\n| + TCD (Pruning Ratio 0.667)     |      2000      |     4:46:32     | 1738 | 84.3  | 58.3  |  55.0   | 68.6  |    60.0    |    48.9    |  311   |  51.2  |    45.1     |  68.1  |     57.5     |\n||\n| LLaVA v1.5 7B |      3000      |     5:51:42     | 1720 | 85.8  | 60.7  |  56.5   | 67.5  |    63.2    |    55.6    |  313   |  55.1  |    46.8     |  70.1  |     59.5     |\n| + Direct SFT (Pruning Ratio 0.667) | 3000 | 5:51:42 | 1698 | 84.3 | 59.0 | 54.9 | 66.1 | 62.3 | 54.1 | 299 | 53.5 | 45.2 | 69.2 | 58.1 |\n| + TCD (Pruning Ratio 0.667)     |      3000      |     7:22:53     | 1733 | 85.7  | 60.6  |  56.0   | 67.8  |    62.5    |    55.2    |  318   |  57.0  |    46.4     |  70.5  |     59.6     |\n||                |\n| LLaVA v1.5 7B |      4000      |     7:49:30     | 1750 | 85.9  | 62.2  |  57.8   | 68.4  |    64.4    |    55.8    |  317   |  54.5  |    47.0     |  71.7  |     60.2     |\n| + Direct SFT (Pruning Ratio 0.667) | 4000 | 7:49:30 | 1721 | 83.2 | 60.3 | 56.3 | 66.9 | 63.2 | 54.8 | 306 | 52.6 | 46.2 | 69.3 | 58.6 |\n| + TCD (Pruning Ratio 0.667)     |      4000      |     9:43:38     | 1793 | 84.7  | 60.8  |  56.9   | 68.8  |    64.9    |    56.1    |  314   |  56.0  |    45.9     |  72.2  |     60.2     |\n||\n| LLaVA v1.5 7B |      5000      |    10:43:29     | 1775 | 86.0  | 62.5  |  58.1   | 68.5  |    64.5    |    56.3    |  317   |  53.4  |    47.1     |  72.3  |     60.4     |\n| + Direct SFT (Pruning Ratio 0.667) | 5000 | 10:43:29 | 1720 | 84.3 | 60.6 | 57.9 | 66.4 | 63.5 | 54.8 | 305 | 52.5 | 46.0 | 70.2 | 58.9 |\n| + TCD (Pruning Ratio 0.667)     |      5000      |    11:58:44     | 1800 | 84.4  | 61.2  |  57.6   | 68.9  |    65.4    |    56.3    |  309   |  56.6  |    46.4     |  72.2  |     60.4     |\n\n\nBest Regards,\n\nThe Authors"}, {"review_id": "dmyCF3Yx9N", "reviewer": "Reviewer_BQCk", "summary": "This paper focuses on the challenge that visual tokens in multimodal large language models (MLLMs) consume significant computational resources, presenting Efficient MLLMs via Progressive Consistency Distillation (EPIC), a novel progressive learning framework. The proposed method decomposes feature space perturbations induced by token compression along token-wise and layer-wise dimensions, introducing token consistency distillation and layer consistency distillation respectively. By leveraging guidance from a teacher model and following a progressive learning trajectory, EPIC aims to mitigate training difficulties. Experimental results across diverse datasets validate the approach’s effectiveness, demonstrating its capability to achieve high efficiency, robustness, and generalization performance.", "questions": "1. For the ablation study section: the direct training without distillation loss, does it use the same dataset and the same training approach? Regarding the configuration without progressive compression, which layers are set? We understand that the selection of layers will affect the effect after compression. Have appropriate and consistent layers been set to maintain the validity of the ablation experiments?\n2. Would it be possible to provide more comparative experimental results on different models across varying parameter scales?\n3. I noticed that in the experimental section, some compressed models unexpectedly outperform the original models. What could be the reasons behind this phenomenon?\n4. Why isn't the result of the complete EPIC method combining TCD and LCD included in the main Table 1 of the paper? Is it possible that the combined effect of the two methods is not ideal?\n5. For other vision-language (VL) models, is this method EIPC equally effective?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The distillation framework proposed in the paper, tailored for MLLMs, demonstrates advantages over direct training in compressing image tokens.\n2. The effectiveness of the proposed approach is validated on multiple datasets.\n3. The paper provides detailed training parameters, execution time, and machine resource specifications.\n4. The paper's figures are clearly rendered, effectively conveying the key concepts and enhancing readability.\n\nWeaknesses:\n1. The paper proposes a potential engineering implementation of distillation for multimodal models, without broader generalization or substantial methodological innovation: LCD based on the previous experimental finding that token compression in deeper layers of MLLMs is more difficult; TCD does not seem to be strongly related to MLLMs, but rather a general strategy for adapting distillation and compression. To the best of my knowledge, similar ideas have been proposed in multiple papers, such as 《Data-Free Knowledge Distillation For Image Super-Resolution》[1].\n2. I think the derivation in the theoretical part is based on several preconditions and assumptions, leaving some room for improvement in terms of rigor. Therefore, the validation in the ablation experiment section is particularly critical, which is also what I focus on.  The experimental setup in the ablation study section is not clearly described. For the direct training without distillation loss, does it use the same dataset and the same training approach? Regarding the configuration without progressive compression, which layers are set? We understand that the selection of layers will affect the effect after compression. Have appropriate and consistent layers been set to maintain the validity of the ablation experiments?\n3. The formulae and variables in the methodology section are presented in a relatively dense manner. It might be beneficial to optimize the presentation by using a more structured approach, such as an algorithmic flow, to enhance readability.\n4. The experiments are primarily centered around LLaVA-v1.5-7B. To validate the effectiveness of the method more comprehensively, it is advisable to include additional models with varying architectures and parameter scales for evaluation. This would help demonstrate the generalizability of the approach across different model configurations.\n5. The detailed descriptions of datasets and models in the appendix occupy a relatively large proportion of the content. It might be necessary to further optimize the presentation to better highlight the core findings and key details.\n\nReference:\n- [1] Zhang, Yiman, et al. \"Data-free knowledge distillation for image super-resolution.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "was_revised": true, "final_justification": "I still hold my initial view on the novelty of the paper.\nWhile I will not oppose acceptance if the other reviewers are in favor, I believe the motivation and justifications could benefit from further clarification and refinement.", "rebuttal": "Dear Reviewer BQCk,\n\nThank you for your insightful feedback. We appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.\n\n> **W1: Similar ideas in DFKD[1]**.\n\n**RW1:** Thank you for the thoughtful feedback. We appreciate the opportunity to clarify the distinctions between EPIC and prior KD methods. While there are conceptual connections to data-free KD methods such as the one proposed for image super-resolution (DFKD[1]), we would like to highlight some key methodological and architectural differences that set EPIC apart.\n\n1. **Model Architecture & Modality**\n   - DFKD for SR employs a **GAN-based generator** to synthesize low-resolution images for a **CNN super-resolution network**.\n   - EPIC operates on **Transformer-based MLLMs**, where vision and language tokens interact via cross-modal attention.\n2. **Task Nature**\n   - SR is a **pixel-level regression task** from LR → HR images.\n   - EPIC targets **multi-modal understanding**, where visual tokens must retain semantic fidelity **within a language modeling objective**—a fundamentally different optimization landscape.\n3. **Distillation Paradigm**\n   - Traditional KD transfers knowledge from teacher to **separate student networks**.\n   - EPIC introduces **weight-sharing consistency distillation**: the same mllm acts as both teacher and student, eliminating the need for extra parameters or model replicas.\n4. **Progressive Strategy Design**\n   - While both EPIC and DFKD adopt progressive mechanisms, their motivation and implementation are fundamentally distinct. DFKD's progressive strategy focuses on stage-wise expansion of student CNNs, suitable for pixel-level SR tasks. EPIC introduces a progressive mechanism tailored for MLLMs, which gradually adjusts (i) token compression ratios and (ii) the token compression layer to construct a progressive learning trajectory. This mechanism is designed to ensure training stability and semantic preservation, which, to the best of our knowledge, has not appeared in prior distillation literature, including DFKD.\n5. **Theoretical Contribution**\n   - We provide the **first formal analysis** that progressive consistency paths yield lower total variation under convexity, specifically derived for **Transformer token compression**—a result inapplicable to GAN-CNN pipelines.\n\nWe will include a comparative discussion with DFKD in the revised manuscript to clarify these aspects and hope our response is helpful.\n\n> **W2&Q1: Theory can be improved and detailed ablation setup**\n\n**RW2&Q1:**\n- **Theoretical part:** We believe those assumptions are standard in convex optimization literature and very common for theoretical tractability[2]. Those assumptions are essential to our theory, and we could explore how to safely remove them in our future work.  Also we believe our theory serves as an intuitive motivation and shows that progressive curriculum learning makes a smoother optimization trajectory (Line 153-156) and makes our work rigorously grounded.\n- **Detailed ablation setup:** We sincerely apologize for any confusion caused. The experiment without the distillation loss was trained using the same LLaVA-665K data, with all other training parameters and procedures kept identical to those of EPIC. For the variants without progressive compression ratio or progressive compression layer, we fixed the second layer as the compression layer. Overall, our ablation studies were conducted under a strictly fair and consistent experimental setup. We will include these details in the revised manuscript, and we would be happy to add any further information if needed.\n\n> **W3: Clarify method via structured/algorithmic presentation.**\n\n**RW3:** Thank you sincerely for your valuable suggestion. We will include an algorithmic flow and provide a more detailed methodological diagram in the revised version to enhance clarity and presentation.\n\n> **W4&Q2&Q5: More results on different models across varying parameter scales.**\n\n**RW4&Q2&Q5:** Thank you for this insightful suggestion. We have conducted experiments on LLaVA-v1.5-13B (a larger MLLM), Qwen-VL series (different model architecture). All experiments are trained on the same LLaVA-665K and have the same training configuration. As shown in the table below, **even when 77.8% of visual tokens are pruned, our approach retains over 95% of the average performance of the vanilla model, and with a 66.7% pruning ratio, it achieves performance on par with the vanilla model. Under the same pruning ratio, our method achieves up to a 4.9% higher average performance compared to direct SFT.**. These results demonstrate the effectiveness of EPIC across different MLLMs and parameter scales and we will include these results in the revised manuscript.\n\n|                   Model                   |  MME  | POPE  |  GQA  | TextVQA |  SQA  | MMbench_en | OCRBench | VizWiz | VQA_v2 |    Average    |\n| :---------------------------------------: | :---: | :---: | :---: | :-----: | :---: | :--------: | :------: | :----: | :----: | :-----------: |\n|               Qwen2.5-VL-3B               | 1738  | 86.6  | 62.5  |  58.4   | 74.2  |    66.8    |   315    |  48.7  |  72.0  |     62.5    |\n| + Direct SFT (Pruning Ratio 0.667) | 1702 | 82.3 | 60.1 | 55.8 | 73.1 | 64.8 | 300 | 48.1 | 69.3 | 60.5 (96.8%) |\n| + TCD (Pruning Ratio 0.667) | 1732  | 85.1  | 62.0  |  57.1   | 74.1  |    66.8    |   301    |  48.3  |  71.3  | 61.9 (99.0%) |\n| + TCD (Pruning Ratio 0.778) | 1711  | 83.7  | 61.4  |  55.6   | 73.5  |    65.3    |   287    |  47.6  |  70.1  | 60.8 (96.9%) |\n||\n|                Qwen2-VL-7B                | 2212  | 85.9  | 67.3  |  80.1   | 82.1  |    79.2    |   664    |  53.6  |  77.9  |     74.6      |\n| + Direct SFT (Pruning Ratio 0.667) | 2110 | 82.8 | 66.1 | 77.3 | 80.1 | 78.5 | 649 | 52.0 | 75.9 | 72.6 (97.3%) |\n| + TCD (Pruning Ratio 0.667) | 2199  | 84.2  | 67.3  |  78.9   | 82.3  |    80.1    |   657    |  53.7  |  78.0  | 74.3 (99.6%)  |\n| + TCD (Pruning Ratio 0.778) | 2157  | 83.8  | 65.6  |  78.1   | 81.6  |    78.7    |   647    |  52.3  |  76.5  | 73.1 (98.0%)  |\n||\n|              LLaVA-v1.5-13B               | 1850  | 86.1  | 62.3  |  61.5   | 72.3  |    67.2    |   332    |  57.5  |  74.7  |     52.8      |\n| + Direct SFT (Pruning Ratio 0.667) | 1715 | 84.2 | 61.4 | 66.0 | 67.3 | 65.3 | 518 | 61.1 | 73.2 | 65.7 (96.9%) |\n| + TCD (Pruning Ratio 0.667) | 1835  | 84.9  | 62.4  |  60.5   | 72.7  |    67.6    |   328    |  58.5  |  74.4  | 52.7 (99.8%)  |\n| + TCD (Pruning Ratio 0.778) | 1800  | 83.4  | 61.6  |  59.1   | 72.8  |    67.3    |   300    |  58.4  |  71.2  | 51.6 (97.7%)  |\n\n**(Due to space limits, more results on LLaVA-Next and Qwen2-VL-0.5B can be found in RW1 to Reviewer aYx1, or we can share the results in the discussion thread upon request. Thanks for understanding)**\n\n> **W5: Dataset/model appendix length issue**\n\nThank you for your feedback. In the revised version, we will simplify and relocate the dataset and model descriptions to better highlight our key findings and essential details.\n\n> **Q3: Why some compressed models outperform original models**\n\nWe appreciate the insightful question. The observation that compressed visual token models occasionally outperform their original counterparts is not an isolated phenomenon, but one supported by several plausible mechanisms:\n- **Reduced redundancy improves focus**: Compression removes redundant tokens (e.g., background, textures), preserving semantically salient regions and helping the model attend to key visual cues.  \n- **Implicit regularization**:Excessive visual tokens may introduce noise or task-irrelevant details, leading to overfitting on non-generalizable patterns during training. Compression acts as an implicit regularizer, forcing the model to focus on more robust and generalizable features.\n- **Optimization benefits from the progressive training strategy (EPIC)**: Our method employs progressive consistency distillation, gradually guiding MLLM to adapt to compressed visual representations. This \"from-easy-to-hard\" learning paradigm facilitates smoother optimization and convergence to better minima.\n- **Task-specific needs**: Many tasks require only a few critical regions; excessive tokens can distract, harming performance.\n\n> **Q4: Why isn't the combined method included in Table 1?**\n\n**RQ4:** We sincerely thank you for raising this important point. The reason we did not include the combined version of TCD and LCD—termed *Integrated Progressive Consistency Distillation* (ICD)—in Table 1 is due to the following considerations, rather than any suboptimal performance of the combined approach:\n- **ICD results are fully reported**: The performance of ICD is presented in **Table 7 of Appendix D**. Notably, with only 192 tokens retained (a 66.7% reduction), ICD achieves an average score of 61.7% across 10 benchmarks—surpassing the vanilla LLaVA (61.4%)—demonstrating its effectiveness.\n- **Table 1 is designed for fair comparison with existing methods**:The primary purpose of Table 1 is to compare our method with existing token-compression approaches, most of which employ only a single compression strategy. To ensure a fair and consistent comparison, we present results for TCD-only and LCD-only, demonstrating the effectiveness of each strategy and highlighting the general applicability of the EPIC framework.\n- **Space and clarity:** Table 1 already includes 10 benchmarks and over 10 methods; adding ICD would increase clutter. To maintain clarity, we present the core components (TCD and LCD) in the main text and defer ICD—an integrated variant—to the appendix for a logical progression.\n\n**Reference**\n\n[1] Zhang, Yiman, et al. \"Data-free knowledge distillation for image super-resolution.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. \\\n[2] Boyd, Stephen P., and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.\n\nBest Regards,\n\nThe Authors"}, {"review_id": "g5OJfH1qI6", "reviewer": "Reviewer_Guzs", "summary": "This paper proposes EPIC, a framework that addresses excessive computational resource consumption by visual tokens in multi-modal large language models. The method decomposes token compression perturbations into token-wise and layer-wise dimensions, introducing Token Consistency Distillation (TCD) and Layer Consistency Distillation (LCD) with progressive learning trajectories. Extensive experiments show efficiency improvements.", "questions": "Refer to weak points.", "limitations": "Refer to weak points.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strong points: \n1.The progressive consistency distillation approach proposed in this paper demonstrates certain innovation, with progressive learning from both token and layer dimensions, which is inspiring.\n2.To prove the effectiveness of the proposed method, this paper conducts a series of comparative and ablation experiments, along with efficiency analysis of the model. The experimental design is reasonably.\n3.The proposed method achieves significant improvements in computational efficiency for multi-modal large models: using only 64 tokens achieves merely 2% performance degradation compared to the original model, while substantially improving inference efficiency.\n\nWeak points: \n1.The visualization of learning trajectories shown in Figure 1 is rather abstract and difficult to illustrate the problem emphasized in this paper. Quantitative experiments are needed for detailed explanation.\n2.To my knowledge, both Token Consistency Distillation (TCD) and Layer Consistency Distillation (LCD) operate on tokens, just from different perspectives. What are the essential differences between the two approaches? A detailed analysis is needed.\n3.Layer Consistency Distillation (LCD) follows an easy-to-hard learning paradigm. What are the differences and connections with commonly used curriculum learning methods? There is a lack of deeper analysis on why progressive learning is effective.\n4.How are the hyperparameter λ in Section 3.2 selected and determined? The selection lacks sensitivity analysis.\n5.This paper demonstrates the effectiveness of the proposed method on visual understanding benchmarks, but the task formats are relatively limited. Additionally, experiments are only conducted using LLaVA, lacking validation of the method's generalizability on other mainstream MLLM architectures (such as BLIP-2[1], InstructBLIP[2], etc.).\n[1]Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.\n[2]Dai W L, Li J N, Li D X, et al. InstructBLIP: Towards general-purpose vision-language models with instruction tuning[C]//37th Conference on Neural Information Processing Systems (NeurIPS 2023). 2023-05-11.", "was_revised": true, "final_justification": "I would be maintain my score.", "rebuttal": "Dear Reviewer Guzs,\n\nThank you for your insightful feedback. We appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.\n\n> **W1: Quantitative experiments are needed for detailed explanation on Fig.1.**\n\n**RW1:** Thank you for your feedback. While Fig.1 qualitatively illustrates the difficulty of directly training under high compression ratios, we fully agree that quantitative evidence is important. To address this, we conduct controlled experiments comparing direct training and progressive learning under the same training budget (4000 steps) on LLaVA-665K. For direct training, we randomly drop 60% of tokens and fine-tune the pretrained LLaVA-v1.5-7B for 4000 steps. For progressive training, we divide training into four stages (0%, 20%, 40%, 60% pruning), each lasting 1000 steps. In each stage, the student adopts the current pruning ratio, while the teacher (shared weights) retains the previous stage’s pruning ratio to provide a consistency signal. Both methods use random pruning to isolate the effect of progressive learning. **Experimental results in Table below show that progressive distillation significantly outperforms direct training, supporting the motivation of our method.**\n\n|                      Model                      |  MME  |  GQA  | TextVQA |  SQA  | MMbench_en | vizwiz | Average |\n| :---------------------------------------------: | :---: | :---: | :-----: | :---: | :--------: | :----: | :-----: |\n|    LLaVA-v1.5-7B (Random Pruning Ratio 0.6)     | 1700  | 58.5  |  53.8   | 66.7  |    60.7    |  52.8  |  58.9   |\n| Progressive Learning (Random Pruning Ratio 0.6) | 1763  | 60.2  |  55.4   | 68.8  |    62.9    |  54.7  |  60.8   |\n\nWe hope this helps better illustrate our motivation, and we will include these experimental results in the final manuscript.\n\n> **W2: A detailed comparison of TCD and LCD**\n\nThank you for the insightful question. TCD and LCD are two complementary designs within our unified EPIC framework: TCD effects a temporal progression by gradually elevating the compression ratio, whereas LCD embodies a spatial progression by shifting the compression layer from deep to shallow, inspired by the observation that visual tokens receive diminishing attention in deeper layers. We will enrich the revised manuscript with further comparative details to highlight their distinct yet synergistic roles.\n\n> **W3: Differences and connections with curriculum learning and deeper analysis on effectiveness of progressive learning**\n\nThank you for the insightful comment. We appreciate the opportunity to clarify the conceptual differences and underlying mechanisms.\n\n**Connections and Differences with Curriculum Learning.** We agree that Layer Consistency Distillation (LCD) aligns with the easy-to-hard philosophy inherent to curriculum learning (CL). Both frameworks aim to reduce training difficulty and enhance convergence by gradually increasing task complexity. However, LCD diverges from standard CL in the following ways:\n- **Different Learning Objects:** Traditional CL operates on input samples or labels, selecting or ordering training data based on their estimated difficulty. In contrast, LCD progressively shifts the *token compression layer* from deep to shallow Transformer layers, which affects the model’s internal feature space rather than external data complexity.\n- **Mechanistic Basis:** LCD is grounded in empirical observations (e.g., [1]) that visual tokens receive minimal attention in deeper layers, meaning early compression in these layers causes less disruption. The progressive schedule is thus designed to gradually introduce stronger perturbations in a *structurally informed* manner, rather than relying on heuristic difficulty measures.\n- **Distinct Objective:** While CL generally aims to improve generalization or training efficiency, LCD is specifically tailored to *mitigate the feature-space perturbations* caused by token compression. It allows the model to transition smoothly from the uncompressed optimum to the compressed one, which we find critical for performance under high compression.\n  \n**Deeper Analysis on the Effectiveness of Progressive Learning.**\nIn Section 3.2, we provide a theoretical prototype demonstrating the benefits of progressive consistency distillation in a simplified 1D setting. We model the shift in optimal outputs (centers) induced by increasing compression ratios, and show that incorporating a teacher with a slightly lower compression ratio leads to a smoother optimization path with *lower total variation*. This translates into better training stability and convergence.\n\nEmpirically, we support this with ablations in Section 5.1 (Table 4), where removing progressive layer scheduling leads to performance drops up to 1.6% under heavy compression (64 tokens). This highlights that progressive learning is not only theoretically grounded but also essential in practice.\n\nWe hope this addresses your concerns, and we are happy to clarify further.\n\n> **W4: The sensitivity analysis about hyperparameter λ.**\n\n**RW4:** Thank you for raising this insightful question. As mentioned in the main text (line 178), TCD and LCD default to λ = 0.7. Following your suggestion, we conducted a sensitivity analysis on λ by evaluating token consistency distillation with λ ∈ {0.1, 0.3, 0.5, 0.7, 0.9}. As shown in the table below (evaluations performed at 77.8% pruning ratio with DART), we observe that model performance degrades significantly when λ is either too large or too small. When λ approaches 0, the distillation loss is almost disabled, depriving the model of valuable guidance from the teacher. Conversely, when λ approaches 1, the SFT loss is suppressed, which can impair the model's autoregressive generation capability. **Our experiments demonstrate that λ in [0.5, 0.7] yields the most favorable results, which balances the KL guidance from the teacher with the task-specific supervision from SFT.**\n\n| Hyperparameters λ |  MME  | POPE  |  GQA  | TextVQA | MMbench_en | VizWiz | Average |\n| :---------------: | :---: | :---: | :---: | :-----: | :--------: | :----: | :-----: |\n|        0.1        | 1749  | 83.2  | 59.8  |  57.2   |    63.8    |  53.2  |  63.3   |\n|        0.3        | 1845  | 83.8  | 59.4  |  56.4   |    65.3    |  54.6  |  64.2   |\n|        0.5        | 1829  | 84.1  | 60.6  |  57.3   |    66.4    |  55.6  |  64.9   |\n|        0.7        | 1861  | 84.5  | 59.9  |  56.6   |    65.6    |  54.9  |  64.7   |\n|        0.9        | 1770  | 81.8  | 58.2  |  53.9   |    63.1    |  54.1  |  62.4   |\n\nWe will incorporate these insights into the revised version of our manuscript. Once again, we sincerely thank you for your valuable feedback, which has significantly strengthened the quality and clarity of our work.\n\n> **W5: Add validation on other MLLM architectures**\n\n**RW5:** Thank you for the insightful suggestion. We have evaluated our method on a range of MLLMs, including LLaVA-Next-7B (featuring a different architectural design), and the Qwen-VL series (distinct architecture). All models are trained on the same LLaVA-665K, under identical training configurations. **As shown in the table below, our approach maintains over 95% of the vanilla model's average performance even when 77.8% of visual tokens are pruned, and achieves nearly full performance at a 66.7% pruning ratio, surpassing direct SFT by up to 4.9% under 66.7% pruning.** These results clearly demonstrate the effectiveness and generalization capability of EPIC across diverse model architectures and scales. We will incorporate these findings into the revised manuscript.\n\n|                   Model                   |  MME  | POPE  |  GQA  | TextVQA |  SQA  | MMbench_en | OCRBench | VizWiz | VQA_v2 |    Average    |\n| :---------------------------------------: | :---: | :---: | :---: | :-----: | :---: | :--------: | :------: | :----: | :----: | :-----------: |\n|               Qwen2.5-VL-3B               | 1738  | 86.6  | 62.5  |  58.4   | 74.2  |    66.8    |   315    |  48.7  |  72.0  |     62.5     |\n| + Direct SFT (Pruning Ratio 0.667) | 1702 | 82.3 | 60.1 | 55.8 | 73.1 | 64.8 | 300 | 48.1 | 69.3 | 60.5 (96.8%) |\n| + TCD (Pruning Ratio 0.667) | 1732  | 85.1  | 62.0  |  57.1   | 74.1  |    66.8    |   301    |  48.3  |  71.3  | 61.9 (99.0%) |\n| + TCD (Pruning Ratio 0.778) | 1711  | 83.7  | 61.4  |  55.6   | 73.5  |    65.3    |   287    |  47.6  |  70.1  | 60.8 (96.9%) |\n||\n|                Qwen2-VL-7B                | 2212  | 85.9  | 67.3  |  80.1   | 82.1  |    79.2    |   664    |  53.6  |  77.9  |     74.6      |\n| + Direct SFT (Pruning Ratio 0.667) | 2110 | 82.8 | 66.1 | 77.3 | 80.1 | 78.5 | 649 | 52.0 | 75.9 | 72.6 (97.3%) |\n| + TCD (Pruning Ratio 0.667) | 2199  | 84.2  | 67.3  |  78.9   | 82.3  |    80.1    |   657    |  53.7  |  78.0  | 74.3 (99.6%)  |\n| + TCD (Pruning Ratio 0.778) | 2157  | 83.8  | 65.6  |  78.1   | 81.6  |    78.7    |   647    |  52.3  |  76.5  | 73.1 (98.0%)  |\n||\n|               LLaVA-Next-7B               | 1783  | 87.1  | 63.6  |  67.5   | 68.7  |    67.6    |   540    |  63.1  |  74.7  |     67.8      |\n| + Direct SFT (Pruning Ratio 0.667) | 1715 | 84.2 | 61.4 | 66.0 | 67.3 | 65.3 | 518 | 61.1 | 73.2 | 65.7 (96.9%) |\n| + TCD (Pruning Ratio 0.667) | 1809  | 85.9  | 63.1  |  66.9   | 68.7  |    68.7    |   537    |  62.7  |  73.4  | 67.5 (99.6%)  |\n| + TCD (Pruning Ratio 0.778) | 1740  | 85.4  | 62.2  |  65.1   | 68.3  |    67.9    |   523    |  60.9  |  74.4  | 66.5 (98.1%)  |\n\n**(Due to space limits, more results on LLaVA-v1.5-13B and Qwen2-VL-0.5B can be found in RW1 to Reviewer aYx1, or we can share the results in the discussion thread upon request. Thanks for understanding)**\n\n**Reference**\n\n[1] Chen, Liang, et al. \"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\nBest Regards,\n\nThe Authors"}]

Author rebuttal:

Meta-review: The paper proposes a progressive distillation framework for compressing the vision tokens in MLLMs. The key insight is that naively training a MLLM with token compression leads to quality drop. To avoid this, the paper proposes to rely on co/self-distillation where multiple forward passes are made through the same model at different token compression rates and the joint loss is optimized. The information flow from the teacher helps smooth the optimization landscape and leads to better solutions. In addition to co-distillation, the paper also proposes a curriculum for teacher and student to make sure the teacher model is not too far from the student model in terms of complexity. Experimental results on Vicuna-7B, Qwen2-VL-7B show that the proposed technique achieves a better quality-efficiency tradeoff

While the empirical evidence looks promising, the paper does a poor job in citing related work, and positioning the paper. For instance, it is claimed that **EPIC introduces weight-sharing consistency distillation: the same mllm acts as both teacher and student, eliminating the need for extra parameters or model replicas.** 

As **Reviewer BQCk** pointed out, this is inaccurate and doesn't properly credit prior works which introduced self distillation. For instance, [1] also performs weight sharing between teacher and student during training. There are multiple works which build on [1] and applied it to various settings [2]. And I believe the distillation community has many more works on this topic.  It is important to cite these works and compare against them either empirically or at a conceptual level.  

While I'm recommending accepting the paper, I'd encourage the authors to properly position their work within the broader area of distillation. 


[1] Matryoshka Representation Learning https://arxiv.org/abs/2205.13147
[2] Matryoshka Quantization https://arxiv.org/abs/2502.06786
Decision: Accept (poster)