Title: Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Efficiency

Multi-model Large Language Model

Token Compression
Abstract: Visual tokens consume substantial computational resources in multi-modal large models (MLLMs), significantly compromising their efficiency. Recent works have attempted to improve efficiency by compressing visual tokens during training, either through modifications to model components or by introducing additional parameters. However, they often overlook the increased learning difficulty caused by such compression, as the model’s parameter space struggles to quickly adapt to the substantial perturbations in the feature space induced by token compression. In this work, we propose to develop Efficient MLLMs via Progressive Consistency Distillation (EPIC), a progressive learning framework. Specifically, by decomposing the feature space perturbations introduced by token compression along the token-wise and layer-wise dimensions, we introduce token consistency distillation and layer consistency distillation, respectively, aiming to reduce the training difficulty by leveraging guidance from a teacher model and following a progressive learning trajectory. Extensive experiments demonstrate the superior effectiveness, robustness, and generalization capabilities of our proposed framework.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EPIC, a framework that addresses excessive computational resource consumption by visual tokens in multi-modal large language models. The method decomposes token compression perturbations into token-wise and layer-wise dimensions, introducing Token Consistency Distillation (TCD) and Layer Consistency Distillation (LCD) with progressive learning trajectories. Extensive experiments show efficiency improvements.\n\nSTRENGTHS AND WEAKNESSES:\nStrong points: \n1.The progressive consistency distillation approach proposed in this paper demonstrates certain innovation, with progressive learning from both token and layer dimensions, which is inspiring.\n2.To prove the effectiveness of the proposed method, this paper conducts a series of comparative and ablation experiments, along with efficiency analysis of the model. The experimental design is reasonably.\n3.The proposed method achieves significant improvements in computational efficiency for multi-modal large models: using only 64 tokens achieves merely 2% performance degradation compared to the original model, while substantially improving inference efficiency.\n\nWeak points: \n1.The visualization of learning trajectories shown in Figure 1 is rather abstract and difficult to illustrate the problem emphasized in this paper. Quantitative experiments are needed for detailed explanation.\n2.To my knowledge, both Token Consistency Distillation (TCD) and Layer Consistency Distillation (LCD) operate on tokens, just from different perspectives. What are the essential differences between the two approaches? A detailed analysis is needed.\n3.Layer Consistency Distillation (LCD) follows an easy-to-hard learning paradigm. What are the differences and connections with commonly used curriculum learning methods? There is a lack of deeper analysis on why progressive learning is effective.\n4.How are the hyperparameter λ in Section 3.2 selected and determined? The selection lacks sensitivity analysis.\n5.This paper demonstrates the effectiveness of the proposed method on visual understanding benchmarks, but the task formats are relatively limited. Additionally, experiments are only conducted using LLaVA, lacking validation of the method's generalizability on other mainstream MLLM architectures (such as BLIP-2[1], InstructBLIP[2], etc.).\n[1]Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.\n[2]Dai W L, Li J N, Li D X, et al. InstructBLIP: Towards general-purpose vision-language models with instruction tuning[C]//37th Conference on Neural Information Processing Systems (NeurIPS 2023). 2023-05-11.\n\nQUESTIONS:\nRefer to weak points.\n\nLIMITATIONS:\nRefer to weak points.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the challenge that visual tokens in multimodal large language models (MLLMs) consume significant computational resources, presenting Efficient MLLMs via Progressive Consistency Distillation (EPIC), a novel progressive learning framework. The proposed method decomposes feature space perturbations induced by token compression along token-wise and layer-wise dimensions, introducing token consistency distillation and layer consistency distillation respectively. By leveraging guidance from a teacher model and following a progressive learning trajectory, EPIC aims to mitigate training difficulties. Experimental results across diverse datasets validate the approach’s effectiveness, demonstrating its capability to achieve high efficiency, robustness, and generalization performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The distillation framework proposed in the paper, tailored for MLLMs, demonstrates advantages over direct training in compressing image tokens.\n2. The effectiveness of the proposed approach is validated on multiple datasets.\n3. The paper provides detailed training parameters, execution time, and machine resource specifications.\n4. The paper's figures are clearly rendered, effectively conveying the key concepts and enhancing readability.\n\nWeaknesses:\n1. The paper proposes a potential engineering implementation of distillation for multimodal models, without broader generalization or substantial methodological innovation: LCD based on the previous experimental finding that token compression in deeper layers of MLLMs is more difficult; TCD does not seem to be strongly related to MLLMs, but rather a general strategy for adapting distillation and compression. To the best of my knowledge, similar ideas have been proposed in multiple papers, such as 《Data-Free Knowledge Distillation For Image Super-Resolution》[1].\n2. I think the derivation in the theoretical part is based on several preconditions and assumptions, leaving some room for improvement in terms of rigor. Therefore, the validation in the ablation experiment section is particularly critical, which is also what I focus on.  The experimental setup in the ablation study section is not clearly described. For the direct training without distillation loss, does it use the same dataset and the same training approach? Regarding the configuration without progressive compression, which layers are set? We understand that the selection of layers will affect the effect after compression. Have appropriate and consistent layers been set to maintain the validity of the ablation experiments?\n3. The formulae and variables in the methodology section are presented in a relatively dense manner. It might be beneficial to optimize the presentation by using a more structured approach, such as an algorithmic flow, to enhance readability.\n4. The experiments are primarily centered around LLaVA-v1.5-7B. To validate the effectiveness of the method more comprehensively, it is advisable to include additional models with varying architectures and parameter scales for evaluation. This would help demonstrate the generalizability of the approach across different model configurations.\n5. The detailed descriptions of datasets and models in the appendix occupy a relatively large proportion of the content. It might be necessary to further optimize the presentation to better highlight the core findings and key details.\n\nReference:\n- [1] Zhang, Yiman, et al. \"Data-free knowledge distillation for image super-resolution.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nQUESTIONS:\n1. For the ablation study section: the direct training without distillation loss, does it use the same dataset and the same training approach? Regarding the configuration without progressive compression, which layers are set? We understand that the selection of layers will affect the effect after compression. Have appropriate and consistent layers been set to maintain the validity of the ablation experiments?\n2. Would it be possible to provide more comparative experimental results on different models across varying parameter scales?\n3. I noticed that in the experimental section, some compressed models unexpectedly outperform the original models. What could be the reasons behind this phenomenon?\n4. Why isn't the result of the complete EPIC method combining TCD and LCD included in the main Table 1 of the paper? Is it possible that the combined effect of the two methods is not ideal?\n5. For other vision-language (VL) models, is this method EIPC equally effective?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces EPIC (Progressive Consistency Distillation), a training framework that makes multi-modal LLMs (MLLMs) more efficient by progressively compressing vision tokens. EPIC decomposes the perturbation caused by token pruning along two axes: (1) Token Consistency Distillation (TCD) gradually increases the compression ratio during finetuning, with a weight-sharing “teacher” that always keeps ≈ 5–10 pp fewer tokens than the “student”. (2) Layer Consistency Distillation (LCD) begins compressing only in the deepest Transformer layer and then shifts the compression layer toward shallower layers as training progresses.

STRENGTHS AND WEAKNESSES:
Strengths

- Clear algorithm description and theoretical intuition.
- Comprehensive benchmark (10 vision–language tasks) and ablations on teacher loss & schedules.
- Shows inference-time gains (KV cache ↓ 89 %, FLOPs ↓ 84 %, latency 1.6×).
- Demonstrates that moderate token reduction (64-128) is a better speed/accuracy trade-off than extreme compression.

Weaknesses

- All experiments use a single 7 B LLaVA backbone. It can be beneficial to add larger MLLMs. 
- Efficiency measured on POPE only; no wall-clock comparison against state-of-the-art token-reduction baselines (TokenPacker, MQT-LLaVA) with identical hardware.

QUESTIONS:
What is the wall-clock and A100-hour overhead of the extra teacher forward pass versus direct finetuning? A cost-accuracy plot would clarify practicality.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of efficient compression of visual tokens in MLLMs. The authors propose a new progressive training-aware token compression that enables various \"scales\" of compression: token level or layer level. The main principle could be considered as a curriculum learning strategy with easy learning tasks with a low token compression ratio at the beginning of the training and more difficult ones after. Their approach is based on a teacher-student architecture with a progressive consistency distillation. The paper contains a theoretical motivation and a formalization of the proposed approach. An experimental validation is provided on 10 visual understanding benchmarks. Experimental results show the interest of the approach. It also includes ablation studies.\n\nSTRENGTHS AND WEAKNESSES:\n** Strengths**\n\n+ The progressive distillation principle and associated framework are relevant and interesting, with an experimental validation that supports the main claim of the paper.\n+ The multi-scale distillation (token and layer-wise) is also a nice idea.\n+ The proposed framework enables the easy integration of  plug-and-play token compression strategies \n+ The paper is well written with a clear formalization of the proposed approach\n+ The attempt to provide a theoretical intuition of the progressive consistency distillation is a positive point of the paper.\n+ Experimental part supports the claims of the paper.\n\n**Weaknesses**\n\n+ Some cognitive motivations for the proposed approach are given without any justifications. For instance, it is not clear to me how the D. Marr works enable us to argue that LLMs only need a small amount of information-dense text tokens. \n+While the motivation is to improve the efficiency of compression of visual tokens in MLLMs, it's not clear to me to what extent the proposed approach is specific to this type of model. Couldn't it be generalized to purely visual models or tokens of other modalities? It's not clear to me to what extent the proposed approach is specific to this type of model. Couldn't it be generalized to purely visual models or tokens of other modalities?\n\nQUESTIONS:\n+ The authors propose to use a linear increase of the compression ratio in the EPIC framework. The choice is relevant, but alternative progressive schemes could also be proposed. For instance, recent approaches have been proposed for learning a curriculum temperature of KD distillation that could be relevant. How the proposed approach could be enriched with other curriculum strategies?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-11"}