Title: DyMU: Dynamic Merging and Virtual Unmerging for Efficient Variable-Length VLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dynamic Length Visual Encoding

Vision-Language Models

Token Merging
Abstract: We present DyMU, an efficient, training-free framework that dynamically reduces the computational burden of vision-language models (VLMs) while maintaining high task performance. Our approach comprises two key components. First, Dynamic Token Merging (DToMe) reduces the number of visual token embeddings by merging similar tokens based on image complexity, addressing the inherent inefficiency of fixed-length outputs in vision transformers. Second, Virtual Token Unmerging (VTU) simulates the expected token sequence for large language models (LLMs) by efficiently reconstructing the attention dynamics of a full sequence, thus preserving the downstream performance without additional fine-tuning. 
Unlike previous approaches, our method dynamically determines token length based on the *image content*—not just resolution—and operates completely training-free, making it readily applicable to most state-of-the-art VLM architectures. Extensive experiments on image and video understanding tasks, demonstrate that DyMU can reduce the average visual token count by 32%-85% while achieving comparable performance to full-length models, across diverse VLM architectures.  Furthermore, qualitative analyses show that the adaptive token reduction from DToMe aligns well with human perception and enables users to better control computational costs through flexible integration with additional vision tools and models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DyMU, a training-free token compression framework for vision transformers. DyMU introduces two key contributions: (1) a dynamic merging strategy that adaptively controls token reduction based on image complexity, and (2) a novel virtual token unmerging mechanism that improves performance degradation caused by aggressive token merging. The method is evaluated on various models across multiple downstream tasks and benchmarks, demonstrating consistent improvements over prior training-free approaches, particularly ToMe. The work is well-motivated, clearly written, and practically relevant.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Well-motivated design: The paper offers a clear rationale for adapting token budgets to image complexity, as opposed to using a fixed schedule. The proposed dynamic merging strategy is intuitive and practically motivated.\n\n2. Robust empirical evaluation: DyMU is validated on a wide range of benchmarks and models, showing strong generalization and consistent improvements.\n\n3. Virtual Token Unmerging: This is a novel contribution not present in prior methods like ToMe. It enables accurate RoPE computation with merged tokens, and empirical results confirm that VTU substantially reduces accuracy degradation.\n\n4. Cost-accuracy trade-off: DyMU supports controllable granularity, letting users dynamically balance efficiency and accuracy. It is also compatible with domain-specific preprocessing tools (e.g., OCR-based cropping), enabling flexible deployment.\n\n**Weaknesses**\n\n1. Limited baselines: While ToMe comparisons are thorough, recent token-merging or pruning methods such as DiffRate[r1] and PiToMe[r2] are not quantitatively reported. Including them (even if training-required) would contextualise DyMU's gains more convincingly.\n\n2. Missing qualitative visualizations: ToMe originally visualized merged-token regions; similar maps for DyMU would help readers see how token allocation differs for simple vs. complex images, beyond giving only the number of tokens after merging in Figs. 1 and 7.\n\n**References**\n\n[r1] Mengzhao Chen, et al. DiffRate: Differentiable Compression Rate for Efficient Vision Transformers. In ICCV 2023.\n\n[r2] Hoai-Chau Tran, et al. Accelerating Transformers with Spectrum-Preserving Token Merging. In NeurIPS 2024.\n\nQUESTIONS:\n1. Could you report DyMU's performance against DiffRate or PiToMe under comparable token budgets? Even partial results would help position the method relative to the latest merging approaches.\n\n2. Would you consider adding merged-token maps (akin to ToMe's Fig. 4) to illustrate where tokens are retained or merged in simple vs. complex scenes?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training-free token reduction method for MLLMs by adaptively adjusting the number of vision tokens based on the complexity of the image. The proposed DYMU framework is comprised of two modules, Dynamic Token Merging and Virtual Token Unmerging. The dynamic token merging module continuously reduces the number of vision tokens by merging similar ones based on image complexity in the different layers of the vision encoder. In the LLM, the designed Virtual Token Unmerging module approximates the full vision sequence operations using the merged tokens. Experiments on several general MLLM benchmarks show the efficiency gain brought by the proposed framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe idea to adaptively reduce the token number according to image complexity is important.\n\nWeakness:\n1. DYMU emphasizes adaptive per-image token reduction. When comparing with other token reduction methods, the proper experiment setup should be measuring the actual (average) inference time on different benchmarks and comparing the performance of other token reduction methods **configured to a similar inference time**. \n2. Many more recent token reduction methods are missing, e.g. VisionZip (CVPR2025) and PyramidDrop (CVPR2025).\n3. DYMU merges tokens at every vision-encoder layer, but the paper lacks any quantitative analysis of the computational benefits this provides over a single post-encoder merge. It remains questionable whether early-layer merging—when token similarities may be noisier—yields net gains.\n4. Token merging thresholds rely on key-vector similarity as a proxy for image complexity, yet no vigorous validation is provided that this reliably relates to semantic detail for the later LLM part. For instance, a dense document image (all text) may exhibit high key similarity and thus undergo over-merging, potentially losing critical OCR information.\n5. Although broad VQA and grounding benchmarks are included, the paper omits fine-grained tasks (e.g., DocVQA, InfoVQA, MME-RealWorld) where small details matter most. Without these, the claim of “preserving core information” under aggressive compression is unsubstantiated.\n6. Besides FLOPs measurements, the inference time should also be included in all experiments.\n\nQUESTIONS:\n1. Is the proposed VTU module compatible with FlashAttention?\n2. Do you think designing some metrics measuring how well the original vision information can be reconstructed from the merged tokens (or directly down-sampled ones) can better represent the image complexity compared with the similarity-based one?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
In this paper, the author presents an efficient, training-free framework DYMU that can dynamically reduce the computation of VLMs while maintaining performance with two novel module DtoMe and VTU. A thorough has validate the frameworks’ effectiveness.

STRENGTHS AND WEAKNESSES:
Strengths:
1.The author logically described the current problems in this field and the purpose of the proposed method.
2.The author provided a detailed explanation of the functions and contributions of each module.
3.Sufficient experiments have fully verified the effectiveness of this method.

Weakness:
I don't completely understand what the red and blue squares in Figure 1 represent. If they represent the same characteristic with different values, it is recommended that the author not use only two colors but instead use one/multiple colors. If they represent different types of variables, it is hoped that corresponding explanations can be added beside the figure.

QUESTIONS:
I don't completely understand what the red and blue squares in Figure 1 represent. If they represent the same characteristic with different values, it is recommended that the author not use only two colors but instead use one/multiple colors. If they represent different types of variables, it is hoped that corresponding explanations can be added beside the figure.

LIMITATIONS:
I don't completely understand what the red and blue squares in Figure 1 represent. If they represent the same characteristic with different values, it is recommended that the author not use only two colors but instead use one/multiple colors. If they represent different types of variables, it is hoped that corresponding explanations can be added beside the figure.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DYMU (Dynamic Merging and Virtual Unmerging), a novel framework to enhance vision-language models (VLMs) by adaptively reducing visual tokens based on image complexity. It addresses inefficiencies in fixed-length token outputs of existing VLMs. Key components are:  1. Dynamic Token Merging (DToMe): Selectively merges similar tokens depending on image complexity (not resolution), reducing tokens by 32%-85% for simple images while preserving semantic details for complex ones. 2. Virtual Token Unmerging (VTU): Reconstructs full-sequence representations without retraining, maintaining performance and optimizing attention efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper provides comprehensive quantitative and qualitative experiments across multiple VLM architectures (LLaVA-1.5, LLaVA-OneVision), visual encoders (CLIP, SigLIP), and evaluation benchmarks (GQA, MMBench, MME, etc.), enhancing the credibility of the proposed method.\n\nThe proposed DToMe and DYMU are plug-and-play, requiring no retraining of the original model, making them suitable for resource-constrained scenarios or when training data is unavailable.\n\nDToMe extends the previous ToMe by dynamically adjusting the token count via threshold control (instead of pre-selecting a token budget), effectively balancing computational efficiency and performance.\n\nVTU addresses the mismatch between variable-length visual token sequences and fixed-length pre-trained LLMs through Virtual Token Unmerging, efficiently computing attention matrices without explicitly reconstructing the full sequence.\n\n\nWeaknesses:\n\nOverall computational cost: Although Table 1 details the attention computation cost, the lack of discussion on end-to-end costs makes it unclear whether DYMU outperforms other methods in efficiency. For the DYMU-high w/ VTU configuration, the MFLOPs (1272) show limited reduction versus Full Attention (1359). In low/mid settings, VTU nearly doubles MFLOPs, potentially limiting its applicability.\n\nArchitectural limitations: DToMe targets ViT-based encoders, and VTU is designed for RoPE-based LLMs. Their generalizability to other architectures (e.g., CNNs or non-RoPE models) remains unverified.\n\nHyperparameter search cost: DToMe requires offline threshold calculation (using large image sets), incurring non-negligible preprocessing costs. Appendix C shows that merging strategies significantly affect results, and fine-tuning layer-specific r_i for different models further increases preprocessing overhead.\n\nVTU approximation: Equation 10 uses embedding averages to reconstruct attention outputs—an approximate approach. The paper claims minimal empirical impact but provides no ablation studies or quantitative error analysis.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-19"}