DyMU: Dynamic Merging and Virtual Unmerging for Efficient Variable-Length VLMs
Authors: Zhenhailong Wang
Senthil Purushwalkam
Caiming Xiong
Silvio Savarese
Heng Ji
Ran Xu
Venue: NeurIPS 2025 poster ()
Keywords: Dynamic Length Visual Encoding
Vision-Language Models
Token Merging
Primary area: applications

Abstract: We present DyMU, an efficient, training-free framework that dynamically reduces the computational burden of vision-language models (VLMs) while maintaining high task performance. Our approach comprises two key components. First, Dynamic Token Merging (DToMe) reduces the number of visual token embeddings by merging similar tokens based on image complexity, addressing the inherent inefficiency of fixed-length outputs in vision transformers. Second, Virtual Token Unmerging (VTU) simulates the expected token sequence for large language models (LLMs) by efficiently reconstructing the attention dynamics of a full sequence, thus preserving the downstream performance without additional fine-tuning. 
Unlike previous approaches, our method dynamically determines token length based on the *image content*—not just resolution—and operates completely training-free, making it readily applicable to most state-of-the-art VLM architectures. Extensive experiments on image and video understanding tasks, demonstrate that DyMU can reduce the average visual token count by 32%-85% while achieving comparable performance to full-length models, across diverse VLM architectures.  Furthermore, qualitative analyses show that the adaptive token reduction from DToMe aligns well with human perception and enables users to better control computational costs through flexible integration with additional vision tools and models.

DyMU: Dynamic Merging and Virtual Unmerging for Efficient Variable-Length VLMs
Zhenhailong Wang", Senthil Purushwalkam Caiming Xiong', Silvio Savarese', Heng Ji', Ran Xu²
'University of Illinois Trbana-Champaige SSalesforce Research {wangz3, dengli)@illliii edu, ran xu@salesforce. com
'Equal contribution
Simple
Complex
Simple
Complex
60.
Aw 1000 on Bacodhit 150
Input Images
Fixed Length Visual Tokens +
CLIP
S76
576
576
576
576
576
(Ongaa))
Dynamic Length Visual Tokens
Lare Dydtil-aw
Dybili-zad
DyMil-iggh
DyMU
10
20
S1
68
85
205
204
STc
Visual Tozen Number
Figure 1: Dynamic Merging and Virtual Unmerging (DyMU) adaptively reduces visual token lengths based on image complexity, shown on the left where simpler images represented using fewer tokens. In contrast, existing representations (like CLIP Bd) always use same number of tokens regardless image content. Note that this limitation also exists in dynamic resolution encoders [8] 38 where token length depends solely image dimensions rather than complexity. DyMU applied to recent VLMs (right) maintains competitive performance across different token compression levels. This training- free approach preserves key semantic information, offering more efficient plug and-pu -lay alternative VLMs with fixed-length visual tokens.
Abstract
We present DyMU, efficient, training-free free framework that dynamically reduces the computational burden vision-language models (VLMs) while maintaining high task performance. Our approach comprises key components. First, Dy- namic Token Merging (DToMe)) reduces the number of visual token embeddings by merging similar tokens based image complexity, addressing the inherent inefficiency of fixed-length outputs in vision transformers. Second, Virtual Token Unmerging (VTU) simulates the expected token sequence for large language mod- els (LLMs) by efficiently reconstructing the attention dynamics of a full sequence, thus preserving the downstream performance without additional fine-tuning Unlike previous approaches, our method dynamically determines token length based on the image content not just resolution aad operates completely training-free making it readily applicable to most state-of-the-ar VLM architectures. Extensive experiments on image and video understanding tasks, demonstrate that DYMU can reduce the average visual token count by 32%-85% while achieving comparable performance to full-length models, across diverse VLM architectures. Furthermore, qualitative analyses show that the adaptive token reduction from DToMe aligns well with human perception and enables users to better control computational costs through flexible integration with additional vision tools and models.
Introduction
Recent large vision- language models (VLMs) follow common approach: visual encoder extracts features from images or videos and projects them into the same embedding space as textual features. 39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

These visual embeddings are then processed by LLMs alongside textual query features, enabling complex reasoning tasks while directly benefiting from advancements in LLM capabilities. As expected, the quality of the final predictions from the I.M relies heavily on the richness of the visual features and the amount of semantic detail captured by the encoder. Consequently, research has focused on improving visual encoders to extract increasingly fine- fine-grained features, leading to architectures that can capture intricate details. However, this level of detail comes at a cost-the computational burden during training and inference.
To process high-resolution images while preserving fine- rreeggrined details, modern visual encoders generate large number of tokenized representations. Furthermore, state-of-the-ar VLMs like LLaVA-One Vision [18] and Qwen-2.5-VL [2] use vision transformers (ViTs) that scale the number of tokens with the resolution of the image or number of frames in videos. For example, the visual encoder LLa VA-One Vision would produce 9477 tokens for an image of 1280x960 resolution. In contrast, the number of tokens in the textual queries for vision tasks is relatively low. On common benchmarks that represent real world use cases, textual queries often consist of just a few tokens, e.g., ~24 on MME [II] This stark contrast highlights that the computational burden of processing Vision-language tasks arises primarily from the visual tokens.
We first identify a critical limitation in current visual encoders (including those with dynamic resolution): the number of tokens generated for an image does not depend on the content of the image. As shown in Figure [I existing visual representations, such as CLIP [34,, produce the same embedding size for a blank image with a small circle and for a complex scene depicting buildings. vehicles, and people. To address the redundancy of visual tokens, previous work has shown initial success with token pruning, merging, and distillation. However, these approaches remain limited in one or more of the following key aspects: (1) They rely on pre-defined compression rates regardless of the image complexity 23 35 5 20 14 488; (2) They require full model retraining, which is often
infeasible for recent mainstream weightiight VLMs [21 20 43 48) ; (3) They depend on
extra textual conditions, rather than implementing a natively dynamic-length visual encoder [49
In this work, we propose Dynamic Merging and Virtual Unmerging (DyMU), which comprises two key methods for modifyyng existing pre-trained Vision-Langura Models (VLMs). First, we introduce Dynamic Token Merging (DToMee ($B.I), which allows the visual encoder to generate variable-length token sequences based on the complexity- not just resolution-or of the image. The core idea is to perform batch-level, rather than instance-level as in ToMe 3] bipartite merging to determine layer-wise thresholds for dynamic token reduction.
Second, we propose a novel Virtual Token Unmerging (VTU) algorithm ($ 3.2). which allows the LLM decoder process shorter, dynamically reduced visual token sequences while efficiently approximating the full-length representation. This method leverages the tracked positions of redundant tokens to reconstruct the full attention matrix without actually performing full attention. VTU enables the direct integration of DYMU into LLM backbones, offering a favorable trade-off between performance and efficiency.
Importantly, both of these modifications do not require additional fine-tuning of the pre-trained VLM. The threshold finding for D'ToMe only needs to be done once, offline, using any sufficiently diverse image-only datasets. Furthermore, D'ToMe is compatible with any Vision Transformer (ViT)- based visual encoder, and VTU can be applied to any LLM that utilizes Rotary Position Embedding (RoPE) 37
We show that VLMs modified with our methods can maintain comparable performance of the full model while reducing the average token count by 32%-85% Through comprehensive quantitative analysis ($ ). verify that our method works effectively across different VLM architectures, visual encoders, and finndingdinidigggg datasets. We show additional insights on why variable length reduction is favorable in terms of both performance and efficiency. Our approach also offers users greater control over token costs compared to existing AnyRes models [88 38 2], which incur a fixed token count per image based solely on resolution. In $ 4.22 we demonstrate example applications on how the number of visual tokens can be further reduced by combining DYMU with various task-specific tools such background removal, object detection, etc.
2

---

2
Related Work
Efficient Vision-Language Models Recent efforts in large vision-language models (VLMs) have primarily focused on reducing computational overhead during the pre- pre-filling and VLM decoding phases. That is, given full sequence of visual tokens from visual encoder, such as CLIP, these approaches perform token pruning and merging [35 S
40 49 15
23 43 distillation [44], or
resampling 21
20
to improve efficiency in either the projectors or the VLM decoder blocks.
However, we identify several key limita- tions: (1) Most existing methods prede- fine a fixed compression ratio for any
Compoment
Dynumic No Adda. Training Granularity
Extra
Improved
Leugth
Modules
Free
Coutrol
Cond.
LLaMA-VID
None
input image regardless of its content.
Fast V
5
Decoder
None
SpumsVV.M
Decoder
Test
While 49] and 43 propose adaptive token pruning frameworks that enable variable-lengee compression, they re- quire either additional textual conditions or retraining the backbone LLM with ex- tra modules. Such training can be costly
MQTELaVA
Projector
X
X
X
None
LlaVA-Prmeer-
65
Projector
None
TokenPackr
20)
Projector
XX*
None
AIP LLaVA
1
Decoder
Tex
LLaVA-mini
Projedor
None
DYMU
Encoder &
Decodar
None
or infeasible as mainstream VLMS rarely open-source their full training recipe and data. (2) All existing methods retain frozen, fixed-length visual encoder, overlooking the potential for further efficiency improvements within the visual encoder itself. In this work, we aim to explore simple training-fres algorithm for building natively variable-length visual encoders, which can be directly applied to cutting-edge VLM architectures including AnyRes models and RoPE embeddings.
Efficient Vision Transtormers We also draw inspiration from separate line of research [29 39 45 26 aimed at improving the efficiency of Vision Transformers (ViTs) themselves, which is
still the main go-to architecture for visual encoders [3 47 33 In particular, ToMe [3] merges predefined number of tokens within each VIT block using bipartite soft matching. However, the effectiveness of such methods in coordination with LLM backbones remains largely unexplored. Our experiments in $4ow that naively applying ToMe to visual encoders in pretrained VLMs results in significant drop in performance. To address this issue, further propose the Virtual Token Unmerging algorithm to boost the performance of VLMs without training with the modified encoders that output reduced token numbers.
Method
first deseribe how to convert a fixed-length visual encoder into a natively dynamic-length encoder using Dynamic Token Merging (DToMe). In $3.2 we then introduce Virtual Token Unmerging (VTU), a method that achieves a better trade-off between performance and efficiency when integrating the dynamic-length encoder into a language model backbone. The combination of these two techniques is referred to as MYMU-an entirely training-free approach compatible with any VLM that uses a biT-based visual encoder and RoPE-based language model. An overview of the method is illustrated in Figure 2
In this section, we present the main technical details of proposed method. In 3.1
3.1 Dynamic Token Merging (DToMe)
Our approach draws inspiration from ToMe a prior work which reduces the number of output tokens of ViT-based visual encoder a predefined fixed number. However, predefining the reduction ratio can still lead to misalignment between the information of an image and the number of tokens needed for representing it. Here we propose D'oMMe, an extension of ToMe that adaptively merges similar tokens in ViT layers, ensuring the output token count aligns with image complexity. DToMe merges tokens based on a similarity threshold while maintaining a record merged tokens to ensure their influence is properly propagated through subsequent transformer layers. To find the thresholds, we propose a inference-only batch-level bipartite merging algorithm which leverages the natural variance of image complexity in randomly sampled images.
3

---

VLM Decoder Blook
Simpler
Megel
Threshols
SB
Alta
MLP
Mooe Complex
DyMI Enc
Am
MLP
Chepaaz
Magad
Intage
Rank Bipartite Edges
Menge
Unmerge
Vintal
Remerge
DyMU Encoder Blocks during Threshold Finding
DyMU during
Figure Method Overview, DYMU, composed of two key ideas: Dynamic Token Merging (DToMe) and Virtual Token Unmerging (VTU). DTToM first determines per-layer thresholds (left) by feeding a large hatch of images into the vision transformer and computing hipartite token similarities. Unlike B. we rank these edges across the entire batch rather than within each instance. This naturally leads to more merges in simpler images (with more redundancy). During inference, DToMe merges tokens per-image basis using these pre-computed thresholds. We then introduce VTU (right), an efficient algorithm for reconstructing the full attention matrix without actually performing full attention. VTU enables a better trade-off between performance and efficiency, facilitating direct integration of dynamic-length embeddings into VLM backbones. The overall process is entirely training- free and allows more flexible control of token budget (Figure
Let us represent the output of the self-attention layer in the ViT layer i as I2 E R. where N₁ is the sequence length and D) is the embedding dimension. Similarly, let the keys computed in the self-attention layer be represented by k; E RN Dr, In each transformer block, we apply an additional DToMe operator to x,, Following []] we use a bipartite soft matching strategy to identify which tokens need to be merged. First, we divide the N; tokens into two sets (say A and BB) by assigning alternating tokens in sequence them. We then compute a bipartite assignment between the two sets of tokens by assigning source token t E A to target token = arg mnax (k:(4)² ( ki [m]) (token with the most similar key). This gives us | edges with scores NEB
Identifying Redundant Tokens
( [k,[t]T A,(tT) fort A. We then apply a threshold r; retain edges + t' where S.[t] T₁- Unlike [3], this thresholding operation leads to a variable number of retained edges depending on the amount of redundancy demonstrated in the key embeddings k; The key question now is: how do we determine the thresholds without training?
Finding Redundancy Thresholds with Batch-level Bipartite Merging In order to determine the layer-wise thresholds T; rely on statistics from large dataset of images i First, we choose hyper-parameter r; for each layer which represents the number of edges we expect to merge in a layer average across images of all complexities. This hyper-parameter serves control for the user The final output would then be expected to have an average of N N. ra tokens, where N is the input full sequence length. Using a dataset of images, we collect large batches of size B which are used to perform forward computation through the layers of the ViT sequentially. For each layer, we compute the B bipartite matching token edge score maps (b)) [t] where b (1, B} as previously described. We then find the threshold T;
granularity
B Σ
T. = max T
Σ (EA())
1(5
>1>7 B
(1)
In words, this finds the largest threshold such that BT; tokens are merged across the batch of images. It is important to note that the number of tokens merged in each image will not necessarily be equal to r; but the average number of tokens merged per image will be ri. Intuitively, since the ranking of edges is over the entire batch, simpler images that have more redundant tokens will be merged more. This process is done sequentially for each layer while only passing the remaining tokens to the next layer to obtain thresholds for every layer. We then average the layer-wie wise thresholds across all
"For standard VIT without any merging, N; is constant across layers "e show Appendix that DToMe is robust different threshold-findhe findid datasets. We include additional analysis of different r; scheduling in Appendix D

---

batches to ensure that they reflect the statistics across diverse set of images. See Figure (left) for an illustration of the proposed batch-level threshold finding.
Tracking and Merging Tokens We now describe how the merged tokens are computed. For each token in the sequence, 2z(t), we also track the set of positions of the tokens that have already been merged into it. P.[ C {1, , .N. For each of the edges between chosen redundant tokens I + I', we compute merged token embeddings and the corresponding position sets as:
[t]
P. P.
+ 34 ( [t
xx [[]
(2)
P.
P. P P()::
(3)
Intuitively, the representation of the target token t' is updated to the average of 2,(t"] and 24(t], weighted by their corresponding merged position set sizes, P.[t"] and P,[t]. The source token is then dropped, thereby reducing the token count in the next layer.
Size Weighted Self-attention To ensure that the self-attention layers weigh each token based on the number of tokens that were previously merged into it, we also adopt the idea of size-weighted self-attention from 33 where the attention is computed as:
P [1]]
QK"
A Softmax
+ log
(4)
Vd
P [.
3.2 Virtual Token Unmerging (VTU)
The language model (LLM) in pre-trained VLM is trained to operate fixed number of embeddings for each image 5 When DTTMe is applied to a visual encoder, this disrupts the optimized VLM and leads to significant drop in performance (see $ In this section, present approach to circumvent this issue while still benefiting from processing fewer number of visual embeddings.
R NxD of which only Nun < N rows are unique. Let Eun E R N. xD the unique embeddings and M {0, IINNNN be a mapping such that = M Cun. Here M is sparse matrix with one-hot rows. We now ask the question: for various operators f in an LLM, can we approximate ((e) using some efficient function of eun and M?
Consider the general case of a sequence of N embeddings
Sequence-independent Operators For any operator f that processes each sequence location independently, we can express f(e) f(e) =M f(cm)) by definition. This means that only need to apply ff to the unique embeddings cun, significantly reducing computational cost while preserving the original outputs. Many key components of modern LLMs fall into this category, including Linear layers, Activation functions (ReLU, GeLU, etc.), and Layer Normalization (along the embedding dimension D). The overall complexity of the MLP layers is reduced from O(ND²) to O D²). resulting in a linear speedup with N < N.
Virtual Token Unmerging for Self-Attention Blocks A common layer in recent LLMs is the Self-Attention operation with Rotary Position Embedding (RoPE). Unlike eequence-independee operators, self-attention considers pairwise interactions between embeddings and assigns a unique position to each of the N locations in e. Consequently, directly applying f(eun) fails to capture the structure of e, generally leading to significant discrepancies in the output.
To address this, provide thcoretical derivation of an efficient method to compute f(e) while preserving the benefits of token reduction. The key insight is to reconstruct the self-attention matrix without explicitly expanding the token sequence. We leverage the linearity of the RoPE transformation to efficiently simulate the appropriate repetitions and the positions of the unique embeddings, significantly reducing computational overhead while maintaining consistency with the full sequence computation.
AnyRee [8 leads multiple fixed length embeddings.
5

---

Q= Wge. K = W&e and V = W,e be the full query. key and value matrices. Similarly, Qun: KKnn and Vun are the unique queries, keys and values satisfying the mapping M defined above. The RoPE Self-Attention similarity matrix is computed as A RoPE(Q) RoPE K)²
For simplicity, let consider the case where D Q1, Q2 E IR We will follow similar notation for all queries, keys and values. This allows us to express each query and key as a complex number i.e. Q[n] Qi + [Q] Let 0 E [0, 2 ) be the rotation angle associated with each position for RoPE. For positions n, m E 1, 2, N, the RoPE-based similarity [37] defined as:
that
write Q [Q1, Q2) where
Afm, n] Re( (000ml Q[m]] [[]] = Re( [[]] ((00m)-)0m))
(5)
where T. Re(x) denote the complex conjugate and the real part of x respectively. This can be expanded as:
A m, n] = (Q1 [m] Kilm - Q2(m] K2 [os[0mm] 0(nl)
(Qi [m] K2 m) Q2 [m] [n]) s0n(0 0n))
(6)
We also have the trigonometric identities:
cos m] 0(nl)) cos ( ml ons (0(m) sin (ml) sin (0[m]
sin 0 m] (n)) sin( (m]) cost ( mn]) cos(0 (m]) sin sin(0(m)
(7)
Let C = diag cos 0)), diag(sin(0)) Using Eq 6 & the matrix form for self-attention similarities is:
A= C+ 5 S(Q x K JC C(Q x K )
(8)
where QK" QIKT + KKK Q KT = K - Q2KT- This formulation can be applied to queries and keys of any dimension D by repeating this for the (D/2) complex numbers obtained by dividing the representation into two parts. In practice, a different 0 is used for each of the (D/2) components.
Using this formulation and the mapping M, we can rewrite the attention matrix in terms of the unique queries and keys
= C.C.M M C KI S SM(Quu KTMM (Qun KT (9)
Observe CM, M, M are highly sparse, each with at most N non-zero entries. These matrices can also be pre-computed and reused across all self-attention layers. Computing Qun K and
Qu KT incurs an (N cost whereas the each the other matrix multiplications in Eq 9 can be efficiently computed using sparse matrix operations in Nun. We can then use the attention matrix to compute the final output of the layer as: f(e) smax(
A VD V
|smax
A MM VD
Unfortunately, the output f(e) E NxD will not necessarily exhibit the same redundancy as This in turn means that the future self-attention layers cannot benefit from the efficiency of virtual token unmerging. In order to remedy this, before passing the output to the future layers, we re-introduce the redundancy by averaging the embeddings in the positions that were originally equal. We denote this re- ne-merged output by f' M) which can be written as:
f'mm, M) = (M M) M f(e) () M smax
V
(10)
While the above averaging operation breaks the exactness of the future operations, we observe empirically (see $14) that this re-merging of tokens, that are known to be redundant, causes minimal drop in performance.
Overall Efficiency The computation attention matrix incurs a cost of O(N2 D + NN (due to the D/2 components). Followed by the softmax and sparse matrix multiplications in Eq 10 which incur a cost ON2 + N2 D). Therefore, the overall complexity for RoPE Self Attentio with Virtual Token Unmerging is O(N.n ND.. For comparison, the full RoPE Self-Attentio

---

Table 1: Comparison of million floating-
64
point operations per second (MFLOPs) the attention blocks on full sequence and DYMU sequence w/ and w/o VTU. N refers to full sequence length, . um refers unique quence length after merging. The statistics are computed with batch size I, head number 32, and head dimension 128. We use the fv- core package for counting FL.OPs
62
6155
Benchmark
55.2
38
56
Full Atrention
DyMti-low w/ VT:
DyMU-low wio VTU
DyMUU-mid wi VTU
DyMU-mid w/o VTU
DyMU-high w/ VTU
52
DyMU-high wio VTU
Methods
Avg Nun N
50
312
1272
159
Full Attention
576/576
1359.0
MFLOPs of the Attention Block
MU-O- w/o VTU
576
32.4
Figure 3: Accuracy FLOPs. Without VTU, DYMU achieves greater reduction FLOPs, but also resulting in more significant drop in performance. Adding VTU achieves a better tradc-off between performance and efficiency, particularly at lower token counts for example, MU-id preserves 99. 99.2% of the original performance while using 4.3 fewer FLOPs.
DyMU-mid w/o VTU
195 576
155.75
Dy MUhigh w/o VTU
394/570
635,85
DYMU-low w/ VTU
576
64.9
MYMi-mid VTU
195 /
311.5
DYMU-high w/ VIU
576
1272.0
Table 2: Comparison with state-of-the-art methods for improving efficiency LLaVA 1.5 25
DYMU-low
achieves 97.5% of the original full-length LLaVA baseline's 's performance while using only ~ 15% of the tokens. Importantly, DyMU entirely training- frreengg-eeeeeeeee and generally outperforms previous fixed-length. training- free methods such as 3 E 49
while also enabling variable-length outputs based image complexity.
Methods
# Visual
Compression Encoder
GQA
MMB
MME
POPE SQA'
SEED
VQA
MMVet LLaVA W
Avg
Tokens
(prep, all)
LLaVA-1.5-7B
576
62.0
64.6
1506.1862
86.9
69.4
66.2
58.3
30.7
63.5
63.1
Fixed Lengih Compression & Traiming-Reou
14
256
No
61.6
64.3
1435,
84.4
67.6
29.8
64.6
Prumerge
32
1350,
Prumerget+
31
144
No
64.9
1462.
84.0
68.3
57.1
LLDMA-VID
49.0
NoCo-I.LaMA
44
1
No
57.0
58.8
1323,
81.4
65.4
TokenPacker
36
L.aVA-Miii
43
No
60.9
65.6
1466,
84.4
70.4
57.0
36.6
68.9
Dunamic Lengrh Comprestion
Training Required
DinRate 6
Yos
57.9
1341.-
66.4
30.6
Fixed Length Compression & Training-Fre Free
Prumerge-no-f
32
1250.
76.2
68.0
54.0
FastV [3
128
No
49.6
56.1
1490
53.4
64.4
50.6
26.3
PDrop 40
128
56.6
61.4
82.3
SpareVLM
49
128
No
57.2
62.3
1721
85.0
67.8
55.8
29.0
PiToMe
~57
VisionZip
4
128
No
57.6
62.0
+, 1762
83.2
68.9
56.8
32.6
64.8
ToMe 3
94
Yes
57.3
59.7
1357, 1673
86.8
68.9
60.5
53.2
25.6
61.0
$9.2
59.2
62.4
1418,
ToMe
4
393
Yes
89.5
64.1
1454, 1769
86.7
68.4
65.1
55.8
30.8
66.0
62.2
Vartable Length Compression & Training Frre
DYMU-low
89 1:77
Yes
60.8
62.1
1438, 1787
86.3
69.3
65.0
53.1
30.0
62.9
61.5
DyMU-mid
195-147
Yes
61.7
62.8
1483, 1862
86.6
69.2
65.9
55.1
30.9
65.1
62.6
DyMU-high
3.4
Yes
61.9
64.3
1498, 1846
86.8
69.9
66.1
58.0
31.5
64.5
63.2
sequence length of N would be O(N2D) operation. Therefore, in theory, efficiency improves approximately linearly with the number of redundant tokens terms of FL.OPs. Table shows the FLOPs comparison for the attention block under different levels of token reduction. Figure [eggureBfrthin further illustrates the trade-off between performance and efficiency. We demonstrate that VTU significantly improves performance while still maintaining lower FLOPs than full attention, especially under more aggressive reduction regimes. In practice, we find that the wall-clock time difference for the attention blocks and the end-to-end inference is less significant due to Py PoTch'''s highly optimized dense matrix-mull matriix-uutipliaationn implementations. We provide an in-depth discussion of this gap in $5)
4 Experiments
We present comprehensive analysis demonstrating the practical benefits and efficacy of utilizing DYMU with various visual encoders and VLMSs. Implementation details on the model variants and threshold-findin datasets can be found in Appendix
7

---

Table 3: DYMU demonstrates similar efficacy different visual encoder, SigLIP 47 We obtain the haseline
by following the same training recipe as I.La VA-1.5 25 DYMU-SSgl.IPPoow achieves 96.1% of the baseline performance while using 15% visual tokens.
Methods
Visual Tokens
GQA
MMB
MME
POPE SQA'
SEED' VQA"
MMVet LLaVA W
Avg
(prep. all)
LLaVA 1.5 SigLIP
576
62.7
65.1
1471,
1770
85.7
68.2
66.7
57.6
30.2
59.8
62.1
ToMe
114
59.3
1717
66.9
61.8
52.1
26.
59.1
90+26
61.3
62.5
1398,
1695
84,9
66.7
64.4
51.8
26.7
58.6
59.7
DYMMU-SigL.I-mmmn
176_43
62.2
1744
67.4
65.2
54.5
26.7
60.7
DyMU- MIMU--ggLIP-high
318-67
62.4
65.0
1449,
1765
86.0
67.6
66.0
56.8
29.4
58.3
61.6
Token Count Complexity
Accuracy Complexily
82
81.3
Takes Nuan Per Etatunce on MMdl
POPE
140
MMD Accunay (6)
MMB
120
75.1
100
-6
75.8
SQA
20.3
4011
36.31
Tobic VTU
80
$9.7
004
ToMe w VIU
DToMe we VTT.
020
40
2
ToMe DyMU
SEED
26,7
42.0
20
L1aVA
0.20
TeaVQN
Image Complexiny Score
Image Complexity
Figure 4: Image Complexity vs Token Count and Accuracy. The scatter plot (left) demonstrates a strong correlation between DyMU's token count and image complexity score. On the right, MME accu- racy at varying complexity levels is compared between ToMe (fixed-length) and DyMU (dynamic-lengthy highlighting the benefit of assigning additional tokens to complex images.
Figure 5: Importance of Virtual Token Unmerg- ing (VTU), We ablate the performance of LLaVA 1.5 with two token reduction methods applied to the visual encoder-ToMe (fixed-enngth) and DToMe (dynamic- length). We observe that applying VTU significantly improves performance on
benchmarks, demon- strating robustness to varied token reduction methods.
4.1 Quantitative Evaluation
Comparing Visual Token Reduction Methods for VLMs To evaluate the efficacy of our approach, we compare against several existing methods that focus on reducing the number of tokens in VI.Ms. To the best of our knowledge, our proposed method is the first to (1) natively produce variable- length visual tokens and (2) require neither further fine-tuning of the VI.M nor additional textual conditions. In Table 2 present quantitative evaluation of all methods applied to pre-trained LLaVA 1.5 (25] architecture on standard VLM benchmarks, including GQA (16), MMBench (27). MME M1 POPE 122, ScienceQA 128). SEED-IMG [19] TextVQA (36.. MM 46) LLaVA- Bench DYMU achieves average performances of 97.5%, 99.2%, and 100.2%, relative to the original pretrained model, while reducing the token number by 84.5% 66.1%, and 31.6%, respectively. DYMU also outperforms previous training- free methods while enabling varied length output per instance. When decreasing the number of tokens, the largest performance drop occurs in TeXtVQA, which aligns with our expectation cantatiionnnnnestnandnnnnnn visual text is highly sensitive to the spatial locations of visual tokens, which are often disrupted by token merging. We leave addressing this issue to future work.
Compatibility with Different LLMs and Visual Encoders DYMU be seamlessly integrated into multiple variants of VLMs featuring different LLMs, visual encoders, and pretrain- we
Table 4: DYMU shows consistent effectiveness on an AnyRes VLM, LLaVA-OneVisin Visiin [18). We additionally show performance two comprehensive video understanding benchmarks.
Visual Tokens
Image Benchmurkks
Video Benchmarka VidMME MMBVid
Methods
MMB MME SEED Math Vista
ing strategies. In Tables 2 and 3 demonstrate that D effectively maintains baseline performance when applied both CLIP 33 to SigLIP [47]
ovaVA-ov-7B
79.3
75.8
75.6
58.0
61.3
ToMe 3
14.4%
71.2
63.1
68.3
46.6
57.6
1.08
DyMU-ov-low
~44.4
73.6
68.0
72.9
47.4
59.3
1.88
DyMU-ov-mid
225.1%
76.0
70.3
73.7
51.7
60.1
1.12
DyMU-ov-high
-46.5%
77.8
73.6
74.2
54.4
60.1
1.16
representations within the LLaVA 1.5
framework, using a Vicuna-7B 7 LLM. Furthermore, in Table 4 we evaluate DYMU on LLaVA-
OneVision [8]. a recent Any-Resolution (AnyRes) model with IPo_--000000M ( as visual encoder
and Qwen2 41 as LLM backbone. AnyRes enables processing images of arbitrary resolutions by
8

---

Query: What kind of hird is this?
Query: What your was the wine mode?
Query: How many women are the insage?
#Tokenn
Anewer
#Token
TTokken
Answer
Llava-OV
+
9477
blue jay
3645
2015
11664
DyMU OV
1295
blue jay
451
2015
1435
36
BG Removal
OCR
Object Detection
Llava-OV
9477
blue jay
3645
2015
11664
5
DyMU-OV
451
blue jay
259
2015
710
DyMU- + Background Removal Tool
DyMU OCR Tool
DyMU + Object Detection Tool
Figure 6: Controllable Visual Token Length. DYMU enables more flexible control over computational cost. In these examples, we combine DYMU with additional task specific tools background removal, OCR, or object detection focus only the relevant regions. As result, token count is substantially reduced without degrading performance, showcasing the flexibility of DYMU to adapt to different tasks.
segmenting them into smaller regions and encoding each individually. Our results show that DYMU remains compatible with this complex operation on both image and video tasks.
Image Complexity vs Number of Tokens In Figure (left), we show how the number of tokens varies with image complexity. We quantify image complexity C(I) by computing the JPEG com- pression ratio, i.e., C(T) S.rRa(f)
H H where SYPEq is the size (in bytes) of the image 7 after JPEG encoding, and H, W are the original height and width. For this experiment, we use CLIP-L/14-336 with DToMe -low to encode images in the MME benchmark. We observe strong correlation between the number of output tokens and image complexity, indicating that DToMM effectively preserves essential details in complex images while reducing redundancy in simpler ones. We include more qualitative visualizations Figure
Fixed vs Dynamic Token Reduction In (right) we categorize images into three bins based on their complexity scores, and compare the performance of ToMe (fixed-length token reduction) and DToMe on the MME benchmark. A key drawback of fixed token reduction its inability to adapt to image complexity, leading to over-compression for complex images and under-compression for simpler ones. While our method outperforms ToMe across all complexity levels, we observe the most significant gains on complex images, where ToMe struggles due to insufficient number of tokens.
Importance of Virtual Token Unmerging VTU efficiently reconstructs the representation of full visual token sequence from a reduced set of visual tokens. To demonstrate its impact, we compare LLaVA 1.5 variants with and without VTU. In the latter, the LLM does not undergo any modifications and directly receives fewer tokens. In Figure we evaluate this effect on two token reduction methods: ToMe [3) which produces fixed-length sequences, and DToMe (ours). Across both cases, we observe that applying VTU significantly improves performance on 8 out of benchmarks, demonstrating its effectiveness in preserving model capabilities despite token reduction.
4.2 Qualitative Analysis
Visualizing Variable Visual Token Length DTToMe facilitates producing variable number of token embeddings for images based on complexity of the content. In Appendix Figure R we visualize the number of visual tokens for various images from nine benchmarks. For each benchmark, we present three images corresponding to the minimum, median, and maximum token numbers output by DYMU-lww. We observe a strong correlation, both within and across different benchmarks, between image complexity and the number of tokens retained by DYMU. In Figure 7 we further visualize the token merging behavior at the patch level, showing that DYMU dynamically merges more tokens in regions of low complexity.
Controllable Visual Token Length DTTMM offers a key advantage over fixed token reduction methods: cost controllability. By dynamically adjusting the number of visual tokens based on image complexity, users gain more control over the computational cost incurred per image. This flexibility allows flexible combination of task-specific tools with DYMU to further boost efficiency while maintaining performance. For instance, in Figure 6. we show example applications of combining DYMU with background removal 4. OCR [10), and object detection 311 models, to extract focused regions and further reduce token count. Unlike existing VLMs, which impose a fixed token budget

---

SIMPLY
Figure 7: Visualization of merged patch tokens by DYMU. Patches with the same color are merged into a single token, and the number inside cach patch indicates the final merged token index. We find that the merging adapts to the complexity of the regions: simpler regions, such the sky a white background, tend to be merged more, while complex regions preserve larger number of unique tokens.
per image regardless of content, our method enables adaptive token allocation, ensuring that simpler regions consume fewer resources while more complex regions retain the necessary level of detail.
5 Improvement Gap Between FLOPs and Inference Time
In Table and Figure we show that DYMU achieves substantial FLOP reductions while pre- serving model performance. In Table [ we further examine inference time metries and observe less pronounced improvements. In this section, we provide deeper analysis of why there exists a non-trivial gap between FLOP reduction and actual inference time gains, particularly for algo- rithms like DYMU that fundamentally alter attention computation. Unlike FLOPs, which serve as hardware-agnostic measure of algorithmic efficiency, wall-clock inference time can fluctuate due to hardware conditions, batching strategies, and shared server workloads. Moreover, implementation details often dominate runtime behavior-low-lewee kernel optimizations and computation scheduling introduce substantial discrepancies between theoretical and practical efficiency. For example, compare two implementations for computing an NxN attention matrix from two (N, D) vectors. In Version 1, single torch. matmul operation produces the full matrix, whereas Version 2 splits the vectors into N/4 chunks and performs matrix multiplications on the corresponding submatrices before combining the results. Both methods yield the same total FLOPS (339. 74 MFLOPs), but Version 2 runs significantly slower (2.311 ms vs. 1.374 ms) because multiple small matrix multiplications underutilize optimized GPU kernels. Similarly, our VTU operation decomposes large multiplications into smaller components, achieving theoretical FLOP efficiency but incurring longer wall-clock times. This example highlights that translating theoretical compute savings into practical speedups often requires non-trivial engineering efforts, such as developing new kernels, efforts comparable to those seen in works like FlashAttention g
6 Conclusions and Limitations
In this work, we introduced DYMU, the first training free framework that dynamically reduces visual token counts in VLMs based on per-image complexity. DYMU can be directly integrated into mainstream VLM architectures that combine ViT-based visual encoders with RoPE-based LLM backbones. Despite its effectiveness, DYMU still faces limitations on information-dense, token- sensitive tasks, such as TextVQA (6), OCR [32], and DocVQA fine-graines visual cue is crucial for maintaining accuracy. Addressing this issue may require future work on complexity-aware token merging mechanisms capable selectively retaining semantically critical regions in information-dense scenes. Additionally, extending DYMU to reduce temporal redundancy videos represents a promising future direction.
30 where preserving every
10

---

Acknowledgments
We would like to express our gratitude to the anonymous reviewers for their insightful comments and suggestions. We would also like to thank our colleagues at Salesforce Research for their valuable internal discussions and feedback. This research is based upon work supported by U.S. DARPA ECOLE Program No. #HR001 10000012390000 The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright.
References
[1] Ibrahim M. Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems, 36: 36:16666-16425, 2023.
[2] Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392, 2025.
[3] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint ar xr1i:221100094,1, 2022.
[4] Bria AL. RMBG-1.4: Background Removal Model. https://huggingface.o///// http://gaaa...ooooraaaaaaaaaaaaa Baaaaai/RRMGG111 1.4 2024.
[5] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baohao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference Computer Vision, pages 19-35. Springer, 2024.
[6] Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 17164 17174, 2023.
[7] Wei-L in Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez. lon Stoica. and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023.
[8] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 2022.
[9] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi. Yue Yang. Jae Sung Park. Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, al. Molmo and pixmo: Open weights and open data for state-of-thh-aa art multimodal models, arXiv preprint arXiv:2409. 17146, 2024.
[10] Felix Ditrrch. Onnxtr: Optical character recogmition made seamless & accessible anyone, powered by onnx. https: //github. htem/fflixxitttrcceeeo wtt.:ttrrrriihh/h/1m/emm 2024.
[11] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, arXiv prepring arXiv:2306. 113223336.133,,,. 2023.
[12] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Flevating the Role of Image Understanding in Visual Question Answering. In 2017 IEEF Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[13] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608 3617, 2018.
[14] Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai- Rei Chang. Matryoshka query transformer for large vision-language models. arXiv preprint arXiv:240S.1 19311 2024.
[15] Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaoshen Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. Dynamic- llava: Efficient multimodal large language models via dynamic vision langugge context sparsification. arXiv preprint Xiv:24122.00866 2024.

---

[16] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6700-6709, 2019.
[17] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Int. J. Comput. Vis.. 123(1):32-73, 2017.
[18] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408 03326, 2024.
[19] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge. Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. arXiv prepring arXiv:2307.16125 2023.
[20] Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal Ilm. arXiv preprint arXiv:2407 023922, 2024.
[21] Yanwei Li. Chengyao Wang. and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[22] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint vxXiv:2305.10355 2023.
[23] Xiaoyu Liang. Chaofeng Guan, Jiaying Lu, Huiyao Chen, Huan Wang, and Haoji Hu. Dynamic token reduction during generation for vision language models. arXiv preprint arXiv:2501. 142044 2025.
[24] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In Computer Vision ECCV 2014 13th European Conference, 2014.
[25] Haotian Liu, Chunyuan Li, Yuheng Li. and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. arXiv prepring arXiv:2310.013
2023.
[26] Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention. arXiv preprint arXiv:2209. 13800, 2022.
[27] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is Your Multi-modal Model an All-around Player? arviv prepring arXiv:2307.0622 2023.
[28] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
[29] Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling vision transformers. arXiv preprint S.Kiv22110003866 2021.
[30] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209,
2021.
[31] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskly, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection. European conference on computer vision. pages 728-755. Springer, 2022.
[32] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirhan Chakraborty. Ocr-vqa: Visual question answering hy reading text in images. In ICDAR, 2019.
[33] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou. Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOV2: Learning Robust Visual Features without Supervision. arXiv prepring arXiv:2304.07193, 2023.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference Machine Learning (ICML), 2021.
12

---

[35] Yuzhang Shang. Mu Cai, Bingxin Xu. Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 153888 2024.
[36] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrhach. Towards vqa models that can read. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
[37]
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:137063, 2024.
[38] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan. Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing laioonlanguuae model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[39] Xinjian Wu, Fanhu Zeng. Xiudong Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers. arXiv preprint arXiv:2310.01812 2023.
[40] Long Xing, Qidong Huang, Xiaoyi Dong. Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelcrating your large vision- ssiounlanguuge models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410. 17247. 2024.
[41] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li. Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng Hc, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mci Li, Mingfeng Xuc, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang. Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024.
[42] Senqiao Yang. Yukang Chen, Zhuotao Tian, Chengyao Wang. Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary vision language models. Proceedings the Computer Vision and Pattern Recognition Conference, pages 19792-19802, 2025.
[43] Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-Ilava: Adaptive token pruning for large vision language models. arXiv preprint arXiv:2412.00447 2024.
[44] Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. Voco-Ilama: Towards vision compression with large language models. arXiv preprint A.Kv:2440.112777 2024.
[45] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809-10818, 2022.
[46] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang, MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv prepring arXiv:2308. 022000 2023.
[47] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- trainignn In Proceedings of the IEEE/CVF international conference computer vision, pages 11975 11986, 2023.
[48] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.032 003895, 2025.
[49] Yuan Zhang, Chun- Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang. Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevim: Visual token sparsification efficient language model inference. arXiv preprint urXiv:2410.04417, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims clearly presented in the abstract and intro, and well supported by empirical results and analysis in the section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are transparently discussed in Section 4.1 and Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

We provide detailed step-by step-p derivation in Section 3.2 for one of our key contribution, the virtual token unmerging (VTU).
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: provide implementation details in Appendix A and also provide the code part of the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used in this work are publicly available. also provide code as part of supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudddeo Coaee//ddeeSSbmmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all details in the Section 4 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information on how we evaluation done and how the scores are computed in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper foundational research and do not directly tied to specific application.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets, code and models are properly cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We provide readme for how use the code and models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy hatt::///eeerrssssse////ffferers for what should or should not be described.
20

---

A Implementation Details
Dynamic Token Merging For DToMe, we find layer-wise thresholds using a diverse dataset of 250k images sampled from the SFT instruction tuning data of LLaVA 1.5 [25 comprising of images from MS-COCO 224] VisualGenome [7]. OCR-VQA B2), TexIVQA 36 and GQA [6]. We also ablate the choice of image datasets in S In general, sufficiently diverse image set suffices, and performance remains robust dataset changes. Importantly, we only use the images to estimate the thresholds (in inference mode) and do not use the associated annotations or text in any way.
DYMU variants For each visual encoder in the experiments, including CLIP 34 and SigLIP A7
we find thresholds for three variants of the encoder by choosing different average number of tokens to drop (r;) in each layer. We represent these variants by e-low; e-mid, ehigh corresponding to
the expected average number of tokens. We also explore different VLMs including fixed-resolution
models, e.g., LLaVA 1.5 25 and any-resolution models, e.g. LLa Ve-One
[18].
Evaluation Details For results on LLaVA-1.5 (as in Tables 2 and 3 we leverage the official evaluation code from LL......5. The results on MME and LLa VA-Bench for DYMU are averaged across three runs, as we observe higher variance these two tasks. We omit VQAv2 12] and VizWiz 13] as the evaluation server was unavailable at the time of writing. For results on LLaVA One Vision (as in 4), we leverage VLMEvalKi for getting the evaluation results. For ToMe [3], we implemented the code for connecting ToMe to different VLMs under the same framework as DYMU. All results for ToMe are evaluated with the same setting DYMU ensure fair comparison. For other prior work, the results are copied from their original papers.
B Additional Analysis on Wall-Clock Inference Time
Table 5: Additional efficiency analysis comparing FLOPs and wall-clock inference times.
Methods
Nun/N
Attn MFLOPs
Attn Inference Time
End-to-Eno Inference Time
Full Attention
576 576
1359.0
9.17
131
DYMU-low VTU
89 576
32.4
1.26
115
DYMU-mid VTU
195 / 576
155.75
121
DYMU-high w/o VTU
576
635.85
2.92
132
DYMU-low w/ VTU
89/ 576
64.9
7.20
131
DYMU-mid w/ VTU
195 / 576
311.5
7.49
123
DyMU-high w/ VTU
394/ / 576
1272.0
7.60
132
C Visualization of Variable Token Length
In Figure [ we present a comprehensive visualization example images along with their encoded visual token counts. We MU-low (based CLIP-L/14-336) the encoder, where the full token length 576. Three images are shown for each benchmark, corresponding to the minimum, median, and maximum number of tokens, respectively. A clear correlation be observed between semantic richness and token count. We also note variations in the token range across different benchmarks. For instance, ScienceQA (28). which primarily contains figures and charts, tends to have fewer tokens than benchmarks featuring complex real-world scenes.
D Impact of Token Merging Schedule
We conduct an additional ablation study on one of the hyperparameters in DToMe, the merging schedule, during threshold finding. As detailed in Section we set target reduction number, r; for each layer. By default, r; set to constant value across all layers. Alternatively, we can vary r; across layers to encourage merging more or fewer tokens at different depths.
1CLIP version: openailclip-vii--argeppuhhhh 14-333 III with I.L.VA-1.5: timm/Vi--B11--Sig.PPP5.
"* SIgLIP version with LLLVA--V: wow.ee/iiiiiiiiiioooooooooaaaaaa patch14-38hh
https:///ti thub. wwtp///////ooocccceeooeee http:///w....pppoc.ccaaaaaa/aaaa
21

---

LLaVA Bench
Scales
GQA
#Token: 31
#Token: 108
WToken: 200
#Token: 86
#Token: 132
iTocen::
MMBench
MME
#Token:
#Token:: 71
MToken:
91
#Token:
MMVet
3.49
3.55
POPE
2+2=
3 65
9+3m
3+8
8 60
#Token:
#Token: 106
#Token: 220
#Token: 33
#Token: 79
#Token: 151
ScienceQA
#Token: 15
ITokem: 102
#Token: 225
#Token:
#Token: 50
#Token: 163
TextVQA
#Token:
Figure DToMe Token Count Across Benchmarks. For each dataset, we show three examples processed by our method those yielding the fewest tokens, the median number of tokens, and the most tokens. Observe that visually simple or nearly blank images consistently require fewer tokens. while more detailed, semantically complex cluttered images produce more tokens. This demonstrates how DToMe effectively adapts image complexity across diverse benchmarks, allocating fewer tokens simpler content and preserving more tokens for complex scenes.
Table 6: Ablation study merging schedules in DToMe. We compare three strategies: constant, linear (more merging in early layers), reverse linear (more merging later layers). Results show that merging fewer tokens in early layers yields better performance, while the constant schedule provides a balanced trade-off
between performance and token count.
Schedule
Visual Tokens
GQA
MMB (rS
POPE
SQA!
SEED
VQAT
MMVet
LLaVA "
Avg
Constant
195
61.7
62.8
1483,
1862
86.6
69.2
65.9
55.1
30.9
65.1
62.6
Linear
16
61.3
62.3
1437. 1767
86.2
69.4
65.3
28.8
58.6
60.8
Reverse Linear
21
61.8
63.8
1491, 1863
86.7
69.3
66.0
57.5
31.8
65.3
63.2
In T we present ablation study two alternative scheduling strategies: (1) linear, which merges more tokens in earlier layers and fewer tokens later layers, and (2) reverse linear, which follows the opposite trend. The results indicate that merging fewer tokens in earlier layers tends to yield better performance, while the constant schedule provides a balanced trade-off between performance and token count. This observation echoes the findings in the ToMe paper [3, where constant schedule was found to be nearly optimal.
E Impact of Dataset for Threshold Finding
The DToMe thresholds computed using images from the LLaVA instruction tuning dataset. Here, we investigate the sensitivity of DToMe the threshold estimation dataset. In Figure 9 we evaluate DYMU-LLaVA 1.5 with DToMe thresholds estimated the Pixmo-Cap [9] imagc-captioning dataset.
22

---

Threshold Data: LLaVA
Threshold Data: Pixmo- Cap
+30
Token Count
100.0
100
250
80.0
60.0
40.0
20.0
GQ4
MMB
MME
POPE
SQA
SEED
FQ4 MMVet
LLaVA
Avg
Figure 9: Comparing thresholds using LLaVA Instruct Data vs Pixmo-Cap. Although both methods use the per-layer merging hyperparameter (r, the Pixmo-base thresholds lead to fewer tokens (top)-like due domain differences. However, performance indicating robustness of DToMM on different image datasets.
range of benchmarks shows minimal drop (bottom),
Table 7: Impact of Virtual Token Unmerging. Full results for Figure
Method
# Visual Tokens
GQA
MMB
(MB(erspal)
POPE
SQA'
SEED
VQA
MMVet
LLaVA
Avg
ToMe 3
94
57.3
59.7
1357, 1673
86.8
68.9
60.5
53.2
25.6
61.0
59.2
+ VTU
94
60.6
63.7
1464, 1815
85.4
69.1
64.9
54.8
28.7
62.5
61.6
DYMU-low
89gu7
60.8
62.1
1438, 1787
86.3
69.3
65.0
53.1
30.0
62.9
61.5
w/o VTU
80
58.2
56.0
1346, 1639
67.7
60.9
51.3
25.2
58.8
58.2
We observe minimal performance change across all the benchmarks, highlighting the robustness of our method to dataset variation. Interestingly, we observe that the thresholds estimated using the Pixmo-Cap dataset lead fewer tokens during inference on the benchmarks. We hypothesize that this is due to the domain shift between the Pixmo- Cap images and more diverse LLaVA- LLaVA-instruce insttructt dataset which covers diverse real-world use cases.
F Full Results for Figure
We present the complete results of the ablation experiments on the effect of our proposed Virtual Token Unmerging, shown in Figure [5] The results are provided in Table 7
G
Full Results for Figure
We present the complete results of the ablation experiments threshold-finding datasets, as shown Figure The results are provided Table
23

---

Table 8: Impact of dataset for threshold finding. Full results for Figure
Model
Thresh Finding
A Visual
GQA
MMB
MMB(arsselt
POPE
SQA!
SEED
VQA
LLaVA
Avg
DvMU-mii
Llava
195
61.7
62.8
1483, 1862
36.6
69.2
65.9
55.1
30.9
65.1
62.6
DYMU mid
Pixmo
120
61.1
64.4
1474, 1808
86.0
69.4
65.3
56.2
30.5
63.7
62.4
24


Reviews:
[{"review_id": "VUvOTR9eYu", "reviewer": "Reviewer_TS2b", "summary": "The paper introduces DYMU (Dynamic Merging and Virtual Unmerging), a novel framework to enhance vision-language models (VLMs) by adaptively reducing visual tokens based on image complexity. It addresses inefficiencies in fixed-length token outputs of existing VLMs. Key components are:  1. Dynamic Token Merging (DToMe): Selectively merges similar tokens depending on image complexity (not resolution), reducing tokens by 32%-85% for simple images while preserving semantic details for complex ones. 2. Virtual Token Unmerging (VTU): Reconstructs full-sequence representations without retraining, maintaining performance and optimizing attention efficiency.", "questions": "No", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper provides comprehensive quantitative and qualitative experiments across multiple VLM architectures (LLaVA-1.5, LLaVA-OneVision), visual encoders (CLIP, SigLIP), and evaluation benchmarks (GQA, MMBench, MME, etc.), enhancing the credibility of the proposed method.\n\nThe proposed DToMe and DYMU are plug-and-play, requiring no retraining of the original model, making them suitable for resource-constrained scenarios or when training data is unavailable.\n\nDToMe extends the previous ToMe by dynamically adjusting the token count via threshold control (instead of pre-selecting a token budget), effectively balancing computational efficiency and performance.\n\nVTU addresses the mismatch between variable-length visual token sequences and fixed-length pre-trained LLMs through Virtual Token Unmerging, efficiently computing attention matrices without explicitly reconstructing the full sequence.\n\n\nWeaknesses:\n\nOverall computational cost: Although Table 1 details the attention computation cost, the lack of discussion on end-to-end costs makes it unclear whether DYMU outperforms other methods in efficiency. For the DYMU-high w/ VTU configuration, the MFLOPs (1272) show limited reduction versus Full Attention (1359). In low/mid settings, VTU nearly doubles MFLOPs, potentially limiting its applicability.\n\nArchitectural limitations: DToMe targets ViT-based encoders, and VTU is designed for RoPE-based LLMs. Their generalizability to other architectures (e.g., CNNs or non-RoPE models) remains unverified.\n\nHyperparameter search cost: DToMe requires offline threshold calculation (using large image sets), incurring non-negligible preprocessing costs. Appendix C shows that merging strategies significantly affect results, and fine-tuning layer-specific r_i for different models further increases preprocessing overhead.\n\nVTU approximation: Equation 10 uses embedding averages to reconstruct attention outputs—an approximate approach. The paper claims minimal empirical impact but provides no ablation studies or quantitative error analysis.", "was_revised": true, "final_justification": "The rebuttal mostly addressed my concerns. I raised my score.", "rebuttal": "**We thank Reviewer TS2b for the detailed and constructive comments! We appreciate the reviewer’s recognition of the strengths of our comprehensive experiments and the novelty of both DToMe and VTU in enabling training-free dynamic-length token merging. We hope the following response addresses the remaining concerns and fosters meaningful discussion.**\n\n# W1: Overall Computational Cost\n## W1.1: Additional Analysis on Wall-clock Inference Times\nWe thank the reviewer for the suggestion! We provide additional analysis on wall-clock inference times, summarizing our findings below.\nWe continue to see speedup in terms of inference times in most settings, however, we find that the inference time reduction is less consistently correlated with token count and FLOPs, particularly when measuring end-to-end duration. \nThis discrepancy has also been observed in prior works, such as Table 5 in [1] and Table 7 in [2].\nWe provide **a detailed discussion in the next section on the underlying reasons and practical challenges.**\n\n|  | Avg N_un / N | Attn MFLOPs | Attn Inference Time (ms) | End-to-End Inference Duration on MME (s)\n|-|-|-|-|-|\n| Full | 576/576   | 1359.0 | 9.17 | 131\n||\n|DyMU-low |\n| DyMU-low w/o VTU |  89/576  | 32.4 | 1.26 | 115\n| DyMU-low w/ VTU | 89/576 | 64.9 | 7.20 | 131\n||\n|DyMU-mid |\n| DyMU-mid w/o VTU |  195/576  | 155.8 | 1.29 | 121\n| DyMU-mid w/ VTU | 195/576 | 311.5 | 7.49 | 123\n||\n|DyMU-high |\n| DyMU-high w/o VTU | 394/576 | 635.9 | 2.92 | 132\n| DyMU-high w/ VTU | 394/576 | 1272.0 | 7.60 | 132\n\n\n## W1.2: Deep Dive into the Gap between FLOPs and Inference Time\nIn this section, we elaborate our findings on why there is a **non-trivial gap between FLOP reduction and actual inference time gains, especially for algorithms like DyMU that fundamentally alter the computation of attention.** We also explain why we believe **FLOPs is a more robust metric for comparing theoretical/algorithmic efficiency**, as is also a standard practice in prior works [3][4][5][6].\n- **Controllability**: From definition, FLOPs quantifies algorithmic compute independent of hardware, batching, parallelism, and system load. Wall-clock inference time can vary significantly due to deployment configurations and parallel jobs running on the same node, which are practically difficult to control on shared school servers.\n\n- **Implementation Gap**: In practice, wall-clock inference time is also **heavily dependent on the actual implementations and low-level optimizations of the computation functions.** This may lead to significant discrepancies between FLOPs and inference time.\n    - Here we show a **concrete example**, where we compare two implementations of computing a N×N attention matrix from two (N,D) vectors. In Version 1, we simply do one matrix multiplication using `torch.matmul` to get the N×N matrix. In Version 2, we first split the input vectors into N/4 chunks, do matrix multiplication on the (N/4, N/4) chunks and add them back; As shown below, this results in **identical FLOPs but very different wall-clock times**. This is because `torch.matmul` is heavily optimized for large matrix multiplications, leading to longer inference times when multiple `torch.matmul` calls are made on smaller matrices, despite having the same total FLOPs.\n\n        |  | # torch.matmul | matrix size per multiplication | MFLOPs | Wall-Clock Time (ms)\n        |-|-|-|-|-|\n        |Version 1| 1 | 576 | 339.74 | 1.374\n        |Version 2| 16 | 144 | 339.74 | 2.311\n\n    - In our case, the VTU operation requires **decomposing the matrix multiplication** into smaller sub-matrices, which **inflates the inference time although having a better theoretical efficiency in terms of FLOP**. (See the “Implementation Notes” section in the supplementary code for more details.) \n\n    - This demonstrates that, to fully translate the benefit of theoretical FLOPs improvements to actual inference time improvements may require **non-trivial work such as implementing new kernels, which themselves can be a separate paper such as Flash-Attention[7].**\n\nTherefore, in this work, we focus on **formally deriving the theoretical efficiency gains and presenting empirical results on FLOPs, token length and end-task performance.** But we do **acknowledge that optimizing wall-clock inference time of DyMU through dedicated low-level kernel development is an important follow-up work.**\n\n\n**Reference:**\n- [1] Xing, Long, et al. \"Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.\"\n- [2] Wen, Zichen, et al. \"Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?.\"\n- [3] Bolya, Daniel, et al. \"Token merging: Your vit but faster.\"\n- [4] Chen, Mengzhao, et al. \"Diffrate: Differentiable compression rate for efficient vision transformers.\"\n- [5] Chen, Liang, et al. \"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.\"\n- [6] Shang, Yuzhang, et al. \"Llava-prumerge: Adaptive token reduction for efficient large multimodal models.\"\n- [7] Dao, Tri, et al. \"Flashattention: Fast and memory-efficient exact attention with io-awareness.\"\n\n\n\n## W1.3: How VTU can achieve better tradeoff in performance and efficiency\nWe elaborate on how VTU can effectively help achieve a better trade-off in both DyMU-low and DyMU-high settings. \n- As shown in the Table below, in **DyMU-low settings** (where token reduction is large) **VTU shows more significant improvement** in performance and **introduces moderate increases in absolute FLOPs** (although nearly doubling); \n- On the other hand, in **DyMU-high settings**, **removing VTU results in much smaller performance drop**; so it’s safer to remove VTU under this setting to gain more efficiency.\n\n|  | GQA | MMB | MME (prcp,all) | POPE | SQA-I | SEED-I | VQA-T | MMVet | LLaVA-W | AVG\n|-|-|-|-|-|-|-|-|-|-|-|\n| DyMU-low w/o VTU | 58.2| 56.0 | 1346,1639 | 86.9 | 67.7 | 60.9 | 51.3 | 25.2 | 58.8 | 58.2\n| DyMU-low w/ VTU | 60.8 | 62.1 | 1438,1787 | 86.3 | 69.3 | 65.0 | 53.1 | 30.0 | 62.9 | 61.5 (↑3.3)\n||\n| DyMU-high w/o VTU | 60.1 | 64.7 | 1460,1798 | 86.8 | 68.7 | 65.2 | 55.8 | 30.4 | 65.2 | 62.3\n| DyMU-high w/ VTU | 61.9 | 64.3 | 1498,1846 | 86.8 | 69.9 | 66.1 | 58.0 | 31.5 | 64.5 | 63.2 (↑0.9)\n\n---\n\n# W2: Applicability to CNN and Non-RoPE models\nWe agree that this would be an interesting extension: to explore whether the high-level idea can be applied to CNNs and non-RoPE models. \nFor DToMe, since the fundamental similarity measurement relies on the key vectors of visual tokens, it is not directly applicable to CNN architectures. \nFor VTU, it is not directly applicable to non-RoPE layers due to the use of absolute positional embeddings.\nNevertheless, **within the claims of this paper**, we emphasize that the proposed method remains **broadly applicable to most mainstream VLMs, where ViT-based vision encoders and RoPE-based LLM backbones are widely used.**\n\n---\n\n# W3: Hyperparameter Search Cost is Minimal, No Finetuning Needed\nWe would like clarify that:\n- **The offline threshold calculation is inference-only**; no fine-tuning is performed to find the layer-specific r_i (Line 122).\n- In practice, threshold finding on a CLIP ViT-L encoder using 250K images can be completed **within 1 hour on 2 H100 GPUs**.\n- There is **no strict restriction on the size or type of dataset**. Any sufficiently diverse dataset is acceptable, and only images (without annotations) are required.\n\n---\n\n# W4: VTU Approximation in Equation 10\nThe reviewer raised an interesting discussion on the approximation of the “re-merging” part in the VTU algorithm. \nTo clarify, this approximation is required for obtaining the expected efficiency derived in S3.2, so **no direct alternatives can be compared under the same computational complexity**. And we show empirically in Figure 5 and Table 6 that this VTU operation successfully leads to performance gains.\nHowever, we agree that **exploring more sophisticated re-merging methods**, such as using similarity-based weighted averaging, would be an **interesting direction for future work**, albeit at the cost of some additional computation."}, {"review_id": "RACQgEeuMn", "reviewer": "Reviewer_2zYg", "summary": "In this paper, the author presents an efficient, training-free framework DYMU that can dynamically reduce the computation of VLMs while maintaining performance with two novel module DtoMe and VTU. A thorough has validate the frameworks’ effectiveness.", "questions": "I don't completely understand what the red and blue squares in Figure 1 represent. If they represent the same characteristic with different values, it is recommended that the author not use only two colors but instead use one/multiple colors. If they represent different types of variables, it is hoped that corresponding explanations can be added beside the figure.", "limitations": "I don't completely understand what the red and blue squares in Figure 1 represent. If they represent the same characteristic with different values, it is recommended that the author not use only two colors but instead use one/multiple colors. If they represent different types of variables, it is hoped that corresponding explanations can be added beside the figure.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.The author logically described the current problems in this field and the purpose of the proposed method.\n2.The author provided a detailed explanation of the functions and contributions of each module.\n3.Sufficient experiments have fully verified the effectiveness of this method.\n\nWeakness:\nI don't completely understand what the red and blue squares in Figure 1 represent. If they represent the same characteristic with different values, it is recommended that the author not use only two colors but instead use one/multiple colors. If they represent different types of variables, it is hoped that corresponding explanations can be added beside the figure.", "was_revised": false, "final_justification": "", "rebuttal": "**We thank Reviewer 2zYg for the constructive and encouraging feedback! We appreciate the reviewer’s recognition of our detailed explanations and comprehensive experiments on the proposed methods. We hope the following response addresses the remaining concerns and questions.**\n\n# W1: Clarification on the token illustration in the method figure\nWe thank the reviewer for the suggestion. We believe the reviewer is referring to the red and blue squares in the method overview figure (Figure 2), which indicate the “visual tokens” in the transformer layers. \nThese tokens indeed represent the same characteristics but with different values. We use two colors to better illustrate the bipartite matching algorithm, which divides the tokens into two groups for matching. The red and blue tokens indicate tokens in different groups which will potentially be merged together.\nWe will **add additional explanation in the figure caption** to clarify this in the next revision."}, {"review_id": "AID5fVk5bb", "reviewer": "Reviewer_iv6c", "summary": "This paper proposes a training-free token reduction method for MLLMs by adaptively adjusting the number of vision tokens based on the complexity of the image. The proposed DYMU framework is comprised of two modules, Dynamic Token Merging and Virtual Token Unmerging. The dynamic token merging module continuously reduces the number of vision tokens by merging similar ones based on image complexity in the different layers of the vision encoder. In the LLM, the designed Virtual Token Unmerging module approximates the full vision sequence operations using the merged tokens. Experiments on several general MLLM benchmarks show the efficiency gain brought by the proposed framework.", "questions": "1. Is the proposed VTU module compatible with FlashAttention?\n2. Do you think designing some metrics measuring how well the original vision information can be reconstructed from the merged tokens (or directly down-sampled ones) can better represent the image complexity compared with the similarity-based one?", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\nThe idea to adaptively reduce the token number according to image complexity is important.\n\nWeakness:\n1. DYMU emphasizes adaptive per-image token reduction. When comparing with other token reduction methods, the proper experiment setup should be measuring the actual (average) inference time on different benchmarks and comparing the performance of other token reduction methods **configured to a similar inference time**. \n2. Many more recent token reduction methods are missing, e.g. VisionZip (CVPR2025) and PyramidDrop (CVPR2025).\n3. DYMU merges tokens at every vision-encoder layer, but the paper lacks any quantitative analysis of the computational benefits this provides over a single post-encoder merge. It remains questionable whether early-layer merging—when token similarities may be noisier—yields net gains.\n4. Token merging thresholds rely on key-vector similarity as a proxy for image complexity, yet no vigorous validation is provided that this reliably relates to semantic detail for the later LLM part. For instance, a dense document image (all text) may exhibit high key similarity and thus undergo over-merging, potentially losing critical OCR information.\n5. Although broad VQA and grounding benchmarks are included, the paper omits fine-grained tasks (e.g., DocVQA, InfoVQA, MME-RealWorld) where small details matter most. Without these, the claim of “preserving core information” under aggressive compression is unsubstantiated.\n6. Besides FLOPs measurements, the inference time should also be included in all experiments.", "was_revised": true, "final_justification": "I appreciate the authors' detailed and thoughtful rebuttal. However, my main concerns remain:\n\nInference Time vs. FLOPs: While I understand the challenges in aligning FLOP reductions with end-to-end latency, actual inference time is the most critical metric for real-world deployment. The current method shows limited or no gains in end-to-end latency, especially when VTU is enabled, which undermines the practical efficiency benefits claimed by the paper.\n\nEvaluation on Fine-Grained / Information-Dense Tasks: The lack of evaluation on tasks such as DocVQA, InfoVQA, or MME-RealWorld limits confidence in the method's robustness for semantically dense images. \n\nTherefore, I will keep my original rating.", "rebuttal": "**We thank Reviewer iv6c for the detailed and constructive comments! We appreciate the reviewer’s recognition of our novelty in complexity-conditional token reduction. We hope the following response addresses the remaining concerns and fosters meaningful discussion.**\n\n# W1, W6: Inference Time Comparison\n## W1.1: Additional Analysis on Wall-clock Inference Times\nWe thank the reviewer for the suggestion! We provide additional analysis on wall-clock inference times, summarizing our findings below.\nWe continue to see speedup in terms of inference times in most settings, however, we find that the inference time reduction is less consistent with token count and FLOPs, particularly when measuring end-to-end duration. \nThis discrepancy has also been observed in prior works, such as Table 5 in [1] and Table 7 in [2].\nWe provide **a detailed discussion in the next section on the underlying reasons and practical challenges.**\n\n||Avg N_un/N|Attn MFLOPs|Attn Inference Time (ms)|End-to-End Inference Duration on MME (s)\n|-|-|-|-|-|\n|Full|576/576|1359.0|9.17|131\n||\n|DyMU-low|\n|DyMU-low w/o VTU|89/576|32.4|1.26|115\n|DyMU-low w/ VTU|89/576|64.9|7.20|131\n||\n|DyMU-mid|\n|DyMU-mid w/o VTU|195/576|155.8|1.29|121\n|DyMU-mid w/ VTU|195/576|311.5|7.49|123\n||\n|DyMU-high|\n|DyMU-high w/o VTU|394/576|635.9|2.92|132\n|DyMU-high w/ VTU|394/576|1272.0|7.60|132\n\n## W1.2: Deep Dive into the Gap between FLOPs and Inference Time\nIn this section, we elaborate our findings on why there is a **non-trivial gap between FLOP reduction and actual inference time gains, especially for algorithms like DyMU that fundamentally alter the computation of attention.** We also explain why we believe **FLOPs is a more robust metric for comparing theoretical/algorithmic efficiency**, as is also a standard practice in prior works [3][4][5][6].\n- **Controllability**: From definition, FLOPs quantifies algorithmic compute independent of hardware, batching, parallelism, and system load. Wall-clock inference time can vary significantly due to deployment configurations and parallel jobs running on the same node, which are practically difficult to control on shared school servers.\n\n- **Implementation Gap**: In practice, wall-clock inference time is also **heavily dependent on the actual implementations and low-level optimizations of the computation functions.** This may lead to significant discrepancies between FLOPs and inference time.\n    - Here we show a **concrete example**, where we compare two implementations of computing a N×N attention matrix from two (N,D) vectors. In Version 1, we simply do one matrix multiplication using `torch.matmul` to get the N×N matrix. In Version 2, we first split the input vectors into N/4 chunks, do matrix multiplication on the (N/4, N/4) chunks and add them back; As shown below, this results in **identical FLOPs but very different wall-clock times**. This is because `torch.matmul` is heavily optimized for large matrix multiplications, leading to longer inference times when multiple `torch.matmul` calls are made on smaller matrices, despite having the same total FLOPs.\n\n        ||#torch.matmul|matrix size per multiplication|MFLOPs|Wall-Clock Time (ms)\n        |-|-|-|-|-|\n        |Version1|1|576|339.74|1.374\n        |Version2|16|144|339.74|2.311\n\n    - In our case, the VTU operation requires **decomposing the matrix multiplication** into smaller sub-matrices, which **inflates the inference time although having a better theoretical efficiency in terms of FLOP**. (See the “Implementation Notes” section in the supplementary code for more details.) \n\n    - This demonstrates that, to fully translate the benefit of theoretical FLOPs improvements to actual inference time improvements may require **non-trivial work such as implementing new kernels, which themselves can be a separate paper such as Flash-Attention[7].**\n\nTherefore, in this work, we focus on **formally deriving the theoretical efficiency gains and presenting empirical results on FLOPs, token length and end-task performance.** But we do **acknowledge that optimizing wall-clock inference time of DyMU through dedicated low-level kernel development is an important follow-up work.**\n\n**Reference:**\n- [1] Xing, Long, et al. \"Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.\"\n- [2] Wen, Zichen, et al. \"Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?.\"\n- [3] Bolya, Daniel, et al. \"Token merging: Your vit but faster.\"\n- [4] Chen, Mengzhao, et al. \"Diffrate: Differentiable compression rate for efficient vision transformers.\"\n- [5] Chen, Liang, et al. \"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.\"\n- [6] Shang, Yuzhang, et al. \"Llava-prumerge: Adaptive token reduction for efficient large multimodal models.\"\n- [7] Dao, Tri, et al. \"Flashattention: Fast and memory-efficient exact attention with io-awareness.\"\n\n---\n\n# W2: Additional Comparison Baselines\nWe thank the reviewer for pointing out additional baselines for comparison! \n- A kind reminder that we have **already included PyramidDrop in Table 2**, Row 11, labeled as “PDrop.”\n- In the following table, we show additional comparison for the following baselines: VisionZip[1], DiffRate[2], and PiToMe[3]. We show that DyMU **achieves competitive overall performance while being both training-free and dynamic in length.**\n\n|Method|#Token|Dynamic?|Training-Free?|GQA|MMB|MME (prcp,all)|POPE|SQA-I|SEED-I|VQA-T|MMVet|LLaVA-W|AVG\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\nVisionZip[1]|64|✗|✓|55.1|60.1|-,1690|77.0|69.0|-|55.5|31.7|62.9|-\nVisionZip[1]|128|✗|✓|57.6|62.0|-,1762|83.2|68.9|-|56.8|32.6|64.8|-\nVisionZip[1]|192|✗|✓|59.3|63.0|-,1783|85.3|68.9|-|57.3|31.7|67.7|-\nDiffRate[2]|~57|✓|✗|57.9|-|1341,-|-|66.4|-|30.6|-|-|-\nPiToMe[3]|~57|✗|✓|59.9|-|1448,-|-|69.0|-|43.0|-|-|-\n||\nDyMU-low|89±27|✓|✓|60.8|62.1|1438,1787|86.3|69.3|65.0|53.1|30.0|62.9|61.5\nDyMU-mid|195±47|✓|✓|61.7|62.8|1483,1862|86.6|69.2|65.9|55.1|30.9|65.1|62.6\nDyMU-high|394±57|✓|✓|61.9|64.3|1498,1846|86.8|69.9|66.1|58.0|31.5|64.5|63.2\n\n**Reference:**\n- [1] Yang, Senqiao, et al. \"Visionzip: Longer is better but not necessary in vision language models.\"\n- [2] Chen, Mengzhao, et al. \"Diffrate: Differentiable compression rate for efficient vision transformers.\"\n- [3] Tran, Chau, et al. \"Accelerating transformers with spectrum-preserving token merging.\"\n\n---\n\n# W3: Merging Per-Layer vs Merging Last-Layer\n\nCompared to post-encoder merging, per-layer merging offers two key benefits: \n- (1) Additional efficiency gains within the ViT encoder (as demonstrated in the ToMe paper); \n- (2) Progressively merging a smaller number of tokens per layer helps preserve more visual concepts better than merging a large number of tokens in the final layer.\n\nTo further validate (2), we provide **an additional ablation study** as follows, in which we introduce a new DyMU variant that performs merging in the last three layers. \n*Note: Due to the nature of bipartite matching, each layer can merge at most half of all tokens.* Therefore, merging only in the last layer is insufficient to meet the target number of remaining tokens. Instead, we merge across the last three layers to reach the same average target token count, i.e., 94 for DyMU-low. \nThis setting can be considered a *“less extreme”* version of post-encoder merging. \nEven under this configuration, as shown in the table below, we observe a **notable performance drop compared to per-layer merging**, particularly on the hallucination task (POPE).\n\n||GQA|MMB|MME (prcp,all)|POPE|SQA-I|SEED-I|VQA-T|MMVet|LLaVA-W|AVG\n|-|-|-|-|-|-|-|-|-|-|-|\n|DyMU-low w/ Per-Layer Merge|60.8|62.1|1438,1787|86.3|69.3|65.0|53.1|30.0|62.9|61.5\n|DyMU-low w/ Last-Layer Merge|58.3|63.2|1421,1755|81.1|69.4|62.9|50.3|25.1|58.2|59.0 (↓2.5)\n\n# W4,W5: Limitations on Semantically-Dense Images\nWe are **fully aware of this limitation and have acknowledged it in Section 5**, where we refer to such cases as token-sensitive tasks, exemplified by TextVQA. In fact, as shown in Table 2, we find this to be a common limitation across all training-free methods. We also agree that incorporating additional benchmarks, such as DocVQA, could further validate this limitation. And future work is needed to explore complexity-aware token merging for semantically dense images, such as multimodal documents.\n\nNevertheless, we would like to **highlight a unique advantage of DyMU** not present in previous work: its **compatibility with external tools**. As illustrated in **Figure 6**, DyMU can be used in conjunction with OCR tools to address text-dense tasks, while still introducing efficiency gains through dynamic-length token merging. Incorporating agentic tool use into the token merging framework may offer a promising direction for handling the diverse merging needs across different tasks.\n\n# Q1: Compatibility with FlashAttention\nWe currently provide two implementations of VTU:\nV1 uses the exact matrix decomposition, as derived in Section 3.2, which is more efficient in terms of FLOPs but empirically slower in wall-clock time;\nV2 uses a direct matrix expansion approximation, which has higher FLOPs but achieves faster wall-clock performance.\n**Among the two, V2 is compatible with FlashAttention. \nMore details can be found in the “Implementation Notes” section of the supplementary code.**\n\n# Q2: Reconstruction Metric for Image Complexity\nWe believe this is an interesting idea and a very promising future direction! Specifically, a reconstruction-based metric captures more low-level visual details, which could be particularly useful for identifying complexity in semantically dense images, as discussed above. \nMoreover, we believe that **combining similarity-based metrics** (which focus on high-level semantics) **with reconstruction-based metrics** (which emphasize low-level details) could result in a more general and effective metric for diverse tasks."}, {"review_id": "PwijTnjb4L", "reviewer": "Reviewer_1uHD", "summary": "This paper presents DyMU, a training-free token compression framework for vision transformers. DyMU introduces two key contributions: (1) a dynamic merging strategy that adaptively controls token reduction based on image complexity, and (2) a novel virtual token unmerging mechanism that improves performance degradation caused by aggressive token merging. The method is evaluated on various models across multiple downstream tasks and benchmarks, demonstrating consistent improvements over prior training-free approaches, particularly ToMe. The work is well-motivated, clearly written, and practically relevant.", "questions": "1. Could you report DyMU's performance against DiffRate or PiToMe under comparable token budgets? Even partial results would help position the method relative to the latest merging approaches.\n\n2. Would you consider adding merged-token maps (akin to ToMe's Fig. 4) to illustrate where tokens are retained or merged in simple vs. complex scenes?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. Well-motivated design: The paper offers a clear rationale for adapting token budgets to image complexity, as opposed to using a fixed schedule. The proposed dynamic merging strategy is intuitive and practically motivated.\n\n2. Robust empirical evaluation: DyMU is validated on a wide range of benchmarks and models, showing strong generalization and consistent improvements.\n\n3. Virtual Token Unmerging: This is a novel contribution not present in prior methods like ToMe. It enables accurate RoPE computation with merged tokens, and empirical results confirm that VTU substantially reduces accuracy degradation.\n\n4. Cost-accuracy trade-off: DyMU supports controllable granularity, letting users dynamically balance efficiency and accuracy. It is also compatible with domain-specific preprocessing tools (e.g., OCR-based cropping), enabling flexible deployment.\n\n**Weaknesses**\n\n1. Limited baselines: While ToMe comparisons are thorough, recent token-merging or pruning methods such as DiffRate[r1] and PiToMe[r2] are not quantitatively reported. Including them (even if training-required) would contextualise DyMU's gains more convincingly.\n\n2. Missing qualitative visualizations: ToMe originally visualized merged-token regions; similar maps for DyMU would help readers see how token allocation differs for simple vs. complex images, beyond giving only the number of tokens after merging in Figs. 1 and 7.\n\n**References**\n\n[r1] Mengzhao Chen, et al. DiffRate: Differentiable Compression Rate for Efficient Vision Transformers. In ICCV 2023.\n\n[r2] Hoai-Chau Tran, et al. Accelerating Transformers with Spectrum-Preserving Token Merging. In NeurIPS 2024.", "was_revised": true, "final_justification": "I appreciate the authors' response and the additional experiments, which effectively address my concern regarding comparisons with more recent baselines. I acknowledge that DyMU's performance is competitive relative to these newly added baselines, and as a dynamic approach, it also holds potential advantages in terms of efficiency. Taking into account the authors' responses to the other reviewers as well, I find the current experiments sufficiently solid, and consider the proposed method to be meaningful and of practical value. I will raise my score to absolute acceptance, and I hope the authors incorporate their additional clarifications and discussions with the other reviewers into the final version of the paper.", "rebuttal": "**We thank Reviewer 1uHD for the detailed and constructive comments! We appreciate the reviewer’s recognition of our strengths in well-motivated design, comprehensive evaluations, novel VTU algorithm, and more controllable tradeoff in efficiency and accuracy. We hope the following response addresses the remaining concerns and fosters meaningful discussion.**\n\n# W1,Q1: Comparison to additional merging/pruning methods\nWe thank the reviewer for pointing out additional baselines to compare with. Below we add the comparison with DiffRate[1] and PiToMe[2], together with VisionZip[3] which is suggested by reviewer iv6c. \nWe show that DyMU **achieves competitive overall performance while being both training-free and dynamic in length.**\n\n| Method | #Token | Dynamic? | Training-Free?| GQA | MMB | MME (prcp,all) | POPE | SQA-I | SEED-I | VQA-T | MMVet | LLaVA-W | AVG\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\nDiffRate[1] | ~57 | ✓ | ✗ |  57.9 | - | 1341, - | - | 66.4 | - | 30.6 | - | - | -\nPiToMe[2] | ~57 | ✗ | ✓ | 59.9 | - | 1448, - | - | 69.0 | - | 43.0 | - | - | -\nVisionZip[3] | 64 | ✗ | ✓ | 55.1 | 60.1 | -, 1690 | 77.0 | 69.0 | - | 55.5 | 31.7 | 62.9 | -\nVisionZip[3] | 128 | ✗ | ✓  | 57.6 | 62.0 | -, 1762 | 83.2 | 68.9 | - | 56.8 | 32.6 | 64.8 | -\nVisionZip[3] | 192 | ✗ | ✓  | 59.3 | 63.0 | -, 1783 | 85.3 | 68.9 | - | 57.3 | 31.7 | 67.7 | -\n||\nDyMU-low | 89±27 | ✓ | ✓ | 60.8 | 62.1 | 1438, 1787 | 86.3 | 69.3 | 65.0 | 53.1 | 30.0 | 62.9 | 61.5\nDyMU-mid | 195±47 | ✓ | ✓ | 61.7 | 62.8 | 1483, 1862 | 86.6 | 69.2 | 65.9 | 55.1 | 30.9 | 65.1 | 62.6\nDyMU-high | 394±57 | ✓ | ✓ | 61.9 | 64.3 | 1498, 1846 | 86.8 | 69.9 | 66.1 | 58.0 | 31.5 | 64.5 | 63.2\n\n**Reference:**\n- [1] Chen, Mengzhao, et al. \"Diffrate: Differentiable compression rate for efficient vision transformers.\"\n- [2] Tran, Chau, et al. \"Accelerating transformers with spectrum-preserving token merging.\"\n- [3] Yang, Senqiao, et al. \"Visionzip: Longer is better but not necessary in vision language models.\"\n\n\n\n# W2,Q2: Visualization of the merged token regions\nThis is a great suggestion! Specifically, such a visualization can help further validate the dynamic-length merging by showing that less complex areas are more aggressively merged.\nSince **OpenReview does not allow figures or external links** in the rebuttal, we are unable to include it at this stage. **We will add this visualization in our next revision.**"}]

Author rebuttal:

Meta-review: This paper introduces DyMU, which combines dynamic token merging with virtual unmerging to improve the efficiency of vision–language models in a training-free and plug-and-play manner. The method adaptively adjusts token counts based on image complexity while preserving compatibility with RoPE, achieving substantial token reductions with minimal accuracy loss across multiple benchmarks. Reviewers appreciated the novelty, clarity, and broad validation, though they raised concerns about missing end-to-end latency results, the limited task scope (e.g., lack of document-heavy data), architecture dependence, and the absence of certain recent baselines and visualizations. In rebuttal, the authors added stronger comparisons (DiffRate, PiToMe, VisionZip), clarified FLOPs versus runtime discrepancies, and acknowledged limitations in text-heavy scenarios, which addressed most concerns. Overall, the contribution is clear, original, and practically useful, with strengths outweighing the remaining limitations, so I recommend acceptance.
Decision: Accept (poster)