PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models
Authors: Mouxiao Huang
Borui Jiang
Dehua Zheng
Hailin Hu
Kai Han
Xinghao Chen
Venue: ICLR 2026 Poster (2026)
Keywords: positional encoding
visual token merging
token clustering
token compression
cascade compression
multimodal large language models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks, yet often suffer from inefficiencies due to redundant visual tokens. Existing token merging methods reduce sequence length but frequently disrupt spatial layouts and temporal continuity by disregarding positional relationships. In this work, we propose a novel encoding operator dubbed as **P**ositional **P**reservation **E**mbedding (**PPE**), which has the main hallmark of preservation of spatiotemporal structure during visual token compression. PPE explicitly introduces the disentangled encoding of 3D positions in the token dimension, enabling each compressed token to encapsulate different positions from multiple original tokens. Furthermore, we show that PPE can effectively support cascade clustering --- a progressive token compression strategy that leads to better performance retention. PPE is a parameter-free and generic operator that can be seamlessly integrated into existing token merging methods without any adjustments. Applied to state-of-the-art token merging framework, PPE achieves consistent improvements of 2\%~5\% across multiple vision-language benchmarks, including MMBench (general vision understanding), TextVQA (layout understanding)  and VideoMME (temporal understanding). These results demonstrate that preserving positional cues is critical for efficient and effective MLLM reasoning. Our code is available at https://github.com/MouxiaoHuang/PPE

Published as a conference paper at ICLR 2026
PPE: POSITIONAL PRESERVATION EMBEDDING FOR TOKEN COMPRESSION IN MULTIMODAL LARGE LAN- GUAGE MODELS
Mouxiao Huang; Borui Jiang" Dehua Zheng, Hailin Hu", Kai Han, Xinghao Chen Huawei Technologies
R [jiangborui, hailin.hu, xinghao. Shen]]huawei.... com
ABSTRACT
Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks, yet often suffer from inefficiencies due to redundant visual tokens. Existing token merging methods reduce sequence length but fre- quently disrupt spatial layouts and temporal continuity by disregarding posi- tional relationships. In this work, we propose a novel encoding operator dubbed Positional Preservation Embedding (PPE), which has the main hallmark of preservation of spatiotemporal structure during visual token compression. PPE explicitly introduces the disentangled encoding of 3D positions in the token di- mension, enabling each compressed token to encapsulate different positions from multiple original tokens. Furthermore, we show that PPE can effectively support cascade clustering -a progressive token compression strategy that leads to bet- performance retention. PPE is parameter-fres and generic operator that can be seamlessly integrated into existing token merging methods without any adjust- ments. Applied to state-of-the-art token merging framework, PPE achieves consis- tent improvements of 2% ~ 5% across multiple vision-language benchmarks, in- cluding MMBench (general vision understanding). TextVQA (layout understand- ing) and VideoMME (temporal understanding). These results demonstrate that preserving positional cues is critical for efficient and effective MLLM reasoning. Our code is available at httts:::/ggghhhhh com/ www.:/Moxxiioooio wwwp:///ooo.iaaoouuaaaa/aa/.
INTRODUCTION
Multimodal Large Language Models (MLLMs) have recently achieved remarkable success across range of Eision-languae understanding tasks Bai et al. (2025); Lei et al.] (2025): [i et al. (024a); Yuan al. (025):: Zhang al.) (2025). A common paradigm involves encoding images or video frames into dense visual tokens, which are then fed into the language model for joint understanding. However, this dense representation is often highly redundant, leading inefficiencies in computa- tion and inference Song et al. (2024), To address this, recent works Dhouib et al. (2025); Jin et al. (2024) Ma et al. 2023); Zeng al. (2022); Zhang et al. (2024a) have explored visual token com- pression, which merges similar tokens to reduce visual sequence length while preserving semantic information, thereby accelerating inference and lowering memory usage.
Despite the efficiency, existing compression methods often disrupt the spatial and temporal structure of visual inputs, limiting their applicability layout-sensitive tasks such as counting, temporal grounding and sequential understanding. As shown in Figure (a). clustering-
Chat-UniVi Jin
ctt al. (2024) token compression may discard fine- mne-grained spatial or temporal cues. Figure (b) illustrates the recent methods like PACT Dhouib al. (2025) which have attempted to preserve layouts during compression, but still remain constrained by the insufficient and imprecise positions.
In this work, we introduce Positional Preservation Embedding (PPE), novel positional encoding strategy which explicitly retains the different spatiotemporal layout into one compressed token. Our design is motivated by two key principles. First, we aim to preserve spatial and temporal positions during similar token merging. To this end, PPE firstly assigns each visual token a positional ID (e.g..
"qqull contribution.

---

Published as a conference paper at ICLR 2026
Answer: The doge with diamond helmet and sword.
MLLM
Token
Compression
Token Embeddings
Text tokens
Visual Token Clustering
RoPE-ID Injection
Question: What in the image?
without extra computation
layout destruction
osiiiin IDs
Position IDs
Splited Position
layout preserv
details missing
R Random Position
Unique Position (b) PACT
PPE Position
(a) ChatUniVi
(c) Ours
Figure 1: Comparison between PPE and other token merging methods of processing positional
IDs. To simplify, the components such the visual encoder omitted. (a) ChatUniVi [Jin et al.
(2024) mainly assigns randomize ID value to the clustered visual tokens. (b) PACT Dhouib et al.
(2025) retains the ID of the cluster center for the clustered visual tokens. (c) Proposed PPE splits
the IDs of compressed token on different dimensions, so that each compressed token could contain
several original position IDs.
2D spatial for images and 3D spatiotemporal for videos). During compression, each merged token retains several positional IDs of their constituents. This ensures that most of the visual scene layouts are still accessible to MILM at high compression rates, as shown Figure] (c).
Secondly, we support the widely adopted cascaded compression strategy, which performs token merging progressively across the Transformer Vaswani et al. (2017) layers. This design is inspired by the observation that different layers capture increasingly abstract representations with higher similarity (Boly et al. (2023a); Dhouib et al. (2025); Song et al. (2024); 22024) et al. (2024a). Thus, merging tokens in multi-stage manner enables higher compression ratios without collapsing the shallow semantics prematurely. Since PPE decoupled from the merging algorithm and operates solely on position IDs, it naturally extends to multi-stage compression. This allows our method to preserve fine-grained spatiotemporal layouts across multiple compression stages, resulting in higher compression ratio, IDs retention (Section 4.4.5) and improved performance (Section
To our best knowledge, PPE is the first work that explores an effective and lightweight positional preservation solution during visual token compression of MILM. In contrast existing positional embedding methods that only preserves one position for single token and lead to the loss of de- tailed layouts, PPE enables each single token represent multiple positions, thus preserving the vision layout more completely. Moreover, PPE is parameter-free and can be used in a plug-and-ple fashion for easily implanting into existing visual token compression methods without any additional computational costs.
In the experiments, we apply PPE for tackling a variety of lasion-language tasks, including gen- eral vision-language understanding on MMBench Liu et al. (2024)). text-based visual reasoning TextVQA Singh et al. (2019), temporal understanding on VideoMME Fu et al. (2024), etc. The reported performances consistently outstrip previous compression methods Dhouib al. (025); Jin et al. (2024) by significant margins after fine-tuning We strongly believe that PPE can make inroads into domains of sparse token representing in MILM where dense visual representation had previously reigned supreme.
The main contributions of this work follows:
2

---

Published as a conference paper at ICLR 2026
We identify critical limitation in existing visual token merging methods namelly, the neglect of spatial structure preservation and temporal coherence whherence-which which leads to distortion of intra-frame layouts and disruption of inter-frame temporal relations.
We propose Positional Preservation Embedding (PPE), novel, plug-and-play approach that explicitly preserves spatiotemporal integrity during token merging, effectively address- ing both spatial and temporal challenges.
We further show that PPE can be applied in cascade compression manner within multiple transformer layers of the MLLM, enabling substantial compression with minimal perfor- mance degradation.
We conduct extensive experiments across serveal image and video benchmarks, demon- strating that PPE maintains accuracy while reducing the visual token count by 90% and outperforms other visual token compression methods at comparable reduction ratios.
RELATED WORK
2.1 MULTIMODAL LARGE LANGUAGE MODELS
MLLMs extend traditional LLMs by incorporating visual inputs, enabling unified vision-language understanding and generation. Recent models such as Flamingo Alayrac et al. (2022), the LLaVA series Li et al. (204a); Liu al. (003); Zhang al. (002b),, and the Qwen- VL series Bai et al. 22023 2025) Wang et al. (2024) achieve strong performance across captioning, VQA, and instruc- tion following. These models typically align vision and language via cross-attention, projection modules, or lightweight multimodal transformers. However, most rely on dense visual tokens, pos- ing efficiency and scalability challenges for high-resolution such as LLaA Lin et al. (2024), VideoChat [i et al. (023b), and Video-LLaMA Zhhag al. (2023) mitigate this via frame sampling or sparse memory. contrast, our method introduces a Positional Preservation Embedding that enables substantial token reduction while maintaining spa- tiotemporal coherence, offering a scalable alternative for multimodal reasoning.
long-form visual inputs. Extensions
2.2 VISUAL TOKEN MERGING
To improve the efficiency of MLLMs, recent works have explored visual token reduction strate- gies Kong et al. (2025) such as clustering and merging [Bolya et al. (2023a): Xu al. (2022): Ma ct al. ( et al. (2022): Dhouib et al. (2025) [Song et al. (024):Un Jin et al. (2024): Yang et al.
(2025) These methods significantly reduce the sequence length of image video inputs, enabling
faster inference and reduced memory consumption. Techniques include token matching Boolya et al.
(2023a). hierarchical clustering Ma et al. (2023): Zeng al. (2022). group-based representation Xu et al. (2022). and pruning-clustering hybrids Dhouib et al. 2025). ). with extensions to long-video understanding via sparse memory representations Song et al.) (2024) and unified token compres- sion across modalities Jin al. (2024). Despite their effectiveness, these methods typically discard original positional information, which can disrupt spatial layouts and temporal inttiuutt-limmitee performance ime-grained visual reasoning tasks. In contrast, approach explicitly preserves spatiotemporal position cues throughout compression, maintaining structural fidelity while achiev- ing substantial token reduction.
2.3 POSITIONAL ENCODING IN MLLMS
Positional encoding Vaswani et 2017)) is essential in MLLMs maintain spatial and temporal relationships across vision and language tasks. The Rotary Position Embedding (RoPE) Su et al. (2024) is widely adopted method, which captures relative token positions in sequences using a ro- tational mechanism. This has been extended to 2D RoPE Heo et al. (2024), which is well-suited for image tasks, enabling models to understand spatial locality in vision transformers (ViTs). For video and temporal tasks, the Qwen series introduced 3D MRoPE Bai et al. 2003); Wang et al. (024)) which integrates rotary encoding across both spatial and temporal dimensions, preserving continuity across frames. However, these methods often face challenges in visual token compression, as reduc- ing token counts can lead to the loss of crucial positional information. In contrast, we propose PPE: Positional Preservation Embedding, novel approach that maintains spatiotemporal positional cues
3

---

Published as a conference paper at ICLR 2026
during token reduction, preserving structural integrity while enabling aggressive compression. This method, to the best of our knowledge, is new exploration in the field.
Answer:
The Cheems equips with leather helmet and diamond sword.
Transformer Layer
11
Compression
12
pos
cascade compression with PPE
Transformer Layer
16
Compression
2
9
10
14
7
13
14
16
Tokenize
Question:
15
What is in the image?
pos IDs
pos IDs
Token Merging
w. PPE
w.o. PPE
Initial compression with/without PPE
MLLM pipeline with cascade token compression
The overview pipeline of the proposed PPE with cascade compression. Left: Main idea of Positional Preservation Embedding (PPE) integrated in token compression. For each RoPE ID in compressed token embedding, PPE splits the dimension into chunks to prefill multiple position IDs. The IDs of tokens with high importance scores reserved preferentially. Right: The MLLM pipeline integrating PPE and cascade compression. Token compression is applied multiple layers, each with PPE. See main text for more explanation.
Figure 2:
METHODOLOGY
In this section, we first recap the preliminaries of PPE, and then analyze why previous methods fail to hold the positional information. Afterward, we introduce our proposed PPE and cascade compression, illustrated in Figure 2
3.1 PRELIMINARY
Rotary Position Embeddings in MILM. RoPE Su et al. (2024) is positional encoding technique designed to enhance the Transformer Vaswani al. (2017) architecture by integrating relative posi- tional information directly into the self-attention mechanism. Unlike traditional absolute positional encodings, RoPE applies a rotation the query and key vectors in multi-head attention, enabling the model to capture relative positions effectively. Assumed that token vector z E ND in multi-head attention, the rotation operation can be represented as:
RoPE( (aPE(aa,0n m) e emma 2d, d 1.....
where m indicates the position ID of z. For capturing spatiotemporal layouts, Qwen2.5-VL. Bai et al. (2025) introduces M-RoPE, a structured embedding scheme that partitions the embedding di- mension, each encoding positional information along different visual axis such temporal order, image height, and width. Formally, the original RoPE rotation operation is modified to:
M-RoPE(za, ma) = e imz za, d 1...,
where m E D is pre-filled 2D or 3D visual position IDs. Assumed that the 3D visual token is at position (t, hi w), the {ma} could be indicated as:
d 1... D1.
m&
h
d Di + ... D + D2,
(3)
w,
d Dj + D2 + 1 ... Dj + D2 + Da.
4

---

Published as a conference paper at ICLR 2026
where D1. D2, D3 are human-craftee integer to control the size of mrope sections Bai et al. (2025) satisfying that D1 + D2 + D3 D.
Visual Token Compression. ChatUni Jin al. (2024) adopts lightweight, parameter-free clus- tering algorithm for token compression to mitigate the computational overhead of long visual token sequences. The key idea is merge tokens with ity. Assumed that N visual token embeddings {z, E ] indicates the token embeddings in group j. The compressed token embeddings zi defined
al. (2016) based token similar- 1i=1 are clustered to M groups, {z, hec,
z
1
Σ IECs
Zi j 1...M.
C2
3.2 POSITIONAL PRESERVATION EMBEDDING
While similarity- compression effectively reduces sequence length, it disrupts fine-grained spatiotemporal layouts. To mitigate this, we propose Positional Preservation Embedding (PPE), which retains multiple positional cues per merged token to preserve structural information.
PPE builds on the principle behind RoPE Su et al. (2024), in which the rotation of position em- beddings is independent at the dimension. As a result, the m in Equation [ can be totally different on the dimension D represent different positions at the same time. For instance, M-RoPE Wang jet al. (2024) partitions embedding dimensions into several groups to store spatiotemporal positions, formally in Equation g Inspired by this, PPE merges different positions one merged token by splitting into more groups. Formally, the merged PPE ID m could be indicated as:
D mJ mi,d, d = -k- +1
D K
(5)
where K is a fixed hyper-parameter to represent the maximum capacity of PPE, and {m3) K the set of different position IDs in ID RoPE manner before merging. Note that K is always divisible by dimension D. In M-RoPE manner, is set to the greatest common divisor of the mrope sections, guaranteed that each dimension is evenly cut into K groups:
(k 1) D
+1
k
3D m mi
3D
3D
(k 1) D2
D1+1
h
D1,
K
(k: 1)
R
D1 D2 +1
D1 D2-
The key insight shared of PPE is that similar token embeddings share their feature embeddings during token merging. Rather than assigning a single position ID to a merged token, we extend this idea to multi-position formulation that better reflects the internal diversity of token cluster con- taining more positional information of original input visual tokens. Specifically, consider a cluster group C, containing token embeddings (z, JeC; with corresponding position IDs (m,. Rather than choosing one representative ID, we select the top-K IDs per cluster which are scored by the distance from the cluster center. Note that the score is higher if the token is closer to the cluster center Jin et al. (2024). If |Cy| < K, high-weight tokens are repeated to fill the slots. Denote the merged IDs of PPE m, the PPE rotation of vector is simply follow the RoPE which could be formulated as:
PPE(za, zd, ma)
e eimgla zd: d L...D.
3.3 CASCADE COMPRESSION WITH PPE
In this section, we further investigate the effectiveness of cascade compression in conjunction with our proposed Positional Preservation Embedding (PPE) strategy. Cascade compression a widely adopted technique in token compression [Bolya al. (2023a); Dhouib et al. (2025); Song et al. (2024); Zhang et al. (2024a), motivated by the observation that deeper Transformer layers exhibit greater representational redundancy Dong et al. (2021). whereas earlier layers encode critical low- level semantics that are less amenable to aggressive compression.
To this end, we implement cascaded PPE-based compression pipeline built upon ChatUniVi Jin jet al. (2024), as illustrated in Figure In this design, visual token clustering is applied not only prior to feeding tokens into the LLM,, but also within selected L.MM layers. Leveraging PPE, we preserve
5

---

Published as a conference paper at ICLR 2026
original positional information, enabling efficient computation of new cluster center positions after each merge.
For PPE ID assignment, we follow the standard top-K selection strategy described in Section 3.2 choosing token IDs with minimal distance to the cluster center. As shown in Figure 2 during repeated merges, the number of preserved position IDs reduces gradually (e.g., retaining only four IDs when two previously merged tokens are merged again).
Experimental results demonstrate that this cascaded PPE design maintains fine- tine-grained spatial struc- ture across compression stages and achieves higher compression ratio without performance loss. Moreover, this design significantly improves both ID retention (Section 4.4.5) and downstream task performance (Section 4.3) These findings confirm the strong compatibility of PPE with standard cascaded compression frameworks.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. To demonstrate the effectiveness of PPE, we construct our training datasets for super- vised fine-tuning (SFT) referring to the public projects: LLaVA-Video-1758 Zhang et al. (2024b) and VA-Oe Vision Li et al. (2024a). Limited by the computational costs, we adopted even down-sampling on these datasets. To highlight the improvements of the model on image and video benchmarks, further up two different settings: one for images and another for videos. Specif- ically, we utilized about 120K video samples to fine-tune the models for multimodal video bench- marks, while 300K image samples to the models for the multimodal image benchmarks.
Evaluation Benchmarks. We use VideoMME Fu et al. (024)) as our primary benchmark for mul- timodal video analysis. To assess generalization, also report results on NeXT-QA (multi-choice and open-ended) Xiao et al. (2021). (2024b). For image understanding, we evaluate MMBench (CN/EN) Liu al. (0024a) and SQAel et al. (2017) to assess general visual comprehension. To further examine text-rich layout and document understanding. we include TextVQA Singh al. (2019), Doc MMthhew et al. 22221. OCRBench Liu et al. (2024b), and ChartQA Masry et al. 2022 All reported metrics fol- low the evaluation criteria of their respective benchmarks, and unless otherwise noted, higher values indicate better performance.
Li et al. (2023a). and MVBench Li et al.
Implementation Details. We conduct our default experimental settings by using the Qwen2.5- VL- 3B-Instruct model Bai et al. (2025). All models are fine-tuned in fully supervised manner on the down-sampled datasets with all parameters unfrozen. The training setting mainly refer- ring to the public project Lee (2024), in which the gradient accumulation step is 4, the learning rate 1e-5 along with the warm-up ratio of 0.03, while the learning rate of visual encoder and patch merger is 2e-6 and 1e-5, respectively. We train for only 1 epoch. To maintain the opti- mal performance and computational costs for both training and evaluation, we mainly follow the official input size configurations. Specifically, we set image_min.pixee imaagemmaa
512 28 28 and and mww.pixels = x280x28 x 28 for image inputs, and videovievooo www.d/ee..i..iiieeeeeee.e 128x 28 x 28 768 x 28 x 28 for video inputs. Additionally, the maximum number of video frames is set to 64 due to the limited memory usage. The 3D M-RoPE section is set to [16, 24, 24], while [32, 32] for the 2D manner. The number of preserved token IDs is set to K for 3D and K = 32 for 2D, which corresponds to the greatest common divisor (GCD) of M-RoPE sections.
mixxppixell
Token compression settings. The proposed PPE can be integrated into most existing compression methods. For our experiments, we adopt the SOTA clustering-basee framework Chat-UniVi et al. (2024) and follow its default training strategy, with slight modifications to adapt to the Qwen2.5-Vi model. In Chat-UniVi Vi. the number clustered tokens is controlled by a predefined clustering ratio due to the native resolution technique. Specifically, the spatial clustering ratio is set 0.45, while the temporal clustering ratio is 0.0625, consistent with the original paper. For fair comparison, token clustering is applied the interface between the vision encoder and the L.M by default.

---

Published as a conference paper at ICLR 2026
Table 1: The overall performance comparison across different benchmarks. The Dense model is simply Qwen2.5-VL-3B --nt----tttt model fine-tuned on our SFT datasets. The subscript S/ST indi- cates spalial-only and spatiotemporal compression, respectively. Bold denotes the best performance among compressed variants, and underline denotes the best overall including the Dense baseline. The image average score calculated without OCRBench.
Benchmarks
Dense
+Chat-UniVis
+PPEs (Ours)
+Chat-UniVisr
+PPEST (Ours)
MMBench (EN)
85.89
84.92
84.73 (-0.19)
MMBench (CN)
86.07
83.71
84.87 (+1.16)
SQA
76.90
77.30
77.88 (+0.58)
TextVQA
79.50
57.66
77.14 (+19.48)
DocVQA
89.44
52.48
76.79 (+24.31)
OCRBench
761
535
691 (+156)
ChartQA
79.96
49.60
74.52 (+24.92)
Image Average
82.96
67.61
79.32 (+11.71)
Vi (wlo subs)
57.81
57.22
58.70 (+1,48)
56.07
57.41 (+1.34)
VideoMME (w subs)
57.96
57.22
59.07 (+1.85)
56.15
57.78 (+1.63)
NeXT-QA (MC)
78.20
77.63
78.42 (+0,42)
77.59
77.99 (+0.4)
NeXT-QA (OE)
31.65
25.37
32.61 (+7.24)
26.55
31.95 (+5.4)
BEED-Bennch-Ve Video
57.60
56.08
55.98 (-0.10)
53.47
54.19 (40.72)
MVBench
67.90
66.90
67.38 (+0.48)
64.38
66.42 (42.04)
Video Average
58.52
56.74
58.69 (+1.95)
55.70
57.62 (+1.92)
Reduction Ratio
0%
55%
55%
94%
94%
4.2
MAIN RESULTS
We evaluate the performance of our proposed method on a range of image and video benchmarks. The results show that PPE consistently outperforms previous approaches, demonstrating its effec- tiveness both image and video understanding.
Image Tasks. Table illustrates the strong performance of PPE on image understanding bench- marks. Despite reducing visual tokens by 55%, PPE achieves competitive or superior performance across all tasks. It significantly outperforms Chat-UniVi in overall accuracy (79.32% % vs. 67.61 %) and on TextVQA (77.14% vs. 57.66%) while maintaining comparable results on other tasks. Com- pared to the full-token Dense baseline model, PPE shows only minor acceptable performance drop in overall accuracy (82.96% % vs. 79.32 %) despite using less than half the visual tokens.
Video Tasks. As shown in Table under 55% token reduction with spatial-only compression, PPE consistently outperforms Chat-UniVi across most tasks and in overall accuracy (58.69 % vs. 56.74%), even exceeding the Dense baseline. With more aggressive 94% reduction using spa- tiotemporal compression, PPE again leads in overall score (57.32 % vs. 55.7 %,, demonstrating strong robustness under heavy compression. Notably, challenging benchmarks like VideoMME and NeXT-QA (OE), PPE shows clear improvements over Chat-UniVi.
Summary. In summary, PPE achieves substantial token reduction- 555% for spatial-only compres- sion and 94% for spatiotemporal compression-whtee preserving, or in some cases even enhancing, downstream task performance. This demonstrates that preserving visual token IDs enables more ac- curate reconstruction of the spatial layout and temporal order, even under aggressive compression, thereby improving reasoning capabilities, as illustrated in Fig.
Comparison with M-RoPE. As reported in Table Dense model adopts M-RoPE by default, and thus Chat-UniVi directly inherits this design. In contrast, our PPE strategy compresses visual tokens while still retaining the positional information of merged tokens within single representation. This design leads to consistently stronger results under identical reduction ratios, highlighting that PPE more effective than M-RoPE in preserving positional cues during token compression.
Comparison of MILMs with Intact and Compressed Tokens. We compare our PPE with repre- sentative dense MLLMs, including LLaVA- Vision Li et al. (2024a), Intern VL2.5 Chen et al. (2024), and Qwen2.5-VL Bai et al. (2025). as well as token compression approaches such as PACT Dhouib al. 2025 and Sparse VL. Zhang et al. (2024a). as shown in Table 2 All baseline
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of MLLMs with intact tokens and compressed tokens.
VideoMME
VideoMME
MMBench
MMBench
Reduction
Model
MVBench
TextVQA
(w/ suhs)
(w subs)
(EN)
(CN)
Ratio
LLVA-One LLVVA-OmeVisionO..55
44.00
43.50
45.50
52.10
InternV1.2.55444
62.30
63.60
71.60
81.10
79.30
76.80
0%
Qwen2,5-VL-3B
61,50
67.60
67.00
79.10
78.10
79.30
0%
PACT-7B
57.60
80.30
75.00
67%
SparscVLM-7B
64.10
57.80
66%
PPE-3B (Ours)
58.70
59.07
67.38 67.35
84.78
84.85
77.08
PB (Ours)
58.48
58.52
90%
Table 3: Performance of cascade compression, PPE integrated Before and/or Within- LLM.
VideoMME (w/o subs)
VideoMME
NeXT-QQ (MC)
NeXT-QA (OE)
SKEDD-Bnnc -Video
Reduction Ratio
Before
Wäthin
MVBench
Average
(w subs)
56.41 58.70
$6.49
78.07
32.99
55.12
68.25
57.91
0%
78.42
32.61
55.98
67.38
$8.69
55%
/
57.41
57.70
77.98
31.06
55,36
66.65
57.69
90%
x
58.48
58.52
78.20
32.20
56.11
67.35
58.48
90%
results are reported by their respective original papers. Note that Chat-UniVi Jin et al. (2024) is excluded due missing benchmark results. Despite using only 3B parameters, PPE achieves com- petitive performance compared to both dense and compressed models, with a 55% token reduction ratio. PPE* refers cascade compression variant applied within the LLM (details in Section 4.3) which further improves the reduction ratio to 90% while maintaining comparable performance.
4.3 CASCADE COMPRESSION
In addition to applying our PPE compression before the LLM, we further investigate its integra- tion within the LLM to enable multi-layer cascade token compression. Specifically, we conduct layer-wise insertion study using the 2wen2.5-VL-33 -nnsrutt model, which contains 36 transformer layers. Specifically, insert PPE-based clustering modules at layers 11, 23, and 35, while keeping all other configurations unchanged.
As shown in Table applying PPE within the LLM using per-layer clustering ratio of 0.45 achieves a 90% token reduction while maintaining performance comparable to the 55%-rrdducton case. Un- der the same 90% reduction budget, the within-LLLM configuration outperforms applying PPE only before the LLM, demonstrating that cascade compression across multiple stages enables higher com- pression ratios without prematurely collapsing shallow semanties.
4.4 ABLATION STUDY
In this section, we study the insight of PPE, its compatibility with other glustering-based compres- sion methods, inference efficiency, and the effect of retaining different numbers of token ID posi- tions to verify its ability to preserve positional information. Further studies different reduction ratios A./ aggregation stages A.2 model size generalization A.4 backbone generalization A.3 task generalization/A.3 etc, are provided in the Appendix
4.4.1 INSIGHT INTO WHY PPE BETTER PRESERVES TOKEN SEMANTICS
The key insight of PPE lies in its ability to simultaneously encode multiple positions within single token. Specifically, unlike traditional RoPE where multiple frequencies are rigidly bound single positional index, PPE introduces flexibility by allowing different frequencies bind to different positional indices-an as long as the corresponding tokens are considered similar the clustering. As result, during attention computation, each token attend to multiple combinations of frequencies and positional indices. This enables each token pair to perceive a richer set of relative positional relationships, thereby preserving the global positional layout despite compression.
We randomly selected two samples and evaluated rext-visual attention at the final LLM layer under a 55% token compression ratio. As shown in Figure 3 b)),, both Uni and PPE reduce entropy
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation of PPE integration with clustering- based compression and inference efficiency.
MMBench (EN)
MMBench
LLM Generation (s)
Peak Memory Usage (GB) blue.
Reduction Ratio
Method
TextVQa
(CN)
Time (s)
PACT
74.14
74.17
73.73
0.08
15.82
89%
PACT + PPE
74.48
75.00
73.87
0.09
15.82
89%
ToMe
74.31
73.63
74.94
0.90
15.81
57%
ToMe PPR
74.57
74.74
76.16
0.91
15.81
57%
Dost
Case M6X
Clse 3000
(a) Entropy comparison
(b) Variance comparison f (c) Chat-UniVi: "d" (d) PPPP:Pdakoooo digital"
Figure 3: Attention statisties and visualizations of samples from TextVQA (a-b) Quantitative
comparison of entropy and variance. (c-d) Qualitative attention score visualizations of case 34602.
compared to the dense baseline, while PPE achieves similar entropy but consistently higher variance, indicating sharper and more confident grounding under compression. This is further illustrated in Figure Segureeccddo d-)) for case 34602 (question: "What is the brand of this camera?"). both models attend the correct region, but Chat-UniVi's focus is narrowly confined due to positional information loss and answers incorrectly, whereas PPE preserves coverage across the text, recovering the full brand name. More samples and analyses are provided Appendix A.7
4.4.2 INTEGRATION WITH OTHER CLUSTERING-BASSE METHODS
PPE is compatible with other clustering based methods. When integrated into the training teieing-free PACT Dhouib et (2025) and ToMe [Bolya al.) (2023b), experiments on Awen2-VL-7B. Instruct Wang et al. (2024) show consistent improvements over the baselines (Table 4.. This com- patibility arises because PPE preserves the correspondence between tokens and their RoPE compo- nents: cluster averaging keeps embeddings close, and associating merged tokens with all original RoPE indices maintains positional information. When training is allowed, PPE further improves RoPE allocation, yielding more robust joint token-position representations and enhancing perfor- mance (e.g., Chat-UniVi + PPE, Table
To further demonstrate PPE's generality across different compression frameworks, conduct tensive experiments on VisionZip Yang et al. (2025). which combines pruning and merging and supports both training-free and SFI settings. We simply integrate PPE into VisionZip on Qwen2.5- VL-3B-Instruct following the official instructions (dominant_rates and contertual ratio are set 0.40 and 0.05 respectively to achieve 55% reduction ratio), randomly sampling 1/10 of our cu- rated dataset for the SFT setting and only fine-tuning the projector for one epoch. The results are shown in Table In both settings, PPE delivers improvements in most cases, especially on layout- and OCR-sensitive benchmarks, demonstrating that PPE retains more spatial information under the same reduction ratio. Moreover, when training is allowed, PPE achieves larger gains. Our method is decoupled from model size., extensive experiments are provided in Appendix A.4
4.4.3 SFT OR TRAINING-FREE
PPE is ald--an,-,,a, parameter-free method. Whether SFT required depends on the underlying compression method, not PPE itself: 1) Chat-UniVi is a merging-based method that requires SFT, and performs bad training-free setting especially on layout-sensitive benchmarks. 2) PACT and ToMe are fully training-free, and PPE is directly applied to the official pre-trained models without any additional training. 3) VisionZip supports both training-fre and SFT usage, and PPE improves performance in both settings without modifying their token selection aggregation mechanisms. This supports our key claim that PPE can enhance existing token compression pipelines in a plug- and-play manner and further improve RoPE allocation when training is allowed.

---

Published as a conference paper at ICLR 2026
Table 5: Extensive comparisons under both training-free free and SFT using Qwen2.5-VL-3B-Imsste
Setting
Benchmarks
Dense
Chat-UniVi
Chat-UniVi+PPE
VisionZip
VisionZip+PPE
Training- Free
MMBench (EN)
79.10
81.50
82.28 (+0.78)
83.48
83.18 (-0,30)
MMBench (CN)
78.10
80.06
81.43 (+1.37)
81.75
81.64 (-0.11)
TextVQA
79.30
37.60
73.78 (+36.18)
79.00
79.62 (+0.62)
DoA
93.90
19.58
66.16 (+46.58)
83.98
85,63 (+1.65)
OCRBench
797
307
598 (4:291)
713
725 (+12)
ChartQA
84.00
18.72
67.08 (+48.52)
79.72
80.72 (+1.00)
MMBench (EN)
85.89
84.92
84.73 (-0.19)
83.99
83.78 (-0.21)
MMBench (CN)
86.07
83.71
84.87 (+1.16)
82.12
82.63 (+0.15)
SFT
TextVQA
79.50
57.66
77.14 (+19.48)
80.02
82.06 (+2.04)
Doc
89.44
52.48
76.79 (+24.31)
84.84
90.52 (+5.68)
OCRBench
761
535
691 (+156)
711
780 (+69)
ChartQA
79.96
49.60
74.52 (+24.92)
80.20
82.36(+2.166
Reduction Ratio
0%
55%
55%
55%
55%
Table 6: Ablation study performance and retained ID ratio with varying K.
VideoMME
VideoMME
NeXT-QA
NeXT-QA
SEED-Bench -Video
Reduction Ratio
K
MVBench
Average
(w/o subs)
(w subs)
(MC)
(OE)
Retained
1
$7.74
$8.04
77.88
28.77
$5.07
68.08
$7.97
55%
45%
58.70
59.07
78.42
32.61
55.98
67.38
58.69
77%
24
58.19
58.56
78.02
31.64
55.52
67.73
58.28
55%
84%
4.4.4
ANALYSIS OF INFERENCE EFFICIENCY
PPE introduces no additional parameters and incurs negligible computational overhead. Since op- erates by redistributing existing RoPE dimensions, the computational and memory savings primarily result from token reduction itself. To ensure fairness and reproducibility. we reused the official PACT code report runtime and memory usage, as presented in Table With same reduction ratios, PPE does not need extra parameter and introduce very few inference time demonstrating its efficiency.
4.4.5 EFFECT OF K ON PERFORMANCE AND RETAINED ID RATIO
The hyperparameter K determines how many token ID positions are retained after merging, thus affecting spatiotemporal preservation. As shown in Table 6 and Figure [bb,, K = causes severe degradation due to insufficient positional information, while K 24 introduces redundancy and slightly underperforms K = The choice K = 8 achieves the best trade-off, aligning with the GCD of the 3D M-RoPE dimensions [16, 24, 24] to preserve essential signals without over-retention, and is adopted as our default. Moreover, Table 6 shows that larger K values yield higher valid ID) retention which refers the ratio of distinct positional embeddings preserved after compression; for K > 1, the retention rate exceeds the visual token retention ratio (45%), confirming that PPE captures positional information beyond token counts. Further visualization analyses for different choices of K are presented in Appendix A.10
CONCLUSION
In this work, we propose Positional Preservation Embedding (PPE), simple yet effective opera- tor for retaining spatiotemporal positional information during visual token compression in MILMs. PPE encodes fine-grained spatial and temporal cues into each compressed token and parameter- free, plug-and-play, and compatible with existing pipelines without architectural changes. Exper- iments on MMBench, TextVQA, and VideoMME show consistent 2% ~ 5% gains, validating its effectiveness and generality. PPE also supports cascade clustering for progressive compression, offering a flexible trade-off between efficiency and performance. Our results underscore the impor- tance of positional information in improving MILMM efficiency.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. Annual Conference on Neural Information Processing, NeurIPS, 2022. htL.tpp: papppp Files/paper/20222 /hash/ 96Ca1 70Ca172be7Bi 1 www.:o0obbsstttttttt
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile lasgun-anguuag model for understanding, local- ization, text reading, and beyond. CoRR, 0bs/2200811966. 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502. 13923, 2025.
Daniel Bolya, Cheng- Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In International Conference Learning Repre- sentations, ICLR, 2023a.
Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. 20233. net/ net/ forum? id= JrwZRaRw7Es
URL htt.tttss///ppeeeeeeee
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang. Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. CoRR, abs/2412.05271, 2024. doi: 10.48550/ 241..24122.00 05271
ARXIV.2412.05271. URL http::htt////ttt
Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, and Aymen Shabou. Pact: Pruning and clustering-based token reduction for faster visual language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2025.
Yihe Dong, bean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Proceedings of the 38th International Conference on Machine Learning. ICML, volume 139 of Proceedings of Machine Learning Re- search, pp. 2092277922880 2803. PMLR, 2021.
Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k- nearest neighbors and principal component analysis. Knowl. Based Syst., 99:135-145 145, 2016. doi: 201 6. 02. 001
URL http::::/doo..rrg//// 10t.///1...1111111 knosys.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms in video analysis. CoRR, abs/2405.21075. 2024. doi: 1044550/ARRXI.224552220 48550/arxiv. 2405. 21075
URL https: //oo/.//ooi... org/1C.
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference of Computer Visi 2024.
11

---

Published as a conference paper at ICLR 2026
Jie Huang, Xuejing Liu, Sibo Song, Ruibing Hou, Hong Chang, Junyang Lin, and Shuai Bai. Revis- iting multimodal positional encoding in Dision-language models. CoRR, abw/2510,23095, 2025. doi: 1044555/ARRXI.2550022099 23095
/RL/ttps: httts:://ooii w2gg10.44550/rr211...
Mohit lyyer, Wen-tau Yih, and Ming- Wei Chang. Search- neural structured learning for quential question answering. In Annual Meeting of the Association for Computational Linguistics, ACI., 2017.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2024.
Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, and Marinka Zitmik. Token reduction should go beyond efficiency in generative models from vision, language to multimodality. CoRR, abs/2505, (esp22555.1227,,. 2025. doi: 10.48550/ARXIV. 0tt.://..rr/..../0/////11 00tg:/...555////222050......
1822.
Yuwon
Lee.
Qwen2-vl-fneetuee 2024.
URL
https://githuu: com/U017
Qwen2 VL- Finnttee
Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of vision-language learning with a sin- gle transformer. CoRR, abs/2504.10465 2025. URL httepa:/aarxiv.oorg/aa///// 10462
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326,, 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal iims with generative comprehension. CoRR, abs/2307.16112. 2023a. doi: 1048550/ARXXV.2277.6222 URL
16125
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. CoRR, abs/2305.06335 2023b. doi: 10. 48550/ARXIV.2305.0655 06355. https: wttp://oo.oorg...oo 48550 arxiv. 2305 06355
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2024b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui. Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Annual Conference on Neural Information Processing. NeurIPS, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In European Conference of Computer Vision,ECCV, 2024a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of OCR in large multimodal models. Sei. China Inf. Sci., 67(12), 2024b. doi: 10. 1007/11 4235-24 URL.
lnttps://dot
wrg/10...0.0................. 1007 10:007/11130 11012114--110 024 4235-6
Xu Ma, Yuqian Zhou, Huan Wang, Can Qin, Bin Sun, Chang Liu, and Yun Fu. Image as set of points. In International Conference on Learning Representations, ICLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqu: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Com- putational Linguistics: ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 2263-2279. Associa- tion for Computational Linguistics, 2022. doi: 10. (6653////022FFIIIIIGG. ht
URL
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for VQA on doc- ument images, In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pp. 2199-2200 IEEE, 2021. doi: 10. 10.1109/WACV4660 00225. URL Utthhtts::///// /oi. org 10. 109
TT2.:///WAAAM Mt////CC4483333 2021, 00225
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR, 2019.
Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye. Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2024.
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Annual Conference on Neural Information Processing, NeurIPS, 2017.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. CoRR, abs/2409. 121911 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas M. Breuel, Jan Kautz, and Xi- aolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Ji- aya Jia. Visionzip: Longer better but not necessary vision language models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pp. 19792-19802. Computer Vision Foundation / IEEE, 2025. doi: 2.110000VPP5527442225550000 http//aaangviiiiiinaiaaaaa VisionZip Longer_ Necessary in Vision Language _VPRaagee__eeeeeee CVPR 2025 pwppr.htp...
wwwcch.ecccccc com/ URL Longer_is_Bettee__bttt
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67-78, 2014. doi: 10. 10.1162/TACL) A
( 00166. URL /ttpps://doi.
org/10..11 tac 00166
Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying SAM2 with llava for dense grounded understanding of images and videos. CoRR, abs/2501. 2025. doi: 10.48550/RRXV.2501. 2501. 00000.0040011 04001
04001. URL htth:tt::: / doi. //5p:///00o.l////4/5///////////0
48550
Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: cumman-centrin visual analysis via token clustering transformer. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
13

---

Published as a conference paper at ICLR 2026
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction perruction-tuned visual language model for video understanding. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2023.
Tao Zhang, Xiangtai Li, Zilong Huang, Yanwei Li, Weixian Lei, Xueqing Deng, Shihao Chen, Shunping Ji, and Jiashi Feng. Pixel-sail: Single transformer for pixel-grounded understanding. CoRR, abs/2504. 10465, 2025. URL battps://arxi..oo wwtpa/wwwww.g..///////a///////// 10465
Yuan Zhang, Chun-Kai Fan, Jumpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A. Gudovskiy. Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Sparsevim: Visual token sparsification for efficient vision-language model inference. CoRR, abs/2410.04417,
2024a. doi: 10.45550ARRXV.2410004411 24100.4411
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. CoRR, abs/2410.02713, 2024b.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ANALYSIS OF DIFFERENT REDUCTION RATIOS
We conduct experiments under different reduction ratios, as shown in Table Overall, we observe that moderate compression tends to preserve the most useful positional information while eliminat- ing redundancy. In particular, reduction ratio of 55% consistently yields the best results across most benchmarks. When the reduction is aggressive (e.g.. 90%), the performance drops sig- nificantly due to the loss of critical visual content. On the other hand, smaller reductions do not fully exploit the potential for compression and efficiency. Based on these observations, we adopt default clustering ratio of 0.45 (i.c., 55% reduction) in all our experiments, balancing efficiency and effectiveness.
Table 7: Ablation study comparing different reduction ratios.
VideoMME (wlo subs)
VideoMME subs)
NeXT-QA (MC)
NeXT-QA
SEED-Bench
Reduction Ratio
MVBench
Average
(OE)
-Video
%
58.41
58.22
78.53
29.37
56.77
68.03
58.22
40%
57.30
57.52
78.15
30.99
56.59
67.40
57.99
55 %
58,70
59.07
78.42
32.61
55.98
67.38
58.69
70%
57.74
57.78
78.61
32.17
55.90
67.38
58.26
90%
57.41
57.70
77.98
31.06
55.36
66.65
57.69
A.2 ANALYSIS OF AGGREGATION STAGES
We conduct experiments to compare two aggregation strategies: a three-stage setting (with spatial clustering ratios 0.25/0.5/0.5) and single-stage setting (with clustering ratio of 0.45). As shown in Table S] the single-stage approach consistently outperforms the three-stage counterpart under comparable reduction ratios. Based this observation, adopt single-stage spatial clustering with a ratio of 0.45 as the default configuration throughout our experiments.
Table 8: Ablation study comparing different aggregation stages.
Aggregation Stages
VideoMME (wlo subs)
VideoMME
NeXT-QA (MC)
NeXT-QA
SEED-Bench -Video
Reduction Ratio
MVBench
Average
(w subs)
(OE)
Three-Stage
$7.30
$7.04 59.07
78.14 78.42
32.33
$6.06
66.95
57.97
57%
Single-Stage
58.70
32.61
55.98
67.38
58.69
A.3 ANALYSIS BACKBONE GENERALIZATION
To evaluate the generalizability of PPE across backbones, we additionally test it on the LLaVA- OV-0.5B model. As shown in Table the Dense baseline reaches 44.8%, while PPE achieves 44.74% with 96% token reduction, showing strong generalization. PPE also outperforms Chat- UniVi (43.07%), especially in preserving spatiotemporal information.
Table 9: Ablation study on backbone generalization, conducted on the VideoMME (w/o subs) bench- mark using the LLaVA LLaVA-OV-0.5B (SI) model.
Method
Short
Medium
Long
Average
Reduction Ratio
Dense
54.70
43.20
36.20
44.80
0%
Chat-UniVi
53.00
40.40
35.80
43.07
96%
PPE (Ours)
56.40 (+3.40)
42.10 (+7.70)
35.70 (-0.10)
44.74 (+1.67)
96%
A.4 ANALYSIS OF MODEL. SIZE
Our method is decoupled from model size. To further demonstrate that PPE scales with model size, additionally evaluate PPE V0wen2.5-VL-7B-Inntte results are shown in Table TO showing
15

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study on model size using Qwen2.5-VL-7B-Instrue
Setting
Benchmarks
Dense
Chat-UniVi
Chat-UniVi+PPE
VisionZip
VisionZip+PPE
Training- Free
MMBench (EN)
83.50
83.23
83.58 (+0.35)
85.24
85.19 (-0.05)
MMBench (CN)
83.40
80.18
82.35 (+2.17)
83.62
83,92 (+0.30)
TextVQA
84.90
35.82
63.44 (+27.42)
84.64
85.02 (+0.40)
Doc
95.70
27.06
66.42 (+39.36)
87.30
88.52 (+1.22)
OCRBench
864
479
577 (+98)
780
794 (+144
ChartQA
87.30
19.92
46.72 (+29.90)
60.56
59.88 (-0.68)
MMBench (EN)
86.90
86.23
86.26 (+0.03)
86.14
87.24 (40.16)
MMBench (CN)
85.35
84.25
84.85 (+0.60)
85.12
85.15 (0.03)
SET
TextVQA
87.20
54.92
82.46 (+27.54)
87.08
87.24 (+0.16)
Doc
92.97
50.01
85,84 (+35.83)
93.16
93.57 (+0.41)
OCRBench
826
584
764 (+180)
828
828 (+0)
ChartQA
86.32
43.96
78.88 (+34.92)
85.40
86.08 (+0.68)
Reduction Ratio
0%
55%
55%
55%
55%
that PPE provides improvements regardless model capacity. For the SFT setting, we trained on only 1/5 of the data used for the 3B model due to time and resource constraints.
We also note that in the training-free setting of VisionZip, PPE shows a slight performance drop on ChartQA for the 7B model, while improving the 3B model. This may be because the stronger 7B baseline already leverages fine-graing aineggrained local patterns for such task, and without SFT, the additional positional details introduced by PPE can slightly perturb the attention distribution. After SFT, PPE consistently brings positive gains. For instance, the slight drop on VisionZip+PPE for ChartQA becomes improvement, indicating that PPE benefits more when SFT is applied.
A.5 ANALYSIS OF TASK GENERALIZATION
PPE is inherently task-agnostic and be seamlessly integrated into any clustering-based token merging method, regardless of downstream task. While our main experiments focus on MLLM QA tasks largely due their popularity and the convenience of evaluation we emphasize that PPE is not limited to QA scenarios.
To further demonstrate its generalization capability, we conducted additional experiments on the image captioning task using Flickr30k Young et al. (2014) benchmark. We reused the same models from T and directly evaluated their zero-shot captioning performance without any further fine- tuning. The results are in Table []]
Bleu_1
Bleu.2
Bleu.3
Bleu.4
METEOR
ROUGELL
SPICE
CIDE:
Dense
0.311
0.188
0.120
0.079
0.147
0.322
0,219
0.722
Chat-UniVi
0,280
0.167
0.104
0.068
0.137
0,311
0,214
0.649
PPE:
0.313 (40.033)
0.185 (+0.018)
0.117 (-0.013)
0.076 (40.008)
0.144 (40.007)
0.312 (40.001)
0.211 (-0:003)
0.690 (+0.941)
Table 11: Ablation study on task generalization beyond QA.
From the table above, expected, Dense achieves the best overall performance. Chat-UniVi, with- PPE, suffers noticeable performance drop. For instance, CIDEr decreases from 0.722 0.649, reflecting reduced content relevance. BLEU scores also decline, suggesting degraded coherence due to spatial misalignment caused by compression. SPICE shows a slight decrease (0.219 - 0.214), indicating minor semantic loss. With PPE, performance is largely restored across most metries: BLEU-1/4 improve from 0.280/0.068 to 0.313/0.076. surpassing Dense in BLEU-I, and CIDEr rises from 0.649 0.690, narrowing the gap with Dense. Additionally, qualitative cases further illustrate PPE's effectiveness, as presented in Appendix A.6
Both quantitative and qualitative results confirm PPE's ability to recover structural and positional cues often lost during naive token merging. While it does not fully close the gap with Dense, PPE consistently mitigates performance degradation.
16

---

Published as a conference paper at ICLR 2026
A.6 QUALITATIVE CASES ON IMAGE CAPTIONING
We randomly select two representative cases from Flickr30k, as shown in Figure
(a) Case 1: 10287332, roof scene
(b) Case 2: 900114335, marathon scene
Figure 4: Qualitative examples for the image captioning task, from Flickr30k benchmark.
Case 1 (10287332.jpg). The task prompt is: "Provide one-sentence caption for the provided image. Do not provide any explanation." The ground truth caption is: "Two men sitting on the roof stands ladder." The Dense model predicts: "Three men work
of house while another a roof with a blue sky in the background." Under compression, Chat-UniVi simplifies it to: "Three roof," omitting background and clothing cues. In contrast, PPE preserves richer
men work details: "Three men working on roof with one wearing an orange shirt," showing stronger spatial recall under compression.
Case (9001 44366..43...). The ground truth caption is: "Marathon runners are racing on a city street, with other people standing around." The Dense model predicts: "A woman with number 1397 is running in race." " Chat-UniVi predicts: "A woman with number 1947 on her shirt is running in race." PPE outputs: "A woman with the number 247 on her shirt is running in a race." All models focus primarily on the runners' numbers. However, there are multiple runners, including men and women, making the predictions less accurate. In the foreground, two women runners are the main focus of the image. The number of the front runner is only partially visible as "27", while the runner behind wears "1597". Consequently, the Dense model appears to focus on the runner behind and all models misrecognize the number.
A.7 ATTENTION VISUALIZATION ANALYSIS
Using the same models from Table I we extract the attention maps of the final LLM layer (layer_id=35) between all text and visual tokens and project them back onto the original images as heatmaps, revealing each question's 's focus across image regions. Both Chat-UniVi and PPE are under reduction ratio of 55%. We randomly selected some cases demonstrate our PPE can preserve valid positional information and focus more on the key region of images.
Official QA index: 1505. Question: "How many types of fruits are there in the image? Please give the answer directly. A. 3 B. 2 C. 5 D. 4 E. None." The image contains five fruit types: a central banana, surrounded by an apple, avocado, pear, and a partially visible orange in the background. The correct answer is C. PPE predicts correctly, while Chat-UniVi outputs D.
Case MMBench (EN), (a-c) in Figure 5
Attention heatmaps show that Chat-UniVi partly focuses the background and fails to capture the orange, likely leading to the incorrect prediction. In contrast, PPE successfully attends to all fruit regions, including the occluded orange, demonstrating better spatial alignment under token compression and improved grounding for object counting in complex scenes.
jpg. ques- tion ID: 37890. Question: "What number is visible on the shorts in front?" " The image depicts three football players on grassy field. The player in the foreground, jumping mid-air, wears shorts with
Case 2. TextVQA, (d-f) in Figure A.7. Official image name: 2.6277a445044
17

---

Published as a conference paper at ICLR 2026
(a) Original
(b) Chat-UniVe
(c) PPE
gen
(d) Original
(e) Chat-UniVi
(I) PPE
(g) Original
(h) Chat-UniVi
(i) PPE
Figure 5: Qualitative comparison of attention visualizations. Each row corresponds different
sample: (a, d, g) original input, (b, e, h) Chat-UniVi outputs, and (c, f, i) PPE outputs.
18

---

Published as a conference paper at ICLR 2026
the number "2". Behind him, another player shows partially visible "14", of which only the "I" is clear. The correct answer is 2. Both Chat-UniVi Vi and PPE incorrectly output 1.
Heatmaps indicate that models attend to the "2" the foreground player, the partial "I" from the second player, and even the text "gen" on an advertising board. PPE places more emphasis on the "2", while Chat-UniV is more distracted by background text. Although both fail in this case, the richer attention of PPE may sometimes introduce ambiguity, but also reflects broader contextual awareness.
Official image name: 23183bf2dbs 6d88c. jpg, ques- tion ID: 37376. Question: "Whaf kind of drink is this?" The image shows orange beverage bottle labeled with the brand maaza. The ground-truth answer is maaza. Both the Dense model and Chat-UniVi predict correctly, while PPE outputs "orange juice".
Case 3. TextVQA, (g-i) in Figure A.7
Visualizations reveal that PPE's attention covers both the brand logo and the broader bottle region. Although its answer deviates from the ground truth, it remains semantically reasonable, suggesting that PPE's broader positional retention can capture additional contextual cues beyond the exact label.
A.8 LIMITATIONS AND FAILURE CASES
gipuzkoa zkoa
manza
deba A REC
(a) Case
(b) Case
Figure 6: Representative failure cases of PPE, from TextVQA benchmark.
As shown in Tables and 7 PPE underperforms Dense in certain cases, highlighting the trade- off between compression and fidelity. We visualize representative failure to illustrate PPE's limitations.
Using the same models from (0be2.[[Owe22.55-VLL-33
we extract attention maps from the
final LLM layer (layer id=35) between text and visual tokens and visualize them as heatmaps. Under identical compression ratios (55% and 90%), we analyze two failure examples from the TextVQA benchmark:
Case (55% compression). Official image: 23183bf2dbl 23188br22bb668888.jjpe Question: "What kind of drink is this?" (question ID: 37376). The image shows orange beverage labeled "maaza". Ground truth: "maaza"" Dense and Chat-UniVi predict correctly, while PPE outputs "orange juice". Heatmaps indicate that PPE attends to both the logo and the bottle, producing a semantically relevant but less precise answer. This suggests that PPE's positional retention captures broader context, though at the cost of exactness.
Case 2 (90% compression). Official image: 0b9a499c78985373.jppp Question: "What word is written the collar of the jacket?" (question ID: 36877). The image contains multiple words: "kutxa" " (collar), "menabi" (chest), "gipuzkoa", and partial "deba". Ground truth: "kutxa". Dense predicts correctly; Chat-UniVi outputs "menabi"; PPE outputs "deb?". Heatmaps show that both
19

---

Published as a conference paper at ICLR 2026
PPE Gains and Recovery Resolution with Numbers
25-
fecsery Deme prage Sunbers
3300
Imagee Numbers
35
200g
1000
365
643"2
Remsttution zwtee"))
Figure 7: PPE obtains consistent improvements across different resolutions.
Chat-UniVi and PPE attend to larger central regions, missing the smaller collar text. This demon- strates limitation of PPE under aggressive compression, where maintaining spatial precision be- comes challenging.
A.9 PPE GAINS ACROSS DIFFERENT IMAGE RESOLUTIONS
To evaluate the effect of PPE under varying image resolutions, calculated the average resolution of TeXIVQA, MMBench, VOA, OCRBench, and ChartQA, based on the resolution statistics of their samples. As illustrated in Figure 7 the horizontal axis denotes the average image resolution for each dataset, while the left vertical axis presents the performance difference (A). This metric reflects both the improvements introduced by PPE over Chat- Uniii and its ability to recover performance relative to the dense model. For OCRBench, whose raw score range is not from 0 to 100, scores are normalized by factor of 10 to ensure consistent visualization. The right vertical axis represents the number of image samples in each dataset.
From the figure, we observe that PPE consistently provides notable performance gains across a wide spectrum of image resolutions: from 385" to 1924 These results demonstrate that PPE imposes no limitations on input resolution and remains robust and effective across images of different scales.
A.10 VISUALIZATION OF THE EFFECT OF K
To better understand how different values of K influence attention behavior, we conduct visual analysis and support it with guantitative indicators such as entropy and variance. Following the analytical framework introduced Section 4.4.I and Figure we compare different choices of K directly on the images.
As shown in Figure when K 1, entropy is slightly higher and variance is lower, indicating that the model's attention becomes more dispersed compared with K 16 or K 32. The visualizations further reveal that K text (with brighter activations in red). Meanwhile, tokens on the camera backplate also become more activated, showing that attention is distracted by background regions and fails to capture the full brand string, unlike the cases with K 16 K 32. For images, set K 32 based on the GCD of M-RoPE configuration [32, 32), which maximizes the preservation of positional information per merged token. The same rationale applies to video settings, K 8 of [16, 24, 24].
the model mainly focuses on the first half of the brand
M-RoPE splits embedding dimensions into multiple periodic components (temporal, vertical, hori- zontal), assigning each axis contiguous frequency bands. Existing analysis (c.g.. Figure 3 and 4 in Reference Huang et al. (2025) shows that uneven allocation can lead to some axes occupying only high- low-frequency channels, impairing long-range and spatial modeling. Choosing K based
20

---

Published as a conference paper at ICLR 2026
K16
K32
K16
1.0
0.8
Entroyy
0.6
0.4
0.2
Case 34602
Case 34603
Case 34602
Case 34603
(a) Entropy comparison
(b) Variance comparison f
(c) PPE (K=1)
(d) PPE (K=16)
(e) PPE (K=32)
Figure 8: Attention statistics and visualizations of samples from TextVQA (a-b) Quantitative comparison of entropy and variance. (c-e) Qualitative attention score visualizations of case 34602.
the GCD ensures that each merged token preserves multiple positional anchors evenly across the frequency spectrum, preventing collapse into only high- or low-frequency bands. In short, K is grounded in M-RoPE's frequency structure, providing theoretical justification for PPE's ability to preserve positional information after token merging.
In short, refer to the analysis of positional encoding VLMs Huang et al. (2025). choosing an ap- propriate K ensures that positional indices distributed evenly across frequency bands; otherwise, some indices may collapse into only high- or www-freuency ranges, degrading positional fidelity.
21


Reviews:
[{"review_id": "q9VmBs1FXd", "reviewer": "Reviewer_gfHj", "summary": "This paper proposes PPE to enhance visual token compression in vision-language model. Instead of assigning a single positional ID to a merged token, PPE encodes multiple positional cues by splitting embedding dimensions and preserving RoPE structure. The method enables high-ratio token compression and supports cascade compression across transformer layers.\n\nExperiments show consistent improvements over clustering-based compression (e.g., Chat-UniVi), especially under aggressive reduction settings. PPE is simple, model-agnostic though current results are limited to clustering-based methods and evaluation breadth.", "questions": "1. PPE relies on RoPE’s dimension-wise rotational independence. How would this method generalize to non-RoPE architectures (e.g., ALiBi, learned absolute embeddings)?\n\n2. Experiments focus on clustering-based token merging. Have you evaluated PPE with learned token selection/aggregation methods?\n\n3. Have you explored other position grouping strategies?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Motivation is clear: positional degradation under token compression.\n\n2. Good solution: leveraging RoPE dimension independence.\n\n3. Strong efficiency gains with competitive accuracy at high compression ratios.", "weaknesses": "1. Evaluated mainly with clustering (DPC-KNN); no results with learning-based compression.\n\n2. Unclear applicability to ALiBi or other positional schemes.\n\n3. More evaluation is needed, in the paper, layout-heavy or OCR-centric tasks partially explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LP5VXHK686", "reviewer": "Reviewer_aRNh", "summary": "This paper addresses the limitations of existing visual token compression methods in MLLMs that disrupt spatial layouts and temporal continuity. The authors propose Positional Preservation Embedding (PPE), a parameter-free positional encoding operator that explicitly retains multiple spatiotemporal positions within each compressed token by splitting RoPE embedding dimensions into K groups, each encoding different position IDs from merged tokens. The method integrates into existing token merging frameworks and supports cascade compression across transformer layers. Experiments on Qwen2.5-VL-3B demonstrate 2-5% improvements across multiple benchmarks (MMBench, TextVQA, VideoMME) while achieving 55-94% token reduction.", "questions": "1. Insufficient theoretical justification: The choice of K based on GCD of M-RoPE sections is presented as a design principle but functions more as a constraint, and there's no formal analysis of information preservation or a clear definition of what \"77% IDs retained\" means.\n\n2. Unclear Result Presentation and Incorrect Marking: The tables in the paper fail to specify the meaning of bold and underlined markings. Additionally, there is no explanation of whether higher values or lower values of the metrics are better. As a result, the results in many tables are extremely difficult to understand. For instance, in Table 8, the highest result for the \"Short\" metric is bolded, while the highest results for other metrics are underlined.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper clearly identifies that existing token compression methods lose fine-grained positional information, with compelling evidence (Figure 1, attention visualizations) showing how this impacts layout-sensitive tasks.\n\n2. The authors provide thorough ablation studies covering K values, reduction ratios, cascade compression strategies, and include attention visualizations and failure case analyses.\n\n3. PPE supports cascade compression, enabling 90% token reduction while maintaining performance through multi-stage processing.", "weaknesses": "1. Critically incomplete baseline comparisons: The paper lacks comparisons with several mainstream token compression methods like FastV, VisionZip, MustDrop, and TokenCarve[1-4].\n\n2. Unvalidated position correspondence assumption: The core assumption that compressed tokens retain a meaningful correspondence to original image positions after vision encoder processing (ViT layers, pooling) is not validated and is questionable for many architectures. This is particularly relevant for models like InternVL-3 with its “Tiles+Thumbnail” approach or cross-attention models, where position IDs might not directly map to original spatial locations, fundamentally questioning the meaningfulness of \"preserved\" positions for the model.\n\n3. Limited architectural generalization: All primary experiments are confined to Qwen2.5-VL-3B and LLaVA-OV-0.5B. On LLaVA-OV-0.5B, experiments show PPE underperforms the Dense baseline (44.74% vs 44.80%). There is no evaluation on other mainstream MLLMs, or larger models (7B+), making PPE’s general applicability uncertain.\n\n4. Confusing experimental setup: The authors claim PPE to be “plug-and-play”, but conduct SFT to obtain a dense fine-tuned model. The paper lacks the validation of PPE on the official pre-trained Qwen2.5-VL-3B model and may compromise the reproducibility and fairness of the experimental comparisons.\n\n[1] An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models\n\n[2] Visionzip: Longer is better but not necessary in vision language models\n\n[3] Multi-stage vision token dropping: Towards efficient multimodal large language model\n\n[4] Tokencarve: Information-preserving visual token compression in multimodal large language models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ny36xNK5On", "reviewer": "Reviewer_vydH", "summary": "This paper introduces Position Preservation Embeddings (PPE), a novel position encoding mechanism designed to preserve spatio-temporal structure during visual token compression in multimodal large language models (MLLMs). PPE explicitly encodes multiple position identifiers into a single compressed token by partitioning the embedding dimension into groups of independent storage spaces. This enables the merged token to simultaneously represent multiple original positions. This method requires no parameter tuning, is plug-and-play, and is compatible with existing token merging algorithms like Chat-UniVi and  PACT. Across various visual-language benchmarks, PPE achieves 2–5% performance gains over state-of-the-art methods, maintaining advantages even under extreme compression rates.", "questions": "Refer to Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novel Concept**: PPE preserves the spatio-temporal integrity of visual markers by encoding multiple position identifiers within a single compressed marker. Even under high compression ratios, it significantly reduces computational and memory overhead while maintaining performance.\n2. **Parameter-free Compatibility**: The PPE strategy requires no additional training parameters or architectural modifications and can be seamlessly integrated into existing large-scale language models and token compression pipelines. Furthermore, PPE demonstrates strong adaptability and generalization capabilities across architectures and foundational models, achieving compatibility with various clustering compression frameworks such as Chat-UniVi, PACT, and ToMe.\n3. **Cascade Compression**: This paper systematically extends PPE to multi-level cascaded compression within large language models, demonstrating that the progressive merging technique maintains high accuracy even at token compression rates exceeding 90%. Furthermore, experiments reveal that PPE achieves stable quantization gains while preserving overall computational efficiency, offering significant practical value for large-scale multimodal inference.\n4. **Comprehensive Evaluation**: Across extensive assessments of image and video tasks, PPE demonstrates remarkable effectiveness in preserving accuracy while maintaining inference efficiency. This paper designs rich ablation experiments including K-value analysis, rate trade-offs, and attention visualization, which robustly validate the design decisions underlying PPE's token compression process.", "weaknesses": "1. **Benchmark Coverage for Layout-Sensitive Tasks**: The current evaluation primarily targets general QA-type tasks (e.g., MMBench, VideoMME). While TextVQA is included, the paper lacks comprehensive validation on benchmarks specifically designed for fine-grained spatial and layout understanding, such as OCR-intensive tasks. It is recommended to incorporate dedicated OCR benchmarks (e.g., OCRBench and DocVQA) to rigorously validate PPE's capability in preserving precise spatial relationships for text-heavy scenarios, which is a core claimed advantage of the method.\n2. **Ablation Analysis**: Table 5 presents results for the number of retained positions K = 1, 8, and 24. However, it does not discuss the more granular effects of K on computational complexity or changes in attention distribution. Furthermore, the choice of K lacks universality across different tasks. While the attention visualization in ablation experiments demonstrates qualitative improvements for PPE, a more nuanced interpretive discussion could be developed by integrating spatial contextual relevance.\n3. **Related Work Could Be More Comprehensive**: The discussion of visual token merging in Section 2.2 would benefit from incorporating several recent and highly relevant works (e.g., Shang et al. 2025 - Llava-prumerge; Li et al. 2024 – TokenPacker; Yuhang Han et al. 2024 - Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration; Mark Endo et al. 2024 - Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration; and Chen et al. 2024 - Recoverable compression: A multimodal vision token recovery mechanism guided by text information). Engaging with these methods would help to better position PPE's contributions within the current landscape.\n4. **Lack of Detailed Efficiency Analysis**: The paper would be significantly strengthened by a more thorough analysis of computational and memory efficiency. While token reduction inherently implies speedup, quantitative evidence is currently lacking. We recommend providing comprehensive comparisons on key metrics including computational cost (e.g., FLOPs reduction in LLM self-attention modules), memory footprint (peak GPU memory usage during inference), and inference latency (end-to-end inference time or throughput). A comparative analysis against other token reduction methods on these metrics would provide a more complete picture of PPE's practical advantages.\n\nMy overall stance on this paper is weak accept. If the authors could provide additional details or point out areas I may have overlooked, I would be willing to further increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWXvcGmnvq", "reviewer": "Reviewer_jviM", "summary": "The paper proposes Positional Preservation Embedding, a positional encoding scheme for multimodal LLMs that compress visual tokens without destroying spatial or temporal structure. Standard token merging reduces sequence length by clustering similar visual tokens, but it typically drops most positional IDs, so the merged tokens lose fine-grained layout and temporal continuity. PPE instead packs multiple positional IDs (2D for images, 3D for video) into different embedding sub-dimensions of each merged token, so a single compressed token keeps location/ordering information from several original tokens. PPE is parameter-free, attaches to existing token merging, and supports cascade compression across multiple transformer layers. The authors evaluate PPE on Qwen2.5-VL-3B (fine-tuned) for both image and video understanding.", "questions": "1. How does PPE compare quantitatively to other popular token reduction approaches such as ToMe, FastV, SparseVLM, PyramidDrop, and VisionZip?\n2. Can the authors expand the evaluation to include more common benchmarks?\n3. Does PPE maintain its claimed efficiency gains when applied to larger MLLMs or higher-resolution inputs beyond the 3B model setting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Problem formulation is clear and technical idea is simple and clean, reuse RoPE’s per-dimension independence to “multiplex” multiple spatial / temporal position IDs into one merged token. \n2. Plug-and-play, no new trainable parameters, negligible runtime overhead.\n3. Ablations are thoughtful, like where to insert compression, compatibility with other methods, compute/memory cost.", "weaknesses": "My main concern is the empirical evaluation. The current experiments are not sufficient. The method is not compared against several widely used token reduction / compression baselines such as ToMe, FastV, SparseVLM, PyramidDrop, VisionZip, etc. The set of baselines is very limited, which makes it hard to judge relative progress. Also, the set of benchmarks is also limited, many common benchmarks are not included. Please address these points in the rebuttal. I am willing to raise my score if the empirical evaluation is strengthened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)