Sparsity Forcing: Reinforcing Token Sparsity of MLLMs
Authors: Feng Chen
Yefei He
Lequan Lin
Jing Liu
Chenhui Gou
Bohan Zhuang
Qi Wu
Venue: ICLR 2026 Poster (2026)
Keywords: reinforced sparse attention
token sparsity
Primary area: foundation or frontier models, including LLMs
TL;DR: Sparsity Forcing is an inference-aligned post-training method that optimizes a joint efficiency–performance reward with multi-budget Top-p exploration via GRPO.

Abstract: Sparse attention mechanisms aim to reduce computational overhead with minimal accuracy loss by selectively processing salient tokens. Despite their effectiveness, most methods merely exploit a model’s inherent sparsity and thus plateau at moderate budgets (about 50\% token reduction), with little headroom to push budget lower without hurting accuracy. 
Other approaches attempt to enforce sparsity through trainable sparse attention or sharpness-inducing regularizers, but these either fix rigid patterns that ignore input and layer dynamics, or optimize proxy objectives without direct control over token budgets.
In this paper, we explicitly reinforce token sparsity in well-posed multimodal large language models (MLLMs) through a simple RL-based post-training framework named  $\textit{Sparsity Forcing}$. 
Our method explores the efficiency-accuracy trade-off by running multiple rollouts with different token budgets, where both efficiency (token reduction ratio) and performance (answer correctness) are formulated as joint rewards.
By contrasting rollouts within each group, the more efficient and correct answer is rewarded while less efficient or incorrect ones are penalized, thereby turning token saving into an end-to-end, inference-consistent optimization objective.
Across thirteen image and video benchmarks, Sparsity Forcing raises token reduction ratio on Qwen2-VL/Qwen2.5-VL from 20\% to 75\% with minimal accuracy decline, 
significantly reducing long-context inference memory by up to 3$\times$  while speeding up decoding by up to 3.3$\times$.

Published as a conference paper at ICLR 2026
SPARSITY FORCING: REINFORCING TOKEN SPARSITY OF MLLMs
Feng Chen't Yefei He2
Lequan Lin
Chenhui Gou
Jing Liu
Bohan Zhuang":
Qi Wu
AIML, University of Adelaide, Australia University of Sydney, Australia
2 ZIP Lab, Zhejiang University, China
Monash University, Australia
ABSTRACT
Sparse attention mechanisms aim to reduce computational overhead with minimal accuracy loss by selectively processing salient tokens. Despite their effectiveness, most methods merely exploit model's inherent sparsity and thus plateau moderate budgets (about 50% token reduction), with little headroom to push budget lower without hurting accuracy. Other approaches attempt to enforce sparsity through trainable sparse attention or sharpness-inducing regularizers, but these either fix rigid patterns that ignore input and layer dynamics, or optimize proxy objectives without direct control over token budgets. In this paper, we explicitly reinforce token sparsity well-posed multimodal large language models (MLLMs) through simple RL-based post-training framework named Sparsity Forcing. Our method explores the efficiency- accuracy trade-off by running multiple rollouts with different token budgets, where both efficiency (token reduction ratio) and performance (answer correctness) are formulated as joint rewards. By contrasting rollouts within each group, the more efficient and correct answers are rewarded while less efficient or incorrect ones are penalized, thereby turming token saving into an end-to-end, anference-consisten optimization objective. Across thirteen image and video benchmarks, Sparsity Forcing raises token reduction ratio on Qwen2- VI /1 VI. from 20% 75% with minimal accuracy decline, significantly reducing long-context inference memory by up to 3 while speeding up decoding by up to 3.3x.
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Chen et al., 2024c; Li et al., 2024a; Yang et al., 2024a) have achieved impressive results in tasks such as image captioning and visual question answering (Fu et al., 2023; Li et al., 2024b). However, when processing high-resolution images or long videos, the visual encoder often produces an excessive number of visual tokens, severely constraining the generative efficiency of MLLMs (Chen et al., 2024a). Existing sparse attention methods (Chen et al., 2024a; He et al., 2024a; Jiang et al., 2024) exploit the sparsity of attention maps to prune redundant tokens, thereby reducing memory and latency. The core idea is that many tokens receive negligible attention and can be safely discarded, as in FastV (Chen et al., 2024a), which drops half of the visual tokens after the second layer with little accuracy loss. However, further budget reduction-for instance, to 20% or even 10% of the total trmansss challenging, since these approaches merely leverage naturally emergent sparsity of MLLMs rather than actively enforcing it, which fundamentally affects the achievable inference-time acceleration.
Early attempts enhance token sparsity typically rely on either new attention architectures or sharpness-inducing regularizers on attention maps. For instance, MOBA (Lu et al., 2025) and NSA (Yuan et al., 2025) enforce semi-structured sparsity through trainable sparse attention in LLMs. However, such approaches predefine rigid sparsity patterns and ignore the dynamics across inputs, layers, and training stages. They also require training from scratch, making them less practical when
1 These authors contributed equally. ' 'Correponding author. Email: bohan, zhuangggmai ccom

---

Published as a conference paper at ICLR 2026
adapted to MLLMs in post- setting. Alternatively, works such as (Gong et al., 2024; Peruzzo et al., 2024; Araabi et al., 2024) introduce structured regularization losses that concentrate attention on semantically relevant regions, thereby reducing token redundaney. Yet these methods optimize proxy objectives like attention sharpness, which does not necessarily translate into consistent end-to-end token savings. Crucially, both streams of work are implemented in the SFT regime under teacher forcing, where sparsity is enforced on ground-truth tokens rather than generated outputs. This creates a mismatch with inference and often limits the realized end-to-end efficiency gains.
In this paper, we propose a reinforcement learning-based post-training method, Sparsity Forcing that directly optimizes the cerformannc--ffficiinn trade-off of MLLMs via the Group Relative Policy Optimization (GRPO) (Shao et al., 2024). For each dision-languaa query, we optimize a joint reward that increases with final-answer correctness and with token reduction ratio, turning token saving into end-to-end objective rather than proxy. Specifically, as shown in Figure 1, we treat an MILMM with sparse attention (e.g.. Qwen2-VL++Zp'> VL+ZipVL. (Wang et al., 2024; He et al., 2024a)) the policy model and the original MLLM with standard causal attention the reference model; this anchoring stabilizes learning and preserves task fidelity, limiting accuracy loss even at high sparsity. To discover the smallest budget that preserves correctness, execute multiple rollouts under adaptive token budgets to generate answers a progressive budget sweep that dynamically tests whether low-salience tokens are actually needed for answer correctness. For each rollout, we compute the joint
reward, then contrast rewards within the group to define the advantages: rollouts that are both correct and more efficient tend to receive positive advantages, while less efficient or incorrect rollouts receive lower or negative advantages; then the policy is updated accordingly. Over training, this update progressively concentrates computation on the most informative tokens. Compared with previous SFT-based methods, our RL-based method applies an dnterece-allinee token pruning policy and KV-cache management for deployment, thereby yielding real end-to-end savings on MLLMs with minimal performance decline.
Our contributions can be summarized as follows:
(1) We propose novel post-training framework named Sparsity Forcing that explicitly promotes token sparsity in MILMs, aiming to enhance their achievable efficiency during inference.
(2) We cast efficiency-performance as an explicit joint reward rather than proxy, producing deployment-aligned sparsity while requiring neither architecture changes nor training from scratch.
(3) Experimental results demonstrate that Sparsity Forcing can enhance the sparsity of Qwen2/2.5-VI. from 20% to 75% with a minimal performance decline on thirteen benchmarks, significantly reducing long-context inference memory by up to 3x while speeding up decoding by up to x.
RELATED WORK
Sparse attention of MILMs. Sparse attention is designed to alleviate the computational constraints during deployment, which arise from the extended visual sequence. A line of work achieves it through token selection. ZipVL (He et al., 2024a) proposed a dynamic ratio allocation strategy of important tokens, which is adaptively determined based the layer-specific distribution of attention scores. VisionZip (Y (Yang et al., 2024b) is a text-agnostic visual token compressor that selects dominant tokens by attention of the visual encoder and merges the rest by semantic similarity. Sparse (Zhang et al., 2024b) selects relevant text "raters" via attention, adaptively prunes visual tokens with rank-based per-layer ratio, and recycles pruned tokens. SeerAttention (Gao al., 2024) developed a self- diftiistillation training scheme to efficiently train the AttnGate as binary classification of token selection. Another line assigns sparsity patterns to attention heads. For example, Minference (Jiang et al., 2024) and MMinference (I.i et al., 2025) determined the optimal pattern for each attention head and dynamically built sparse indices based on the assigned pattern during inference. However, these methods primarily leverage inherent attention sparsity at inference rather than actively encouraging token sparsity during optimization, which can lead to accuracy degradation under extremely low budgets. In this paper, we propose Sparsity Forcing, a post-training RL-based framework that explicitly encourages token sparsity by formulating token selection and answer correctness as an efficiency-perrormee performance reward-driven decision process that aligns with the actual inference pipeline.
Encouraging token sparsity. Recently, trainable sparse attention methods exploit the sparsity patterns of attention during training, which can also be deemed encouraging token sparsity toward
2

---

Published as a conference paper at ICLR 2026
KL.
Orriut Tokes Rabio
Advintuue
Reforcuce
Wher the
Queai/VI
FL
Policy
Dwea2-11-7ipV
A
Compatation
:
Attentoon Robersiion Tureduuids
Pt
FX
Ax
Model
Figure I: Overview of the proposed Sparsity Forcing. We use an MLLM with sparse attention as policy model, e.g., Owen2-VL+ZipyV and the original model with standard causal attention the reference model. The sampling group is to explore the minimum token ratio required to maintain the current answer under different attention score retention thresholds
predefined budget with end-to-end training. For example, NSA (Yuan et al., 2025) designed dynamic hierarchical sparse strategy, combining coarse-graine grained token compression with fine-grained token selection to preserve both global context awareness and local precision. MOBA (Lu et al., 2025) proposed block-wise attention probing method and reformulated the sparse attention Mixture of Experts. However, these methods typically overlook the dynamics across inputs, layers, and training stages. They also require training from scratch, which makes them less practical and often suboptimal when directly adapted to well-posed MLLMs in post-training setting. Another line of work explicitly sharpens attention by adding entropy -ospmmmziiiiing or norm- based regularizers to the attention map, e.g., ceriimum-entropyy//
penalties for NMT (Zhang et al., 2018) and -patial-ntroppy regularization for ViTs (Peruzzo et al., 2024), which yield sharper, more selective attention distributions. Nevertheless, such sharpening losses optimize proxy objective: they expose no explicit control over the retained- token budget, and the induced sharpness does not reliably translate into end-to-end token savings or stable accuracy when post-training MLLMs. In this paper, we reformulate the objective as a joint efficiency-perfommance performance reward with GRPO and train with multi-budge exploration.
efficiency-peer
3
OUR METHOD
3.1 PRELIMINARIES
Token-level sparse attention. Let Q, E gdd be query, key, and value matrices in the attention mechanism, respectively, where is the sequence length and d denotes the embedding dimension. In token-level sparse attention, the attention computation is restricted exclusively to subset of selected tokens in the query and key matrices. Formally, two binary indicator vectors mQ: m {0, 1) introduced to specify the selection, where the value of indicates a selected token, and O otherwise. For the query and key matrices, the corresponding masks MQ. Mæ Exd constructed
MQ mQ I, M = mx 1_
where 1, E R is vector of ones. The masks then have rows of ones for selected tokens, and zeros elsewhere. Finally, output of the sparse attention mechanism is computed by:
sparse (0
( Vd
(Q) & MK)
V
(2)
where a is the row-wise Softmax function, and denotes the element-wise matrix multiplication.
The token budget of sparse attention is given by b (||Mollo + | (ooloo/ddd where || llo is the zero-norm function that counts the number of nonzero elements. this paper, we adopt top-p sparse attention, mechanism derived from top-p (nucleus) sampling, where the smallest set of tokens whose cumulative attention scores exceeds a threshold p e [0, 1] is retained for subsequent computation. This design enables dynamic adjustment of sparsity across different layers tasks while providing
3

---

Published as a conference paper at ICLR 2026
theoretical upper bound the approximation enror of (1-p) p) V (Lin et al., 2025). Consequently, it supports an online efficiency policy during training with controllable resulting enror. Mathematically, the query and key masks are tailored to solve the following constrained optimization problem:
QK
Mr. M = argmin b, S.L. ΣΣ o
(mQ).(mk); 2exp.
(3)
Vd
Mg.Mx
where the of attention scores equals due the row-wise Softmax function.
Inference pipeline with sparse attention. Given a rision-language question x ~ X, where X' is the joint input space of multimodal data, MILMs first prefill the input with sparse attention and save the pruned KV cache KK @ MR. V @ Mx) into memory for decoding, Ideally, the final answer is expected to be the same as that without using sparse attention. Since the answer is usually short, that is, of < x the final KV cache usage is approximately [/MM/ll/// /d. The token ratio is defined as by for sample with the average proportion of selected tokens per sample, computed as Ej=1 1 E token length l over all transformer layers j = J.....J
3.2 SPARSITY FORCING
Overview. As shown in Figure 1, we use an MLLM (e.g., Qwen2-VL. VL (Wang et al., 2024)) with sparse attention (ZipVL. (He et al., 2024a)) our policy model me, and the same model with frozen parameters and standard causal attention as the reference model Teef- For each question x ~ X, we perform N independent rollouts through the policy model to generate a of distinet responses {or. ) where each response On, is generated with random threshold pr, U(0, 1), for n I,.., N. In addition, we record the token ratios {TI, ... TN. associated with these responses. Then we compute the reward T, of each answer with a specially designed reward function that simultaneously considers the performanne-efficieenn constitutes progressive test of whether low-salience tokens are necessary: correctness changes across p reveal the smallest budget that preserves accuracy. Such a more efficient and correct rollout receive a positive advantage in GRPO, while all other rollouts-eitee even a negative advantage. Finally, we use the corresponding advantages to update the policy model. trade-off GRPO jointly rewards token reduction and answer accuracy. (2) Multi- budget rollouts obtained by sweeping thresholds p enable dynamic exploration of the minimum tokens needed for correctness across layers, inputs, and training dynamics, avoiding rigid, hand-crafted positive/negative labels for RL. (3) The training loop mirrors inference: the same token pruning policy and KV-cache management are applied during training and deployment, yielding predictable, budget-aware behavior across tasks and hardware.
trade-off. As illustrated in Figure 2, sweeping p
incorrect or less efficient-rrceiiee less
Here are three key benefits of Sparsity Forcing: (1) We make the fffiieecc--eeffommnn
an end-to-end objective rather than a
Dynamic sparse attention. Our approach can be in- tegrated with any sparse attention mechanism. this paper, we build up our method on top of VI, which a typical top-p sparse attention. Consider an attention layer with input tokens, where the standard attention score matrix is denoted as A E gixl, The accumulated attention score for each token j is calculated by summing the corresponding column:
P4000
00130
Gamaatte Sore
am
a100
oxes
axes
130
176
0
A..j.
(4)
Toler indox Lastal by atintiseooee
Figure 2: Progressive top-p sampling as low-salience token test. As p increases, additional tail tokens are included; cor- rectness is then evaluated at each p to identify the minimal budget that preserves accuracy.
These accumulated attention scores subsequently sorted in descending order, such that (sorte(j) represents the j-th highest attention score. Then, the number of im- portant tokens b is determined by preserving the majority of attention scores with minimal number of tokens, which can be expressed as: b b= min(p Σ
P Σ
asonted(j) 2 P x 1).
(5)
4

---

Published as a conference paper at ICLR 2026
After determining the number of important tokens b for each layer, normalized attention scores to assess token importance, calculated as follows:
E Aed
a nnz
(6)
Here, anz(A..)) denotes the number of non-zero elements in the j-th column. Important tokens T are then selected using the top-k indexing method, while the remainder are considered less important:
T topk_ index(āj,
(7)
Inference is then carried out using only the important tokens in T for each layer, which participate in subsequent self-attention during both prefilling and decoding, while unimportant tokens are skipped. For implementation, we keep full KV cache and perform dynamic sparse token selection only at decoding time by fetching the corresponding KV entries on the fly (Kw (Kwon et al., 2023).
Reward function. The reward of our method is composed of performance and efficiency rewards. Specifically, the performance reward function Fper assigns binary reward where correct answer assigned to and otherwise 0 (Guo et al., 2025). The efficiency reward Taff given by the token reduction ratio 1-77, 73, where higher value indicates more extensive pruning of redundant tokens, thus leading to better efficiency. However, a naive additive combination of them becomes efficiency-biant when rollout in the group is correct, because the accuracy signal collapses to constant while the efficiency term still pushes budgets downward. To probe the fficiency-accuraan collapsing to trivial ultra-sparse policies, promote efficiency only when the group contains least one correct answer. Concretely, given sampling group {o), ON }, we define a group-level indicator:
landscape without
C' 1 (L....,N) Correct(o,) 1}.
(8)
The per-sample reward is then defined as:
r per, C Teff.is
(9)
Let {ri,.., JN} be the reward the N generated answers, then the advantage of each answer is obtained by normalizing its reward as:
mean (ri, T2,,,,,.... FN)
A std (T1, T2,,,,...,77
(10)
Eventually, the policy update follows the clipped surrogate objective of GRPO (Shao et al., 2024):
[( (min
ne (On
x)
(On, x) TTRas (On x)
x)
[(0)= =E nEM(NN)
x-x
Ai,
Ai
3DDL ( [|
Th
(On
(11)
where 9 denotes all learnable parameters to be updated: ([N)) the discrete uniform distribution from to N; (-(-) and wa. (-) are the probabilities under the updated and previous policies, respec- tively; x(() the cl ip operator that constrains the input within narrow range small-valued parameter to prevent large policy updates; IDxL. (To (TTll [( measures the KI. divergence between the updated and reference policies; and 3 is the weight coefficient of the KL divergence term. The first term facilitates better responses while ensuring stable updates through clipping. In addition, the second term penalizes deviation from reference model, hence minimizing the transfer entropy caused by the sparse attention.
<, + <) with
4 EXPERIMENTAL SETUP
Implementation. We adopt VL-seiess models (Qwen2-VL.) VL-7B (Wang al., 2024), Qwen2.5- VL-3B (Yang et al., 2024a), and (w-n2.5-VL-7BB) and LLaVA-series models (LLaVA-Video- 7B (Zhang et al., 2024c)) as base MLLMs, owing to their strong adaptation recent R R1-based
5

---

Published as a conference paper at ICLR 2026
Table Performance comparisons with training-free sparse attention on image benchmarks. Here, "Ratio" denotes the average proportion of tokens participating in attention computation over all benchmarks.
Method
MMBencheN
MMStar
CharQQA
Tex:VQ
OCRBench
Avg.
Full
100
2305
78.9
57.9
81.3
82.0
807
32.9
70,9
Owen2-VL-7B
ZipVL
79.7
2294
Spansity Forcing
23.6
2308
78.4
58.2
81.2
82.1
807
32.9
70.8
Full
100
2157
78.8
55.5
83.4
78.7
784
31,5
69.1
75.3
52.1
78.5
76,9
30.2
Qwen2.5.VL..
VisionZip
50
2026
76.6
53.5
80.5
76.4
721
29.2
65.8
Zip
74.1
2142
78.1
$4,3
77.6
776
30.2
68.0
Spansity Forcing
22.9
2146
78.4
55.5
83.2
78.6
774
30.9
68.7
Full
100
2303
83.9
62.2
84.0
82.9
845
36.7
73.8
52.1
2115
81.9
61.2
80.2
79.6
760
34,5
69.9
Qwen2.5-VL-7B
VissonZip
50
2209
83.3
61.8
80.6
79.9
796
34.6
71.2
Zip
79.5
83.9
60.4
82.6
36.2
72.9
Spursity Forcing
24.7
2286
84.1
62.5
83.1
82.6
847
36.7
73.6
projects. For VLenei-ss calibration, we employ the Video-R1-266 (Feng et al., 2025) dataset, which covers both image and video modalities to ensure broad domain generalization. For LLaVA- sideoo-78k 178k (Zhang et al., 2024c) maintain its capability in the video domain. The parameter p is set within the range of [0.94, 0.975] with a step size of 0.005 for training. We train model 8 NVIDIA A A100 GPUs with 883 GPU hours for Qwen2.5VI.-7b and 164 GPU hours for -V--Video-7h,, limiting the number of video frames to 8 for efficiency. The Adam optimizer used with a learning rate of le-6. In the KL divergence term of the GRPO algorithm, the hyperparameter 3 is set to 0.04. During inference, Qwen2.5-V and Qwen2-VL are evaluated video benchmarks with approximately 120 frames corresponding to 16,384 tokens, while LL- Video is evaluated with 110 frames corresponding to 20,240 tokens. Notably, during training we employ multiple-budget rollouts to explicitly probe the sanciincc--peformmaae so that the model learns to enhance token sparsity. At inference, however, we simply fix to 0.975, which the upper bound of optimization range, to guarantee accuracy while still reaping the efficiency gains learned during training.
Video,
10k subset of LL Video-
trade-ofi off,
Benchmarks. To assess the effectiveness of our approach, we evaluate on 13 benchmarks: image tasks (MME (Fu et al., 2023), MMBench (Liu et al., 2024a), MMStar (Chen et al., 2024b), ChartQA (Masry et al., 2022), TextVQA (Singh et al., 2019), OCRBench (Liu et al., 2024c), MMMU- Pro (Yue et al., 2024)) and video tasks (VideoMME (Fu al., 2024), MLVU (Zhou al., 2024), VideoMMMU (Hu et al., 2025), Perception (Patraucean et al., 2023), Egoschema (Mangalam al. 2023), TempCompass (Liu al., 2024b)). We additionally use HallusionBench (Guan et al., 2024) to assess robustness under low token budgets. Together, these benchmarks span diverse modalities, domains, and difficulty levels for a rigorous evaluation of both efficiency and accuracy. The results are evaluated using LMMs-Eval (Zhang et al., 2024a).
Baselines of enhancing token sparsity. We compare our Sparsity Forcing with the existing two baselines to verify its effectiveness in the post-training scenario. Trainable sparse attention: We adopt MOBA (Lu al., 2025) and Zi into existing MLLMs to explicitly enforce token sparsity toward a predefined budget/attention retention ratio. The block size of MOBA is set to 256, and the budget set to 25% for salient token selection. Attention sharpness loss: We implement Too sharpness-inducing regularization term on attention maps (Gong et al., 2024), encouraging the model focus its attention distribution on smaller set of highly relevant tokens while suppressing low-importance ones. We adopt the block-wise attention map probing strategy MOBA to obtain the attention map. During inference, we select 25% salient tokens for sparse attention.
MAIN RESULTS
Performance comparison with training-free free sparse attention. As training-based enhancement of Zi we first compare against Z. As shown in Tables and 2, our Sparsity Forcing can further reduce the token retention ratio of ZipVL from approximately 80% to 25% in Qwen series MLLMs and from 45% 29% LLaVA- Video across 13 image and video benchmarks with minimal performance decline. Moreover, compared to other training-fret free sparse attention methods, Sparsity Forcing achieves much lower token ratio while maintaining accuracy comparable

---

Published as a conference paper at ICLR 2026
Table Performance comparisons with training-free sparse attention on video benchmarks. For Minference (Jiang et al., 2024), we report FLOPs-equivalent token ratio.
Model
Method
Ratio
VideaMMME
MLXUUsus
VideoMMMI
PerceptionTest
EgnSchhema
TempCompuss
Avg.
Full
100
62.9468.2
66.4
42.2
61.6
66.2
67.1
62.
Qw
ZipVL
71.2
62.1/67.8
64.9
42.1
61.3
66.0
66.8
61.6
Spunsity Forcing
61.86883
61 61,2/66.8
67.3
64.2
64.0
61.7
Q-
ZipVI.
69.4
60.66665
67.3
41.4
65.2
63.7
63.9
61.2
Sparsity Forcing
22.8
61,066.5
67.2
41.9
662
64.0
63.7
61.5
Full
100
61.5711
69.7
46.3
69.0
61.3
71.2
65.2
Qwen2.5-VL-7B
VL
64.1/7088
Spansity Forcing
64/0/1..
Full
100
64.5/70.8
70.4
35.2
66
$7.01
66.
61.4
Fasty
51.5
64.0/70.5
69.7
31.8
65.8
56.2
65.4
60.9
VisionZip
42.2/60.6
$2.9
31.7
42.0
39.7
46.2
45.0
ILaMa-Vidoo78
Minference
6/.3/
VipVI.
3/703
70.2
Spansity Forzing
29.6
64. 1/70.2
70.3
35.2
66.0
57.0
66.2
61.3
(pwn2255-31-7
S
P1y
Spuriily
10
DJ
10
05
20
30
10
E20
40
10 du
Sulget
Figure 3: Adjustment under low budgets across different models and benchmarks.
MaVAAVideo-7h VideoMME
ILaVA-AVdde-775 on VideoMME
Selected Toloen Batjo (%)
Accuraty Budget Hatt
Hotto Accuracy (%)
60
FlashAltentson
Zip/t
Flshhtterca00 Ztph
Sparsily Burtaa
40000
Spanity Fondra
63
50
130
- Buatge Radio
Monnryy
E22
-de Accommy
20000
25
30
20
0.08
Sequence Length
Sequence Length
Sequesse Length
Figure 4: (a) The effect of attention scores retention threshold p token ratio and performance. (b) Accuracy and token budget with respect to increasing token sequence. (c) Prefill latency and decoding memory usage under varying sequence lengths ILAVA-Video-77.
to the standard-attentio baseline, thus demonstrating superior balance between efficiency and performance. On LLVVA-Video-7B 29.6% tokens vs. 46.1 16.1% % for Minference. Moreover, it clearly outperforms VisionZip (Yang et al., 2024b) and FastV (Chen et al., 2024a), which both require around 50% tokens but achieve lower or similar accuracy. These results show that Sparsity Foreing can significantly reduce computation by enhancing token sparsity while maintaining performance across image and video benchmarks.
Video-7B, it matches Minference's (Jiang et al., 2024) accuracy while using
Performance comparison with post-training base- line methods in enhancing token sparsity. As shown in Table Sparsity Forcing achieves su- perior efficiency-performance trade-off compared to baseline methods on Qwen2.5-VL-77 The standard- attention model reaches an average score of 73.2, while MOBA and sharpness loss, both operating 25% token ratio, suffer from noticeable drops to 66.6 and 67.8, respectively. ZL provides a mod- erate balance, obtaining 71.5 at higher token ratio of 61.7%. In contrast, Sparsity Forcing attains 72.8 with only 26.4% tokens, outperforming all baseline approaches tasks such as MMStar (Chen et al., 2024b) and ChartQA (Masry et al., 2022). These results highlight the effectiveness of our RI--based optimization that reformulates the aediiency-pefformaan tradeoff explicit rewards.
Table 3: Comparisons with baseline methods of enhancing token sparsity on Qwen2.5VL. 7b. t denotes post-training MLLMs with ZipVL.
MeBled
Ratio
MME
MIMStcr
ChurQA
VidecMME
Arg
V1
Sharpness
37.0
7

---

Published as a conference paper at ICLR 2026
100
30
20
Eufurn Imiining
Selectad Token Ratio (%)
After training
103-
60
123
192
142
192
183
203
20
W9
15 Layer Index
taining
Insiing
Figure Token sparsity evolution of Qwen2.5-VL-77 Left/center: attention maps of the same sample before vs. after training, suggesting our method reinforces MILM to concentrate smaller subset of tokens. Right: layer-wise selected token ratio before and after training, showing that the achievable sparsity differs markedly across layers.
Table 4: Ablation study different sparse attention with top-k, top-p, and threshold-based pruning.
Table 5: Ablation study of different ranges of and group sizes for training.
Model
Sparse atlention
Ratio
MME
VideoMEE
Range (p)
Group size
Ratio
MME
VideoMME
Full
100
2303
64.5
1..94, 0.975
24.1
2286
64.0
Qwen2.5-VI.77
Top-k
25
2160
60.2
0.94. 0.975
23.7
2290
63.9
Threshold
37.8
2218
61.6
[0.80, 0.95
29.0
2285
63.6
Top-p
24.1
2286
64.0
[0.80,
25.7
2285
63.7
Adjustment under different budgets. We further evaluate Sparsity Forcing under varying token budgets illustrate its adaptability. As shown in Figure Sparsity Forcing enables flexible budget adjustment across tasks and models. On commonsense benchmarks (Liu et al., 2024a), it maintains strong performance even under an extremely low budget of about 10% tokens. On OCR-related tasks (Liu et al., 2024c; Masry et al., 2022), it can further compress token ratio while still preserving better accuracy than other methods, demonstrating its ability focus on fine- aeee-graied textual regions. Moreover, this property generalizes well across both image and video benchmarks and different model backbones, outperforming other sparse attention baselines under the same token ratio.
Besides, further discuss the influence of the attention retention ratio p during inference. As shown in Figure 4 (a), increasing p smoothly raises the token ratio while also improving accuracy, even for the value outside the optimization range, showing that Sparsity Forcing enables controllable efficiency-accuracy trade-offs without abrupt performance drops.
Runtime speed comparison. As shown in Figure 4 (c), Sparsity Forcing delivers clear efficiency gains over existing methods. It achieves up to 3.3x faster inference and 3.0x lower memory usage than FlashAttention-2 (Dao, 2023) at 200k sequence lengths. These results highlight that Sparsity Forcing not only reduces token budgets but also substantially improves runtime efficiency for scage multimodal inference.
ABLATION STUDY AND DISCUSSION
Sparsity Forcing with different sparse attention mechanisms. Table 4 compares Sparsity Forcing using various sparse attention strategies, including top-k sampling, threshold-baun pruning, and dynamic top-p sampling. While top-A and threshold-based methods can reduce token usage, they rely on fixed sparsity patterns or constraints and thus act as offline policies, which may not adapt optimally to input-specific variations. In contrast, top-p sampling enables online adjustment of token retention according to attention score distributions, achieving the best trade-off between efficiency and performance (e.g., 24.1% % token ratio with minimal accuracy drop). This highlights the advantage of coupling grouped-rollour RL with a dynamic, inference-time controllable sparse attention mechanism.
Token sparsity change. Figure 5 shows how training reshapes both the attention distribution and the retained token ratio. After training, the model concentrates attention on a more compact subset of tokens, illustrated in the left and center figures, demonstrating effective pruning. The layer-wise
8

---

Published as a conference paper at ICLR 2026
curves (right) reveal substantial variation in attainable sparsity across depth, indicating that single global retention rate is suboptimal and a dynamic, layer-aware policy is preferable.
Robustness using a low token budget. Aggressive token pruning drop essential evidence and increase hallucination risk. To assess robustness, we conduct a sensitivity analysis on HallusionBench (Guan et al., 2024) under striet token budget. As shown in Figure 6, Sparsity Forcing closely tracks the original model across all accuracy, figure accuracy, and question pair accuracy (aAcc, fAce, and Acc) with only minor fluctuations, indicating that our dynamic sparse attention preserves key evidence and does not amplify hallucinations in low-budget settings.
Scalable token sparsity. We examine how learned token sparsity scales with model capacity and context length. As shown in Tables and 2, we find models of different sizes and capabilities in the same MILM family exhibit broadly similar retained token ratios, a phenomenon that holds across both image and video domains. Moreover, in Figure 4 (b), as the input sequence grows from 4k to 20k tokens, the retained-token ratio steadily decreases while accuracy remains nearly unchanged. This suggests that learned token sparsity adapts to longer sequences by safely discarding more redundant tokens yet preserving essential information, making it particularly effective for high-complexity. long-context video understanding.
Puil Spursay Feetieg
Accmmyy
43
Figure 6: Sensitivity analy- sis of Qwen2.5-VL-7b on Hal- lusionBench using low token budgets.
Why grouped-rollout RL? Our method requires comparing multi- ple budgets per query to explore the efficiency-peffommant trade- off, where the notion of "positive
negative" is inherently non- stationary: as training progresses, sparsity patterns and the minimal correct budget shift, so predefined preference pairs of DPO (Rafailov al., 2023) and rigid sparse pattern of trainable sparse attention quickly become misaligned with the current trade-off frontier. In contrast, grouped-rollout RL methods (e.g., GRPO, DAPO (Yu et al., 2025)) operate on-policy and define relative advantages within each sampled group of rollouts. This design naturally provides exploration and credit assignment directly tied to budget choices, and avoids maintaining a large, ever-updating preference dataset.
Budget sampling strategy. As shown in Table 5, the ablation of the range of p and group size can be interpreted as probing the group size enables denser coverage of candidate token budgets, allowing the model to explore the budget-accuracy landscape more thoroughly and providing stronger contrastive signals for learning sparsity. However, this benefit comes at the cost of higher training overhead, since more rollouts must be generated per sample. In addition, setting the p range relatively high improves the likelihood that at least one rollout within the group is both correct and efficient, which is crucial for contrastive learning to effectively guide sparsity.
trade-off through grouped rollouts. Increasing the
7 CONCLUSION
In this paper, we presented Sparsity Forcing, an RL-based post-training method that explicitly encourages token sparsity well-posed MLLMs, delivering substantial inference-time acceleration. We treat the token-reduction ratio as the efficiency reward and answer accuracy as the performance reward. Concretely, Sparsity Forcing performs multi-budget rollouts and applies GRPO with group- wise advantages to favor the smallest budget that still yields correct answer. Compared with previous SFT-based methods, our RL-based approach enforces an diference-aligneee token pruning policy and KV-cache management, thereby delivering real end-to-end efficiency gains in MILMs while keeping performance degradation minimal. Experiments across thirteen image and video benchmarks show that Sparsity Forcing increases the token reduction ratio Qwen2-VL/ WL/Owen2..-W V- from 20% to as high as 75% with minimal accuracy loss, while cutting long-context inference memory by up to 3x and speeding up decoding by up to 3.3x.
Limitations and future work. Our current work mainly focuses on single-turn hardware -ggossii token sparsity. In our future work, we plan to extend reinforced efficiency broader applications and objectives-seeeeeeet such as hardware-aware targets (aaeenyyiemmory/eeeryy) multi-turn dialogue, tool- /retrieval-call budgets, head/layer/Moo expert gating, and K KV/ctiivation quantization and retention.

---

Published as a conference paper at ICLR 2026
REFERENCES
Ali Araabi, Vlad Niculae, and Christof Monz. Entropy-and distance-regularizee attention improves low-resource neural machine translation. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pp. 140-153, 2024.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Play-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we the right way for evaluating large vision-language models? arXiv preprint arXiv:2403. 20330, 2024b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c.
Tri Dao. Flashattention-22 Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2.3070869)) 2023.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.2177,, 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306. 13394, 2023.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang. Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of mudti-modal llms video analysis. arXiv preprint arXiv:2405. 10055, 2024.
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok- Hay So. Ting Cao. Fan Yang. et al. Seerattention: Learning intrinsic sparse attention in your Ilms. arXiv preprint arXiv:2410. 13276, 2024.
Shizhan Gong, Qi Dou, and Farzan Farnia. Structured gradient-based interpretations via norm- regularized adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1100--10018, 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, et al. Hallusionbemch: an advanced diagnostic suite for entan- gled language hallucination and visual illusion in large vision-language models. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14485,, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang. and Bohan Zhuang. Zipvl: Efficient large vasion-language models with dynamic token sparsification and kv cache compression. arXiv preprint urXiv:2410.0858. 2024a.
Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. Advances in Neural Information Processing Systems, 37:68287-68300 2024b.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. 2025.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre- prefiiling for long-contens LLMs via dynamic sparse attention, In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
10

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposhum on operating systems principles. pp. 611-626, 2023.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03322 03366 2024a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 52299-1330, 2024b.
Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Mminference: Accelerating pre- filling for long-context vlms via modality-aware permutation sparse attention. arXiv preprint arXiv:2504.16083, 2025.
Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, lon Stoica, Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-p pruning. arXiv preprint (rXXh:25002027700, 2025.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024a.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video ilms really understand videos? arXiv preprint arXiv:2403.00 00476, 2024b.
Yuliang Liu, Zhang Li. Mingxin Huang, Biao Yang, Wenwen Yu, Chumyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 6711)222010,, 2024c.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang. Zhiqi Huang, Huan Yuan. Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502. 2502 13189, 2025.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212 46244, 2023.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.1022 10244, 2022.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, al. Perception test: diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36: 42748 42761,22111122 2023.
Elia Peruzzo, Enver Sangineto, Yahui Liu, Marco De Nadai, Wei Bi, Bruno Lepri, and Nieu Sebe. Spatial entropy as an inductive bias for vision transformers. Machine Learning, 113(9):6455-6755 2024.
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53747 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03330 2024.
11

---

Published as a conference paper at ICLR 2026
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 8317-8326. 2019.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12//11 2024.
An Yang, Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15111 2024a.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467 2024b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint 2tti:25003.1477770 2025.
Jingyang Yuan, Huazuo Gao, Damai Dai. Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang. Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.
2025.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.0288 02813,, 2024.
Jiajun Zhang, Yang Zhao, Haoran Li, and Chengqing Zong. Attention with sparsity regularization for neural machine translation and summarization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(3):507-518, 2018
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevim: Visual token sparsifica- tion for efficient vision-language model inference. arXiv preprint arXiv:2410.044 2024b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuming with synthetic data. arXiv preprint arXiv:2410.02713, 2024c.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tięjun Huang, and Zheng Liu. Mlvu: comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.044 04264, 2024.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ALGORITHM OF OUR DYNAMIC SPARSE ATTENTION
As shown in Algorithms I and 2, we summarize the sparse attention mechanisms used in our method for the prefill and decoding.
Algorithm 1: Prefill phase of our dynamic sparse attention.
Algorithm 2: Decoding phase of our dy-
namic sparse attention.
Input: Input embedding X, attention
Input: Input embedding stored KV cache (Kin, Vin).
retention ratio p. Output: Attention output O, KV cache (K, V).
Output: Attention output o, updated KV cache (Kout, Vout).
Calculate query, key and value states (Q.
Calculate query, key and value states (q.k.v)
// Fetch KV cache from memory and update
// Probe attention map Select a subset of tokens Q' from query states and compute attention scores
Kout = Concat (Km,,
Vout
Concat(V in: v)
A' Softmax QQK²T)
// Compute attention output
Determine the number of important tokens
o = FlashAttention(( q. Kout, Vout)
per Eq. (5)
return o, (Kout, V Vout)
Calculate the normalized attention scores for
each token as per Eq. (6)
Select a set of important tokens T as per
Eq. (7)
// Token-level Sparse
Attention with
FlashAttention
0 FlashAttention ( V(T)
/ KV Cache
K KTT
V V
return O, (K, V)
A.2 IMPLEMENTATION OF ATTENTION SHARPNESS LOSS
Following prior works such as MOBA (Lu et al., 2025) and NSA (Yuan et al., 2025), we adopt block-wise probing strategy by applying average pooling over queries and keys with block size B to approximate the attention map. Given the visual input X, with query Qu and key K, E Bixd we first compress the sequence via average pooling:
mean pool B), K, mean (rraan_poo((,....
where mean_pool is applied along the sequence dimension, yielding compressed representations Qu, K, R2/Bxd; The block-wise attention map is then computed
Q₂ + +M))
A=o
d;
where a denotes the softmax function and M is the causal attention mask.
To promote sparsity, we encourage each query concentrate few salient keys by introducing Los sharpness-inducing regularization:
I/B Σ
Leharp T/B
|
where A, denotes the i-th row of A. This loss increases the peak mass of each query's attention distribution, enforcing sharper focus and reducing redundancy among tokens. At inference time, we apply sparse attention by retaining only the top 25% salient tokens according A.
13

---

Published as a conference paper at ICLR 2026
Question: Measuring from the closest each object, what the distance between the weshing machine and garbage bin?
Video
Ground Truth: 3m
Original Modeli 4,2m
Sparsity Forcing(26%)
Question: size room (in square meters)? If multiple rooms shown, estimate combined space.
Video
Ground Truth: 26.2
Original Model: 20
Sparsity Forcing(33%) 15.0
Figure 7: Failure cases. Our method still struggles with spatial reasoning, particularly in estimating relative object distances and available space. Results are shown on Qwen2.5-VL-7B.
A.3 IMPLEMENTATION OF SPARSITY FORCING WITH DIFFERENT SPARSE ATTENTION
To implement Sparsity Foreing with different sparse attention mechanisms, we first compute the accumulated attention score of each token as in Eq. 4, and normalize it to à using Eq. 6, since early tokens accumulate higher values than later ones (He et al., 2024b). For top-k sparse attention, we retain the & most salient tokens ranked by for subsequent attention. For threshold-basee sparse attention with threshold т, we keep all tokens with à т.
During training, we adopt multiple budget rollouts to probe the efficiency accuurayy trade-off. For top-k sampling, we set within [25%, 50%] at intervals of 5%, where 25% reflects the target efficiency budget while 50% corresponds to the accuracy-peeserving range of most sparse attention methods. For threshold-based attention, we empirically vary within [10 3, 10-2] with a step of 0.002. At inference time, fix the budget by retaining the top 25% salient tokens for top-k sparse attention and set r = 10-3 for threshold-bot based sparse attention.
A.4 FAILURE CASE
These failure cases in Figure 7 highlight key limitation of token pruning, especially with enhanced sparsity: while it effectively reduces redundancy, may inadvertently disrupt cross-frame correspon- dences that are critical for spatial reasoning. Unlike commonsense reasoning, which often relies on text- -roundeddddd cues, spatial understanding demands aligument of visual features across frames such as consistent tracking of object positions, relative distances, and scene layouts.
A.5 MORE RESULTS DISCUSSION
Table 6: Ablation of different efficiency reward designs on LLa VA-VA-iide--77
Efficiency
Ratio
VideoMME
MLVUstay
VideoMMMU
PerceptionTTss
EgoSchema
TempCompass
Avg.
Token reduction mini
29.6
64. 17702
70.3
38.2
66.0
$7.0
66.2
61.3
Lateney token reduction ratio
30.5
63.9/70.3
70.4
35,0
66.4
56.4
66.0
61.2
Hardware efficiency reward. Hardware latency is important practical metric. We therefore conducted an additional ablation in which we explicitly include measured hardware latency in the efficiency reward. Concretely, compare two variants LLaVA-Video-75 (i) using only the
14

---

Published as a conference paper at ICLR 2026
token reduction ratio as the efficiency term, and (ii) using combined efficiency reward that depends on both latency and token reduction ratio. As shown in Table 6, incorporating latency into the reward does not yield noticeable benefits: both variants achieve almost identical accuracy (61.3 vs. 61.2 Avg.) with very similar token ratios. These results suggest that the token reduction ratio is already a sufficiently good surrogate for hardware efficiency in our setting.
Table Training and rollout cost comparison with baseline methods. Latency measured under 128k context.
Method
Wall-clock training time (hour)
Latency(s)
MOBA
75.6
11.6
ZLpVL
81.0
18.7
Sharpness loss
88.4
11.9
Sparsity Forcing (ours)
110.4
12.0
Training and rollout cost comparison with baseline methods. As shown in Table 7, note method is moderately slower than other SFT-based baselines but still within an acceptable range. This is mainly because GRPO-based fine-tuning requires the MLLM generate multiple rollouts for each sample, whereas other methods only perform standard fine-tuning, In terms of rollout efficiency, our method is 1.6x faster than the SFT-based ZipVL. and remains comparable other methods, since MOBA and sharpness loss use manually fixed token budgets. However, as shown in Table 3, our method achieves a superior performance-efficiinng tradeoff compared with SFT-based counterparts on both image and video benchmarks, making the additional training cost worthwhile.
Table 8: Additional comparison with temperature adjustment enhance token sparsity. t denotes post-traing pust-training MLLMs with ZL
Method
Ratio
MME
MMStar
ChartQA
VideoMME
Avg
Full MOBA
100
2303
62.2
84.0
64.5
70.2
25.0
1906
58.6
77.3
62,6
66.2
Zip
61.7
2264
62.0
78.9
64.2
68.4
Sharpness loss
25.0
1965
59.6
77.0
63.7
66,8
Temperature adjustment
25.0
1960
59.8
78.6
61.3
67.4
Sparsity Forcing(ours)
26.4
2286
62.5
83.1
64.0
70.0
Additional comparison with temperature adjustment baseline method. Intuitively, tempera- ture adjustment in the Softmax function can also sharpen the attention distribution. As shown in Table temperature- sharpening performs better than the sharpness- baseline, yet still underperforms our method by 2.6%.
15


Reviews:
[{"review_id": "zx7Ecj4e5S", "reviewer": "Reviewer_htVe", "summary": "The paper identifies a real and important limitation of current sparse-token methods for MLLMs and they rely on emergent sparsity and plateau around ~50% token reduction.", "questions": "see above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Transforms token sparsity into an explicit joint reward (accuracy + sparsity) and performs GRPO over multiple budget rollouts, effectively avoiding the token-sparsity mismatch seen in SFT-based methods.\n\n2. General and practical framework that requires no architectural modifications and remains fully compatible with ZipVL and other sparse-attention approaches.", "weaknesses": "1. Is the reward shaping universally effective, or does it depend on dataset and model scale? Additional experiments or analysis on scalability and robustness would strengthen the claims.\n\n2. Although the method’s goal is to improve inference efficiency, the paper does not disclose critical training cost details. Specifically:\n\n    2.1 Total training time?\n\n    2.2 GPU resources (e.g., A100 hours)?\n\n    2.3 Wall-clock comparison vs. ZipVL, LoRA, or R1-style RL fine-tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d8nBse0qCP", "reviewer": "Reviewer_JKcD", "summary": "This paper proposes Sparsity Forcing, an RL-based post-training framework that promotes token sparsity in multimodal LLMs using GRPO. The method treats token reduction and correctness as joint rewards. Applied on QwenVL and LLaVA-Video, it improves sparsity with minor accuracy loss.", "questions": "Do baseline methods also involve training? If so, what is the training and rollout cost for each? If not, how do you ensure comparison fairness?\n\nDoes this approach support multi-turn conversations with incremental KV cache usage, or does each turn require full recomputation?\n\nIs the model retrained for each desired sparsity level, or can sparsity be adjusted at inference time without retraining?\n\nHow does the method handle video temporal redundancy, and could temporal attention provide additional efficiency gains?\n\nWhy not include real hardware latency as part of the efficiency reward instead of only token reduction?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Demonstrates sparsity gains across many MLLM tasks.\n\nClear implementation using grouped rollouts and ZipVL backbone.\n\nExperiments across benchmarks covering both image and video. There are some interesting discussions in ablation study.", "weaknesses": "However, while the results are promising, I find the methodological novelty to be somewhat limited. The work mainly applies the existing GRPO framework to a known sparse attention mechanism (ZipVL/MOBA style) and does not clearly articulate what new algorithmic insights it introduces beyond this combination. \n\nThe paper also does not sufficiently discuss how sparsity behaviors differ between text-only LLMs and multimodal models, even though modality-specific sparsity patterns and cross-frame visual dependencies are central challenges in MLLMs.\n\nThe comparison to baselines leaves unanswered questions. Although the related work section lists many sparse attention approaches, only a small subset is used in experiments. It is not clear whether the baselines are retrained or used as training-free methods, nor whether the training and rollout costs are comparable. Since the method uses RL and multiple rollouts with a reference model, training cost is a key factor, and it would be helpful to discuss fairness and overhead. \n\nIn addition, the improvements over ZipVL are sometimes modest (Fig 4 cd), particularly in latency, which suggests that the marginal efficiency gain relative to the additional training cost may be less significant in some settings.\n\nThere are also concerns regarding generality and deployment. It appears that the method largely supports single-turn inference and requires recomputing context when multiple iterations occur, since the KV cache pruning changes across runs. This limits applicability to interactive or long-horizon reasoning settings. \n\nThe method also seems to require training for each sparsity level, which affects flexibility in real-world deployment scenarios where dynamic sparsity adjustment may be desired. \n\nFor video tasks, the approach applies the same sparsity policy as images and does not attempt to leverage temporal redundancy or cross-frame consistency, which misses an opportunity.\n\nFinally, although the paper reports latency and memory results, the reward signal itself uses token ratio rather than actual measured runtime efficiency. This gap raises questions regarding hardware alignment, especially given that sparse attention can have non-linear speed behavior depending on the implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCvSU6FbEq", "reviewer": "Reviewer_92KG", "summary": "The paper proposes Sparsity Forcing, a post-training, RL-based framework that explicitly trades off answer correctness and token savings for MLLMs. The method performs grouped, multi-budget rollouts using top-p sparse attention; a joint reward promotes correctness and token reduction, and GRPO (group-relative PPO) updates a sparse policy model while anchoring to a reference (full-attention) model for stability. Experiments on 13 image/video benchmarks claim to raise token reduction from ~20% to ~75% with minimal accuracy loss, yielding up to 3.0× memoryand 3.3× decoding speedups in long-context settings.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed effieicency-aware RL post-training is important and novel.\n2. The experimental results show the proposed method is very promising. \n3. The paper writing is very clear and easy to follow.", "weaknesses": "1. The method needs to compute a_sort, nnz, topk_index. Some of these functions have linear complexity. It is hard to compute in parallel. How did the authors compute this function efficiently on GPU in Algorithm 1&2 ?\n2. MOBA keeps 25% of full attention. Sparsity Forcing keeps 26.4%. It seems dynamic sparsity cannot achieve lower compression rate. Can the proposed method be applied to block-based patterns to achieve the best of both worlds?\n3. How did the authors deal with the irregularity introduced by the dynamic sparsity? It is not very friendly to system deployment, especially considering the skewness of KVCache, tensor core optimization and continuous batching.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTHdQNLdIX", "reviewer": "Reviewer_DEv6", "summary": "The authors propose a token-sparse attention for VL with RL to enhance sparsity. It can be combined with different sparsity methods.", "questions": "1. What's the motivation behind \"sharpness-inducing regularization/loss\"? It makes no sense to me as you can tweek temperature (softmax scale) to adjust shapeness. The reason we use softmax scale in attention is that we want it to be \"less sharp\" otherwise zero attention score will lead to no grad for certain tokens and you miss the chance to correct your attention distribution when the model makes mistakes. Besides, in LLM, people use all kinds of tricks like QK norm or QK clip to reduce max-attn-logits (prevent too sharp) for training stablity. \n2. How is moba baseline being trained? I believe moba is a pre-training method. If you apply its attn prob on a dense pretrain model, it should have large accuracy degradation. \n3. Have you tried any non-pruned kv dynamic sparse method? For example. using SeerAttention like methods in post-training setting.\n4. What's the cost of RL training-speed slowdown and GPU memory increase in current design? Similar to previous discussion, if you do not prune KV, your RL should also be efficient with a shared copy of KV cache. You only need to have differnt sparse index selections on different rollouts. Correct me if I am wrong.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Sparsity is combined with RL, which is a new direction of sparsity training.\n2. Evaluation of methods are solid, with different sparsity methods and models.", "weaknesses": "Personally I do not believe in token-sparsity with pruned KV cache as you never know what information is going to be used in the future. The situation is more severe when you have interaction with human, e.g. multi-round questioning/chating. Once you prune your KV cache, it's going to be lossy. As a result, it would be better either to use sparse attention kernel with full kv cache or you have certain ability to retrieve KV cache.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)