VideoNSA: Native Sparse Attention Scales Video Understanding
Authors: Enxin Song
Wenhao Chai
Shusheng Yang
Ethan J. Armand
Xiaojun Shan
Haiyang Xu
Jianwen Xie
Zhuowen Tu
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Video Understanding
Sparse Attention
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: VideoNSA shows that native sparse attention can scale video-language models reliably and efficiently.

Abstract: Video understanding in multimodal language models remains limited by context length: models often miss key transition frames and struggle to maintain coherence across long time scales. To address this, we adapt Native Sparse Attention (NSA) to video-language models.  **Our method, VideoNSA, adapts Qwen2.5-VL through end-to-end training on a 216K video instruction dataset. We employ a hardware-aware hybrid approach to attention, preserving dense attention for text, while employing NSA for video.** Compared to token-compression and training-free sparse baselines, VideoNSA achieves improved performance on long-video understanding, temporal reasoning, and spatial benchmarks. Further ablation analysis reveals four key findings: (1) reliable scaling to 128K tokens; (2) an optimal global–local attention allocation at a fixed budget; (3) task-dependent branch usage patterns; and (4) the learnable combined sparse attention help induce dynamic attention sinks.

Published as a conference paper at ICLR 2026
VIDEONSA: NATIVE SPARSE ATTENTION SCALES VIDEO UNDERSTANDING
Enxin Song', Wenhao Chai?, Shusheng Yang', Ethan Armand', Xiaojun Shan', Haiyang Xu!, Jianwen Xie*, and Zhuowen Tu
University of California, San Diego LLambda, Inc
Princeton University
New York University
ABSTRACT
Video understanding in multimodal language models remains limited by context length: models often miss key transition frames and struggle to maintain coher- ence across long time scales. To address this, we adapt Native Sparse Attention (NSA) to video-language models. Our method, VideoNSA, adapts Qwen2.5- VL through end-to-end training on a 216K video instruction dataset. We em- ploy hardware-aware hy hybrid approach atfention, preserving dense atten- tion for text, while employing NSA for video. Compared to token compression and training-free sparse baselines, VideoNSA achieves competitive performance on vodg-video understanding, temporal reasoning, and spatial benchmarks. Fur- ther ablation analysis reveals four key findings: (1) reliable scaling to 128K to- kens; (2) an optimal global-local attention allocation at a fixed budget; (3) task- dependent branch usage patterns; and (4) the learnable combined sparse attention help induce dynamic attention sinks.
Project page: hwapg///////eiiiooooooom////////
Code: http://gshhooooooomyrrreelllll/s Model: http://hhnggnggaaee...///nniiiia
INTRODUCTION
Key moments of a video can occur at any time, exemplified by soccer where game deciding moments typically span seconds of 90 minute game. Within those game deciding moments split second actions define the outcome: assist, missed tackle, the movement of the keeper. Multimodal large
language models (MLLMs)(Team)) 2025 Team et al. 2025b a Qiu et al.) 2025 Yuan et al.. 2025a
Gui et al.) 2025 Chung al. 20225] Li al.) 2025b) have achieved substantial progress in vision- language perception and reasoning, but still cannot match humans ability to extract and reason about salient moments in videos. While humans naturally sample color visuals around 60hz, (Kalloniatis & Luu 2007) across large contexts, existing VLMs often sample a single frame per second. Intuitively, increasing the context for these models by sampling more frames improves accuracy (Cai et al.) 2024 Wu et al. 20244), particularly for long videos and complex reasoning tasks. However, this approach pays for improvement with additional tokens, increasing computational complexity and pushing against fundamental limits of model context.
To address these challenges, many approaches (Wang et al. 2024 Li al. 2024b Jin
2024
Wang et al., 2025a; Yang et al., (224) adopt token compression to reduce redundancy and increase informative context. However, when applied complex reasoning tasks, these compression based models perform worse compared to full-token methods (Song et al. 2025a). Moreover, compression strategies often limit generalization through reduced perception and reasoning capacity (Wen et al.) 2025). In contrast, sparse attention mechanisms preserve tokens, but focus the models capabilities on relevant dependencies between tokens. Numerous sparse attention methods have already been employed in large language models (LLMs), but most are inadequate for video complexity (detailed in Appendix A).. Therefore, we present VideoNSA, which adopts Native Sparse Attention Yuaan et al. 2025c). learnable hardware-aware sparse attention mechanism proven to be effective in long-

---

Published as a conference paper at ICLR 2026
context modeling. VideoNSA is the first learnable and hardware- sparse attention framework tailored for video understanding, effectively scaling to ultra-long vision-text context. We apply the learnable sparse attention to video token sequences, while preserving grouped querry attention for text tokens. Following this pattern, our experiments show that using only 3.6% the attention budget 128K context length while improving performance on various tasks
We further conduct massive experiments and analyses of VideoNSA. revealing several impor- tant findings: (1) VideoNSA extrapolates effectively to contexts beyond its training length, and the optimal balance between temporal density and spatial resolution is highly task dependent. (2) VideoNSA also sensitive by attention scaling, with results remaining strongest near the training configuration. (3) The gating distribution evolves dynamically across layers, and the selection and sliding-window branches gradually lose importance in deeper layers. (4) The compression branch emerges the main computational bottleneck. Mo)ovr,,,,, the learned sparse attention weights remain beneficial even under dense attention settings. (6) Learnable sparse attention induces dis- tinctive attention sink behaviors across branches, with very few sinks in the selection branch and periodic sink formation in the compression branch.
In particular, our paper makes the following contributions:
We propose VideoNSA, a hardware-aware native sparse attention mechanism, and systematically investigate its effectiveness for video understanding, scaling up to 128K vision context length.
We introduce hybrid sparse attention in VideoNSA. enabling flexible allocation of information and attention budgets to achieve optimal performance across diverse task.
dynamically combine global and local attention through three complementary branches, which effectively reduce attention sinks in long vision contexts.
2
VIDEONSA
2.1 PRELIMINARIES
Native sparse attention. Existing training-free sparse attention methods are rarely hardware aligned, and typically don't increase training efficiency. Native Sparse Attention (Yuan et al. (025c) (NSA) avoids computing attention between all key-value pairs (K,, Vt). instead, for each query qt- NSA dynamically constructs information-dense KV cache subset. NSA combines three comple- mentary cache branches with learnable gate of adaptively weighting each branch yielding o,:
02 Σ
gi E-Attn(q qt K, V.)).
cE cE(emp, slc, win}
Token Compression (CMP) branch aggregates sequential blocks of keys into more coarse-grained, single block-level representations K.m via learnable MLP
kemp
10
S << 11
where is the block length, d the stride.
Token Selection (SLC) branch preserves the most salient key-value blocks by computing importance scores slee and selecting the indices of the top-n blocks:
I {i rank (t]) S n).
(3)
The final set of selected keys is formed by concatenating these top-ranked blocks:
dlle Cat( jim (1:+4:(
It)),
where 1, is the set of selected indices, is the number of blocks to retain.
Sliding Window (SWA) branch simply applies the standard sliding window attention, which retains the fixed most recent kayuvalue pairs:
K swa
K, -i-tt Viwa V Vi-
(5)
2

---

Published as a conference paper at ICLR 2026
VideoNSA
Prefilling
Frame KV Blocks
Text Tokens
Projection Layer
Comprossa
= Sakeddaa
9:11
Ocmp
Osic
Vision
Encoder
Armp
Aste
Frames
Decoding
Figure Overview of VideoNSA. Video frames are encoded into frame-level KV blocks. VideoNSA utilizes three sparse attention branches during prefilling stage: compression branch reduces redundancy via token averaging, selection branch identifies top-k important to- kens, and sliding window branch enforces local temporal coverage. The outputs combined through dynamic gating before integration with text tokens for LLM decoding.
Grouped query attention. In Multi-Head Attention (MHA), each query head has dedicated key- value (KV) projections, which makes the KV cache scale with the number of heads and increases inference cost. Grouped-Query Attention (GQA) (Ainslie et al. 2023) mitigates this by letting multiple query heads share fewer KV heads. For each input (x, J 2=1- GQA partitions the h query heads into groups (1 S S h). At a given timestep i, the output of
(a) for the s-th query head with
group index m(s) = sg/ h] is computed by applying attention to the shared keys and values as:
() TTTTK! (m(())
o," Attention(g?) (9) K Ve
(m(()) vim(())) softmax
(m(())
(6)
vdl.
where (+) xz w.(, 2.(m()) [m(())
(m))) (m((s) x. x:WW((())) (m((s))
The outputs from all heads (1)) VideoNSA utilizes Qwen2.5-VL-7B (Bai et al,
are concatenated by Og = (o .22) 2025) as the backbone, with Qwen2.5-7B (Qwen et al. (2025) the LLM decoder, which employs GQA for efficient KV cache utilization using 28 query heads and 4 shared key/value heads.
2.2 ARCHITECTURE
Existing token compression methods (Yang 2025d Zhang al. 2025c Hyun al. 2025 Zhang et al., 2025h suffer from irreversible information loss on complex tasks and don't 't address computational and latency bottlenecks in LLM video understanding. From the perspective of tention as a message passing in Graph Neural Network [Joshi] 2025 Pappone (025), it's clear this bottleneck fundamental. Standard attention propagates information between nodes (tokens) through edges (attention weights), with each token being updated by aggregating features from its neighbors, weighted by attention scores. Training-free free sparse attention often imposes a static adja- cency matrix whose fixed subgraph connectivity restricts information flow. Conversely, NSA Yuann et al. 2025c) provides data-dependent sparsity that preserves edges necessary for particular task.
VL-7B (Qwen et al. 2025), which incorporates vision encoder and adopts Qwen2.5-7B (Bai et al., 2025) as the LLM. As illustrated in Figure VideoNSA intro- duces hybrid attention mechanism in the LLM across different modalities. At each layer we split (f-1) and text tokens x! (1-1) according to their position
We build VideoNSA upon Qwen2.5 20wen2.5VVL-7B
the input tokens (I-1 into vision tokens x y IDs. For vision tokens, VideoNSA applies NSA YYuan al., 2025c) with dedicated gate of on each
head. We set the block size s equal to the token number per frame, and obtain the block-level rep- resentation by averaging all tokens within the block. The vision attention output oy dynamically
3

---

Published as a conference paper at ICLR 2026
weighted by the compression, selection, and sliding window branches as:
si Atto( yt. K2 Vi),
(emp.sle,win)
where of is implemented as a two-layer MLP with a sigmoid activation.
The text attention output of is computed using standard GQA (Ainslie et al. 2023 to preserve instruction following capabilities. We obtain the final output o!) of the layer by concatenating:
o ( [0
(9 o.
2.3 TRAINING RECIPE
We conduct end-to-end training adapt vision features for data-dependent sparse connectivity in the language model. The training dataset of VideoNSA is constructed from -LaVA-Viiee--17 (Zhang al. 2024d) by filtering for question answer pairs fps and retaining videos with 350-550 Trames, for a subset of 216K pairs. To emphasize sparse attention for temporal redundancy, we constrain the maximum pixels per frame to 50,176, and the maximum context length per training instance to 36K tokens. In VideoNSA, block size 8 is set to 64, block b is set to 32, and sliding window size w is set to 256. We trained using SWIFT (Zhao et al. 2024), adapting the NSA (Yuan) et al. 2025c) implementation from FLA (Yang & Z (2024) and (Pai et al.] 2025b). The complete training process requires 4600 H100 GPU hours. More training details including hyper-ppaaanen selection can be found in Appendix B
EXPERIMENTS
3.1 EFFECTIVENESS ON VIDEO UNDERSTANDING
Baselines Our primary baseline is Qwen2.5-Vi VL-7B @Qwn al. 2025) with dense FlashAtten- tion (Dao 2023). We compare VideoNSA against several strong baselines, including the quan- tization model AWQ (Team] 2024). training-free token compression models (Yang al. 2025c Zhang et al.] 2025b Chen et al. (024a), and training-free sparse attention methods (Jiang et al. 2024 Xu al.] 2025a Lai al. 2025 Li al. 2024c). All methods employ their official config- uration without additional training and using Qwen2.5-VL-7B (Qwen et al.) (025) as a base. For token compression baselines, we use the token kept ratio and sampling fps from the original pa- pers that yield the best accuracy, while for sparse attention baselines, we use the same configuration as VideoNSA. In addition, we fine-tune Qwen2.5-VLL7B (Qwen et al.) (025) using the same train- ing dataset VideoNSA to serve as competitive baseline. We also include models with different backbones for a broad comparison.
We evaluate VideoNSA across three domains including long video understanding, tem- poral reasoning, and spatial understanding using LMMs-Eval (Zhang et al.) (2024a) and VLMEvalKit (Duan et al. 2024). Table[]indicates that sparse attention methods consistently outper- form token compression approaches. We empirically evaluate the effectiveness of VideoNSA based several popular long video understanding benchmarks, including LongVideoBench (Wu al.; 2024a), MLVU (Zhou et al. 2024). TimeScope (Zohar et al. 2025) and Long Tingeepee (Zohar al. 2025). VideoNSA achieves competitive results, narrowing the gap with state-of-the-art meth- ods. We observe that VideoNSA shows clear advantages on tasks involving order-sensitive tem- poral reasoning and ultra-long video settings (10 hours in LongTimeScopp (Zohar al., 2025)) ). To evaluate the visual temporal reasoning capbility of VideoNSA, we evaluate VideoNSA on Tomato (Shangguan et al., 2024). benchmark spanning six reasoning types and three video sce- narios. VideoNSA attains the highest accuracy on Tomato (Shangguan et al.) (2024), substantially outperforming compression-ber based methods, underscoring their limitations in fine-grained temporal inference. VSIBench (Yang et al. 2025a) focuses on spatial reasoning allowing us to test whether efficient models can preserve local fidelity while achieving efficiency. VideoNSA matches the strongest sparse attention baselines and significantly surpasses token compression methods in spa- tial understanding, confirming that it preserves spatial fidelity. All detailed evaluation settings and subset results can be found in Appendix C Appendix D. Appendix E and Appendix
4

---

Published as a conference paper at ICLR 2026
Table I: Results on long video understanding, temporal reasoning and spatial understanding tasks.
LVB, LTS for Long Videobeeeen (Wu et al. 2024a) and LongTimeScope Zohar et al. 2025).
Long-form Video
Temporal
Spatial
Model
LVB
MLVU TimeScope
LTS
Tomato
VSIBench
LLaVA-Ome (Li et al.
2024a
56.3
25.5
32.4
LLaVA-Video-778 ZZang et al. 2024c
58.2
74.1
34.0
35.6
V (Zhang
2025a
59.8
47.7
69.5
IntemVL2..58B (Chen al.
2024c
55.8
Video-XL-2 Qin et al. 20256
61.0
52.2
Qwen2.5-VI--BB (Qwen
2025
58.7
51.2
81.0
40.7
22.6
29.7
-wen2.55VLL-7B-ANB
(Team
2024
59.0
46.0
35.0
Qwen2.5-VL-7B-SFT
57.8
51.2
76.8
40.2
21.7
30.5
Token Compression Methods
FastV (Chen 2024a
57.3
41.8
46.5
35.6
21.6
32.0
+ VScan Zhang al. 2025b
48.1
80.3
31.1
19.1
34.4
+ VisionZip Yang et al. 2025c
52.4
33.1
43.5
40.4
23.6
32.1
Sparse Attention Merhody
Tri-Shape (Li al. 2024c
59.5
49.2
82.7
28.4
22.1
34.9
+ Minference Uiang al. 2024
49.2
82.7
44.4
23.0
36.5
+ FlexPrefil LLai cf al. 2025
58.4
46.0
83.0
39.1
23.7
34.0
+ XAttention
Xu et 2025a
59.1
50.2
83.1
41.1
21.4
36.6
VideoNSA
51.8
83.7
44.4
26.5
36.1
Table 2: Ablation study on branch selection across different tasks. Lideohehhhench (Wu et al. 2024a) and LongTimeSSope (Zohar et al. 2025).
LVB, LTS for
Branch
Long Video Understanding
Temporal Reasoning
Spatial Understanding
CMP SLC SWD LVB
MLVU lext
TimeScope
LTS
Tomato
VSIBench
48.1
43,9
41.5
25.1
23.3
29.2
48.4
47.7
63.7
37.1
24.0
27.6
49.1
40.2
59.3
29.8
24.0
29.8
49.4
42.7
57.3
32.4
23.5
29.4
49.3
42.4
65.2
34.4
23.0
29.1
48.8
43.4
57.3
31.6
24.5
30.3
60.0
51.8
83.7
44.4
26.5
36.1
3.2 ABLATION STUDY
To further analyze the components of VideoNSA, visualize attention pattern in each branch in Appendix H and assess the effectiveness different branches. Table 2 shows that single-branch models suller significant degradation, and even two-branch combinations remain inferior to the full VIdeoNSA, highlighting the necessity integrating all three branches with dynamic gating. Detailed results of different branch combination can be found in Appendix
SCALING ANALYSIS AND FINDINGS
Finding 1. Do learned sparse attention weights remain beneficial in dense attention settings?
Table 3: Ablation study on transferring sparse attention weights to dense attention across tasks.
Long Vidco Understanding
Temporal Reasoning Spatial Understanding
Model
Beng
MLVU
TimeScope
LonoTe
Tomato
VSIBench
Qwen2.5 S-LL2B
$8.7
51.2
81.0
40.7
22.6
29.7
Dense-SFT
(-1.5%)
51.2 (40.0%)
76.8 (-5.2%)
40.2 (-1.2%)
21.7 (-4.0%)
30.6 (42.19)
Dense-NSA
56.1 (-4.4%)
51.6 (40.8%)
(32.55)
40.9 (+0.5%)
(+3.5%)
33.1 (+10.7%)
VideoNSA
(+1.1%)))
51.8 (+1.2%)
82.7 (42.1%)
44.4 (+9..1 %)
26.2 (+15.9%)
36.1 (420.3 %)
5

---

Published as a conference paper at ICLR 2026
Token Frame (TPF)
500
Token Budget Constraint
Token Frame (TPF)
500
Token Budget Constraint
80
400
400
512
Performance
70
Performance
300
300
128K
45
200
128K
200
2048 Frames, 64 TPF
100
40
100g
200
400
600
800
1000
400
800
1200
1600
2000
Total Frames
Total Frames
(a) Information Scaling of LongVinn VideoBench
(b) Information Scaling of TimeScope
500
Token Frame (TPF)
500
Token Budget Constraint
26.0
Token Frame (TPF)
Token Budget Constraint
400
400
Ferformance
300
300
34
200
4 FPS, 256 TPF
24.5
33
256 Frames, 128 TPF
100
24.0
1000
1.0
1.5
2.0
2.5
3.0 3.5
4.0
200
400
600
800
1000
Frame Per Second (FPS)
Total Frames
(c) Information Scaling of Tomato
(d) Information Scaling VSIBench
Figure 2: Scaling Performance of VideoNSA under Different Context Allocation Strategies. We highlight the Token Budget Constraint to indicate settings with equal context length, and annotate the best-performing configuration under each benchmark. Since vidcos in Tomato (Shangguan et al. 2024).
vary FPS instead of total frames, with FPS x TPF 128 denoted as Ko-
We further examine whether the learned QKV weights of VideoNSA can imrpove performance in dense attention inference. Table 3 reports the relative performance change over the Qwen2.5-VL- 7B (Qwn al. 2025). Due the limited quality of the training data, our fine-tuned Qwen2.5- VL-7B (Dense-SFr) exhibits slight performance drops on most benchmarks. We observe that the transferred model (Dense-NSA) allows the dense variant several benchmarks suggesting that sparse-trained weights provides inductive bias towards more ef- fective attention distributions. However, the effect remains limited on Long VideoBench (Wu al.] 2024a). VideoNSA significantly outperforms Dense-NSA most tasks, highlighting the impor- lance runtime sparsity and dynamic gating.
recover and surpass the baseline on
Finding 2. How far can VideoNSA scale in context length?
The effective vision context length L is jointly determined by the number of vision tokens per frame and the total number of input frames F. VideoNSA is trained with a maximum context length of I, = 36K tokens, corresponding to T 64 tokens per frame. We conduct an information budget study under a fixed context length, by varying tokens per frame and frame rate. We then scale up the context length beyond the training budget, evaluating up the maximum 128K tokens supported by the language model. As observed in Figure[] the model consistently achieves higher performance when scaled to longer contexts beyond its training length across benchmarks. However, the ideal al- location of same token budget is highly dask-dependent. Long VideoBench (Wu et al. 2024a) favors allocating more tokens per frame, while Tomato (Shangguan al.) (024) and TimeScope (Zohar et al. 2025) benefit more from increasing the number of frames, emphasizing temporal coverage. VSIBench Yang et al. 2025a) shows mixed preferences depending context length, reflecting a balance between spatial and temporal sampling. Additional results on information scaling are reported in Appendix
Finding 3. How to allocate attention budget?

---

Published as a conference paper at ICLR 2026
14.47
7.2%
Training Setting
7.2%
3.6%
15
1.8%
Sading Window log2 scaes
Si ding Wardow oog scale)
(a) Attention Scaling of MLVU
(b) Attention Scaling of LongTime TimeScope
02
7.2%
Training Setting
Training Setting
30
0
Stiing Window
Window scale)
(c) Attention Scaling of Tomato
(d) Attention Scaling of VSIBench
Figure 3: Scaling Performance of VideoNSA under Different Attention Allocation Strategies. Scat- ter points from small to large and from light to dark indicate increasing performance. We annotate the point corresponding to the same attention allocation strategy as used during training and connect configurations with equal attention budgets using solid orange lines. We further scale the best con- figuration using dashed lines. Percentages show attention relative to full attention.
We define the Attention Budgel as the total number of key-value pairs visible to each query, denoted by Kuis- It is composed of a global sparse component and local sliding-window component as: Kattn = x s + w, where and s denote the number and size of global blocks, and w is the sliding- window width. With context length L, compared to causal dense attention with L(L-1) edges, the fraction of attention used y is
L(eS + w) 2(c5 + w) L(L-1)
L-1
To determine the optimal attention allocation, we first fix the total sequence length L, the atten- tion budget Kvis, and the block size S 64, while systematically varying the local attention ratio We then employ the optimal allocation ratio a* for attention budget scaling. As shown in scatter points denote different allocation strategies, with their size and color reflecting per- formance. We highlight the point corresponding the training configuration, connect equal-budget settings with solid orange lines, and extend the best-performing configuration with dashed lines, where the annotated values indicate the fraction of attention used y. Results show that model perfor- mance is highly sensitive to attention allocation. Although the optimal ratio between global and local attention varies across tasks, configurations close to the training allocation generally yield better re- sults. Under the same budget, fine-tuning around the training setting often improves performance, whereas simply enlarging the overall budget does not consistently bring further gains. Moreover, across most benchmarks, increasing global attention (enlarging the block count) tends to outper- form increasing local attention (enlarging the sliding window). Remarkably, VidcoNSA achieves leading performance using only 3.6% of the full attention budget. More results are in Appendix L
10
Figure
3
Finding 4. What roles do compression, selection, and sliding- winddww gates play VideoNSA?
We analyze the gating distribution of VideoNSA across Tomato (Shangguan et al. 2024], VSI- Bench (Yang et al. 2025a), and LongVideoBench (Wu et al.) 2024a), and aggregate the average routing gate weights over 100 examples from each. As illustrated in Figure where shaded bars de- note the interquartile range and horizontal lines represent mean values, each head in VideoNSA hibits distinct and diverse preferences across branches throughout its full depth. The diversity al- lows different layers to specialize in distinct modes of the context-dependent information flow. The
7

---

Published as a conference paper at ICLR 2026
Comoreseen Iner!
Salecios Prien)
dliceg Kindon Jrone
Combono of
Gate We Value
0.6
Sdutko
Lome Lave
Comdstin
10
L10
L32 Layer Index
L14 L16
Lia
120
122
L24
126
127
Figure 4: Gate weights across layers in VideoNSA. Compression remains dominant, while selection and sliding-window weaken in later layers.
Figure 5: Inter-head similarities of gates in VideoNSA. Selection and sliding-window gates show high similarity in middle layers.
compression branch maintains relatively high average weights across most layers, underscoring its primary role in reducing redundancy while preserving salient features. The selection and sliding window gates fluctuate more strongly, occasionally surpassing the compression branch in early and middle layers. However, their contributions diminish in the final layers (c.g.. L22-L26), demon- strating that the focus shifts towards aggregating high-level features. We also note strange behavior in the last layer, where all three branches are fully active despite selection and sliding window being inactive in the layers before. Full gate values distribution in Appendix
We further dive into the inter-head gate similarity of each layer in Figure 5 In the middle layers, both selection and sliding window gates exhibit pronounced increases in inter-head similarity. This indi- cates that multiple mid-layer heads converge to highly consistent gating behaviors when the model performs block selection and local temporal integration. However, the compression gate shows con- sistently low inter-head similarity, indicating that it operates largely in a hhadd hhead-iddependent manner. At both the initial and final layers VideoNSA, inter-head similarity remains weak across all gates, reflecting the need to maintain diversity in early representations and to support mixing information higher-level abstractions. More inter-head gate similarites visualization Appendix O
Finding 5. Where does the efficiency bottleneck come from?
We measure the inference latency of each branch in VideoNSA using wall-clock time across varying con- text lengths from 1K to 128K. The compression branch dominates runtime as the context grows, while the selection and sliding window branches contribute relatively little at longer contexts. Ideally, the compression branch grows approximately linearly with L, and the sliding window branch has complexity of O(L w). which results in linear scaling for fixed window size w. The selection branch requires computing importance scores over all L//b blocks per query, leading to computational complexity of O(T2 /b). However, wall-clock latency deviates from these estimates due to hardware parallelism, memory access patterns, and kernel launch overheads. Overall, the compression branch emerges as the primary bottleneck, highlighting the need for further optimization of its kernel design and memory efficiency.
2.0
selection
compression
sliding window
1.0
0.5
0.0
16
32
64
128
Context Length
Figure 6: Inference latency of each branch in VideoNSA.
Finding 6. Do learnable sparse mechanisms induce dynamic attention sinks?
8

---

Published as a conference paper at ICLR 2026
In decoder-only transformers, a disproportionate amount of attention is often allocated to the first few tokens, which act as attention sinks and absorb excessive attention mass as a byproduct of softmax normalization. Prior studies (Gu al., 2024 Xiao et al. 2023) show that attention sinks arise from massive activations and unusually small key and value norms, so attention directed to these tokens contributes little to the residual state. This raises an important question in learnable sparse attention: whether sparsity patterns amplify or mitigate such sinks.
Compression
Selection Sinks: 0.1%
Sliding Vindow
VideoNSA
0.5
0.8
2.0
0.4
0.6
10-
2.
3.0
7.3
15.0
15
Figure 7: Attention sinks distribution of different branches. VideoNSA maintains low overall sink ratio, with pink points indicating identified sinks.
1:ch
one
1.
Layer Index
Reavivee Pos
2.4%
Compresion
Sink Denity
Comprisioon
1%
5liligg Widoo
Index
Relativn Position
Figure 8: Layer-wise attention sink ratio distri-
Figure 9: Relative positions of attention sinks in different branches and Flash Attention.
bution in different branches and Flash Attention.
We follow the attention sink defination in (Pai al. 2025a
Attention Sink = i( > 0.1 A |o|| < median(Ilu]) - 2 IQR([Ui)))
where a is the average attention score received by the key, and ||||| the value norm of the token.
Figure Jillustrates the average distribution of attention sinks across the three branches of VideoNSA. Each frame is encoded into 256 tokens, and we adopt the same sparse attention configuration as used during training. The three branches exhibit markedly different sink behaviors. The compres- sion branch produces the most sinks, with distinct banded concentrations along the value norm axis caused by token merging that amplifies some token norms while suppressing others. Con- versely, the selection branch yields almost no sinks, as its top-k block filtering mechanism enforces smoother value norm distribution. Notably, the sliding window branch demonstrates clearer separation between sink and non-sink tokens along the value norm axis. Critically, dynamic gating allows VideoNSA to counteract the negative effects of the compression branch, achieving stable model with low overall sink ratio 0.3%.
Figure indicates that VideoNSA maintains low sink ratios overall, with only minor fluctuations across layers. However, Flash Attention exhibits a gradual increase in sink ratios toward deeper layers. The compression branch maintains relatively high sink levels across most layers. The lection branch remains consistently close to zero, while the sliding window branch occasionally shows higher peaks in the middle-to-late layers, indicating that locality constraints may still intro- duce bias in sequence settings. From the perspective of positional distribution in Figure 9 Flash Attention produces sinks that are uniformly spread across the entire sequence due to its fully connected dense attention. Under dynamic gating, VideoNSA achieves smoother temporal cover- age, alleviating over-reliance on early positions while avoiding the global diffusion characteristic of dense attention. In contrast, the compression branch exhibits strong accumulation at the beginning with an even steeper decay, indicating that token merging exerts its strongest impact on early- representations. The selection branch yields very few sinks across the sequence, while the sliding window branch produces sparse peaks at periodic boundaries local neighborhoods. More analysis about attention sinks various sparse attention settings can be found in Appendix S

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this work, we present VideoNSA, a hybrid hardware-aware sparse attention model that sig- nificantly advances video understanding across various tasks. By dynamically fusing block-wise compression, salient block selection, and a sliding window, VideoNSA effectively preserves criti- cal information while achieving near-linear scalability in efficiency and memory. Our experiments demonstrate that VideoNSS consistently outperforms existing methods on key tasks including long video understanding, temporal reasoning, and spatial understanding. While the prefill stage remains the primary bottleneck, our findings confirm that this hybrid sparse approach provides a powerful and scalable framework, paving the way for more capable video foundation models.
ACKNOWLEDGEMENT
This work is supported by NSF award IIS-2127544 and NSF award IIS-2433768. We thank Lambda, Inc. for their compute resource help on this project.
7 ETHICS STATEMENT
This research on video understanding utilizes publicly available datasets, ensuring that all data com- plies with privacy regulations. We acknowledge the potential biases that can arise in automatic answer generation, particularly concerning gender, race, or other characteristics. We have taken measures to evaluate and minimize such biases, while remaining committed to further improve- ments. Additionally, recognize the potential risks of misuse, such as generating misleading answers, and have checked the training dataset with safeguards against such applications.
REPRODUCIBILITY STATEMENT
We have made several efforts ensure the reproducibility of our work. All the key implementa- tion details, including the architecture of our model, the training procedures, and hyperparameter settings, described in supplementary meterial Section B The settings of the used evaluation benchmarks are in Section too further support reproducibility.
THE USE OF LARGE LANGUAGE MODELS
Large language models (LLMs) were used only for light editorial purposes, such as minor grammar checking and language polishing. They were not used for generating scientific content, research ideation, experiment design, or analysis. The authors take full responsibility for the entirety of the paper, and LLMs are not considered contributors or eligible for authorship.
REFERENCES
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305. 13245, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 2025.
Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov. Rock- etkv: Accelerating long-context 1lm inference via two-stage ky cache compression, 2025.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 1500 2020.
Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine-grained temporal under- standing for multimodal video models. arXiv preprint arXiv:2410. 10818, 2024.
10

---

Published as a conference paper at ICLR 2026
Jiawang Cao, Yongliang Wu, Weiheng Chi, Wenbo Zhu, Ziyue Su, and Jay Wu. Reframe anything:
Llm agent for open world video reframing, ArXiv, abs/2403.0000000 2024. URL WtLhhtps::////
semant somaantcscholart org/ ott.:/o.......oooooooooooa2aa32.
Wenhao Chai, Enxin Song, Yilun Du Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal Jenq-
Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark. arXiv preprint arXiv:2410.03051, 2024.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer. 2024a.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 2023.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for 2024b.
long videos. arXiv preprint arXiv:2408.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c.
Hyungjin Chung. Hyelin Nam, Jiyeon Kim, Hyojun Go, Byeongjun Park, Junho Kim, Joonseok Lee, Seongsu Ha, and Byung-Hoon Kim. Video parallel scaling: Aggregating diverse frame subsets for videollms. ArXiv, abs/2509.0800 2025. URL httph::t/pppppp semant ccscoo ar .rr.or/
CorpusID:28124423
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint 1.Xiv:2.300.88691, 2023.
Sixun Dong, Juhua Hu, Mian Zhang, Ming Yin, Yanjie Fu, and Qi Qian. Mmtok: coverage maximization for efficient inference of vlms, 2025.
Multimodal
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang. Jiaqi Wang, et al. VImevalkit: An open-source toolkit for evaluat- ing large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pp. 11198- 11201, 2024.
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijic Cao, Peiyuan Zhou, Jiaxing Qi. Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, et al. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint 2rxXiv:2400.332777 2024.
Yizhao Gao, Shuming Guo, Shijie Cao, Yuqing Xia, Yu Cheng, Lei Wang, Lingxiao Ma, Yutao Sun, Tianzuu Ye, Li Dong, Hayden Kwok-Hay So, Yu Hua, Ting Cao, Fan Yang, and Mao Yang. Scerattention-r: Sparse attention adaptation for long reasoning, 2025.
Xiangming Gu. Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410. 107781 2024.
Zhongrui Gui, Junyu Xie, Tengda Han, Weidi Xie, and Andrew Zisserman. Character-centric un- derstanding of animated movies. Proceedingg of the 33rd ACM International Conference on Mul- timedia, 2025. URL htRLthtps:// //api. www.ntiiassst.ooo
lar wwtp:/oggg/ooooroeeeooo 281311222211124400
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. 2023.
Zhiyuan He, Yike Zhang, Chengruidong Zhang. Huiqiang Jiang, Yuqing Yang, and Lili Qiu. Trian- glemix: A lossless and efficient attention pattern for long context prefilling, 2025.
Yu Huang, Zixin Wen, Aarti Singh, Yuejie Chi, and Yuxin Chen. Transformers provably learn thain-of-thought reasoning with length generalization. arXiv preprint arXiv:2511.07377 2025.
11

---

Published as a conference paper at ICLR 2026
Jeongseok Hyun, Sukjum Hwang, Su Ho Han, Taeoh Kim, Inwoong Lee, Dongyoon Wee, Joon- Young Lee, Scon Joo Kim, and Minho Shim. Multi-granull spatio-temporal token merging for training-fre free acceleration of video llms. arXiv preprint arXhv:2507.07990 2025.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin- Lin, al. Minference 1.0: Accelerating pre-filling for long-con contecontext Iims via dynamic sparse attention. Advances in Neural Information Processing Systems, 37:52481-5255 2024.
Jindong Jiang, Xiuyu Li. Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, Sungjin Ahn, Jan Kautz, Hongxu Yin, Yao Lu, Song Han, and Wonmin Byeon. Storm: Token-efficient long video understanding for multimodal 1lms, 2025a.
Pengfei Jiang, Hanjun Li, Linglan Zhao, Fei Chao, Ke Yan, Shouhong Ding, and Rongrong Ji. Visa: Group-wise visual token selection and aggregation via graph summarization for efficient milms inference. 2025b. doi: 10. 10.1145/374602.33755777
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700- 13710, 2024.
Chaitanya K Joshi. Transformers are graph neural networks. arXiv preprint arXiv:2506.22884, 2025.
Michael Kalloniatis and Charles Luu. Temporal resolution. In Helga Kolb, Eduardo Fernandez, Ralph Nelson, and Bryan Jones (eds.), Webvision: The Organization of the Retina and Visual System. University of Utah Health Sciences Center, Salt Lake City (UT), 2007. URL httpss:::
/ wwww.ncbb. nlm. nih got/books/NBKK15555555
Accessed September 24, 2025.
Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. Infinipot-v: Memory- constrained kv cache compression for streaming video understanding, 2025.
Henry Ko. Optimizing nsa for tpus kernel worklog.
https://hennrhhno..gihhbbb io/
post //ss//aaaaaa http//ssaappp// hat__ttuussssa_aaeeeeeeee hts_nnsaa___ss html 2025. Blog post.
Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, and Marinka Zitnik. Token reduction should go beyond efficiency in generative models from vision, language to multimodality. ArXiv, 0bb/22555.1821, 2025. URL htttps://api. Ctrrg/CorpssI:::: 27890015199
semanticscho...
Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. arXiv preprini arXiv:2502.20766, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint N4i::2408.033226, 2024a.
Bozheng Li, Yongliang Wu, Yi Lu, Jiashuo Yu, Licheng Tang, Jiawang Cao, Wenqing Zhu, Yuyang Sun, Jay Wu, and Wenbo Zhu. Veu-bench: Towards comprehensive understanding of video editing. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1 13671-13680, 2025a. URL https://api. wew.ntiisccooar.orgg//ooooo
278130017
Chenglin Li, Feng Han, Feng Tao, Ruilin Li, Qianglong Chen, Jingqi Tong, Yin Zhang, and Jiaqi Wang. Videopro: Adaptive program reasoning for long video understanding. 2025b. URL
Handong Li, Yiyuan Zhang, Longteng Guo, Xiangyu Yue, and Jing Liu. Breaking the encoder barrier for seamless video-language understanding. 2025c.
12

---

Published as a conference paper at ICLR 2026
Hongyu Li, Songhao Han, Yue Liao, Junfeng Luo, Jialin Gao, Shuicheng Yan, and Si Liu. Re- inforcement learning tuning for videollms: Reward design and data efficiency. arXiv preprint arXiv:2506.01908, 2025d.
Li Li, Yongliang Wu, Jingze Zhu, Jiawei Peng, Jianfei Cai, and Xu Yang. Unveiling effec- tive in-context configurations for image captioning: An external & internal analysis. ArXiv, abs/2507.08021, 2025e. URL https://apss remanticschon ar. wtt.:orrg///ooooo 28C282985
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang. et al. Videochat flash: Hierarchical compression for oong-context video modeling. arXiv preprint arXiv:2501.000 000544, 2024b.
Yixuan Li, Changli Tang, Jimin Zhuang, Yudong Yang. Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. Improving IIm video understanding with 16 frames per second, 2025f.
Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412. 2024c.
Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and Lili Qiu. Mminference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention, 2025g.
Manlai Liang, Wanyi Huang, Mandi Liu, Huaijun Li, and Jinlong Li. Lag-relative sparse attention in long context training, 2025.
Chenfei Liao, Wensong Wang, Zichen Wen, Xu Zheng, Yiyu Wang, Haocong He, Yuanhuiyi Lyu, Lutao Jiang, Xin Zou, Yuqian Fu, Bin Ren, Linfeng Zhang, and Xuming Hu. Are we using the right benchmark: An evaluation framework for visual token compression meth- ods. ArXiv, abs/2510.07143, 2025. URL https://aa http:/pp....mmtttcccocco....ro
Xiangrui Liu, Minghao Qin, Yan Shu, Zhengyang Liang, Yang Tian, Chen Jason Zhang, Bo Zhao, and Zheng Liu. Timescope: Towards task-oriented temporal grounding in long videos. ArXiv, abs/2509.26360,, 2025. URL https:/ happs://api... www.nnticcccoooa...oooo.
Corpus1D:221661422
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. Videoeval-pro: Robust and realistic long video understanding evaluation. arXiv preprint arXiv:2505.14640, 2025.
mdy666. Sealhheffaa--maiiiespprreatttttd
https:://itthb. comm////66
Scalable-Flash- Nattss--aaiv---paaaaaa SoulaaleePPlaa---aateee--earrr Attenseiottnntiooo 2025. GitHub repository.
Zhenyu Ning, Guangda Liu, Qihao Jin, Wenchao Ding, Minyi Guo, and Jíeru Zhao. Livevlm: Efficient online video understanding via streaming streaming-oriented oriented kv cache and retrieval, 2025.
Dhruv Pai, Timor Averbuch, Mason Wang, and Ben Keigwin. Sparsity is cool. Blog post, Tilde Re- search, June 2025a. URL httpt:::t/ttt..ttt www. www..teeeeeeeecccc...o.caaeo wwt.c/oooooooocoeeeeeoooooeoooee Tilde; correspondence to dhruv @ileeeeeileeeeecc..oomm.
Dhruv Pai, Timor Averbuch, Mason Wang, and Ben Keigwin. Sparsity is cool. https://www tilderesearch http://wwo//.o..///e/ee////////e attn June 25 2025b. Tilde Research Blog.
Francesco Pappone. Attention sinks from the graph perspective. lattps://ubbiish
o August 2025. Blogpost.
http:///e..atatet....oooooaaaaar 8 biiesssftaaartttrrrggppppgpgpppp
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108, / /2409, 2021.
13

---

Published as a conference paper at ICLR 2026
Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu. Video-xl-2: Towards very long-vided video understanding through task- kv sparsification, 2025a.
Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu. Video-xl-2: Towards very long- understanding through task-aware kv sparsification. arXiv preprint arXiv:2506. 1iii:22506.12255., 19225, 2025b.
Tianheng Qiu, Jingchun Gao, Jingyu Li, Huiyi Leong, Xuan Huang, Xi Wang, Xiaocheng Zhang, Kele Xu, and Lan Zhang. Intentvenet: Bridging spatio-temporal gaps for intention-oriented con- trollable video captioning. In Proceedings of the 33rd ACM International Conference on Multi- media, pp. 13822-13829, 2025.
Tianyuan Qu, Longxiang Tang, Bohao Peng, Senqiao Yang, Bei Yu, and Jiaya Jia. Does your vision- language model get lost in the long video sampling dilemma? arXiv preprint 1rXiv:2503.12490 2025.
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: www.v.orgg/o...oaaa........ooo.r abs/ 2 2212.2144 15115
Weiming Ren, Wentao Ma, Huan Yang. Cong Wei, Ge Zhang, and Wenhu Chen. Vamba: Under- standing hour-long videos with hybrid mamba-transsfommrs, 2025.
Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Ar- man Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. arXiv preprint drXiv:2410.23266, 2024.
Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios, 2025.
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Patlern Recognition Conference, pp. 26160-26169, 2025.
Enxin Song. Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18232. 2024.
Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video-mmlu: A massive multi-discipline lecture understanding benchmark. arXiv preprint arXiv:2504.1465 2025a.
Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025b.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063. 2024.
Hongyuan Tao, Bencheng Liao, Shaoyu Chen, Haoran Yin, Qian Zhang, Wenyu Liu, and Xinggang Wang. Infinitevl: Synergizing linear and sparse attention for highly-efficient, mulimited-input vision- lisiun-Language models. ArXiv, abs/2512.088829, 2025. httthttts:://e/ semant i eschol 1 ar.. wwwpooooooootoa03333
14

---

Published as a conference paper at ICLR 2026
Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao Song, Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang, Xu Wang, Xinzhe Xu, Xingchen Song, Xing Zhang, Xing Yong, Xin Zhang, Xiangwei Deng, Wenyu Yang, Wenhan Ma, Weiwei Lv, Weiji Zhuang, Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang, Peng Wang, Nuo Chen, Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang, Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang. Chong Ma, Chang Liu, Can Cai, and Bingquan Xia. Mimo-vl technical report, 2025a. URL
org/ abs 2506. 03569
Kimi Team, Angang Du Bohong Yin. Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-yl technical report. arXiv preprint arXiv:2504.0749), 2025b.
Kwai Keye Team. Kwai keye-vl technical report, 2025. URL https://arxiv...
2507. 01949
Qwen Team.
Qwenn:5-wl-7b--nsstuutttee --V 7B- Instruct--Wtg 2024.
Abhishek Tyagi, Arjun Iyer. William H Renninger, Christopher Kanan, and Yuhao Zhu. Dynamic sparse training of diagonally sparse networks, 2025.
Pavlo Vasylenko, Marcos Treviso, and André F. T. Martins. Long-context generalization with sparse attention, 2025.
Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Retake: Reducing tempo- ral and knowledge redundancy for long video understanding, arXiv preprint arXiv:2412.20504, 2024.
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvide02. 5: Empowering video milms with long and rich context modeling. arXiv preprint arXiv:2501.12382 2025a.
Yixuan Wang, Huang He, Siqi Bao, Hua Wu, Haifeng Wang, Qingfu Zhu, and Wanxiang Che. Proxyattn: Guided sparse attention via representative heads. arXiv preprint arXiv:2509.2474 24745, 2025b.
Ziyi Wang, Haoran Wu, Yiming Rong, Deyang Jiang, Yixin Zhang, Yunlong Zhao, Shuang Xu, and Bo XU. Lve: A lightweight compression framework for enhancing vims in long video under- standing, 2025c.
Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem? arXiv preprint arXiv:2502.1150/ 2025.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideohench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828-28857, 2024a.
Yongliang Wu, Bozheng Li, Jiawang Cao, Wenbo Zhu, Yi Lu, Weiheng Chi, Chuyun Xie, Haolin Zheng, Ziyue Su, Jay Wu, and Xu Yang. Zero-shot long-form video understanding through screenplay, ArXiv, abs/2406.1 17309, 2024b. URL.ttts:: :/appi semanticscholar org/
CorpusID: 27:0:11::2000000
Yongliang Wu, Wenbo Zhu, Jiawang Cao, Yi Lu, Bozheng Li, Weiheng Chi, Zihan Qiu, Lirian Su, Haolin Zheng, Jay Wu, and Xu Yang. Video repurposing from user generated content: A large-scale dataset and benchmark. ArXiv, abs/2412.08879, 2024c. URL
semanti cschol aar ortooaororroorrrrt
15

---

Published as a conference paper at ICLR 2026
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13754-13765, June 2025a.
Yongliang Wu, Yizhou Zhou, Ziheng Zhou, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Mingzhuo Yang, and Xu Yang. On the generalization of sft: reinforcement learning perspective with reward rectification. ArXiv, abs/2508. abs/2500.00629, 2025b. URL https://api.
semanticscholern cow.:aarrrrrroooooo:o 280
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations, 2024.
Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. arXiv preprint arXiv:2503. /6428, 2025a.
Weili Xu, Enxin Song, Wenhao Chai, Xuexiang Wen, Tian Ye, and Gaoang Wang. Auro- ralong: Bringing mns back to efficient open-ended video understanding. arXiv preprint arXiv:2507.02591, 2025b.
Ran Yan, Youhe Jiang, and Binhang Yuan. Flash sparse attention: An alternative efficient imple- mentation of native sparse attention kernel, 2025.
Chenyu Yang. Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang. Lewei Lu,, and Jifeng Dai. Pvc: Progressive visual token compression for unified image and video processing in large vision-language models. arXiv preprint arXiv:2412.09613 2024.
Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces, 2025a. URL
https://arxiv. org. http:///b///2/a0 2tt.b///2221..44..
Lijie Yang, Zhihao Zhang, Arti Jain, Shijic Cao. Baihong Yuan, Yiwei Chen, Zhihao Jia, and Ravi Netravali. Less is more: Training-fres sparse attention with global locality for efficient reasoning, 2025b.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-19802, 2025c.
Sihan Yang, Runsen Xu, Chenhang Cui, Tai Wang, Dahua Lin, and Jiangmiao Pang. Vflowopt: A token pruning framework for Imms with visual information flow-guided optimization. arXiv preprint arXiv:2508.05211, 2025d.
Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-effficinnt implementatiins of linear attention mechanism, January 2024. URL htttt:/://gtthuu.coo///lleee Flash-Iirear-attentiio
Xu Yang, Yongliang Wu, Ming-Hsuan Hsuan Yang. Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. ArXiv, abs/2305. 14800, 2023. URL https:
//api. semant www.eetraccccaa......mmoo crrpchoolll..rgoooorrott...... 2583::::2258555555
Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla, Aashu Singh, Shlok Kumar Mishra, Lizhu Zhang, and Mengye Ren. Streammem: Query-agnostic kv cache memory for streaming video understanding, 2025e.
Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li. Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, Lingpeng Kong, Qi Liu, Yuanxing Zhang, and Xu Sun. Timechat-onling 80
16

---

Published as a conference paper at ICLR 2026
Chao Yuan, Yang Yang, Yehui Yang, and Zach Cheng. Date: Dynamic absolute time enhancement for long video understanding, 2025a. URL htttps://axxi.......m wwwpaawwwwww..aaaaaooooaoaooor org/ abs///bs/22/ 0aabs//509.00m 09263
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang. Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025b.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089 2025c.
Zhihao Zhan, Jianan Zhao, Zhaocheng Zhu, and Jian Tang. Overcoming long-context limitations of state-space models via -eeenxt-eeennent sparse attention, 2025.
Boqiang Zhang, Kehan Li. Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang. Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 2025a.
Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang, Yaqi Xie, Katia Sycara, Haitao Mi, and Dong Yu. Vsean: Rethinking visual token reduction for efficient large vision-language models. arXiv preprint arXiv:2505.22654. 2025b.
Dell Zhang, Xiangyu Chen, Jixiang Luo, Mengxi Jia, Changzhi Sun, Ruilong Ren, Jingren Liu, Hao Sun, and Xuelong Li. Infinite video understanding. arXiv preprint arXiv:2507.09066, 2025c.
Haichao Zhang and Yun Fu. Neural discrete token representation learning for extreme token reduc- tion in video large language models, 2025.
Hanzhi Zhang, Heng Fan, Kewei Sha, Yan Huang, and Yunhe Feng. Dam: Dynamic attention mask for long-context large language model inference acceleration, 2025d.
Jintao Zhang, Rundong Su, Chunyu Liu, Jia Wei, Ziteng Wang, Haoxu Wang, Pengle Zhang, Huiqiang Jiang, Haofeng Huang. Chendong Xiang, Haocheng Xi, Shuo Yang, Xingyang Li, Yuezhou Hu, Tianyu Fu, Tianchen Zhao, Yicheng Zhang, Boqun Cao, Youhe Jiang, Chang Chen, Kai Jiang, Huayu Chen, Min Zhao, Xiaoming Xu, Yi Wu, Fan Bao, Jun Zhu, and Jianfei Chen. A survey of efficient attention methods: Hardware-efficienl, sparse, compact, and linear attention. 2025e.
Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kaiwen Zheng, Haocheng Xi, Ziteng Wang, Hongzhou Zhu, Min Zhao, Ion Stoica, et al. Sla: Beyond sparsity in diffusion transformers fine-tunable sparse- -parse-linear attention. arXiv preprint (tti::5509220000 2025f.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.122 12772 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024b.
Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. Vsa: Faster video diffusion with trainable sparse attention, 2025g.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024c. URL 1o httpbb/io//oooo//2-0o00000
30 alllalavaaaee -ava-nneee next video/
Yuanhan Zhang, Jinming Wu, Wei Li. Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.0271 2024d.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruc- tion tuning with synthetic data, 2024e. URL
02713
17

---

Published as a conference paper at ICLR 2026
Yunzhu Zhang, Yu Lu, Tianyi Wang, Fengyun Rao, Yi Yang. and Linchao Zhu. Flexselect: Flexible token selection for efficient long video understanding. arXiv preprint wxXiv:5066000933, 2025h.
Weilin Zhao, Zihan Zhou, Zhou Su, Chaojun Xiao, Yuxuan Li, Yanghao Li, Yudi Zhang, Weilun Zhao, Zhen Li, Yuxiang Huang, et al. Infllm-v2: Dense-sparse switchable attention for seamless short-to-long adaptation. arXiv preprint arXiv:2509. 24663, 2025.
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swiftra scalable lightweight infrastructure for fine-tuning, 2024.
05517
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv e-prints, pp. 2406, 2024.
Orr Zohar, Xiaohan Wang, Rui Li, Andrés Marafioti, Miquel Farré, Merve Noyan, Leandro von Werra, Serena Yeung-Levy, and Thomas Wolf. Apollo2: Exploring the long-video frontier of large multimodal models, 2025.
18

---

Published as a conference paper at ICLR 2026
Appendix
The supplementary material is structured as follows:
literature review about the related works in Section A
The training settings for VideoNSA Section B
The introduction of the used evaluation benchmarks and settings in Section C
More results on long-form video benchmarks in Section D
More results on temporal reasoning benchmarks Section
More results on spatial understanding benchmarks in Section
Results additional video understanding benchmarks in Section G
Visualization of attention pattern in each branch in Section
More results on branch combination in Section!! !|
More results on information scaling study in Section
Additional context-length scaling results of Qwen2.5-V VL in Section
More results on attention scaling study in Section L
Theoretical analysis context length and attention budget scaling in Section
Full gate values distribution in Section N
More inter-head gate similarites visualization in Section O
Benchmark-level gating analysis and PCA visualization in Section
Additional analysis of training and inference efficiency in Section
Additional analysis CMP latency bottleneck in Section R
More analysis about attention sinks on various sparse attention settings can be found in Section S
Comparison of sparsity patterns between text-only NSA and VideoNSA in Section
Visualization of attention sinks in dense attention in Section
A RELATED WORK
A.1 EFFICIENT VIDBO UNDERSTANDING
Video understanding systems typically convert videos into long sequences of vision tokens, which can easily exceed GPU memory and slow down inference as the video length grows. To address this, existing work mainly address this by token compression, alternative sequence modeling, and KV- cache compression. One important line of work emphasizes token compression. Spatial or temporal token merging methods (Wang et al. [205cc] /Zhag & Fu 2025 Li et al. 20251 (002f)Jiane Jiang et al. (202aa
Li et al. 2025c Shao et al. [2255] [oog et al. 2024 Chai al. 2024 Liao et al. [2025] Wu et al.
2025b 2024b 2025a 2024c Cao al. 2024 Li al. 2025a Yang et al. 2023 Kong al. 2025
progressively discard redundant content, while -uestion-/task-ane aware strategies (hang et al. 2025b
Dong et al., 2025 Yao al.) 2025 Song et al., 2025b) tailor retained tokens to the query. These approaches substantially lower FLOPs but still rely on dense attention once tokens are merged. Beyond pure self-attention, Mamba-based or hybrid architectures (Jang al.) 2025a Ren al.) 2025 Xu et al. 2025b Tao et al. 2025) inject state-space or recurrent modules to approach linear- time inference while preserving long- long-range dependencies. Also, there exists approach to design data efficient systems for further fine-tuning (Li et al., (025d). Another direction targets the key-value cache during decoding via task-aware sparsification and streaming-frieet friendly memory (Qin et al., 2025a] Ning et al.) 2025) Kim et al.] 2025) Yang et al.) 2025e, Liu et al., 2025) reduce memory and improve throughput, yet prefill still scales quadratically with sequence length. In contrast to methods that mostly decide where to drop or compress tokens, our approach systematically probe the effectiveness of native sparse attention Yuaan et al.) (025b) that restructures attention itself to be learnalle and sparse from the ground up. VideoNSA attains near-linear scalability up 128K tokens and processes over 10,000 frames on a single GPU, outperforming compression-onl pipelines on long-video understanding. temporal reasoning, and spatial understanding tasks.
19

---

Published as a conference paper at ICLR 2026
A.2 SPARSE ATTENTION MECHANISM
Sparse attention is a central strategy for efficient long-context modeling in language and multi- modal systems. Surveys (Zhang al. 2025e) categorize approaches into paltern-based vs. dy- namic/learned. Pattern- based sparsity. Methods such as Longformer (Beltagy et al. 2020).
StreamingLLM (Xiao et al. 2024), and TriangleMix (He et al., 2025) preseribe fixed striddt patterns that can be applied training-frees recent multimodal works (Zhang et al. 2025d Yang et al., 2025b) follow similar principles, while hardware-efficient kernels like Flash Sparse Attention (Yan) al. 2025) further reduce prefill latency. InfLLM-V2 (Zhao et al. 2025) uses switchable dense sparse attention to smoothly adapt models from short to long sequences while maintaining consis- tency and achieving efficient acceleration with high performance. ProxyAttn (Wang et al.) 2025b)
uses representative heads for fine-grained block importance estimation, enabling faster sparse at- tention with minimal performance loss. Dynamic and trainable sparsity. Content- gradient- adaptive mechanisms select important connections (e.g., diagonal selection (Tyagi et al. 2025) or
lag-relative strategies (Liang a 2025) ); trainable sparse attention improves long-context reason- ing (Gao et al. 2025 Vasylenko et al. 2025 Gao et al. 2024), diffusion-based vidco generation (Zhang al. (025g), and state-space models (Zhan et al. 2025). SLA (Zhang et al. 2025f) decom-
poses attention weights into critical, marginal, and negligible parts, combining sparse and low-rank acceleration to greatly reduce computation while preserving generation quality. Hybrid approaches such as RocketKV (Behnam et al. 2025) combine token/cache compression with learned sparsity, and MMInference (Li et al., 2025g) accelerates modality-aware sparse prefill for VLMs. Despite these advances, most techniques are optimized for text or short multimodal contexts and do not directly address the ultra-long, highly redundant spatio-temporal structure of videos. VidcoNSA unifies block-wise compression, salient block selection, and a sliding-wi fidding-window branch under learn- able gates that dynamically allocate computation across three native sparse branches (Yuan et al., 20226). This end-to-end, data-driven design preserves critical global/local dependencies while scal- ing nearly linearly in both time and memory.
B DETAILED TRAINING SETTINGS
Training hyperparameters for VideoNSA are shown in Table We filter subset of LLaVA- 178K (Zhang et al. 2024e) as the training data. For each video, we uniformly sample at frames second and retain only those with 350-550 frames, resulting in 216K video answer pairs from the original 961K pairs in LLa VAVViico-178K (Zhang et al. 2024c).
Table 4: Training
for VideoNSA.
Flyper-parametens
Fine-tuning
trainable parameters
ViT + MLP LLM
warmup schedule
linear
warmup start factor
le-5
warmup ratio learning rate schedule
0.1
cosine
optimizer
Adam (Loshchilov Hutter, 2017)
optimizer hyper-parameters
81, B2 = (0.9, 0.999)
weight decay
0.01
max norm
epoch
peak learning rate
1e-6
total equivalent batch size
32
C EVALUATION BENCHMARKS AND SETTINGS
We list all the hyper-parameters and prompt used for evaluation shown in Table
20

---

Published as a conference paper at ICLR 2026
Table Evaluation settings summary for each benchmarks. For all benchmarks we set temperature, top number of beams to O, 0, respectively. # TPF stands for the vision tokens per frame, and # stands for the number of sampling frames.
Benchmark
TPF
F
Max New Tokens
Long Vich Wu cl al.
2024a
512
256
32
LongTin TimeScope Zohar
2025
128
512
16
TimeScope (Zohar al. 2025
2048
16
MI.VUtaat (Zhou et 2024
512
Tomato (Shangguan ct a
20124
4FPS
256
1024
VSIBench (Yang al. 20 12
256
128
16
D MORE RESULTS ON LONG-FORM VIDEO BENCHMARKS
Table Cone results across baselines. Metries include overall accuracy and task -secifc scores across different steps. Flash Attn stands for Sowen2.5-VL-1 (Qwen et al. 2025) accelerated by Flash Infer, and Flash Attn SFT stands our fine-tuning version.
18000
28800
36000
Method
Overall
OCR
QA
Temporal
OCR
QA
Temporal
OCR
QA Temporal
Flash Attn
40.7
54.0
42.0
22.0
48.0
60.0
24.0
48.0
58.0
10.0
Flash Attn + SFT
40.2
46.0
30.0
34.0
46.0
44.0
36.0
52.0
44.0
20.0
AWQ
XAttn
41.1
52.0
56.0
30.0
54.0
52.0
6.0
52.0
64.0
4.0
MInference
44.4
64.0
56.0
26.0
58.0
60.0
8.0
56.0
66.0
6.0
tri-shape
28.4
34.0
36.0
12.0
48.0
48.0
0.0
44.0
32.0
2.0
FlexPrefill
39.1
52.0
46.0
24.0
46.0
56.0
14.0
46.0
66.0
2.0
FastV
35.6
36.0
50.0
16.0
44.0
50.0
44.0
64.0
12.0
VisionZip
31.1
38.0
32.0
14.0
56.0
46.0
0.0
44.0
46.0
4.0
VScan
40.4
48.0
52.0
24.0
50.0
52.0
22.0
46.0
64.0
6.0
VidcoNSA
44.4
50.0
54.0
30.0
54.0
72.0
0.0
48.0
76.0
16.0
Table 7: Long results across baselines. Metrics include overall accuracy and task- specific scores across different steps. Flash Attn stands for Wwvn2.55VL-7B (Qwen et al.) 2025) accelerated by Flash Infer, and Flash Attn + SFT stands for our fine-tuning version.
Method
E3E
Flash Alin
69.9
70.5
37.6
63.2
55.6
$2.4
63.1
63.1
646
72.7
61.7
65.6
Flash Artn + SFT
57.8
55.8
38.4
65.6
61.7
7399
56.9
65.
55
57.5
57.5
56.6
62.5
$1.2
55.4
66-2
302
63.
74.
56.0
64.6
52.0
AWQ
59.0
60.0
34,2
72.0
64.
68.2
59.7
57.6
$2.7
50.7
69.0
$2.6
639
$7.3
61.5
67.7
43.3
62.0
738
63.0
67.7
50.9
XAIln
59.1
59.4
66.0
67.2
$7.3
58.1
55
53.8
64.5
62.2
61.
56.3
65.2
66.7
41.
58.5
75.2
60.7
68.8
50.6
Minierence
mi-shupe
66.0
600
74.6
FlexPrefill
58.4
61.7
31.5
65.6
62.8
71.6
59.7
59.1
SK
52.1
65.5
$1.3
62.5
48.
61.5
42.
633
71.
63.4
58.2
52.1
FastV
57.3
57.3
43.8
64.5
60.6
70.5
52.8
56.1
52.7
48.0
59.8
67.1
56.9
43.
67.7
66.2
402
5
89
61.7
70.9
48.9
VisionZip
52.4
53.2
63.4
66.0
58.0
56,2
50,0
51.
42.5
57.5
47.
583
45.1
56.9
61.5
519
62.
61.7
58.2
46.8
VSean
VidooNSA
60.2
59.9
48.1
65.1
67.6
55.6
56.3
62.2
57.0
63.9
53.3
56.2
71.6
62.7
67.5
72.4
66.3
We take Long VideoBench (Wu et al. 2024a), LongTimeScopp (Zohar et al. 2025), MLVU (Zhou al. 2024), and TimeScope (Zohar al.) (025) representative long-video benchmarks and com- pare against existing token compression and sparse attention methods. As shown in Table Table 7 Table 8. and Table VideoNSA achieves comparable performance without specialized designs. Moreover, we observe that VideoNSA significantly outperforms the baselines on subtasks related to temporal reasoning and on videos of extended length.
21

---

Published as a conference paper at ICLR 2026
Table 8: MLVU results across baselines. Metrics include overall accuracy and task-specific scores across different steps. Flash Attn stands for Qwen2.5-VL VL-7B (Qwen et al., 2025) accelerated by Flash Infer, and Flash Attn SFT stands for our fine-tuning version.
Method
Overall
PlotQA
Needle
Ego
Count
Order
Anomaly Reco
Topic Reason.
SportsQA
TutorialQA
Flash Attn
$1.2
58.0
68.3
52.8
31.7
25.7
46.2
79.1
38.9
48.8
Attn SFT
58.0
58.5
40.0
43.6
81.3
36.1
AWQ
46.0
42.7
53.0
40.9
27.2
50.2
57.0
65.0
38.3
39.2
XAttn
50.2
60.0
64.7
56.5
28.0
29.4
41.6
74.9
39.7
39,9
MInference
49.2
56.0
64.7
48.9
29.7
26.6
41.6
77.1
39.7
39.9
tri-shape
49.2
64.7
48.9
29.7
26.6
41.6
77.1
39.7
39,9
FlexPrefill
42.7
36.6
72.6
29.7
32.2
FastV
41.8
44.0
45.0
47.2
18.3
30.0
46.2
84.6
28.6
32.2
VisionZip
33.1
30.0
26.7
30.2
6.7
22.9
41.0
68.1
19.7
26.4
VScan
48.1
58.0
63.3
50.9
28.3
24.3
43.6
78.0
47.2
39.5
VideoNSA
51.3
34.6
Table 9: TimeScope results across baselines. Metrics include overall accuracy and task-specific scores across different steps. Flash Attn stands for Qwen2.5-VL-7B (Qwen et al. (025) accelcrated by Flash Infer, and Flash Attn SFT stands for fine-tuning version.
Method
Overall
60
120
180
300
600
1200
1800
3600
7200
10800
Flash Attn
81.0
96.7
96.0
96.0
94.7
94.0
88.0
82.0
68.7
52.7
41.3
Flash Attn + SFT
76.8
96.7
96.7
96.0
95.3
90.7
78.0
78.0
54.7
41.3
40.7
AWQ
XAttn
83.1
94.0
93.4
93.4
92.0
92.7
89.4
82.7
72.7
70.7
50.7
MInference
82.7
93.4
94.0
93.4
92.0
92.7
87.4
80.0
74.0
70.0
50.0
tri-shape
82.7
93.4
94.0
93.4
92.0
92.7
87.4
80.0
74.0
70.0
50.0
FlexPrefill
83.0
96.7
96.0
96.7
95.3
96.0
95.3
86.0
77.3
55.3
35.3
FastV
46.5
82.7
76.0 74.0
54.0
32.7
32.7
29.3
29.3
34.0
20.0
VisionZip
43.5
92.0
66.7 60.0
43.3
35.3
26.0
30.7
29.3
28.0
23.3
VScan
80.3
96.7
96.7 96.0
93.3
92.7
89.3
81.3
60.0
55.3
41.3
VidcoNSA
83.7
96.7
96.0 97.4
92.0
85.4
91.6
89.3
73.3
63.3
52.0
E MORE RESULTS ON TEMPORAL REASONING BENCHMARKS
We take Tomato (Shangguan et al. as the representative temporal reasoning benchmark and compare against existing token compression and sparse attention methods. As shown in Ta- ble TO, Aacchvess comparable performance without specialized designs. Moreover, observe that VideoNSA significantly outperforms the baselines on subtasks including object count- ing, shape description, and human actions.
MORE RESULTS ON SPATIAL UNDERSTANDING BENCHMARKS
We take VSIBench YYng et al. 2025a) the representative spatial understanding benchmark and compare against existing token compression and sparse attention methods. As shown in Ta- ble TI VideoNSA achieves comparable performance without specialized designs. Moreover, observe that VideoNSA significantly outperforms the baselines on subtasks including object relative direction, route planning, and object size estimation.
G RESULTS ON ADDITIONAL VIDEO UNDERSTANDING BENCHMARKS
We conduct additional experiments LSDBench Quu al. 2025 and VideoEvalPro Ma al. 2025) to compare VideoNSA and other training-free free sparse attention baselines, demonstrating the consistent advantage of VIdeoNSA in multiple video understanding tasks.
22

---

Published as a conference paper at ICLR 2026
Table 10: Tomato results across baselines. Metrics include overall accuracy and task-specific scores across different steps. Flash Attn stands for Qwen2.5- VL--B (Qwen et al. 2025) accelerated by Flash Infer, and Flash Attn SFT stands for our fine-tuning version.
Method
Overall
Direction
Count
Rotation
Shape Trend
Vel. Freq.
Visual Cues
Human Simulated
Object
23.6
21.9
Flash Atta + SFT
21.7
19.6
23.3
18.2
26.0
18.1
38.6
18.8
18.0
25.6
XAttn
21.4
22.1
22.9
19.6
17.9
17.1
42.9
13.5
21.5
26.8
MInference
23.0
22.6
27.1
22.0
20.0
37.1
16.6
20.6
29.6
FlexPrefill
23.7
23.3
25.0
22.7
22.0
21.4
35.7
17.1
22.7
29.9
FastV
21.6
20.6
26.0
20.3
23.3
12.7
17.1
24.2
25.6
VisionZip
17.6
16.8
21.0
19.3
19.0
30.0
14.8
21.5
22.3
VScan
23.6
25.3
21.9
19.9
24.2
20.5
42.9
18.7
21.9
28.7
VideoNSA
26.5
21.6
31.5
22.0
25.6
23.3
40.0
21.7
23.6
29.3
Table 11: VSIBench results across baselines. Metrics include overall accuracy and task-specific scores across different steps. Flash Attn stands for Qwen2.5-VL-78 (Qwen et al. (025) accelerated by Flash Infer, and Flash Attn SFT stands for fine-tuning version.
Method
Overall
Obj. Order
Abs, Dist.
Counting Rel. Dist.
Size Est.
Room Est.
Route Plan.
Rel. Dir.
Flash Attn
29.7
25.7
16.0
20.5
34.7
49.5
22.5 30.4
30.4
38.5
Flash Atm + SFT
30.6
31.9
14.2
12.3
40.4
46.6
30.9
37.8
AWQ
XAttn
35.0
18.1
37.6
52.1
30.0
32.5
37.4
MInference
36.6
36.5
18.2
43.9
39.4
48.5
38.8
30.0
37.7
tri-shape
36.5
35.7
18.2
44.3
39.8
48.6
38.8
29.0
37.7
FlexPrefill
34.9
34.1
21.6
35.1
39.3
51.8
29.7
30.4
36.8
FastV
34.0
31.7
21.7
26.1
36.2
47.8
35.0
33.5
40.1
VisionZip
32.1
28.8
17.9
28.8
36.5
48.9
26.9
29.4
39.3
VSean
34.4
21.9
40.0
$1.9
30.4
VideoNSA
36.0
25.5
19.0
42.5
35.4
$4.0
30.1
37.5
43.6
H VISUALIZATION OF ATTENTION PATTERN IN EACH BRANCH
We visualize the attention patterns of the last layer across the three branches in Figure 10 Figure
Figure [2] and Figure [3 together with the final attention output, as representative examples. The compression branch reduces redundancy to preserve salient information, the selection branch high- lights task-relevant regions with sparse activations, and the sliding window branch enforces local temporal coverage by focusing short-range dependencies. These complementary roles collec- tively shape the final attention output.
I MORE RESULTS ON BRANCH COMBINATION
In this section, we report detailed results of different branch combinations across three domains, in- cluding long video understanding (Table 14, Tavke 15| Table and Table temporal reasoning (Table 18), and spatial understanding (Table 19). The corresponding performances are summarized in the table, which highlights how the of individual branches their combinations affects down- stream tasks.
J MORE RESULTS ON INFORMATION SCALING STUDY
Figure 16 shows the scaling performance of VideoNSA under different context allocation strategies and MI.UU. Both benchmarks were trained with maximum context length of 32K tokens, yet their performance consistently improves when scaled to 64K, beyond the train- ing budget. On Long cooeS (Zohar al.) 2025), the best results emerge around 512 frames with 128 TPF at 64K tokens, underscoring the dataset's reliance on extended temporal coverage for long-hou mor-zooizon reasoning. contrast, MLVU U (Zhou et al.) 2024) also peaks at 64K with the same al- location, but its contours smoother, and competitive performance extends across broader range of frame-token trade-offs. This suggests that while Long TineTTimepppp demands aggressive temporal scaling, MLVU benefits from balanced distribution of temporal and spatial information.
on LoongTimeSeene
23

---

Published as a conference paper at ICLR 2026
Table 12: Results on LSDBench (Qu al.
2025).
Model
Accuracy
LongVA VA (Zhang et al. 2024b
32.5
Long (Chen al. 2024b
49.8
Intern (Chen et al.
2024c
50.1
Owen2.5-VL-7B (Qwen et al.
2025)
52.2
Qwen2.5-VL-7B-SFFT
52.5
Sparse Attention Methods
Tri-Shape (Li et al.. 2024c
49.5
+ MInference (Jang et al.
2024
49.5
FlexPrefill (Lai et al.. 2025)
52.3
XAttention (Xu et al.
2025a
51.3
VideoNSA
55.2
Table 13: Results on VideoEvalPro (Ma et al. 2025). stands Holistic Perception, HR
stands for Holistic Reasoning, LR stands for Local Reasoning, LP stands for Local Perception.
Model
HP
HR
LR
LP
Overall
Long VA (Zhang et al. 2024b)
20.5
6.8
19.0
9.5
16.5
Video-X0 XL (Shu et al. 2025
22.3
15.0
18.2
10.2
18.6
Intern VL2.5 (Chen et al. 2024c
28.8
19.7
21.5
16.7
24.6
Qwen2.5-VL-7B Qwen et al.
2025
33.9
15.6
24.8
17.8
27.7
Swen2.5-VL-7B-SHP
34.5
15.8
25.3
18.2
28.3
Sparse Attention Methods
Tri-Shape (Li et al. 2024c
34.1
16.3
25.1
20.0
28.4
+ MInference (Jiang el al. 1024
32.3
17.1
27.7
16.7
26.0
+ FlexPrefill (Lai et al. 2025
33.0
15.9
26.3
19.8
28.3
+ XAttention (Xu et al. 2025a
34.5
16.6
25.6
20.5
28.9
VideoNSA
35.4
16.9
26.3
19.1
29.4
Table 14: LongVideoBrande results across different branch selection strategy. Metrics include over- all accuracy and task- aask-specffic scores across different steps.
Madied
600
E3E
120
T2A
VideoNSA
47.6
NSA-SEC
49.0
69
46.0
36.8
48.1
50.3
NSA-SWA
49.1
50.7
37.0
52.7
56.4
59.1
51.4
48.5
43.2
45.
55.2
42.1
48.6
45.1
46.2
61.5
30.9
45.6
54.1
55.4
48.7
46.5
NSA CMPSLC
49.4
49.5
34.3
55.9
61.7
5800
$6.9
48.5
47.3
41.
56.3
35.5
52.8
47.6
46.2
55.4
34,
41.8
54.1
63.0
18.2
48.2
NSA SECSWA
49.3
48.8
329
58.1
61.7
55.7
52.8
47.0
46.0
46.6
54,0
34.2
48.6
47.6
47.7
54.0
35.1
48.1
54.1
64.2
49.2
48.2
NSA CMPSWA
19.3
34.3
59.6
48.7
40.8
55.4
addition to overall scaling trends, we further report detailed subtask-level results under differ-
ent allocation settings in Table 20 Table 21
Table 22
Table 23
Table
24
and Table 25
K ADDITIONAL CONTEXT-LENGTH SCALING RESULTS OF QWEN2.5-VL VL
We include Table 26 and Table 27 to further illustrate the long-context behavior of the base model. Since Qwen2.5-Vi 7B Qwen et al. 2025) has a maximum context window of 128k, its modeling ability tends to become less stable when approaching this upper bound. As shown in Figure K
24

---

Published as a conference paper at ICLR 2026
Layer Compression (o.cmp)
Nere1
Peud
Herd
Houn
Norkoe Feedo
151
Paitinn
X53 130
20 10
Herd
Feed
Feed
He
Fead
2-6
700 AM
700 AM
430
NO
-30
Biy Poulica
Polile
Fead23
Head 2B
720
M0
200
6.00
MC3
420
423
4/0 100
Figure 10: Attention pattern of the compression branch in the final layer of VideoNSA.
Layer Sclection
Fend
Her13
Feed
Held
Head
Head
200
730
R0
3KC
80
200
496
60
240
406
6/1
20
20
400
0/
200
480
+0
Key holiine
Kiw Powit3
Km Pailien
On Priflen
Perlic
Rour14
Pau
Houu
Heud
730
700
300
209
130
80
200
63
260
4/C
20
430
20 180
Figure 11: Attention pattern of the selection branch in the final layer of VideoNSA.
Qwen2.5-Vi VL (Qwn et al., 2025) often peaks at 64k and slightly declines at 128k across several benchmarks. In contrast, VideoNSA maintains stable or stronger performance at 128k, demonstrat- ing that the observed 64k > 128k phenomenon arises from backbone limitations rather than the proposed sparse architecture.
L MORE RESULTS ON ATTENTION SCALING STUDY
Figure e6lealuaee the scaling behavior of VideoNSA under different attention allocation strategies, where the x-axis denotes the sliding window size (log scale), the y-axis shows the block count, and the size and color of each marker reflect performance, with the dashed blue curve indicating config- urations of equal attention budget and arrows marking the training setting well reduced-budee budget configurations (3.6% and 1.8%); on Long VideoBench, performance peaks near the training con- figuration and degrades when allocating excessive budget to local attention through larger sliding
25

---

Published as a conference paper at ICLR 2026
28 Sliding Window (o,sed)
Foad1
NerF1
Peud
Heue
Hoat
200
201
X53
20
Paiteen
Herd
Feed
Feed
Herd
read 14
60%
4-
69
300 FM
PE0 AV
430
6/
4M0
-30
Biy Poulica
Peclaan
Poilac
Head 23
200
6.0
M22
120
423
Figure 12: Attention pattern of the sliding window branch in the final layer of VideoNSA.
Layer 28 Final Output
kay Proliie
Kay
Poifeen
Perlics
Faads
Feel111
Her111
Head 11
209
220
380
203 Poshle
hewd
Fowe
Feed
Hor
Fead
150
4.3
24
80
700
656
455
205
A/C
200
130
4/2
24
Day mudiii
Kay Paolian
Oa Poutlan
Figure 13: Attention pattern of the final vision attention output in the final layer of VideoNSA.
windows, while the best configuration achieves strong results with only 3.6% of the full budget, and on TimeScope, performance even more sensitive, with larger sliding windows quickly reducing accuracy whereas maintaining more global blocks yields superior outcomes, and overall the results confirm that training allocations are well balanced, that prioritizing global attention is consistently more effective than enlarging local windows under equal budget, and that VideoNSA sustains lead- ing performance with as little as 3.6% or less of the full attention cost, demonstrating both efficiency and hardware awareness.
In addition to the overall scaling trends, we further report detailed subtask-leve results under differ-
ent allocation settings in Table 20 Table 21 Table 22 Table 23 Table 24 and Table 25
26

---

Published as a conference paper at ICLR 2026
Table 15: LongTimeScope results across different branch selection strategy. Metries include overall accuracy and task-specific scores across different steps.
18000
28800
36000
Method
Overall
OCR
QA
Temporal
OCR
QA
Temporal
OCR
QA Temporal
VideoNSA
Test SFT
40.9
52.0
42.0
42.0
48.0
62.0
18.0
42,0
50.0
12.0
NSA-CMP
25.1
20.0
22.0
24.0
38.0
40.0
0.0
34.0
34.0
14.0
ASA-SLC
37.1
30.0
38.0
40.0
50.0
58.0
12.0
42.0
44.0
20.0
NSA-SWA
29.8
34.0
34.0
22.0
36.0
46.0
4.0
34.0
46.0
12.0
NSA-CMPSLC
32.4
36.0
34.0
24.0
46.0
54.0
8.0
42.0
36.0
12.0
NSA SSA-SLCSWA
34.4
38.0
46.0
56.0
8.0
38.0
36.0
16.0
NSA AAA-CMPSWA
31.6
30.0
38.0
20.0
40.0
52.0
16.0
36,0
36.0
16.0
VideoNSA
44.4
50.0
54.0
30.0
54.0
72.0
0.0
48.0
76.0
16.0
Table 16: TimeScope results across different branch selection strategy. accuracy and task-specific scores across different steps.
Metrics include overall
Method
Overall
60
120
180
300
600
1200
1800
3600
7200
10800
Full Atin
81.0
96.7
96.0 96.0
94.7
94.0
88.0
82.0
68.7
52.7
41.3
Flash Attn
81.0
96.7
96.0
96.0
94.7
94.0
88.0
82.0
68.7
52.7
41.3
Flash Attn + SFT
76.8
96.7
96.7
96.0
95.3
90.7
78.0
78.0
54.7
41.3
40.7
AWQ
XAttn
83.1
94.0
93.4
93.4
92.0
92.7
89.4
82.7
72.7
70.7
50.7
MInference
82.7
93.4
94.0
93.4
92.0
92.7
87.4
80.0
74.0
70.0
50.0
tri-shape
82.7
93.4
94.0 93.4
92.0
92.7
87.4
80.0
74.0
70.0
50.0
FlexPrefill
83.0
96.7
96.0 96.7
95.3
96.0
95.3
86.0
77.3
55.3
35.3
FastV
46.5
82.7
76.0 74.0
54.0
32.7
32.7
29.3
29.3
34.0
20.0
VisionZip
43.5
92.0
66.7
60.0
43.3
35.3
26.0
30.7
29.3
28.0
23.3
VScan
80.3
96.7
96.7
96.0
93.3
92.7
89.3
81.3
60.0
55.3
41.3
Retake
AdaRetake
SFT + Test NSA
81.0
96.7
96.0
96.0
94.7
94.0
88.0
82.0
68.7
52.7
41.3
NSA + Test SFT
83.0
96.7
95.3
94.0
93.3
94.0
90.7
87.3
76.7
54.7
47.3
NSA-CMP
41.5
82.0
74.0
65.3
59.3
17.3
25.3
19.3
26.7
27.3
18.0
NSA-SLC
63.7
92.0
86.0
86.7
78.0
66.7
57.3
51.3
40.7
38.0
40.0
NSA-SWA
59.3
NSA-CMPSLC
57.3
88.7
80.0
73.3
73.3
46.7
44.7
48.7
42.7
43.3
32.0
NSA-SLCSWA
65.2
92.0
89.3
89.3
79.3
66.0
59.3
50.0
41.3
40.7
44.7
NSA-CMPSWA
57.3
88.7
80.0
73.3
73.3
46.7
44.7
48.7
42.7
43.3
32.0
VideoNSA
83.7
96.7
96.0
97.4
92.0
85.4
91.6
89.3
73.3
63.3
52.0
M THEORETICAL FOUNDATIONS OF SCALING BEHAVIOR
In Section we perform two scaling experiments along context length and attention budget. We observe that VideoNSA exhibits strong extrapolation ability on context length: although trained with only 36K tokens, it can generalize to 128K at test time, achieving the best performance at 64K. In contrast, when scaling the attention budget, even a small reduction 3.6% of attention computation already delivers outstanding performance, and further increasing the visible-token count does not yield additional gains. To clarify these phenomena, we provide theoretical interpretations from routing- path stability and the geometric structure of RoPE (Su al. 2024
Routing-pate Stability. Recent work (Huang al. 2025 indicates that model's 's ability to main- tain performance on long sequences depends critically on the stability of its attention routing struc- ture across positions. In the standard attention mechanism, the attention weight from the query
27

---

Published as a conference paper at ICLR 2026
Table 17: MLVU results across different branch selection strategy. Metrics include overall accuracy and task-specifie scores across different steps.
Method
Overall
PlutQA
Needle
Ego
Count
Order
Anumalyy Reco
Topic Reason.
SpurisQA
TutorialQA
NSA Test SFT
51.6
56.0
61.7
66.0
31.7
28.6
51.3
80.2
36.1
32.6
NSA-CMP
43.9
36.0
35.0
42.9
24.3
30.8
80.2
30.6
NSA-SLC
47.7
50.0
50.0
52.4
22,9
33.3
74.7
33.3
NSA-SWA
40.2
40.0
40.0
41.5
15.0
24.3
30.8
76.9
36.1
34,9
NSA-SLCSWA
42.4
42.0
48.3
25.7
38.5
75.8
33.3
NSA-CMPSWA
43.4
46.0
40.0
43.4
18.3
35.7
33.3
82.4
27.8
32.6
Table 18: Tomato results across different branch selection strategy. Metries include overall accuracy and task-specific scores across different steps.
Method
Overall
Direction
Count
Rotation
Shape & Trend
Vel. & Freq.
Visual Cues
Human
Simulated
Object
NSA Test SFT
23.4
21.3
29.1
17.5
25.1
20.0
40.0
19.3
19.3
28.5
NSA-CMP
23.3
22.1
29.5
17.1
24.7
20.5
34.3
19.2
22.7
27.3
24.0
21.3
32.2
16.4
26.0
22.9
32.9
19.8
28.7
NSA-SWA
24.0
21.3
32.2
16.4
26.0
22.9
32.9
19.8
21.5
28.7
NSA-CMPSLC
23.5
20.8
29.8
18.5
22,9
23.8
34.3
19.0
26.8
25.8
NSA-SLCSWA
23.0
18.5
22.4
32.9
26.8
NSA-CMPSWA
24.5
23.1
30.8
18.9
25.1
22.9
32.9
21.0
23.6
28.0
Token Frame (TPF)
512
Frame (TPF)
Taken Budget Constraint
Taken Budget Constraint
Performance
Performance
35
45
30
128K
25
Token
200
512 Frames, 128 TPF
128
512 Frames, 128 TPF
64
100
400
800
1200
1600
2000
200
600 Total Frames
400
800
1000
Total Frames
(a) Information Scaling of LongTimeScope
(b) Information Scaling of MLVU
Figure 14: Scaling Performance of VideoNSA under Different Context Allocation Strategies. We highlight the token budget constraint indicate settings with equal context length, and annotate the best-performing configuration under each benchmark.
vector Qn position to the key vector Zj at position j is defined as
exp(Z]Q. Qn)
Attn Σ. exp(Z, Q.)
Here, Qn and Zy denote the query and key representations at positions n and j. respectively. the model can consistently focus its attention on the task-relevant target token set T during inference, then (i) Ejer Attn should dominate across different positions; and (ii) the attention assigned to the same key token should remain nearly unchanged under positional shifts, i.e.
A Attm, Attn R O,
where A; measures deviation routing paths across positions long sequences.
When we scale the context length using dense temporal and spatial sampling, the sparse-attention pattern mask structure M remain unchanged, which means the model continues to use the rout- ing structure learned during training while simply facing a larger pool of candidate evidence. Since denser sampling mainly introduces redundant or finer-grained details, the model treats these tokens auxiliary evidence, leaving the target tokens and their relative attention weights essentially unchanged. Consequently. the overall routing-pan path structure is preserved, A; remains small, and the model maintain or even improve its performance longer context lengths.
In contrast, attention- budget scaling explicitly modifies the set visible tokens in the sparse- attention mechanism by replacing the original mask M with new mask M'. The effective query
28

---

Published as a conference paper at ICLR 2026
Table 19: VSIBench results across different branch selection strategy. Metries include overall accu- racy and task-specific scores across different steps.
Method
Overall
Obj. Order
Abs, Dist.
Counting
Rel. Dist.
Size Est.
Room Est.
Route Plan.
Rel. Dir.
Test SFT
24.3
31.2
49.8
32.2
37.2
NSA-CMP
29.2
19.9
16.3
12.6
29.3
48.7
26.7
38.1
41.7
NSA-SLC
27.6
18.0
10.9
17.3
32.0
47.8
24.8
32.0
38.1
NSA-SWA
29.8
22.8
15.6
17.4
32.3
49.8
27.2
33.5
39.4
NSA-CMPSIC
29.4
19.9
16.3
15.1
31.0
51.1
25.5
33.5
42.6
NSA-SLCSWA
29.1
19.9
12.2
18.5
31.4
49.6
26.5
34.0
40.4
NSA-CMPSWA
30.3
22.5
15.6
15.3
31.1
52.5
26.7
35.1
43.3
Table 20: Ablation study results information scaling of LongTimeScope (Zohar et al. 2025
Metrics include overall accuracy and task-specific scores across different steps. # TPF stands for token per frame, and # F stands for sampling frame number.
18000
28800
36000
# TPF
# F
Overall
OCR
QA
Temporal
OCR
QA Temporal
OCR
QA
Temporal
256
128
42.9
54.0
48.0
36.0
46.0
62.0
6.0
40.0
80.0
14.0
512
128
41.1
54.0
60.0
28.0
42.0
62.0
4.0
40.0
78.0
2.0
128
256
42.0
58.0
56.0
26.0
62.0 46.0
2.0
40.0
78.0
10.0
256
256
41.3
58.0
52.0
36.0
48.0
62.0
0.0
40.0
70.0
6.0
512
256
41.6
54.0
56.0
32.0
46.0
60.0
2.0
40.0
78.0
6.0
64
512
40.2
52.0
52.0
26.0
44.0
64.0
2.0
44.0
76.0
2.0
128
512
44.4
50.0
54.0
30.0
54.0
72.0
0.0
48.0
76.0
16.0
256
512
38.7
48.0
50.0
30.0
52.0
56.0
8.0
36.0
60.0
8.0
64
1024
41.6
54.0
56.0
22.0
46.0
66.0
4.0
36.0
72.0
18.0
128
1024
41.1
50.0
46.0
32.0
46.0
62.0
14.0
38.0
54.0
28.0
64
2048
38.4
50.0
62.0
26.0
40.0
60.0
2.0
38.0
42.0
26.0
Onard.5
60.0
One-2.3
83
82
Reformancs
59.5
Welder
80
58.5
Vodee
Over2 VL
79.
128k
64k Context
Context Length
Context
(a) Long Scoe
(b) TimeScope
(c) Long VideoBench
Vide-NAA
Ower2.5 VL
36
3eformannc
50
32
49
30
Conteat Length
Length
(d) MLVU
(e) VSIBench
Figure 15: Performance comparison of Qwen2.5-VL and VideoNSA under different context lengths.
becomes
Qui = Q @ M', where @ denotes elementwise multiplication, and the corresponding new attention weight is
Attn/ exp(2] (Q M').
29

---

Published as a conference paper at ICLR 2026
Table 21: Ablation study results on information scaling of Teme (Zohar al. 2025) ). Metries include overall accuracy and task-specific scores across different steps. TPF stands for token per frame, and # stands for sampling frame number.
# TPF
F
Overall
60
120
180
300
600
1200
1800
3600
7200
10800
256
128
73.1
96.7
94.7
93.4
85.4
72.0
62.6
57.3
56.6
54.0
58.6
512
128
72.5
95.4
94.0 92.7
82.0
72.7
64.6
57.3
56.3
53.3
56.6
128
256
76.5
98.0
97.4 96.0 86.7
78.7
78.0
63.3
58.6
54.0
54.0
256
256
76.1
96.7
96.7 91.4 86.7
76.0
74.6
64.0
62.0
56.0
56.6
512
256
75.8
95.4
94.7 90.7 86.7
76.0
75.3
66.0
62.6
53.3
57.3
64
512
78.5
96.7 95.4 94.7 88.0
80.7
82.6
71.3
62.0
58.0
55.3
128
512
76.5
98.0 97.4 96.0 86.7 78.7
78.0
63.3
58.6
54.0
54.0
256
512
77.3
96.7
96.7 90.7 83.4
76.7
78.0
72.0
66.6
59.3
52.6
64
1024
81.7
96.7
95.4 94.7 90.0
84.7
88.0
78.0
69.3
64.0
56.6
128
1024
81.8
98.0
97.4
94.0
85.4
80.7
92.0
78.0
72.6
66.0
54.0
64
2048
82.7
96.7
95.4 94.7 90.0 82.0 91.3
88.0
73.3
63.3
52.0
Table 22:
Ablation study results on information scaling of Bine (Wu et al., 2024a).
Metrics include overall accuracy and task-specific scores across different steps. # TPF stands for token per frame, and stands for sampling frame number.
Overall
T2E
512
58.3
41.0
666
718
54.0
48.9
599
55.7
63.9
53.
57.8
70.1
37.9
602
68.1
71.
65.7
$1.9
128
128
574
5688
45.4
66.1
66.6
69.5
61.2
55.5
53,0
49.5
59.9
51.7
$4.2
47.
57.8
68.5
40.0
5
68.6
70.0
63.6
52.4
256
128
579
58.5
48.1
69
666
7066
58.4
55.5
53.0
48.1
S76
478
54.2
49.7
59.3
68.5
40.0
65.2
68.1
70.0
63.1
$2.6
312
590
53.0
61.2
56.2
39.0
63.6
$4.2
128
58.7
46.7
69.
57,8
52.5
48.1
564
46.5
51
48.5
57.8
67.0
37.9
652
63.4
70.0
58.3
$2.7
256
256
58.2
58.7
39.9
640
66
718
58.4
540
59.7
52.
599
55.7
564
509
56.2
7U.1
40.0
614
66.9
68.7
63.1
53.5
512
256
594
004
52.2
67.2
66.5
75.2
61.2
540
56.7
52.2
622
53.1
624
49.7
56.2
68.5
35.9
662
67.
724
65.7
$4.0
64
57.7
41.3
672
68.7
622
49.1
5844
533
624
71.6
35.9
68.7
61.5
53.5
256
512
58.
592
41.0
61.0
61.5
718
65.3
52.5
55.7
55.0
610
51.4
62.
19.7
59
71.6
37.9
564
66.9
67.5
63.1
53.5
64
1024
58.4
59
42.6
65
68.7
66.1
62.6
55.5
584
49.5
64.5
$4.4
5
50.9
59.
74.7
36.9
57.6
66.3
68.
61.5
54.2
128
1024
58.7
58.5
41.3
67.2
68.
71.8
65.
600
59.7
52.2
99.
53.1
626
47.2
59.3
71.6
328
602
65.7
67.5
63.6
55,1
Seiting
3.8%
Training Setting
1.8%
50%
stiding Window 0lgg 2c3kd
Sldnig Window Hog2 scale)
(b) Attention Scaling of TimeScope Figure 16: Scaling Performance of VIdeoNSA under Different Attention Allocation Strategies. We highlight the attention budget constraint to indicate settings with equal attention budget, and annotate the best-performing configuration under each benchmark.
(a) Attention Scaling of Long VidcoBench
Even if the modification from M to M' appears small in proportion, it substantially changes the set of candidate evidence accessible to each query and alters the relative logits. First, the newly visible tokens reduce the relative weight allocated the original key tokens, producing dilution effect. Second, in video tasks, the added visible tokens often lie in similar visual-semantic clusters as the original key tokens and thus have non-negligible similarity scores Z.QQQ These tokens directly com- pete with and divert attention away from the originally dominant key tokens. Under the combined influence of these effects, the stable routing-pati structure learned during training overwritten: formerly high-weight key tokens may be diluted or overshadowed by the candidates, causing A, to increase significantly. As result, the model is more likely to follow incorrect reasoning paths, leading to degraded performance.
30

---

Published as a conference paper at ICLR 2026
Table 23: Ablation study results information scaling MLVU (Zouu al. 2024). Metrics include overall accuracy and task-specific scores across different steps. TPF stands for token per frame, and # stands for sampling frame number.
TPF
F
Overall
PlotQA
Needle
Ego
Count
Order
Anomaly Reco
Topic Reason.
SportsQA
TutorialQA
256
128
49.6
46.0
52.7
53.2
24.3
36.0
47.0
87.3
36,6
36.2
512
128
49.2
52.0
51.0
47.5
24.3
37.4
39.3
87.3
42.1
33.9
128
256
50.6
50.0
57.7
60.7
24.3
33.1
39.3
86.2
42.1
36.2
51.2
50.0
56.0
56.9
27.7
38.9
41.9
85.1
42.1
36.2
512
54.0
49.3
49.4
22.7
37.4
39.3
86.2
33.8
29.3
64
50.0
62.7
55.1
24.3
34.6
47.0
84.0
42.1
38.6
128
51.8
48.0
69.3
51.3
27.7
34.6
44.5
86.2
47.7
31.6
256
512
48.6
50.0
51.0
47.5
24.3
33.1
52.2
84.0
47.7
26.9
64
1024
51.8
56.0
66.0
53.2
26.0
36.0
47.0
84.0
42.1
31.6
128
1024
48.0
52.0
51.0
49.4
29.3
33.1
44.5
80.7
44.9
24.6
Table 24: Ablation study results on information scaling of Tomato (Shangguan et al. 2024).
Metries
include overall accuracy and task-specific scores across different steps. TPF stands for token per frame, and # stands for sampling frame number.
FPS
TPF
Overall Direction Count Rotation Shape& Velocity&Freq,
Visual Cues
Human
Simulated Object
64
24.7
22.8
26.7
20.6
25.1
27.1
34.3
21.2
23.2
28.4
128
23.9
20.6
29.8
19.9
24.7
23,8
32.9
19.8
24.0
27.6
256
24.7
22.3
19.6
25.1
25.2
20.8
23.3
28.7
512
23.9
20.6
29.8
19.9
24.7
22.9
34.3
20.3
21.5
27.9
2
64
24.5
21.1
31.8
19.6
22.4
25.7
35.7
20.7
21.9
28.8
128
24.3
20.6
30.5
20.3
24.7
23.3
38.6
20.7
22.3
28.4
24.4
21.3
18.5
26.5
24.8
24.0
28.2
2
512
24.7
19.4
32.2
21.3
25.6
23.8
37.1
20.0
24.0
29.1
4
64
25.1
22.1
31.5
19.6
26.5
23.3
38.6
21.2
25.0
29,0
128
25.8
21.8
33.2
21.3
25.6
25.7
37.1
21.5
25.3
29.9
26.2
23.1
26.9
25.3
30.5
4
512
26.5
21.6
31.5
22.0
25.6
23.3
40.0
21.7
23.6
29.3
Geometric Rotational of RoPE. RoPE (Su et al. 2024 maps the representation at position i into rotation in two-dimensional subspace:
di(i) R(iw)g
k'()) R(jw)k
where R(ww) is a rotation matrix and denotes the frequency parameters. This yields an inner product that depends only on the relative distance between the two positions:
(1),,) (') (R((i j)w)), k))
RoPE (Su et al., 2024) therefore establishes a structured geometric correspondence between relative distance and rotation phase. Under this geometry, when the context length is moderately increased (e.g.. from 36K to 64K), the model only needs to resolve a larger phase difference dw; within this range, the growth of the phase still lies in the extrapolation regime covered by the empirical distribution seen during training. As result, the model can naturally generalize.
LM-Infinite (Han al. 2023) further proves that, in order distinguish the growing clusters of relative distances ann), the attention logit must increase monotonically with sequence length:
(a(n) 2
1/(2r)
sup [w]g, k. g.
4e
where w(q, k; d) denotes the logit relative distance d, and ((n) grows with n. This "logit growth" controlled and beneficial at moderate lengths, expanding the dynamic range of attention and en- abling the model to maintain token separability over larger distances and consistenting with the strong performance observe around 64K.
However, when the effective phase difference du becomes excessively large, the rotation angle may approach or exceed the periodic range of multiple frequency dimensions, giving rise to phase aliasing: tokens that should correspond to distinct relative distances collapse into similar or even in- distinguishable phase regions. In such cases, although attention logits continue grow with length,
31

---

Published as a conference paper at ICLR 2026
Table 25: Ablation study results information scaling VSIBench Yang al. 2025a). Metrics include overall accuracy and task-specific scores across different steps. TPF stands for token per frame, and # stands for sampling frame number.
TPF
Max Frames
Overall
Obj. Order
Abs. Dist.
Counting
Rel. Dist.
Size Est.
Room Est.
Route Plan.
Rel. Dir.
512
32
34,9
27.6
16.2
31.4
35.1
52.2
31.5
40.1
44,6
512
29.1
17.5
34.9
33.0
52.2
31.0
36.5
43.9
256
128
36.0
17.6
41.3
37.5
53.9
39.1
43.3
512
128
34.6
27.4
17.2
37.3
34.4
50.3
30.6
35.5
43.8
128
256
35.6
26.8
17.0
42.0
36.8
51.8
31.2
35.0
44.2
256
256
35.5
27.8
17.0
42.4
33.7
51.3
31.6
35.5
44,5
512
256
28.2
16.5
40.3
33.9
48.8
30.7
36.5
43.3
64
512
34.2
29.1
15.8
42.1
33.9
45.5
27.7
37.0
42.9
128
512
36.0
25.5
19.0
42.5
35.4
54.0
30.1
37.5
43,6
256
512
33.9
28.2
15.8
42.9
31.3
43.6
29.9
36.5
43.1
64
1024
35.8
24.4
18.5
46.4
34.4
52.4
29.7
37.5
43.0
128
1024
18.4
45.3
32.7
50.3
31.7
37.0
43.7
Table 26: Performance of Qwen2.5-VL 7B under different context lengths.
Context
LVB
MLVU
TimeScope
LTSS
VSIBench
32k
58.1
50.5
80.33
34.4
35.1
64k
59.7
51.2
81.13
40.7
29.7
128k
58.7
48.0
81.00
39.7
29.3
the high-frequency components of RoPE lose their discriminative resolution, reducing geometric separability among tokens, which aligns with existing analyses (Press et al. [2021] Chen et al. 2023 showing the degradation of relative positional encoding at extreme distances.
Chen
N FULL GATE VALUES DISTRIBUTION
O MORE INTER-HEAD GATE SIMILARITES VISUALIZATION
tottation Layer
60
Inter-head Similaaikes Visualizaion
LA
Camprouuke Sate
Deleciion Cale
Skding Wineww D01s
Dargresukn Sele
Decection Cote
Shzing Mindbr Srid
Layor
aitnn of Laver
Figure
Visuolization
L3
L9
Comeression Date
Selettion Daae
SSIPe Minlow dane
Oiergnesion Gate
Seeertinn Cate
SEsPo Mi Gana
Cundariia of Logit
Mad
aeal
Aad
Figure 20: More Inter head Gate Similurites Visualicsion
32

---

Published as a conference paper at ICLR 2026
Table 27: Performance of VideoNSA under different context lengths.
Context
LVB
MLVU
TimeScope
LTS
VSIBench
32k
58.7
50.6
78.5
41.1
36.0
64k
59.4
51.8
81.7
44.4
36.6
128k
60.0
48.7
83.7
42.9
35.7
Table 28:
results across different attention budget strategy. Metrics include overall accuracy and task specific scores across different steps.
ConeSeeeScope (Zhaar et al. 2025)
18000
28800
36000
Block Count
Window Size Overall
OCR
QA
Temporal
OCR
QA
Temporal
OCR
QA
Temporal
44.0
56.0
50.0
28.0
46.0
66.0
16.0
46.0
72.0
16.0
35
64
44.0
54.0
58.0
26.0
46.0
68.0
12.0
46.0
74.0
12.0
34
128
41.8
50.0
56.0
28.0
44.0
64.0
6.0
46.0
74.0
8.0
28
512
42.0
50.0
56.0
28.0
48.0
64.0
6.0
46,0
76.0
4.0
20
1024
40.9
52.0
56.0
28.0
48.0
64.0
0.0
44.0
76.0
0.0
1900
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
128
41.6
52.0
26.0
46.0
62.0
10.0
42.0
76.0
4.0
512
42.4
52.0
56.0
28.0
48.0
64.0
8.0
46.0
76.0
4.0
L.12 Selection cate
L13 Screction Cate
Comfrrssionn Sate
59iing Window 5o0e
Cempressionn Date
Skking Me Sane
Dodoon 13
Figure 21: More Inter head Gate Similariees Visualizaion
1.16
L.17
Conporcction Cata
Selection Gate
Sistng Miccow Sons
Corpgractio Cate
Selecdio Cads
Slotngg Nirdow Cana
Lagrer
Figure 22: More Inter head Gate Similarites Visualization
1,20
1.21
Conrercciion Cana
Selection Gate
Sisting Window Sons
Ceregnvction Sate
Selentinn Gabs
Sliting Mirdow Cons
Srm AU Layer
Figume 23: More Inter-head Gate Similurtes Visudization
1.24 Selettion riste
1.25
Conrersciie.co dana
Sistin Minii dans
Ceergvectio Rats
Severton Caty
Sisiin Mirdie Sons
Laver 25
Figure 24: More Inter-head Gate Similurites Visudization
BENCHMARK-LEVEL GATING ANALYSIS AND PCA VISUALIZATION
In this section, provide additional evidence that VidcoNSA's 's routing strategy depends input video content rather than layer depth alone. We collect the layer-head gate vectors for represen- tative videos from three benchmarks with distinet visual properties (LongTimeecoope (Zohar al.
33

---

Published as a conference paper at ICLR 2026
Table 29: TimeScope Zohaar al.
results across different attention budget strategy. Metrics include overall accuracy and task specifcciiic scores across different steps.
2025)
Block Count
Window Size
Overall
60
120
180
300
600
1200
1800
3600
7200
10800
81.6
97.4
97.4
96.7
85.4
82.7
86.3
85.3
71.3
60.6
52.7
35
64
82.4
92.7
91.3
92.7
91.4
83.4
91.6
91.3
74.6
63.3
52.0
34
128
82.2
96.7
96.7
94.7
89.4
82.0
88.3
85.3
76.6
60.0
52.7
512
82.8
94.7
97.4
94.7
90.7
82.7
91.6
88.6
74.0
63.3
50.0
20
1024
83.2
96.7
97.4
97.4
88.7
85.4
89.0
84.6
78.6
58.6
55.3
1900
8.6
4.7
4.7
4.7
4.7
12.3
13.3
13.3
13.3
10.0
10
512
59.9
4.7
4.7
94.7
88.7
79.4
85.6
80.0
68.0
58.0
35.3
40
1024
83.7
96.7
96.0
97.4
92.0
85.4
91.6
89.3
73.3
63.3
52.0
Table 30: LongVideoBench (Wu et al. 2024a) results across different attention budget strategy. Metrics include overall accuracy and task- sesc-specific scores across different steps.
Black Couet Window Size Dvenall
6030
TD8
S2F
F3R
S2A
SAA
010
T10
TIE
OOR
T30
$20
TAA
T2E
F30
SSS
T2A
6O0
SOS
15.0
3600
36
59.9
57.7
45.1
64.0
6666
72.9
55.6
52.5
5K.4
57.7
59,9
54.
70.9
$2.1
56.2
70.1
52N
66.5
67.
72.4
65.2
56.1
35
64
60.1
58.7
48.1
64.0
655
752
$7.0
SSS
59.
90.1
56.7
S5.7
66
49-7
56.2
70.1
348
652
660
73.7
65.7
$5.0
54
178
60.2
599
48.1
65.1
67.6
74.1
55.6
S5.
584
56.3
622
57.0
63.
53.
56.2
716
35.9
62.7
67
72.4
66.3
55.1
28
512
59.4
60.4
56.7
62.9
M6.6
74.1
58.4
52.5
54.3
56.3
6
55.7
59
99.7
56.2
79
54.8
614
67
68.7
64.1
53.3
20
1024
59.5
60.6
45.4
66.1
66.
72.9
59.8
S4.0
54.3
55.0
64.
$7.0
5
50.9
56.2
90.9
528
62.7
69
67.5
636
53.1
4
1900
28.3
27.6
23.4
24.2
35.7
25.
32.0
29.7
27.3
248
31.1
36.7
33.4
35.1
27.9
28.5
26.6
19.7
2
26,
27.1
29.7
17
64
59.4
58.0
46.7
62.9
05.
74,1
35.6
52.
5844
57.7
56.7
57.0
66.7
50.9
56.2
716
30.7
64,9
68.1
72.4
65.7
542
2025)
for multi-shot transitions, Tomato (Shangguan et al.) 2024) for feeu-freeeenyy motion, and VSIBench (Yang al. 2025a) for complex spatial layouts) and project the gate vectors into 2D
space using PCA.
As shown in Figure 25 the gate patterns form three clearly separated clusters, regardless of whether we use the compression branch, the selection branch, or the alding-window branch, which indicates that VideoNSA learns benchmark-specifee routing strategies conditioned on visual content, rather than following fixed depth pattern.
To further isolate the role of input-driven routing, we replace each layer's gate with a static value averaged from 1K training subset, forcing the model to depend only on layer depth. As shown in Table 34 the performance drops across all six benchmarks, especially on tasks requiring long-range temporal integration, confirming that dynamic gating is essential.
Q ADDITIONAL ANALYSIS OF TRAINING AND INFERENCE EFFICIENCY
To complement efficiency discussion the main paper, we provide additional analysis both FLOPs and wall-clock latency across different attention mechanisms and context lengths.
Training Efficiency. Under identical optimization settings, training VideoNSA requires approx- imately 4600 H100 GPU hours, while the dense baseline requires 5280 H100 GPÚ hours. This corresponds 0.87x of the dense baseline, indicating that VideoNSA achieves slightly improved training efficiency despite using a more complex attention mechanism.
Inference Efficiency. Table 35 presents the theoretical FLOPs of different attention mechanisms. In the ideal case, NSA requires only 2.05 PFLOPs. which is 0.24x that of Flash Attention, demon- strating the theoretical computational efficiency of the sparse routing structure. However, the actual FLOPSs and wall-clock latency of VideoNSA are higher than this ideal value due to implementation constraints in the current NSA kernel. The Qwen2.5-VL. 7B (Qwen et al., (023) adopts an unusual head configuration of KV heads and 28 query heads. To satisfy Triton kernel requirements, the query heads must be padded to 64, which introduces additional computation and memory access overhead. As a result, the practical efficiency of VideoNSA deviates from its theoretical FLOPs ad- vantage. As shown in Figure 26 VideoNSA's 's latency grows much more slowly than dense attention, and compared with other sparse baselines, delivers competitive inference speed while achieving stronger model performance.
34

---

Published as a conference paper at ICLR 2026
Table 31: MLVU (Zhou al. 2024 results across different attention budget strategy. Metries include overall accuracy and task lask-specife scores across different steps.
Block Count Window Size
Overall
Direction
Count
Rotation
Shape&Trend Velocity&&Freg
Visual Cues
Iuumn
Simulated
Object
256
36
32
25.9
21.6
32.5
19.2
25.1
25.5
37.1
21.4
21.5
29.2
35
20.6
25.6
25.5
37.1
22.1
24.2
34
128
27.2
23.8
34,2
20.3
25.1
25.5
38.6
21.9
24.2
28
512
26.1
21.8
32.2
19.2
24.7
27.5
37.1
22.1
22.4
28.2
1024
25.1
20.6
34.3
21.4
23.3
64
512
25.3
21.3
30.5
19.6
24.2
27.5
32.9
21.4
22.9
27.4
16
128
21
19.5
17.5
20.2
21.0
30.0
28.8
17.8
17.6
20.6
Table 32: Tomato (Shangguan et al. 2024) results across different attention budget strategy. Metries include overall accuracy and task specifcc scores across different steps.
Block Count
Window Size
Overall Direction
Count
Rotation
Shape&
Velocity&&Prq Visual Cues
Human
Simulated
Object
35
64
27.1
23.8
33.9
20.6
25.6
25,5
37.1
22.1
24.2
30.8
34
128
27.2
23.8
34.2
20.3
25.1
25.5
38.6
21.9
24.2
30,8
28
512
26.1
21.8
32.2
19.2
24.7
27.5
37.1
22.1
22.4
28.2
20
1024
25.1
20.6
30.8
17.5
23.3
29.4
34.3
21.4
23,3
25.6
64
512
25.3
21.3
30.5
19.6
24.2
27.5
32.9
21.4
22.9
27.4
4
2048
26.4
21.8
33.6
20.3
25.6
27.5
32.9
21.8
24.7
29.4
128
28.8
R ADDITIONAL ANALYSIS ON CMP LATENCY BOTTLENECK
In this section, we provide additional analysis supporting the observation findings that the CMP branch becomes the dominant source of latency as the context length increases.
Since the block size determines how many CMP operations are executed, we vary the block size and measure the resulting latency across multiple context lengths. As summarized in Table although increasing the block size reduces the number of CMP executions, the overall latency improvement remains small. Smaller blocks are dominated by memory-ace seceosy-access overhead, while larger blocks incur higher computation per block. As a result, block-size scaling affects latency only moderately within narrow range and does not change the overall scaling trend.
We also observe that the most significant acceleration comes from more efficient NSA implementa- tions instead of architectural hyperparameters. As shown in Table 37 the flash-nsa (mdy666) 2025) implementation runs about twice as fast as our current nsa-impl Pai et al., (025a) in the forward pass and up six times faster in the backward pass. Other teams also developing improved kernels such as optimizing NSA for TPUs (Ko 2025). These findings show that the dominant factor affecting CMP and overall NSA latency comes from kernel efficiency, including memory access patterns and kemee design.
MORE ANALYSIS ABOUT ATTENTION SINKS ON VARIOUS SPARSE ATTENTION SETTINGS
Figure 27a indicates that in the compression branch, smaller blocks produce sharper and higher sink peaks at the sequence start, while larger blocks used in training reduce the initial peak but introduce broader low-density diffusion with periodic boundary spikes. The selection sinks in 27b remain at consistently low densities under different configurations, suggesting that the top-k filtering mecha- nism robustly suppresses sink formation across different settings. Figure [7 shows the distribution of attention sinks under different sparse attention settings. When varying the window size, sinks are concentrated near the beginning and decay rapidly with position. Overall, larger windows yield lower sink density but broader coverage, while the training configuration (w 256) strikes mid- dle ground and exhibits sparse periodic clusters the mid-to-late sequence, reflecting sensitivity to local boundaries learned during training.
35

---

Published as a conference paper at ICLR 2026
Table 33: VSIBench (Yang et al. 2025a) results across different attention budget strategy. Metrics include overall accuracy and fask-speciln scores across different steps.
Block Count
Window Size
Overall
Appearance
Abs, Dist.
Counting
Rel. Dist.
Size Est.
Room Est.
Route Plan.
Rel. Dir.
28
512
36.0
23.9
18.2
45.5
36.9
$4.1
29.8
36.0
43.4
35.9
46.8
53.6
29.0
36.5
42.0
1900
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
35.5
24.7
37.8
36.2
53.9
31.7
36.0
45.2
35.4
20.4
33.2
53.6
31.5
38.1
46.0
36
32
34,9
25.3
20.6
28.4
36.0
$4.4
30.9
38.1
45.9
35.3
20.5
28.3
54.7
31.0
41.1
46.1
16
128
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
64
512
35.8
23.6
18.3
45,9
36.8
$4.2
29.3
36,5
42,3
Comomeaion Imeat
Selection
51dhg (indza Iraard
0.8
Gate Wei Weight Value
0.6
0.2
L10
L16
L1
10
L23
124
125
L28
127
Layer Index
Figure 17: Gate weight distribution of each layer.
Bondhmart sergiimnrion tanaco
Ievfisminge
taratt
3.5
800
-036
-04
Denshraang legiraaccz
-37
-c3
PC
PC
(a) Compression Branch
(b) Selection Branch
(c) Sliding Window
Figure 25: PCA visualization of benchmark-leve gate patterns.
DISCUSSION ON MODALITY-SPECIFIC SPARSITY
To further contextualize the modality-speeific sparsity patterns exhibited by VideoNSA, we compare its behavior with the text-only NSA (Pai al. 2025a) used in language models. As shown in Figure (28] the text-only NSA (Pai et al.) 2025a) displays distinct gating dynamic. The sliding- window branch gradually becomes dominant deeper layers, while the compression and selection branches diminish rapidly and remain almost inactive throughout most of the network. This pattern reflects the miedimensionnl and relatively uniform nature of textual sequences, where long-range interactions sparse and stable, and the model tends converge toward single prevailing routing path. The text-only NSA (Pai et al. 2025a) also presents a noticeable anomaly in the final layer, where all three branches suddenly become active again despite having remained largely inactive in
36

---

Published as a conference paper at ICLR 2026
Table 34: Ablation static gates averaged over 1K training subset.
Long Video Understanding
Temporal Reasoning Spatial Understanding
Model
LongVideeBeench MLVUreat TimeScope
Tomato
VSIBench
Qwen2.5-VI.77B
$8.7
51.2
81.0
40.7
22.6
29.7
VideoNSA
(+1.())
51.8 (+1.2%)82.7 (+2.1%)
44.4 (+9.11))
26.2 (+15.9%)
36.1 (+20.3%)
VideoNSA+ Static 58.4 (-0.5%)
(0.0%) 81.2 (40.2%)
41.5 (+2.0%)
23.7 (+4.8%)
31.8 (47.1%)
Table 35: Theoretical FLOPs comparison among different attention mechanisms. (ideal)" denotes the theoretical FLOPs of NSA without query-head padding.
"VideoNSA
Method
FLOPs Relative
Flash Attention
8.40
1.00x
Tri-shape
7.07 PF
0.84x
MInference
4.13 PF
0.49x
Flexprefill
7.75 PF
0.92x
XAttention
0,23x
VideoNSA (idcal)
2.05 PF 0.24x
VideoNSA
4.68 PF 0,56x
25
Full Attention
Flexprefill
Flash Attention
XAttention
20
Tri-shape
VideoN5A
Minference
Latency
5
4k
8k
16k
32k
64k
128k
Context Length
(a) Absolute prefill latency across attention mechanisms.
Speedup vs Full Attention (X]
4
Attention Trishape
Minference Flexprefill
XAttention
3.5 4.0
41
2.4
3.1
3.
2.6
2.9
2-
1.8
1.4
1-
0.5
6.2
4k
Skk
16k
32k
64k
128k
Context Length
(b) Prefill-tmme speedup over full attention across context lengths.
Figure 26: Inference efficiency comparison across attention mechanisms.
previous layers. This behavior suggests a late-stage shift in inductive patterns that is characteristic of language modeling. In contrast, VideoNSA. as shown in Figure 17 maintains active and balanced usage of all three branches across nearly the entire depth of the network, with the compression branch playing a consistently prominent role.
The inter-head similarities further highlight the divergence between the two modalities. The text- only NSA (Pai et al. 2025a) in Figure 29 exhibits strong correlations across heads in the early layers, which indicates a set of conserved induction-like operations. Later in the model, selection and slid- ing window gate values become decorrelated across heads. VideoNSA, as shown in Section how- ever, displays substantially weaker cross head correlations overall, and only few mid-layer clusters emerge in the selection and sliding- window branches. These findings imply that VideoNSA adjusts its sparse routing behavior accommodate the rich spatiotemporal redundancy and multi-scale
37

---

Published as a conference paper at ICLR 2026
Table 36: CMP latency under different block sizes and context lengths.
Block Size
8k
16k
32k
64k
128k
32
0.868
1.022
1.311
1.982
3.698
8.343
64
0.882
1,036
1.322
1.997
8.323 1.993
3.687
128
0.880
1.027
1.308
3.705
8.353
Table 37: Latency comparison between different NSA implementations at 8k context length.
Implementation
forward backward
nsa-impl (Pai et al. 2025a
5,402 2.429
32.826 5.537
flash-nsa (md6666 2025)
Sink Density
block size 64, block count block black
20
8.0
0.2
0.4
0.6
0.8
Relative Position
Compression sinks across black size and counts.
Sik Density
64,
block size 128, black count 16
20
0.0
0.2
0.4
0.6
0.8
1.0
Relative Position
(b) Selection sinks across block size and counts.
Window Size 128
Window Size 512
Sink Density
Window Size 256
Window Size 1024
20-
10-
11.1
In
8.0
(c) Sliding window sinks across window sizes.
Figure 27: Attention sink distributions across the three branches under different sparse settings.
structure of video inputs, rather than collapsing into a single dominant pathway as observed in the text-only model.
38

---

Published as a conference paper at ICLR 2026
NSA Gate Value Distributions Across Layers
1.0
g_cmp
0.8
0.6
Cate Value
0.4
0.2
0.0
Leper
Layer
Layer
Layer
Layer
Layer
Laper
Laper
Layer 20
Layer 22
Layer 22
Layer 23
Layer 14
Layer 13
Layer 16
Layer 17
Layer 18
Layer 10
20
Layer 23
22
Figure 28: Layer-wise gate distributions of text-only NSA Pai al
2025a
HILL
Figure 29: Inter-head gate similarities of text-only NSA
Pai et 2025a
39

---

Published as a conference paper at ICLR 2026
U DENSE ATTENTION SINK VISUALIZATION
QINEGO 5-VIL Layer Ilends FFlash Attention Analysis (116720 takens]
Beat
Neith mid
QUEEN2.5VV 5-V Layer 34 All Heads Flach Attention Analysis (116720 takens)
Bat R
Pari shall
Sut wwdeee
ment Uwn mot
WWENG.5AL Layer 27 AB lieads Flash Attention Analysi (316728 takens)
40


Reviews:
[{"review_id": "ccWbbuW2sS", "reviewer": "Reviewer_8ijG", "summary": "This paper extends Native Sparse Attention (NSA), originally designed for long-context models, to the multimodal domain, proposing VideoNSA. This method applies NSA's three-branch hierarchical sparse structure and gating mechanism to video tokens, while employing Grouped Query Attention for text tokens. Furthermore, the paper provides a scalability analysis, offering new insights into the behavior of sparse attention in multimodal models.", "questions": "1. The paper lacks a strong motivation for applying NSA, a text-modality method, to the video modality. Many methods exist for long-context modeling; why is NSA a good choice? In other words, do the characteristics of NSA offer unique advantages in the context of video?\n2. The authors state that the model utilizes the selection and sliding-window branches in shallow layers to capture fine-grained details and local information, while relying more on the compression branch in deep layers to integrate and refine high-level global semantics. Do the sparse distributions and gating activations differ across various types of videos (e.g., high-speed motion, shot transitions, static scenes), or is the gating behavior solely dependent on layer depth? If the gating behavior is only correlated with layer depth and is independent of input features, does this imply that the model has limited adaptability to different types of video content?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. Insightful Analysis: The scalability analysis provides a degree of interpretability for sparse attention, while also clarifying the advantages of the sparse mechanism (e.g., extensibility to contexts longer than those seen during training, control over attention sinks).\n2. Comprehensive Experiments: The experimental validation is extensive, covering ablation studies for each branch, visualizations of gating distributions, context extension curves, and performance evaluations on multiple benchmarks.\n3. Practical Guidance: The empirical findings offer practical guidance for deploying hardware-aligned sparse attention in long-context multimodal systems.", "weaknesses": "1. Limited Algorithmic Novelty: The core framework for the vision component (the three-branch sparse structure + learnable gating) is nearly identical to that of NSA (Yuan et al., 2025), meaning the method lacks fundamental innovation.\n2. Lack of Discussion on Modality-Specific Sparsity: The paper does not discuss the differences in sparsity between text and video. In text, sparse attention primarily filters information at the syntactic and semantic levels, where token dependencies are relatively stable and one-dimensional. In video, however, sparsity involves spatiotemporal locality and motion redundancy, implying that the definition of semantic redundancy differs across modalities. This raises questions about whether it is appropriate to directly reuse a text-based sparse attention mechanism for the video modality.\n3. Insufficient Theoretical or Analytical Depth: The scalability analysis is predominantly empirical. While it helps users better utilize the model, it lacks theoretical explanations or modeling to elucidate the underlying causes of the observed trends.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UXm8nj5itU", "reviewer": "Reviewer_evHH", "summary": "This paper introduces VideoNSA, a method for scaling video understanding models to very long contexts by adapting NSA. The core idea is to apply a hybrid attention mechanism to a Qwen2.5-VL-7B. Specifically, text tokens are processed with standard GQA, while video tokens are handled by NSA, which dynamically combines three complementary sparse attention branches: CMP for global aggregation, SLC for salient information, and SWA for local context. The authors fine-tune this model on a 216K video instruction dataset. The resulting model scales effectively to 128K tokens and performs well at a series of challenging long-video benchmarks,", "questions": "In \"Finding 5,\" you identify the token compression (CMP) branch as the main latency bottleneck. Given its importance (as shown in the gate analysis in \"Finding 4\"), what are your thoughts on potential avenues for optimizing this branch to further improve the model's overall efficiency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Extensive Evaluation: The paper evaluates VideoNSA across a diverse set of challenging long-video benchmarks, demonstrating competitive or SOTA performance. The inclusion of strong baselines and thorough ablations validates the design choices.\n- In-depth Analysis: The analysis in Section 4 is a standout feature. The structured \"Findings\" provide clear, actionable insights into how sparse attention behaves when scaled. The study of information scaling, budget allocation, and attention sinks goes far beyond a typical model performance paper and offers significant value to the research community.\n- Efficiency and Scalability: The paper shows that VideoNSA scales effectively to 128K context lengths, far beyond its training regime. The finding that it achieves top-tier performance with only 3.6% of the dense attention budget is a powerful demonstration of the method's efficiency.", "weaknesses": "Novelty: The primary weakness is that the core technical component, NSA, is adapted from a previous work (Yuan et al., 2025b). The novelty is in the application, specific architectural choices for video, and the extensive analysis, rather than a new algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WEKi1WRvSJ", "reviewer": "Reviewer_mLuS", "summary": "This paper targets the challenge of long-video understanding in MLLMs, which is constrained by the quadratic complexity of standard attention. The authors propose VideoNSA, a hybrid attention mechanism that adapts Native Sparse Attention for video-language models. The core of the method is to apply standard Grouped-Query Attention to text tokens while using a learnable, three-branch sparse attention mechanism for the video tokens. This video-specific NSA dynamically combines a Token Compression branch, a Token Selection branch, and a Sliding Window branch using learnable gates. The model, which is an adaptation of Qwen2.5-VL , is trained end-to-end on a 216K video instruction dataset. The authors present experiments showing that VideoNSA achieves competitive performance on long-video understanding, temporal reasoning, and spatial benchmarks. The paper also provides a very detailed analysis of the model's scaling properties, attention budget allocation, internal branch usage, and its effect on mitigating attention sinks.", "questions": "1. Could you elaborate on the \"Dense-SFT\" baseline's performance drop? If the training data is of limited quality, how can you be sure it isn't also limiting VideoNSA's potential? Conversely, do you hypothesize that VideoNSA's gains over the dense baseline would be larger or smaller if trained on a much larger, higher-quality video instruction dataset?\n\n2. Given the CMP branch is the bottleneck, do you have concrete suggestions for optimizing it? The paper states the block-level representation is obtained by \"averaging all tokens\" 29, but the preliminary definition (Eq. 2) mentions a \"learnable MLP\" ($\\varphi$)30.\n- First, please clarify this: is the learnable MLP simply performing a weighted average, or is it a more complex, non-linear projection?\n- Second, if it is a simple average, the latency should be low. If it's a learnable MLP, did you experiment with replacing it with a fixed (non-learnable) pooling operation to see if the performance/latency trade-off improves?\n\n3. The authors note the \"strange behavior\" in the last layer (L27) where all three branch gates become fully active. Do you have any hypothesis for this? Is it a learned behavior for final-layer aggregation, or could it be an artifact of training (e.g., the gates for that layer not receiving a strong gradient)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a highly significant and timely problem: scaling video-language models to handle long contexts (e.g., thousands of frames or 128K tokens) efficiently.\n\n2. The proposed hybrid attention mechanism is well-motivated. Preserving dense attention for text tokens while applying aggressive, learnable sparsity to the highly redundant video tokens  is a sensible architectural choice.\n\n3. The paper's main strength is its extensive analysis section. The authors provide commendable, deep insights into the model's behavior, including:\n- A study of information scaling (spatial vs. temporal trade-offs) .\n- A detailed breakdown of attention budget allocation (global vs. local).\n- An analysis of the dynamic gate usage across layers.\n- A novel and valuable investigation into how different sparse branches uniquely contribute to or mitigate attention sinks.", "weaknesses": "1. My most significant concern is the flawed \"Dense-SFT\" baseline. This baseline, which should serve as the primary control, was fine-tuned on the same 216K dataset as VideoNSA. However, this Dense-SFT model performed worse than the original, pre-trained Qwen2.5-VL on most benchmarks (e.g., LVB, TimeScope, Tomato). The authors attribute this to the \"limited quality of the training data\". This admission severely confounds the paper's central claim. We cannot know if VideoNSA's architectural improvements are genuine or if the VideoNSA architecture is simply more robust to this specific, low-quality training data than a dense model. The experiment fails to demonstrate that VideoNSA is better than a properly trained dense model.\n\n2. While the paper claims \"improved performance,\" the results in Table 1 are more accurately described as \"competitive\" or \"on par\" rather than a significant step forward.\n- On Long VideoBench (LVB), VideoNSA (60.0) is outperformed by Video-XL-2 (61.0).\n- On $MLVU_{test}$, VideoNSA (51.8) is outperformed by Video-XL-2 (52.2) and InternVL2.5-8B (55.8).\n- On Long TimeScope (LTS) and TimeScope, its scores (44.4 and 83.7) are effectively tied with other sparse attention methods like MInference and XAttention.\n- Given the confounding baseline (Weakness 1), these marginal gains are not sufficient to robustly claim superiority.\n\n3. The paper is motivated by efficiency, but its own analysis (Finding 5) identifies the compression (CMP) branch as the dominant latency bottleneck as context length grows. The paper concludes that \"the prefill stage remains the primary bottleneck\". While the analysis is transparent and appreciated, the paper identifies a critical practical limitation of its own method without offering a solution. This undermines the practical efficiency claims of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3FJmyNnEPj", "reviewer": "Reviewer_awwy", "summary": "This paper adapts Native Sparse Attention (NSA) to video-language models, demonstrates its potential in tasks including long video understanding, temporal reasoning, and spatial intelligence, and conducts extensive and thorough ablation and analytical experiments. Additionally, to mitigate attention sinks in long-term visual contexts, this paper further proposes to dynamically integrate global and local attention via three complementary branches, effectively addressing this issue.", "questions": "1. Does \"Dense-NSA\" in Table 3 refer to the use of NSA for all modalities?\n2. Regarding Figure 2, does the performance under the 64k context length appear to be better than that under 128k? Does this indicate that training under a short context length (i.e., 36k) is insufficient to unlock the full 128k performance of VideoNSA?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper attempts to apply Native Sparse Attention (NSA) to VideoMLLMs for the first time, and effectively demonstrates its potential in tasks such as long video understanding.\n\n2. The ablation experiments in this paper are quite comprehensive and rigorous, which analyze the performance of NSA in video understanding tasks from multiple perspectives.", "weaknesses": "1. This paper appears to lack an analysis of the training and inference efficiency of VideoNSA. For instance, regarding Table 1, it would be desirable to know the comparisons between various methods and VideoNSA in terms of inference efficiency, latency, and FLOPs. For training efficiency, a comparison between VideoNSA and full attention under the same context length is also expected.\n\n2. Regarding the application of NSA in video tasks, the primary video-related modification in this paper seems to be employing standard GQA for text while adopting NSA for video attention. It is of interest to understand the impact of this operation on the final performance. Does \"Dense-NSA\" in Table 3 refer to the use of NSA for all modalities? It is suggested that the authors further elaborate on the performance differences of NSA across different modalities (i.e., video and text).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)