FlexSelect: Flexible Token Selection for Efficient  Long Video Understanding
Authors: Yunzhuzhang
Yu Lu
Tianyi Wang
Fengyun Rao
Yi Yang
Linchao Zhu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodel LLM
long video understanding
token reduction
Primary area: applications

Abstract: Long-form video understanding poses a significant challenge for video large language models (VideoLLMs) due to prohibitively high computational and memory demands. 
In this paper, We propose $\textbf{FlexSelect}$, a flexible and efficient token selection strategy for processing long videos.
FlexSelect identifies and retains the most semantically relevant content by leveraging cross-modal attention patterns from a reference transformer layer.
It comprises two key components: (1) $\textbf{a training-free token ranking pipeline}$ that leverages faithful cross-modal attention weights to estimate each video token’s importance, and (2) $\textbf{a rank-supervised lightweight selector}$ that is trained to replicate these rankings and filter redundant tokens.
This generic approach can be seamlessly integrated into various VideoLLM architectures, such as LLaVA-Video, InternVL and Qwen-VL, serving as a plug-and-play module to extend their temporal context length. Empirically, FlexSelect delivers strong gains across multiple long-video benchmarks – including VideoMME, MLVU, LongVB, and LVBench. Morever, it achieves significant speed-ups ($\textit{e.g.,}$ up to 9 $\times$ on a LLaVA-Video-7B model), highlighting FlexSelect’s promise for efficient long-form video understanding. Project page: https://flexselect.github.io

FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
Yunzhu Zhang -f Yu Lu I Tianyi Wang Fengyun Rao
Yi Yang Linchao Zhu 1:21
The College of Computer Science and Technology. Zhejiang University The State Key Lab of Brain-Machine Intelligence, Zhejiang University 3 WeChat Vision, Tencent Inc.
Abstract
Long-form video understanding poses a significant challenge for video large lan- guage models (VideoLLMs) due to prohibitively high computational and memory demands. In this paper, We propose FlexSelect, a flexible and efficient token selection strategy for processing long videos. FlexSelect identifies and retains the most semantically relevant content by leveraging cross-modal attention pat- terns from reference transformer layer. It comprises two key components: (1) training-free token ranking pipeline that leverages faithful cross-modal attention weights estimate each video token's importance, and (2) rank-supervised lightweight selector that trained to replicate these rankings and filter redun- dant tokens. This generic approach VidcoLLM architectures, such as LLaVA-Video, InternVL and Qwen-VL, serv- ing plug-and-play module to extend their temporal context length. Empir- ically, FlexSelect delivers strong gains across multiple long-video benchmarks including VideoMME, MLVU, Long V B, and LVBench. Morever, it achieves significant speed-ups ups (e.g., up to 9 x on a LLaVA- Video 7B model), highlighting FlexSelect's promise for efficient long- form video understanding. Project page: https:///yunzhuuhhag011... ......githut github..... io
be seamlessly integrated into various
f fex.fexx..cc select /feeeettteeeet
Introduction
Long-form video understanding is crucial for applications such analyzing movies, building multimodal web agents (31, assisting video surveillance tasks. Recent Video Large Language
22 (55 have shown impressive results on short video clips, combining vision and language processing to answer questions or follow instructions about video content. However, extending these models process long videos presents significant challenges. Long videos yield a substantial volume of visual tokens, often surpassing the context length of transformer- hhasoeedde----deeee--- L.MMs. Additionally, processing entire long videos with VideoL.LMs leads to excessive computational and memory overhead, making effective long video analysis infeasible.
Models (VideoLLMs) II 53 27
g 25
56,
47
To mitigate this, some recent works [5 33 employ training- based compression modules [8 51
to summarize visual tokens into a compact representation via finetuning on long-video datasets. However, these methods introduce substantial overhead due to additional training. Alternatively, other approaches
[33 35 leverage cross-modal attention scores from pre-trained VideoL.MMs to rank
"Work done during internship Wechat Vision.
'Equal Contribution.
I Cooresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Response Timele)
30
LLaVA Video FlexSelect
Layer 0
20
9 xffaser
10.00
10
2.45
Layer 7
VideaMME Acc(%)
70
06.1%
08.9%
65.2%
65
Layer 19
60
60.4%
-5
Video
LLaVA Video FlexSelect
50
64
128
256
512
Layer
Number of Frames
Figure 1: (a) Visualization of cross- modal attention maps of -i-eeo across layers (user query: "what's 's the color of the cup?"). Attention scores progressively highlight the query-related regions (the cup) with layer depth, and this highlighting is most pronounced at the specific reference layer (layer 19 in example). FlexSelect employs attention scores from this layer to select semantically related visual tokens. (b) VideoMME accuracy and response time (time to generate the first token) of LLaVA- Vid The original model with 64 input frames achieves limited accuracy 64.4% due to inadequate coverage for long video content, while increasing frames will overload the model's context window, reducing accuracy to 58.5% and slowing response time to 38.2s. FlexSelect improves this by filtering irrelevant tokens, achieving 68.9% accuracy at 512 frames with 9x faster response (4.2s).
and prune video tokens without training. While more efficient, these attention based heuristics often suffer from performance degradation due inconsistent relevance patterns across transformer layers, which may not reliably reflect the semantic importance of visual tokens for the task.
In this paper, we present FlexSelect, a general-purpose framework for efficient long-form video understanding. FlexSclect enables VideoLLMs to focus on the semantielly relevant visual tokens for the query by identifying and filtering out less informative visual tokens before heavy multimodal reasoning occurs. Crucially, FlexSelect is architecture-agnosti and does not require any modifi- cations or training of the base VideoLLM. It acts as a preprocessor that significantly extends the model's 's effective temporal context window without compromising its reasoning ability.
A central observation underlying FlexSelect is that well-trained VideoLLMs inherently encode meaningful cross-modd modal relevance signals within their internal attention maps. particular, the cross-attention weights between textual queries and visual tokens progressively reflect semantic alignment across transformer layers and typically peak at an intermediate depth (Fig. [a)). Motivated by this, FlexSelect extracts attention scores from empirically identified optimal reference layer to derive faithful, training-free token importance rankings. These scores enable the selection of semantically critical visual tokens while discarding redundant or irrelevant ones, thus substantially reducing token volume without compromising the model's reasoning capability. Unlike existing methods that rely extensive training or oversimplified heuristics, FlexSelect dynamically balances efficiency and performance by exploiting the latent structure of attention patterns within L.s.
To reduce the computational cost of processing long videos with VideoLLM layers for token ranking, we introduce a lightweight selector network trained to mimic the reference layer's ranking. The selectorin is supervised with ranking loss and learns to assign higher scores to tokens that the reference model would attend to. This allows for flcxible token selection without retraining the full model. Once trained, it efficiently selects the top-ranked visual tokens from long videos, significantly reducing computational overhead. These tokens are then passed into original VideoLLM for final reasoning. In doing so, FlexSelect replicates large model's attention patterns at a fraction of the computational cost.
FlexSelect is generic approach can be seamlessly integrated into various VideoL.MM architectures. such as LLaVA- Video, Intern VL and Qwen-Vie FlexSelect serves module requiring no changes to the base V.deoLLM. We evaluate on four challenging long-video -iiiaarrr--iieMMMM.. MLVU, Long VB, and L.Veench-sinn VideoLL.Ms of varying sizes, from 7B to 72B parameters. Experimental results show that FlexSelect achieving significant speed-ups (e.g., up to 9x faster inference with LLa VA- LLAVA-Vidoo-7B while maintaining
plug-and-pla -lay inference
understanding

---

improving performance. Notably, by filtering out irrelevant content, FlexSelect not only accelerate inference but enhances final answer quality.
Contributions. To summarize, our contributions are threefold:
We propose FlexSelect, flexible and architecture- framework that extracts faithful cross cross-modal modal token relevance from an optimal reference layer in VideoLLMs to select semantically important visual tokens for long-form video understanding.
We design a lightweight rank-supervised selector that mimics the cross-modal attention ranking from a reference model layer, enabling fast and accurate token filtering without modifying retraining the base VideoLLM.
We demonstrate that FlexSelect is generic framework applicable to various VideoLLMs, achieving up 9 speed-up and improved accuracy four long-video benchmarks across multiple model scales.
Related Works
Long-form Video Understanding Current VideoLLMs showcase remarkable video-language understanding abilities. However, is still challenge process long-form videos due to the extensive visual tokens. Recent approaches mainly address this by three ways: (1) applying length extrapolation methods (e.g. YARN (30) to VideoLLMs and training on longer sequences 7 49) to support long context input. (2) adopting trainable compression modules to VideoLLM to compress visual content into fewer tokens [T]
(3) cutting long videos into clips and exploring multi-agent collaboration pipelines to process them [ Different from these approaches, we introduce token selector to directly select semantically relevant visual tokens before L.LM generation, without training the large-scale VideoL.LM, which is more efficient.
[5 14] via post-finetuning.
Attention-based Token Pruning Recent works explore visual token pruning for efficient image-text understanding using cross-modal attention scores. FastV (6) first identifies visual token inefficiency in LLM processing, pruning tokens via second-layer attention scores but suffering significant accuracy drops. PyramidDrop144 [4] observes increasing redundancy with layer depth, applying predefined layer- wise drop ratios to prune more tokens at deeper layers for better results. Sparse VLM (52) dynamically adjusts pruning ratios through attention score ranks at each layer, improving performance yet still suboptimal. Recently, FramcFusion applying similar attention- bbaed token pruning methods to VideoL improving efficiency but also degrading performance. Meanwhile, several studies [50 40 10 show that attention scores fail to reliably indicate semantical relevance of visual tokens because of the attention shift phenomenon, where later tokens tend to have a higher scores due to the autoregressive characteristic of LLM. However, these conclusion are limited because they only discuss layer-averaged attention scores. In this paper, we conduct comprehensive layer-wise analysis on the cross-modal attention pattern, and identify reference layer where attention scores can reliably indicate the semantical relevance of visual tokens.
3 and Dycoke (35] analyse redundancy in video data (39),
3 Methods
We present FlexSelect, a token selection strategy for long-form video understanding. FlexSelect consists of two complementary components: (1) training-free selection pipeline that leverages faithful cross-modal attention scores in VideoILMM to select semantically relevant visual tokens from long video, and (2) a lightweight rank-supervised model trained to replicate the visual token rankings of the faithful cross-modal attention scores from VidcoLLM. This section first analyzes token semantic relevance across transformer layers identify the reference layer, then explains the training- free FlexSelect procedure for long video understanding, and finally details the aunk-supervin token selector, covering architecture, training objective, and integration into the framework.
3.1 Layer-wise Semantic Relevance Analysis
VideoL..Ms employ transformer decoder to process video frames as sequences of visual tokens. However, not all tokens equally relevant to given query. To identify which decoder layer
3

---

best captures semantic relevance, we perform a layer-wise analysis using a sreedle-in-hasster experiment. Specifically, we insert one unique needle frame (image containing distinctive visual content) at random positions in a video sequence. We design query solely about the needle image so that the visual tokens derived from are treated as ground-truth semantically relevant tokens, while all other tokens are considered irrelevant. By passing the augmented video through the VideoLLM and analyzing the attention patterns at each layer, we assess whether the model successfully highlights these needle visual tokens as semantically aligned with the query.
For given transformer layer 1, let r!!") denote the semantic relevance score of visual token i that layer. We derive r!" from the model's cross-modal attention. Formally, if (1.A) is the attention weight from query tokens q to visual token i head of layer I, then:
H
(I)
h=1
where H is the number of attention heads. This score reflects how strongly the model semantically links visual token i to the query. We rank all visual tokens by r in descending order to produce semantic relevant ranking that layer.
To quantify how faithfully each layer's semantic relevance scores identify the ground-trute truth semanti- cally relevant tokens (the needle visual tokens), we use the Recall @ K. metric, which computes the fraction of relevant tokens recovered in the top-K ranked tokens. Denote TopK(() is the set of top- K tokens ranked by ri and R is the set of needle visual tokens considered semantically relevant by construction, then:
Recall @ (1) (T) n N
R
We evaluate Recall for each transformer layer I by choosing K R (so that perfect recovery of all needle visual tokens in the top-K K yields Recall @ K = 1.0). A higher Recall @ K indicates that the layer's relevance more faithful to the query.
Results of Layer Analysis: Figure 2 illustrates the Recall K value across all transformer layers in the LLM decoder of LLaVA-Vidco-7B. We ob- serve substantial variation in the effectiveness of different layers at retrieving semantically relevant tokens. Specifically, early layers exhibit relatively low Recall@ K scores, suggesting that the atten- tion distributions at these stages are less aligned with the semantic relevance of the query. Very deep layers also not highlight the needle frames as the model has already consolidated the critical vi- sual information into the final token for next token generation. Interestingly, an intermediate layer achieves the highest Recall K, indicating that it best identifies the target visual tokens among its top-ranked outputs. In other words, L,e serves as the optimal attention layer that most reliably captures truly semantic-relevan tokens the se- quence. Based this finding, we designate layer Tree as the reference layer for guiding token selec- tion in FlexSelect. All subsequent token semantic- relevance computations in our method will use the reference layer's attention scores the measure- ment of semantic importance. More analysis including more base models and PCA visualization on tokens can be found in appendix A.I
reference layer +
60%
R
40%-
20%
0%
0
8
12 16
1920
24 28
Layer Index
Figure 2: Recall@ K values across different lay- ers LLaVA- ViVAA----eo-BBB... Recall K metric is the recall ratio of ground-truth relevant tokens (e.g., needle-frame tokens) among the top-K tokens ranked by a layer's roos- attention scores. A higher Recall @ indicates the attention scores of that layer can more accurately identify the se- mantically related visual tokens. We choose the optimal layer with the highest Recall @ K as the reference layer for token selection.
3.2 Training-Fres Free FlexSelect Pipeline
Even with the reference layer Tyer identified, processing long video in single forward pass is typically infeasible due to the quadratic cost of self-attention and memory constraints. To address

---

query
query:
frame sets partition
Token Selector
What color is this cup?
projector'
text embedding,
F
P2
videol I layers
lightweight model
encoding
Layer
Layer
Layer
Layer
T2
Tx
reference layer
lost layer
selection
token selection
token selection
cross-modal attention scores
(sel)
2
head
head
aggregate
average
semantic relevance scores
projector
rank
text embedding
L.M Decoder
White.
(inl)
Figure Overview of FlexSelect token selection pipeline. Given a long video and a query, FlexSelect first partitions the video into frame sets and encodes each into visual tokens. For each set, a token selector identifies semantically relevant tokens by ranking cross-modal attention scores from reference layer in pre-trained VideoLLM
lightweight selector network trained approximate it. In this process, the projector, and text embedding, are employed to convert the visual tokens and queries into tokens that match the dimension of subsequent transformer layers. After getting the scores, the top-ranked tokens across all segments are aggregated and projected into the decoder for final reasoning. FlexSelect operates in training-free rank-supervised mode, and serves blug-and-plat module that enables efficient long-video understanding without requiring modifications the base VideoLLM.
this, propose training-free FlexSelect strategy that enables efficient visual token selection while preserving semantic coverage across the entire video. The video is divided into multiple frame sets uniformly, with token selection performed independently within each set. This approach ensures comprehensive temporal coverage without requiring the entire video sequence to be processed at once.
Frame Sets Partition. We partition the extensive frames into frame sets and each most contains S frames to prevent token number exceeding the VideoLLM's maximum context length. Given a video with N' sample frames (f1, f2......
..a.....Jayy fN') }. we construct K [N/S] frame sets F2, FRx
such that each set samples frames at a same stride. Formally, the j-th frame set is defned as F, = (f+ j mod N}, where j (1..... K). This sampling ensures that each frame set spans the entire video with different temporal offsets, capturing diverse temporal dynamics and reducing redundancy between sets.
Each frame set F, is encoded by the VideoLLM's visual encoder. This yields a sequence of visual tokens T₁- (ty.1....,ty.. .t.M, for each set, M means the total number of visual tokens in frame set.
Semantic Relevance Scoring and Token Selection. Within each frame Fj. we compute semantic relevance score f for each token 4,3 using the attention mechanism at layer Lye:
H
1
A
(Laat) x
FJ.
H
where A ()
q the attention weight from the query tokens q token 15.4 in head h, and H is the number of heads. This relevance score reflects the semantic alignment between each token and the query. We then rank the tokens in each T, by Ty. and select the top-k tokens T TopK, ((r,))) yielding set of semantically relevant tokens for each frame set.
5

---

Aggregation and Final Token Composition. form the final visual token input T selected = - T, 2, This merged token set provides globally informed yet compact representation of the video.
The selected tokens from all frame sets are aggregated
By constructing frame sets with uniform sampling and processing them independently. our proposed FlexSelect strategy ensures that the framework scales efficiently to long video sequences. The method minimizes computational overhead by leveraging the parallclizability of processing smaller frame sets, while maintaining temporal fidelity across the video.
3.3 Rank-Supervised Lightweight Token Selector
While the training-free approach described above effectively reduces computational overhead, it still relies on partial forward passes through the large VideoLLM to score visual tokens. fur- ther enhance inference efficiency, we introduce lightweight token selector trained via rank su- pervision to predict semantic relevance scores independently. The selector model is explicitly designed to replicate the token-ranking behavior observed at the reference transformer layer Leef-
Architecture and Input. Our lightweight token selector a compact, shallow transformer-base network intended to substantially reduce inference costs compared to the large-scale VideoLLM. The model receives by the vision encoder and the corresponding tex- tual query. It outputs predicted semantic rele- vance scores for each visual token in the input sequence. Formally, for visual token sequence (2,,62..... far) paired with a textual query q. the selector produces scores {f1, f....... fM) } indicat- ing each token's 's predicted semantic alignment with the query. These scores are subsequently used to select the most relevant visual tokens, sim- ilar to the training-fire method described above.
visual tokens
query
projector
text embedding
projector,
text embedding,
layer
layer
inputs: visual tokens extracted
layer
layer
reference layer
layer
VM layers
lightweight model
gest
Lilla
Spearman
backward
.....
Figure 4: Illustration of our rank-supervised training. We align lightweight model's pre- dicted scores i with the reference layer's ss sematic relevance scores rf by optimizing the spear- man rank correlation coefficient between them. Once trained, the ranking derived from these two scores will follow similar order, enabling the lightweight model rank the visual tokens as the reference layer does and select the related tokens quickly.
To leverage pretrained knowledge and accelerate training convergence, initialize the selector from smaller-scale pretrained VidcoLLM (ap- proximately 0.5B parameters). In our experiments, we separately train selectors for LLaVA- VidA-i-d Qwen2.5VL-7B, and Intern VL2.5-8B.
Sank-Supervised Training Objective. We train the lightweight selector with rank supervision, directly leveraging semantic relevance rankings provided by the reference transformer layer Lnef from the larger Videol.L.. For each training video-query pair, we first compute the semantic relevance scores pe pef gfe ..... rreff at the reference layer Tyer- Then, the lightweight selector predicts
scores f = [1 f2. PM]] for the same tokens.
The training goal is to align the predicted semantic relevance ranking with the reference ranking
rel To quantify this alignment, we use the spearman rank correlation coefficient [34]:
Σ -t=
M (rank(r,*t) (ann( rank (f)
Pspcarman (x f)
M (rank(r;"f) rank(r"of)) Σ M (rank(f.) 2
Pspearman ref reef closer indicates stronger consistency between the ranking orders of re and while closer to 0 suggests no monotonic relationship. The corresponding training loss is defined as:
Crank 1 Pspcarman ee ()).
(4)

---

Model
Size
VideoMME
MLVU M-Avg
LongVB
LVBench
Long
Overall
Val
Test
Proprictary Models
GPT-4o 29
65.3
71.9
64.6
66.7
34.7
Gemini-1..-Pro
67.4
75.0
64.0
33.1
Open-Source VideoLLMs
LongVU
59.3
63.7
52.1
43.5
mPLUG-OwI
45
7B
50.1
59.3
63.7
52.1
43.5
NVILA 25
8B
54.8
64.2
70.1
57.7
VideoLLaMA3
46
7B
66.2
73.0
59.8
45.3
Aria
8x3.5B
67.6
70.6
65.3
Oryx-1.5
26
34B
59.3
67.3
72.3
62.0
30.8
Vidco-XI.-Pa
24
3B
60.0
70.6
56.7
LongVU 32
7B
59.5
60.6
65.4
VF-LL
43
7B
63.9
71.5
62.5
45.3
TPO 19
65.6
60.1
Quato 28
7B
55.7
65.9
71.9
59.0
ViLAMP
9
7B
57.8
67.5
72.6
61.2
45.2
VidcoChatFlash
20
7B
55.4
65.3
74.7
64.7
48.2
VLaVA-Video
53
7B
52.9
64.4
68.6
58.2
43.1
FlexSelect
7B
59.8
16.9
68.9
14.5
73.2
14.6
61.9
13.7
52.9
19.8
+ FlexSelect-Lite
58.3
15.4
68.3
13.9
71.8
f3.2
60.7 12.5
52.2 19.1
Ln2emVL2.5
52.8
64.2
68.9
59.5
43.4
+ FlexSelect
58.1 15.3
67.0
f228
71.9
13.0
60.1 10.6
49.7 16.3
+ FlexSelect-L.iee
8B
57.9
15.1
67.2
13.0
71.9
13.0
61.2 11.7
49.9 16.5
Qwen2.5-VL [
7B
55.6
65.4
70.2
59.5
45.3
FlexSelect
7B
59.3
13.7
68.2 12.8
72.5
12.3
62.4
12.9
51.2 15.9
FlexSelect-Lite
7B
58.6
13.0
67.4 12.0
70.3
10.1
61.9
12.4
50.0
14.7
I.LVA-Video
53
72B
61.9
70.0
71.2
62.4
45.5
FlexSelect
72B
66.1
14.2
73.1 f3.1
76.0 14.8
66.9 14.5
55.5
110.0
Qwen2.5 VL [
72B
63.9
73.4
76.3
66.2
47.3
Flexeelect
72B
66.9
13.0
74.4
1.0
76.6
10.3
66.4
10.2
56.6
19.3
Table 1: Comprehensive evaluation different long video benchmarks. Gray rows show base- line results reproduced from public model weights. FlexSelect employs attention scores from the reference layer in VideoI.IM for token selection, while FlexSelect-L.it utilizes scores from lightweight token selector. Our methods consistently improve performance when integrated into various VideoLLMMs by selecting semantically relevant visual tokens from extensive sampled frames. The implementation details of these results can be found in appendix A.4
Since the ranking operation (i.e. argsort) itself is con-differentiable, we adopt differentiable sorting algorithm (2) to approximate ranks and enable gradient backpropagation. This ensures effective training while strictly supervising the model on ranking quality.
Integration into FlexSelect Pipeline At inference, the lightweight selector directly replaces the transformer layers from the bigger VideoLLM process visual tokens and query inputs and produces semantic relevance scores, eliminating the need for intermediate VideoLLM computation. By using these scores to select only the top-ranking tokens, we significantly reduce computational overhead. enabling efficient long-video understanding at scale. For clarity, we denote the FlexSelect integrated with the lightweight token selector as FlexSelect-L.te.
Experiments
4.1 Models and Benchmarks
We conduct comprehensive evaluations across diverse VideoLLM architectures scales assess the generalizability of our FlexSelect, including: (1) ALaVA-Vides (7B/72B) [53), (2) Intern VL-2.5

---

8B [8] and (3) Qwen2.5VL (7B/72B) M. The models are evaluated on four established long-video understanding benchmarks: (1) Long VideoBench [4T] benchmark designed for accurate retrieval and reasoning in long-context videos, we report the validation set result. (2) MIVU (54, multitask benchmark specifically designed for long-form video understanding, we report the M-avg score. (3) VideoMME [2], comprehensive evaluation across short/medium/long videos, we report the the long and overall results without subtitles. (4) LVBench [38], an extreme-long video benchmark with average video length reaches to one hour.
4.2 Main Results
Compared to SoTAs As shown in Table 1 methods consistently enhance VideoLLLM per- formance across multiple benchmarks. For LiaVA-Video77 FlexSelect delivers a +5.5 points improvements in average (+4.5 on VideOMME, +4.6 on MLVU, +3.7 on LongVB, and +9.8 on LVBench), surpassing other long-form video understanding methods like SF-LLaVA [43] TPO [19], Quato 28 and ViLAMP derstanding. FlexSelect- Lite maintains most of these gains (+3.9 in average) with less computational cost, validating the effectiveness of our rank-supervised token selector. The method's adaptability is further confirmed by similar improvements when integrated into other models: Qwen2.5-VL-77 shows an average gain of +3.5 with FlexSelect and +2.3 with FlexSelect-L.ie while InternVL.2.588B achieves average improvement of +3.2 with FlexSelect and +3.7 with FlexSelect-l.ite.
7B parameter scale, demonstrating its effectiiveness for long video un-
For larger-scale models, FlexSelect continues deliver impressive results: LLaVA-Video-728 with FlexSelect shows a +4.5 improvement on LongVB (62.4 - 66.9), outperforming GPT-4o (66.7) Swen2.5VL-72B with FlexSelect sets new state-of-the-arr results among open-source methods, achieving +9.3 on LVBench (47.3 + 56.6). These consistent improvements across model sizes and benchmarks demonstrate the ability FlexSelect to select semantically relevant tokens, confirming the effectiveness of our token selection mechanism.
Compared to other Token Pruning Methods We compare FlexSelect with FastV (6) FrameFu- sion [13. Dycoke [33 and VisionZip 44 using the same base model. As shown in Table when sampling 32 64 frames, FlexSelect achieves better performance than these methods with the same retain ratio. When sampling 512 or 1024 frames, FrameFusion, Dycoke and VisionZip all encounter Out of Distribution (OOD) issues. FlexSelect, on the other hand, can effectively process these long input frames due to its Frame Sets Partition and Token Selection operation and achieves significantly better results.
Method
Sample Frames
Retain Ratio (%)
VideoMME (%)
LLaVA- -LVV--Vico-7BB
64
100.00
64.4
FrameFusion
64
30.00
61.3
FlexSelect
64
30.00
64.9
FrameFusion
512
6.25
OOM
FlexSelect
512
6.25
68.9
LLaVA-OV-7B
32
100.00
58.5
FastV
32
35.00
57.3
DyCoke
32
14.25
58.3
+ FlexSclect
32
14.25
60.4
Doke
512
6,25
0OM
FlexSelect
512
6.25
63.7
Qwen2.5-VI--7B
64
100.00
63.6
VisionZip
64
50.00
62.4
FlexSelect
64
50.00
62.6
VisionZip
1024
50.00
OOM
FlexSelect
1024
6.25
68.2
Table 2: Comparison of FlexSelect with other token reduction methods the VideoMME.
Efficient Long Video Understanding Our method significantly improves VideoL efficiency when processing long videos with extensive frames. We evaluate the response time (i.e., time cost to generate the first token) on LLaVA-Video-1 As shown in Figure 5 both FlexSelect and

---

40
Maxbeisii
70
30
65
Resporse Time
Irput Veuaa Taken Number
60
VodeoOA
56
3332
50
20000
45
2.50
4.04
2.89
A23
40
LengVe
NUYU
Dillnchh
0
120
35
14
19(L.er
27
Number of frames
Selection
Figure 5: Response time when sampling differ- number of frames. FlexSelect accelerates inference by selecting semantically relevant vi- sual tokens faithfully.
Figure 6: VideoQA Ace with different layers for token selection. The reference layer (19th) yields more faithful cross-modal ranking for token selection.
Input Frames
VideoMME (%)
Training Data Scale
VideoMME (%)
64
65.2
No training 1% (14k samples)
61.9
I 128
67.8
63.3
256
68. 1
2% (29k samples)
63.4
512
68.9
5% (67k samples)
63.9
1024
68.
10% (134k samples)
63.5
Max Selected Tokens
VideoMME (%)
Instruction Type
VideoMME (%)
1,680
67.1
Detail caption Open-ended QA
63.5
3,360
68.4
63.6
6,720
68.9
Multi-choice QA
63.7
13,440
68.
Mixed types
63.9
Table 3: Ablation input frames and max lected tokens of training-fiee free FlexSelect. Evalu- ations are conducted LLaVA-Video-T1
Table 4: Ablation data scale and instruction
type of lunk-supervised training. Our token se- lector boosts performance while requiring only 5% of training data.
FlexSelect-Liic reduce response time for the same number of frames by decreasing the visual token number. FlexSelect-Lite achieves more pronounced acceleration, highlighting the efficiency of our lightweight token selector. This speed advantage grows progressively with more sampled frames: for 512 frames, LLaVA- LLaVA-Video-78 requires 38.24 per sample, while FlexSelect-LLit reduces this to just 4.23 achieving a 9x speedup. More analysis on FLOPPs estimation can be found in appendix A.3
4.3 Ablations
Effectiveness of Reference Layer We compare the performance of FlexSelect on LLaVA- when using attention scores from different layers for token selection. The evaluation is conducted with 512 input frames and 6,720 max selected tokens. As shown in Figure 66 the reference layer (layer 19) performs the best, as it can more accurately select semantically related tokens.
Influence of Input Frames and Max Selected Tokens We evaluate LLa VVV--ido--7B with FlexSe- lect on VideoMME under varying input frames and max selected tokens. As demonstrated in Table 3 when fixing the max selected tokens to 6,720, we observe progressive accuracy improvements as the input frames increase from 64 512, followed by slight degradation at 1,024 frames. Similarly, with input frames fixed at 512, performance improves when increasing max selected tokens from 1,680 to 6,720, but further selecting 13,440 tokens reduces the accuracy. The result shows that insufficient input frames or selected tokens cannot adequately cover the video content, risking overlook critic infromation, while excessive frames or tokens may introduce semantically irrelevant noise both scenarios ultimately degrading performance.
Scales and Types Training Data data scale and video instruction types, and evaluate them with 64 input frames and 1,680 max selected
We train token selectors for VA-VVidd under different

---

tokens. We first randomly sample 4 subsets (1%, 2%, 5%, 10%) from LLaVA- without considering video instruction type, using them as training datasets to train different token selectors. As shown in Table directly initializing the token selector from small-scale VideoL.MM achieves an accuracy of 61.9 61.9%, while after training with just 1% of the data, the accuracy rises to 63.3%. The performance peaks when trained on 5% of the data (approximately 67k samples), and saturates with more data, demonstrating the quick convergency and training efficiency of our rank-supervised training. Subsequently, we fix the data scale at 67k samples while varying the video instruction types. Results show comparable performance across multiple-choice QA, open-ended QA, and video captioning data, suggesting that our training is effective on various instruction types.
[33
Datasets
Token Selector Params
Response Time
VideoMME
LongVB
MLVU
LVBench
0.5 B
4.0
67.2
61.2
71.9
49.9
1.8 B
7.4
67.3
60.6
71.7
49.7
3.0 B
12.0 s
67.4
61.8
72.8
50.1
Table Ablation on token selector parameters of rank-supervised training. We train different param- eter size token selector for Intern VL2 and test their acuracy on four benchmark. Evaluations are conducted with input frames set to 512 and max selected tokens set to 8,256.
Parameter Scale of Token Selector We investigate the impact of token selector parameter scale on performance. We train token selectors with 0.5B, 1.8B, and 3B parameters for VL2.5-VL2..88B. which are initialized from Intern VL2.5-1B, VL2.5-2B and VtterVL22.544P respectively, then we compare their performance on four benchmarks. Our results in Table 5 show that the 3B token selector achieves slightly higher scores than others but incurs significant computational overhead. and the 0.5B model delivers comparable performance while requiring only one- nhe-third of the response time of 3B model(4. (4.00 vs. 12.0s). This indicates that scaling parameter of token selector brings limited gains, and 0.5B a cost-effective choice.
We compare FlexSelect against the majority voting method to verify that the performance gains stem not merely from processing more frames but from the mechanism that locating the query-related visual tokens. For majority voting, temporal bins, randomly sample one frame per bin, and repeated this process 8 times. The final answer is determined by the most frequent prediction. As shown in Table 6 majority voting only requires more time (18.96 seconds) but also achieves significantly lower performance (64.9) compared to FlexSelect (68.9). This result strongly demonstrates that simply increasing the number of processed frames and smoothing noise via ensemble voting is far less effective than FlexSelect, which enables global consideration and fusion of key information across segments.
Compared to Majority Voting
divide each video into 64
Method
Total Sampled Frames
Time Cost
VideoMMEE
LLaVA- VVVe-V Majority Voting
64
2.37 s
64.4
64 x 8
18.96
64.9
FlexSelect
512
13.32
68.9
Table 6: Comparison between FlexSelect and the Majority Voting,
5 Conclusion
This paper presents FlexSelect, flexible and efficient token selection method that leverages cross- modal attention scores in VideoLLMs to identify query-relevant visual tokens. Our approach com- bines: (1) training-free free attention-ban based token ranking, and (2) lightweight selector for fast filtering. Its rrchttcture-agnostt
design enables integration into diverse VideoLLMs without modification. By selecting the most query-relavant visual tokens, Flex achieves SoTA results on VideoMME (74.4), MLVU (76.6), Long Bene (66.9), and LVBench (56.6) while reducing tokens by over 90% (up to 9x speedup). This method demonstrates powerful long-form video understanding capabilities, enabling effective analysis of long-form videos with minimal computational overhead.
10

---

Acknowledgements
This work is partially supported by Major program of the National Natural Science Foundation of China (T2295720/T2293723) This work was supported in part by "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2025C02032), the Fundamental Research Funds for the Central Universities 222-2222500005 This work was also supported by the Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University.
References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jiangiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, http//bbs/5220... 2502 13923.
2025. URL httpt: https:///arxiv org/ www./rgg/////////////..././eee//
[2] Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning, 2020.
[3] Rasmus Bro and Age K Smilde. Principal component analysis. Analytical methods, 6(9): 2812-2831, 2014.
[4] Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng. Vashisht Madhavan, Omer Bar- Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D. Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark. In International Conference on Learning Representations, 2025. URL https://axxxx org/ abs/ 2210/2400000 03051
[5] Boyu Chen, Zhengrong Yue, Siran Chen, Zikang Wang, Yang Liu, Peng Li, and Yali Wang. Lvagent: Long video understanding by multi-round dynamical collaboration of milm agents. In IEEE International Conference on Computer Vision, 2025. URL URL htttts://aaxxiiii org/ abs/
[6] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, 2024. URL org wbwp:////////////e//e/so 2403. 06764
https://arxiv.ct
[7] Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavio Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos, 2024. URL.httss://axxi org/ abs/ abs/2408. 22082 10188
[8] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian a Zhaoyang Liu, Lixin Gu, Xuchui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He. Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong Hc, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URL
abspa///...222/e/e/ /212/4422220 05271
[9] Chuanqi Cheng. Jian Guan, Wei Wu, and Rui Yan. Scaling video-language models to 10k frames via hierarchical differential distillation. In International Conference on Machine Learning, 2025. URL StRL:htts:://al //atpxs org/ h/5//ab///i ab2//44.. 02438
[10] Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. In IEEE International Conference on Computer Vision, 2025.
[11] Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos, 2024. URL https::///// http///xxxv...gaab.aaaa/a wrg.abs///0/3 14023

---

[12] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. In Conference on Computer Vision and Pattern Recognition, 2025.
[13] Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning. and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large visual language models. In JEEE International Conference on Computer Vision, 2025. URL httpl::///// http://aaxvv..oo org/ abs/ 211.220011191
[14] Lishuai Gao, Yujie Zhong, Yingsen Zeng, Haoxian Tan, Dengjie Li, and Zheng Zhao. Linvt: Empower your image-level large language model to understand videos, 2024. [RL https
//arxiv.org/ abs/ 2212 05185
[15] Jihyun Lee, Weipeng Xu, Alexander Richard, Shih-En Wei, Shunsuke Saito. Shaojie Bai, Te-Li Wang, Minhyuk Sung. Tae-Kyun Kim, and Jason Saragih. Rewind: Real-time egocentric whole-body motion diffusion with exemplar- based identity conditioning. In Conference on Computer Vision and Pattern Recognition, 2025.
[16] Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL https: //tp:v/aaiiv.///a orgg aat///ba////// 2408. .0440000.36666
[17] Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang. Yanpeng Li, Chongyan Zhu, Xiaoyi Ren, Chao Li, Yifan Ye, Peng Liu, Lihuan Zhang, Hanshu Yan, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model, 2025. https:: arxiv. org/abs/ 2410 22410.05993
[18] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730-19742. PMLR, 2023.
[19] Rui Li, Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Yeung-Levy. Temporal preference optimization for long-form video understanding. 2025. htt.b/ttss//arxii.rrrg...//
13919
[20] Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang. Videochat- flash: Hierarchical compression for long-contex. vidco modeling, 2025. URL httpss///axxii org/abs 0t////2510.000111
[21] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In Eumpean Conference on Computer Vision 2024. URL htttts:///a/xxi org/ Brg/abs//o abs//bbs/////// 23.1. 17043
[22] Jiajun Liu, Yibing Wang. Hanghang Ma, Xiaoping Wu, Xiaoqi Ma. Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input, 2024. URL /https://arxi: org/ hatgg/bb/24000 15542
[23] Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. Bolt: Boost large vision-language model without training for long-form video understanding. In Conference Computer Vision and Pattern Recognition, 2025. URL btttps://arxivv
h/s//ab//// abs////b/2/0...../// 2503. .21883
[24) Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Recon- structive token compression for extremely long video understanding, 2025. URL https
/ harxiv.or//abst 2ta.s////55..... 18478
[25] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang, Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, and Yao Lu. Nvila: Efficient frontier visual language models, 2025. URIps //arxiv.... www..orgggaoo... /bs/
2412. 2412.04468)
12

---

[26] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-tempon understanding arbitrary resolution. In International Conference on Learning Representations, 2025. URL.
[27] Yu Lu, Ruijie Quan, Linchao Zhu, and Yi Yang. Exploiting unlabeled videos for video-text retrieval via seeudo-uupervieed learning. Trans. Img. Proc., 33:6748 6760, January 2024. ISSN 1057-7149, doi: 10.1 00.100971P.222243545535 URL 3514622
2024
(28) Yongdong Luo, Wang Chen, Xiawu Zheng, Weizhong Huang, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Jieho Luo, and Rongrong Ji. Quota: Query-oriented token assignment via cot query decouple for long video comprehension, 2025. URL https 08689
//arxiv.
[29] OpenAI. Hello GPT-4o, 2024. URL htttpa///opmaa..commiieee/e///// Accessed: 2024-05-20.
[30] Bowen Peng. Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In International Conference on Learning Repre- sentations, 2024. htLp:ttp:://aree http//axxvvv.rg/bbbaa. wttp:///.w.rr///////////////// 00071
[31] Yujia Qin. Yining Ye, Junjie Fang. Haoming Wang. Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li. Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma. Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang. Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL http::
org/ abaaabss////0/ 12326
[32] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. Longvu: Spatiotemporal adaptive compression for long video-language understanding. In International Conference on Machine Learning, 2025. URL https:://///// //axivv org/ /bs/ 2410. 17434
[33] Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In Conference on Computer Vision and Pattern Recognition, 2024. URL
https://arxiv wwrexivv.rg//ee org//br//////a//oeoo abs/ 14485
[34] Charles Spearman. The proof and measurement of association between two things. In James J. Jenkins and Donald G. Paterson, editors, Studies in Individual Differences: The Search for Cantllttt---euurry -roftss 1961. doi: 10.1037/11491-005. URL https://doi. fttps://doi.orr/100 0tte//10.000//1100
Intelligence, pages 45-58. 11191-11911000 103.10009/11900000
[35] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In Conference Computer Vision and Pattern Recognition, pages 18992-19001, 2025.
[36] Gemini Team, Rohan Anil, Sebastian Borgeaud, (ean-Bapticn Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai. Anja Hauth, and et al. Gemini: A family of highly capable multimodal models, 2024. URL https: /tttps://arxiii org/ abs 2312. 11805
[37] Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:/t::///rr wttpaa/wwwwwwpraa/a//aa2//// 15115
[38] Weihan Wang. Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark. In IEEE International Conference on Computer Vision, 2024.
13

---

[39] Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding, 2025. URL https
7/ //arxiv..oo org/ abs 2203. 12559
[40] Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Du- plication matters more. In Conference Empirical Methods in Natural Language Processing,
2025. URL https: /arxivx.vvv/./////////////////// org/ abs/ 11494.
[41] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. In Neural Information Processing Systems, 2024.
[42] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. Pyramiddrop: Accelerating your large vision-languag models via pyramid visual redundancy reduction. In Conference on Computer Vision and Pattern Recognition, 2025. URL http:://aa.x...og///////1.... 17247
[43] Mingze Xu, Mingfei Gao, Shiyu Li, Jiasen Lu, Zhe Gan, Zhengfeng Lai, Meng Cao, Kai Kang. Yinfei Yang, and Afshin Dehghan. Slowfast-llava-I.5: A family of token-efficient video large language models for long-form video understanding, 2025.
/ttpps://axxaae http///xxvvv..rree abs/
2503. 18943
[44] Senqiao Yang. Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Conference on Computer Vision and Pattern Recognition, 2025. URL httth:t//axxiiii http:///axxvvv..gaabsssss/ www./ogg/////aoooooooooooo 2412
04467
[45] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owi ow13: Towards long image-sequence understanding in multi-modal large language models, 2024. URL htt.:: /ttpss///ariiv org/ abs/ 2408. 04840
[46] Boqiang Zhang, Kehan Li. Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang. Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025. URL huttps://araae org/ hatgg/bb//55/ 13106
[47] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025. URL https: //wxxv/aavvvvv//a org/ 2501 ..110011111111111..11
[48] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu, Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URL https://arxiv orgg abs/ 2407 12772
[49] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang. Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision, 2024. URL
httpt:tt:/ /tatps://arxiv.... org.eiiiiorrreeooo / aba
16852.
[50] Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang. Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. In IEEE International Conference on Computer Vision, 2025.
[51] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. In International Conference on Learning Representations, 2025.
[52] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Sparsevim: Visual token sparsification for efficient lasionnlanguage model inference. International Conference Machine Learning, 2025, URL https: / h /ttp///avvvv//////////eee//aaav/ www.wog.///oo.o..oooooo.oo... abs//410./210....//// 04417
14

---

[53] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. URL https:ttt:///rre http://aaaxi.....abbbb//aaaa orgg/bs///100
[54] Junjie Zhou, Yan Shu, Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking multi- task long video understanding. In Conference on Computer Vision and Pattern Recognition, 2025.
[55] Zitang Zhou, Ke Mei, Yu Lu, Tianyi Wang, and Fengyun Rao. Harmonyset: A comprehensive dataset for understanding video-music semantic alignment and temporal synchronization. In Conference Computer Vision and Pattern Recognition, 2025. URL httpss/ts///arii org aba/2503 01725
[56] Orr Zohar, Xiaohan Wang. Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, and Xide Xia. Apollo: An exploration of video understanding in large multimodal models, 2024. URL
/ttxs://arxiv.. arxiv...vv.. hwtp:/oorg/bb///////// 10360
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We carefully described our contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed our limitations appendix A.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
16

---

Justification:
This paper is not about theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided details about our methods and implementation the main paper 4.1 and appendix A.4 We also upload our main code supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

reproducible, they Answer: [Yes]
Justification:
We have released our code and trained model weights.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sewriides/CodSuummiise
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nip. httpcc/ccpllliggicccceeo httg///ddd//ooeuubmmim
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the experimental setup and details in the main paper 4.1 and appendix A.4
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Typically, results of the benchmarks we evaluated on do not include error bars. Morever, we found our results are quit stable across multiple runs. We provide our code, detail configuration and random seeds to facilitate the reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We clarify our computation resources at implementation details in appendix
A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We carefully reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed societal impact in appendix A.6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The data and models used in this paper have been extensively used in the multi-modal understanding community, and have undergone comprehensive safety risk assessments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provided appropriate citations in the reference section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included the code, data, along with detailed usage instructions in anonymous link. We will make it publicly once the review finished.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
21

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method developed in this study was derived through extensive eeooreesis-experinenatttoon cycles, without utilizing LLMs as any essential, novel, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: postps:/nneuripeee httprrppp...c/ooeeeeeneeee////2 for what should or should not be described.
22

---

A Appendix supplemental material
A.1 Details of @ecall@ K Experiment
The needle images and corresponding queries used in our Recall K experiment are directly borrowed from the V-NIAH experiment in LongVA [49]. which provides five needle-query pairs. We randomly sample 128 videos from the VideoMME [12] test set and insert each needle-query pair into them, resulting in a total of 640 test samples. We compute Recall K on these samples.
Needle Images and Queries
Recall @ K of Various Models In addition to LLa we calculate Recall@ K for LLa Video-72B, IntemVL2.5-8B Qwen2.5VL-7B and Owen2.5VL-722 The result is shown at Figure 7 We identify the reference layer of these models: layer 15 for LL2em layer 60 for LLaVA- Vidco-7B, layer 20 for Qwen2.5VL-7B and layer 60 for Qwen2.5VL-72B.
IntemVL2.5-88
LLaVA-Video-1228
Qwen2.5VL-78
Qwen2.5VL-722
70%
reference layer
70%
reference layer
70%
70%
60%
60%
60%
60%
reference layer
Recallex
50%
Recalles
50%
Recallie
50%
Recallex
50%
reference layan
40%
40%
40%
40%
30%
30%
30%
307
20%
20%
20%5
20%
10%
10%
10%.
10%.
0%
10 15 20 Layer Index
0%
20 40 so 70
0%
8
12
20 2427
06%
20
40 So 60 70
Layer Index
Layer Index
Layer Index
Figure 7: Recall across layers of different VideoLLMs.
Layer
Layer
Layer
Layer
10
10.0
7.5
2
3.0
2.5
0.0
25
5.0
10
10
10
20
Layor
Layor 20
Layer 24
Layec 27
10
30
-3
-10-
D-
-15-
-0-
40
Noodle Visual Tokers
coedle Visual Tiloos
Figure 8: PCA visualization of query tokens, needle visual tokens(i.e. semantically related tokens) and non-needle visual tokens from different layers of LLa VA-LVA--iidooo Video-7B. We found that the correlation between destributions of text tokens and needle visual tokens varies with layer depth, matching the trend of Recall @ K: shallow layers, their distributions diverge; intermediate layers, strong linear correlation emerges; while at very deep layers, this linear correlation weakens. This finding demonstrates that semantic alignment between visual and text tokens established at intermediate layers in VideoLLMs, which further explains the emergence of the reference layer.
PCA Visualization We further visualize the query tokens, needle visual tokens and non-needle visual tokens using Principal Component Analysis (PCA) K to examine cross-modal semantic alignment across different layers in VideoLLMs. As shown in Figure g (LLaVA-
23

---

significant distribution discrepancy exists between text tokens and needle visual tokens in shallow layers. As the layer depth increases, these distributions develop strong linear correlation in deeper layers, which subsequently weakens in the deepest layers. This indicates that the cross-modal semantic alignment is initially absent in early layers, gradually strengthens in intermediate layers, and then decreases in the final layers, which is consistent with the trend of Recall K.
A.2 More Evaluation Results
Compared Frame Selection Method We evaluate and compare FlexSelect with BOLT on the Video-MME benchmark. BOLT enhances long-form video understanding by selecting key frames, while FlexSelect operates at the token level. We sample fps (max to 512 frames) and select 32* 196 tokens per video, following the same setting as BOLT. As shown in Table the result demonstrates that FlexSelect with fine- eraigna token-level selection significantly surpasses BOLT that with frame-level selection.
Method
Sample Frames
Visual Token Number
Video-MME:
LLaVA-OV-BB
32
32 196
58.5
BOLT
512
32 196
59.9
+ FlexSelect
512
32 196
63.7
Table 7: Comparision with Frame Selection Methods.
Performance on Caption Tasks VidcoDC 4 is a detailed video captioning benchmark that evaluates the capability of models for caption generation. We compare results of VideoDC with and without FlexSelect on LLaVA-OV-7B. As shown in Table the result indicates that FlexSelect achieves comparable performance with the original model with 23.25% visual tokens retained. When models are prompted with queries like "Describe the video in detail", the reference layer's attention patterns still exhibit meaningful selectivity, focusing on important elements like main subjects and actions while deemphasizing unnecessary background. The cross-modal attention mechanism inherently adjusts its selection strategy based on query sype-focusing narrowly for specific questions while maintaining broader coverage for descriptive tasks. This indicates that FlexSelect's semantic relevance scoring effectively handles various video understanding scenarios. More Visualization can be seen at Figure
Methods
Sample Frames
Retain Ratio (%)
VideoDC Score
LLLVA-OV-7B
32
100.00
3,30
FlexSelect
32
23.25
3.29
Table 8: Performance on Caption Tasks.
Performance on Open-ended Tasks VideoEvvalPro [24] is a benchmark for open-ended video query answering, with an average video length of 38.25 minute. We compare the results of VideoEval- Pro with and without FlexSelect on different models. As shown in Table these results demonstrate that FlexSelect consistently improves performance on open-ended tasks, validating its generalization beyond multi choice tasks.
Methods
Sample Frames
Retain Ratio (%)
VideoEvalPro
LLa Vidco-7B
64
100.00
23.4
FlexSelect
512
6.25
30.7
Qwen2.5VL-7B
512
100.00
21.3
FlexSelect
1024
6.25
25.2
Intern VL2.558B
64
100.00
21.7
FlexSelect
512
6.25
28.2
Table 9: Performance on Open-ended Tasks.
24

---

A.3 FLOPs Analysis
Suppose the transformer decoder VideoLLM has L layers, h heads, and the hidden states size the intermediate size of FFN is m. For input sequence of n tokens, the FLOPs of the prefilling stage can be estimated as:
FLOPs = L x (4nd² + 2n'd + 2ndm) Re 2Ln²d,
since >> d and >> m in typical scenarios.
When using FlexSelect, assuming the M-th layer serves as the reference layer and the input sequence is partitioned into K segments via our frame set partition operation, ultimately selecting n' tokens, the FLOPs can be estimated as:
2ndd
2M
FLOPs' M x (4nd²
+ 2ndm) L (4n'd² + 2n 2n' dm) ~
K
K
since in our configuration, n'' = 0.0625n, making the L x (4n'd 2n + 2n' dm) term negligible in FLOPs'. Consequently, FlexSelect requires only M x K of the original FLOPs. I
Similarly, when employing FlexSelect-Liie the Flops be estimated 2L'n'²d K where L' and d'
are layer num and hidden states dimension of lightweight token selector. This FLOPs is more smaller than FlexSelect because L' < ans d' < d. FLOPs estimation only provides theoretical reference for computational cost. For practical considerations, we recommend referring to the actual time cost analysis in Figure 5
A.4 Implementation Details
The evaluations in main Table are conducted under LMMS-Eval [88] framework on 96G H20 GPUs. We set the input frames number N to 1024, 512, 512 for Dwen2.5VL(7B/72BB) LLaVA- Video(7B/72B), and Intern VL2.5-8B respectively, and max subset frames number S to 64 for all models. Denoting Nimage is the token number required for encoding one frame, we select 7,010 (64 Nimage), 6,720 (32 * Nuaage). 8,256 (32 Namuge) visual tokens for these models respectively, which is 6.25% of original input length. The reference layer L for token selection are set to Layer 15 for Intern VL2.5-8B. Layer 19 for LLaVA- VVVV-V----oBBB... Layer 20 for Owen2.5VVL-7B. Layer 60 for LLaVA-Video-721 and Qwen2.5VL-72B determined by arg max (L)).l@K(L)..
We train lightweight token selectors for 7B/8B models but exclude 72B due to memory constraints. For token selector of Video- we initialize it with the decoder of ALaVA-OneVisionn 0.5B [66] for token selector of V1.2.5-8. we initialize it with the decoder of InternVLL-II In the case of 2wen2.5VI.-7B, where no similarly-sized VideoLLMM .M is available, we directly initialize with Owwe2.5--nstrucctO0.
(37). We randomly select a small (5%) subset of L...V--Viia 178K [53] as the training data, which contains about 67k video instruction samples. We uniformly sample 64 frames from each video during selector training. Token selectors for LLa VA-Video and InterVVL2.. were trained for epoch, while the token selector for Qwen2.5VL was trained for epochs since it was initialized from language model, which is lack of visual prior knowledge, and requires more training steps achieve convergence.
A.5 Limitations
FlexSelect enhances VideoLLMs' long- understanding by selecting semantically relevant tokens, without modifying or retraining the base VideoLLM. Thus, its performance ceiling is bounded by the host VideoLLM's native capabilities.
FlexSelect-L trains a lightweight token selector to predict the reference layer's importance scores in large-scale VideoLLMs. While more efficient, it typically underperforms direct reference-layer token selection. Nevertheless, compared to other heavily trained alternatives and existing sub-optimal token pruning methods, FlexSelect remains efficient and effective solution for boosting diverse VideoLLMs' long-video comprehension.
25

---

A.6 Social Impacts
FlexSelect operates as a preprocessing module that selects semantically relevant visual tokens for diverse VideoLLMs. It is important to note that this method cannot prevent the base VideoLLMs from potentially generating erroneous, biased, or harmful hallucinations.
A.7
Visualization of Some Examples
What is the animal in this video?
FlexSelect
A chimp.
What's the color of the appeared in this video?
NEWAAT
FlexSelect
The color of the cup white.
oo
What is the first celestial object shown in the video?
13
FlexSelect
Earth.
O
Who ultimately the high jump competition the video?
FlexSelect
Athlete wearing yellow top and green shorts.
oo
26

---

Why does the woman need to drink water at the beginning of the video?
FlexSelect
Becouse she is preparing for medical examination.
oo
What is this video mainly about?
FlexSelect
O0
teaches how to tie a bow tie.
Describe what the video depict briefly.
FlexSelect
A virus attacks cell.
oo
What was the overall mood of the people interviewed?
FlexSelect
oo
Sad.
Figure 9; Examples of FlexSelect Local and Holistic Questions LLL We choose frames from the sampled frames for visualization.
27


Reviews:
[{"review_id": "8iCawjAFNO", "reviewer": "Reviewer_TShu", "summary": "This paper focuses on token selection in long video understanding with LLM. The authors analyze the attention distribution in existing VideoLLMs and find the optimal layer that indicates semantic relevance to guide token selection. The experiments show the efficiency of the proposed strategy.", "questions": "Did the authors try soft token merging or adaptive region resize instead of hard token selection through the learned token selector?", "limitations": "See weakness", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The semantic token selection significantly reduces the token number and retrains necessary information.\n2. The experiments show performance improvements with significantly reduced response latency.\n\nWeakness:\n1. The quantitative layer-wise semantic relevance analysis through a needle-in-haystack design is not ideal. The data distribution of the synthesized video is quite different from the natural videos.\n2. The semantic salient tokens alone are not sufficient for all questions. Besides, I am curious how will the model select tokens when given a general instruction like captioning.\n3. The detailed comparison on the number of selected frames, input tokens, performance and inference latency is desired. And the performance comparison on models with large context window is preferred, e.g., both sampling 512 frames w/ and w/o FlexSelect without exceeding the context window.\n4. More visualizations on the selected tokens are required.", "was_revised": true, "final_justification": "The paper provides a view for token selection in video understanding and presents competitive experimental analysis. And more OOD cases would make it more supportive.", "rebuttal": "We appreciate your valuable comments. In the following, we provide responses to the concerns.\n\n**Q1: The Needle-In-Haystack design is not ideal. The data distribution is different from the natural videos.**\n\nThanks for the question. The Needle-In-Haystack task is a well-established benchmark in video understanding. This task assesses whether a Video-LLM can identify visual cues from the extensive visual tokens based on the query, and has been used in recent works such as the NQA (Needle QA) test in MLVU, the V-NIAH experiment in LongVA, and Multi-Hop V-NIAH in VideoChat-Flash. We choose this task because it allows us to more clearly and easily observe the alignment between text and visual tokens across different layers. \n\nTo address your concern, we conducted additional experiments on natural videos using the temporal grounding task. We randomly sampled 128 videos from Charades-STA, where each video is paired with a ground-truth temporal range relevant to one question. We treated tokens within the ground-truth range as semantically relevant and others as irrelevant, then computed the layer-wise Recall@K for LLaVA-Video-7B. Here are our results:\n\n|Layer|Recall@K|Layer|Recall@K|Layer|Recall@K|Layer|Recall@K|\n|-|-|-|-|-|-|-|-|\n|0|0.2086|7|0.3092|14|0.3679|21|0.3638|\n|1|0.2587|8|0.3336|15|0.3155|22|0.3686|\n|2|0.2734|9|0.3423|16|0.3711|23|0.2641|\n|3|0.2790|10|0.3280|17|0.3344|24|0.2391|\n|4|0.2626|11|0.3448|18|0.2867|25|0.2541|\n|5|0.2996|12|0.3191|19|**0.3688**|26|0.2632|\n|6|0.3035|13|0.3021|20|0.3228|27|0.2385|\n\n\n\nThe results on natural videos show a consistent pattern with our synthetic experiments: Recall@K increases through early layers, peaks at intermediate layers (layers 16, 19, 22), and declines in deeper layers. This validates that our reference layer selection methodology generalizes to real-world video data.\n\n\n**Q2: The semantic salient tokens alone are not sufficient for all questions. What about the result of caption tasks.**\n\nThanks for your question. We acknowledge that different task types may require varying levels of visual information. Our experiments demonstrate that FlexSelect adapts effectively across diverse tasks, including captioning.\n\nIn Table. 1, we evaluated FlexSelect on four long video benchmarks (VideoMME, LongVideoBench, MLVU, and LVBench) that include diverse types of questions, such as complex reasoning, spatial temporal perception. Our results demonstrate that FlexSelect consistently enhances the performance of VideoLLM across these tasks.\n\nTo specifically address captioning, we evaluated FlexSelect on the VideoDetailCaption (VDC) benchmark using LLaVA-OV-7B with 32 sampled frames. Retain ratio refers to the keep ratio of visual tokens.\n\n|Method|Retain Ratio (%)|VDC score |\n|-|-|-|\n| LLaVA-OV-7B (Baseline)|100.00|3.30|\n| FlexSelect|14.25|3.11|\n|FlexSelect|18.75|3.20|\n|FlexSelect|23.25|3.29|\n|FlexSelect|30.00|3.30|\n\nThe results reveal that captioning tasks benefit from higher token retention ratios. With 23.25% retention, FlexSelect achieves comparable performance (3.29 vs 3.30) to the baseline while also improves efficiency a lot.\n\nInterestingly, even for open-ended queries like \"Describe the video in detail\" the reference layer's attention patterns still exhibit meaningful selectivity. The model naturally focuses on salient elements (e.g., main subjects, significant actions) while deemphasizing redundant background information. For example, when describing the video in Fig. 3, the model attends to relevant elements like the commentator and Christmas decorations while filtering repetitive backgrounds. The cross-modal attention mechanism inherently adjusts its selection strategy based on query type—focusing narrowly for specific questions while maintaining broader coverage for descriptive tasks. This demonstrates that FlexSelect's semantic relevance scoring is sufficiently flexible to handle diverse video understanding scenarios.\n\n**Q3: The detailed comparison on the number of selected frames, input tokens, performance and inference latency is desired**\n\nThank you for your advice. The effects of the input frame number and selected token number on performance and time cost have been analyzed in Table 2 and Figure 5. Here we provide a more detailed comparison. The tables below show the VideoMME results and time cost at different input frames and selected tokens.\n\n| |64 Frames|128 Frames|256 Frames|512 Frames|1024 Frames|\n|-|-|-|-|-|-|\n|1,680 Tokens|64.5|66.2|66.6|67.1|66.6|\n|3,360 Tokens|64.7|66.9|67.0|68.4|68.1|\n|6,720 Tokens|65.2|67.8|68.1|68.9|68.1|\n|13,440 Tokens|64.4|66.1|66.5|68.1|67.0|\n\n\n| |64 Frames|128 Frames|256 Frames|512 Frames|1024 Frames|\n|-|-|-|-|-|-|\n|1,680 Tokens|1.95s|3.63s|6.55s|12.82s|26.84s|\n|3,360 Tokens|2.01s|3.95s|6.86s|13.10s|27.40s|\n|6,720 Tokens|2.24s|4.64s |7.38s|13.32s|27.58s|\n|13,440 Tokens|2.37s|5.54s|8.54s|24.87s|29.32s|\n\nOur experiments show that using 512 input frames with 6,720 selected tokens (6.25% retention) achieves 68.9% accuracy on VideoMME, improving from the original model's 64.4%. This supports our hypothesis that semantic token selection enhances efficiency and accuracy in long-form video understanding.\n\nAdditionally, FlexSelect enables processing up to to 1,024 frames—avoiding OOM errors in the baseline—while maintaining a competitive 68.1% accuracy, offering crucial scalability for real-world applications needing extended temporal coverage.\n\n**Q4: The performance comparison on models with large context window is preferred, e.g., both sampling 512 frames w/ and w/o FlexSelect without exceeding the context window.**\n\nThanks for the question. Qwen2.5VL series models support long window context. Following the official repository, we use YARN with scaling factor = 4.0 to expand the context window length from 32k to 128K. The result of Qwen2.5VL (original) we reported in the main table was tested under 768 sampled frames (about 107k tokens). Here are the results:\n\n|Model|Sample frames |Visual token number | VideoMME |LongVB|MLVU| LVBench |\n|-|-|-|-|-|-|-|\n|Qwen2.5VL|768| About 107k|65.4|59.5|70.2|45.3|\n|Qwen2.5VL (FlexSelect)|1024|7010|68.2|62.4|72.5| 51.2|\n\nThe results show that FlexSelect with just 7,010 tokens outperforms the baseline with 107k tokens, achieving higher scores on all four benchmarks. By focusing on semantically relevant tokens, FlexSelect reduces the noise and redundancy inherent in dense video representations and improves performance, independent of the model's context window length.\n\n**Q5: Did the authors try soft token merging or adaptive region resize instead of hard token selection through the learned token selector?**\n\nThanks for the insight discussion. We explored both soft token merging and adaptive region resizing as alternatives to hard token selection, though neither achieved comparable results to FlexSelect.\n\nFor soft token merging, we implemented a LinVT-inspired approach on InternVL2-2B, which trains an adapter to compress visual tokens into fixed number of token sets before they are fed into LLMs. The adapter consists of a Spatio-temporal Visual token Refiner module (SVR) to distill the video information into a concise set of visual tokens, and a Text-conditioned Token Aggregator (TTA) module to refine and integrate the visual tokens with the textual information. In our implementation, we use 4 full transformer layers as TTA and 8 spatial-temporal transformer layers as SVR. \n\nThe parameters of the adapter are randomly initialized and we trained it in two stages on video instruction data. The first stage only trains the adapter and the second stage trains both the adapter and the LLM.  We finally found that such training degraded the performance. \nHere are our results:\n\n||Frame Number| Token Number  |VideoMME |\n|-|-|-|-|\n|Original model| 32| 32*256=8192| 51.5|\n|Soft Token Merging Training| 32| 1024| 50.5|\n\nWe hypothesize that learning-based compression introduces information loss that cannot be recovered, whereas FlexSelect preserves the critic information well by selecting semantically relevant visual tokens.\n\nFor adaptive region resizing, we explore a similar training free method on InternVL2.5-8B. The visual encoder of InterVL2.5-8B supports both of 224 * 224 and 448 * 448 resolution inputs. We first resize the video frames into the small resolution and locate the important frames based on the query-to-visual relevance scores. And then we resize the important frames to the higher resolution while remains the other frames as low resolution. Here are our results:\n\n|| Sample Frame Num |High Res Frame Num| Visual Token Num | VideoMME |\n|-|-|-|-|-|\n| original model| 64 | 64 | 16384| 64.2|\n| adaptive resize| 64| 16| 7168| 62.4|\n| adaptive resize| 64| 32| 10204| 63.7|\n| adaptive resize| 128 | 32| 14336| 64.2|\n| FlexSelect | 64 | - | 2458 | 65.6|\n\nResults showed consistent underperformance: with 64 frames (16 high-res), accuracy dropped from 64.2% to 62.4%; even with 128 frames (32 high-res), we only matched baseline performance while using 14,336 tokens, while FlexSelect's 2458 tokens achieving 65.6%. \n\n**Q6:More visualizations on the selected tokens are required.**\n\nThank you for this suggestion. In our revised version, we will include more comprehensive visualizations to fully demonstrate FlexSelect's robustness and adaptability across different tasks.\n\nBesides the visualizations in Appendix A.6 (Figure 9), we observe consistent patterns in different question types: (1) Object-specific queries focus token selection around relevant objects; (2) Action queries select tokens across motion sequences with temporal coherence; (3) Counting queries distribute selection to capture all instances. These visualizations show our cross-modal attention mechanism effectively adapts to query semantics.\n\n**References:**\n1. [LinVT: Empower Your Image-level Large Language Model to Understand Videos](https://arxiv.org/abs/2412.05185)"}, {"review_id": "wemZa1ojNp", "reviewer": "Reviewer_xDCz", "summary": "The work introduces a novel way of subsampling visual tokens for speeding up execution of VideoLLM in VQA settings. The authors observe that textual tokens in the query tend to cross-attend only to a small set of visual tokens, while the vast majority receives low or zero attention. Therefore they propose to use this as a ranking signal for identifying the top-K tokens to keep and discard the rest. This technique is particularly effective when applied at a specific layer of the VideoLLM (identified by the authors and architecture dependant), however it still requires to execute part of the VideoLLM at the full sequence length. As an alternative to further speed up computation the authors propose to learn a standalone lightweight LLM to predict the token ranking, this solution brings down computational cost significantly at a small price in terms of quality. The proposal is tested on top of 5 different VideoLLM across 4 video understanding benchmarks and it can consistently improve performance both in terms of accuracy and speed.", "questions": "* How is Eq. 1 averaged over all tokens composing the question?\n\n* Have you tried to select only temporally? AKA keyframe rather than sparse tokens? \n\n* How is “No training” implemented in Tab. 3? The tiny LLM should not generate ranking scores without re-training.\n\n* In Tab. 4 did you scale the number of samples seen together with the model parameters? Could the bigger models be undertrained?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n+ FlexSelect (the non lite version) is a training free method that promises to be applicable to any off the shelves VideoLLM. The only requirement is identifying which layer is the best to perform the selection which is a one off operation to do. As such the proposal is very flexible and could readily be deployed on any up and coming model in the future.\n\n+ I appreciated the proposal of FlexSelect-Lite as a way of providing additional speedups for a small cost in terms of accuracy. The proposed loss for training is well explained and nicely grounded on differentiable sorting and allows to dynamically select at inference time the computational budget to use since tokens are completely sorted in order of importance. Other methods in the past used binary classification to decide which tokens to keep or drop which would not allow the same flexibility . \n\n## Weaknesses\n\na. **Lack of comparison to other training free token dropping methods**: the method is not compared to state of the art solutions in the field, so it is hard to quantify the effectiveness of the proposal compared to SOTA. In particular works it might be worth comparing against are [1] for frame selection and [2,3] for token selection. If a comparison is not pertinent it should be explained in the paper why. Currently this is not addressed at all. In particular this work seems in my opinion conceptually very similar to [3] with the addition being performing the pruning at a later layer and the “lite” version of the method.\n\nb. **Cross-attention based pruning**: the author propose to prune tokens based on the aggregated cross-attention scores at a certain layer between query and visual tokens, intuitively this makes sense only for queries that require a “needle-in-haystack” like scenario, it does not for queries that would require to observe the full video, e.g., “list down the actions in this video”. Moreover, following the experimental analysis of [2], the cross-attention pattern that matters more is the one between the output tokens that the model will generate and the visual tokens; these might also change at every decoding iteration. The results from this work are somehow contradicting the results from [2] and, in lack of a proper comparison, as a reviewer I tend to agree more with the analysis provided in [2].\n\nc. **Identification of the layer to use for cross-attention computation**: following on weakness [b] the authors base their motivation and their layer selection on a “needle-in-haystack” problem. I’m wondering how much the findings would hold for a different category of problems, like how the attention pattern looks for a captioning task? If the outcome of the analysis would be very different and the attention would be quite distributed across several more tokens then the findings of the work will hold only for a subcategory of problems and this should be clearly explained in the work, which is instead currently presented as a general solution for any kind of video understanding problem. \n\nd. [minor] **Need to train FlexSelect-Lite**: The lite version of the method needs to be trained on certain data and might underperform when tested on tasks different from the one it has seen while training. Moreover since the Lite model uses a tiny LLM might be more exposed to the risk of overfitting the training distribution and not being able to handle a bigger set of training samples that would improve generalization (see for example Tab. 3 where performance deteriorates moving from 5 to 10% of training data). Having said that, there does not seem to be a negative generalization on the reported results.\n\ne. [minor] **Potentially missing baseline**: Tab. 5 reports increasing computational saving when consuming more and more frames. However I feel there is a crucial difference in the set up between the proposed method and the baseline LLava-Video, namely that for the baseline the frames are processed all at once, while the proposed method processes first independent partitions of the video selecting only a subset of frames and only at a later stage aggregates them. I wonder what the performance difference would be if the same “ensembling like” approach would be applied to LLava-video. So for example instead of processing 512 frames in parallel, the model would be used to process 8 partitions with 64 frames each and answers would be aggregated using some form of majority voting. Time would be 2.37 * 8 = 18.96 second, making the gains from FlexSelect less impressive, and the ensemble might still benefit final performance of the model.  \n\nf. [minor] **Fixed token budget**: The method does not incorporate a way of dynamically selecting how many tokens are relevant for the given task. This would be very useful for practical applications.\n\n## References\n\n1. [Liu, Shuming, et al. \"BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.](https://arxiv.org/pdf/2503.21483) \n\n2. [Tao, Keda, et al. \"DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.](https://openaccess.thecvf.com/content/CVPR2025/papers/Tao_DyCoke_Dynamic_Compression_of_Tokens_for_Fast_Video_Large_Language_CVPR_2025_paper.pdf)\n\n3. [Chen, Liang, et al. \"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.](https://arxiv.org/pdf/2403.06764)", "was_revised": true, "final_justification": "The authors have addressed most of my criticisms in the rebuttal and the many new results and comparison help to position this work wrt competitors. For this reason I'm changing from a negative rating to a leaning positive one.", "rebuttal": "We appreciate your valuable comments. Below are our responses:\n\n**Q1: Lack of comparison to other token dropping methods**\n\nWe appreciate this suggestion and will include direct comparisons in the revision. We compare the VideoMME result with BOLT, DyCoke and FastV on LLaVA-OV-7B.\n\nFor comparison with BOLT, we sample at 1 fps (max to 512 frames) and select 32 * 196 tokens per video, which is the same as BOLT. The result shows that FlexSelect with fine-grained token-level selection significantly surpass BOLT that with frame-level selection.\n\n| | VideoMME|\n|-|-|\n|LLaVA-OV-7B| 58.5|\n|BOLT|59.9|\n|FlexSelect|63.7|\n\nFor comparison with DyCoke and FastV, we maintained their experimental settings, sampling 32 frames per video. FlexSelect consistently outperforms both DyCoke and FastV across different retain ratios.\n\n|| Retain Ratio (%)| VideoMME wo Sub | VideoMME w Sub |\n|-|-|-|-|\n|LLaVA-OV-7B|100|58.5|61.3|\n|FastV|35|57.3|60.5|\n|DyCoke|14.25| 58.3|60.7|\n|FlexSelect|14.25| 60.4|62.2|\n|DyCoke|18.75| 59.5|61.4|\n|FlexSelect|18.75|61.0|62.4|\n|DyCoke|23.25|58.8|61.0|\n|FlexSelect|23.25|60.4|63.0|\n\nThe key difference from FastV is our systematic identification of the optimal reference layer through semantic alignment analysis. FastV considers the proportion of visual tokens' attention scores in the whole attention map, suggesting that layers with higher proportions are more suitable for token pruning. It points out that in MLLM, visual tokens garner an average attention score that amounts to only 0.21% of the score attributed to system prompts, but this figure reaches 50% in the initial two layers. Therefore, they choose to prune tokens at layer 2. FlexSelect considers the semantic alignment between the vision and language modalities, suggesting that layers with attention scores that better locate query-related visual tokens are more suitable for token pruning. Thus, we prune tokens at the reference layer. The experiment results show that our method significantly surpasses FastV.\n\n**Q2: Beyond the NIAH-Type queries**\n\nThanks for your question. \n\nOur method generalizes beyond NIAH-Type scenarios. It performs well for both NIAH-type and holistic queries. We evaluate FlexSelect on four well-designed long video benchmarks that include diverse types of questions, such as complex reasoning, spatial temporal perception. Our results show that FlexSelect consistently enhances the performance across these tasks, no matter the query is NIAH-type or holistic-type. For example, MLVU benchmark contains Holistic Task that consists of Topic Reasoning (TP) and Anomaly Recognition (AR). These questions require the models to watch the entire video to answer correctly.\n\n||TP|AR|\n|-|-|-|\n|LLaVA-Video-7B|84.8|66.5|\n|LLaVA-Video-7B+FlexSelect|87.1|70.0|\n\n**Q3: Concerns about contradictions with the results of DyCoke**\n\nThanks for your question. The experimental analysis in Table 5 of DyCoke [12] shows that cross-attention based pruning (w/o DP) performs worse than the original model, which seems to contradict with our results but actually not. There are crucial differences that explain our divergent findings:\n\n1. Attention type is different. DyCoke uses output-to-visual attention while we use query-to-visual attention. This is fundamentally different - we analyze attention during the prefilling stage before generation begins, while they analyze attention during the decoding process.\n2. Layer selection strategy is different. DyCoke performs pruning at layer 3, while we systematically identify and use the optimal reference layer (e.g., layer 19 for LLaVA-Video-7B).\n\nWe compared the results of using these two types of attention scores for token pruning at layers 3 and 19, and the results are below. The experiment is conducted under LLaVA-OV-7B with 32 sampled frames. We found that (1) Query-to-visual attention contains more explicit semantic information and consistently outperforms output-to-visual attention. (2) Regardless of the attention type, the performance at layer 3 is worse than the original model, which aligns with FlexSelect's analysis and is consistent with the results in DyCoke.  (3) Our reference layer selection is crucial for achieving performance gains. The dramatic performance difference between layer 3 and layer 19 pruning (56.7→60.4 for query-to-visual) underscores the importance of our systematic approach to identifying the optimal reference layer.\n\n|Prune Layer|Retain Ratio (%)|Cross-Attn Type|VideoMME|\n|-|-|-|-|\n|Layer 19|14.25|output-to-visual|58.1|\n|Layer 19|14.25|query-to-visual|60.4|\n|Layer 3|14.25|output-to-visual|55.8|\n|Layer 3|14.25|query-to-visual|56.7|\n||||\n|LLaVA-OV-7B|100|-|58.5|\n\nWe ensure that our results are reproducible, and will open source all the code related to the results in the future.\n\n**Q4: Caption Task Result**\n\nThanks for the question, we compare results of VideoDC on LLaVA-OV-7B with 32 sampled frames, and will include it in our revision. Here are the results. \n||Retain Ratio (%)|VideoDC score|\n|-|-|-|\n|LLaVA-OV-7B|100.00|3.30|\n|LLaVA-OV-7B+FlexSelect|23.25|3.29|\n\nThe result shows that FlexSelect achieves comparable performance with the original model. We find that when models are prompted with queries like \"Describe the video in detail\", the reference layer's attention patterns still exhibit meaningful selectivity, focusing on important elements like main subjects and actions while deemphasizing unnecessary background. The cross-modal attention mechanism inherently adjusts its selection strategy based on query type—focusing narrowly for specific questions while maintaining broader coverage for descriptive tasks. This indicates that FlexSelect's semantic relevance scoring effectively handles various video understanding scenarios.\n\n**Q5: Overfitting and generalization of token selector**\n\nThanks for the discussion. Our token selector is trained on a randomly sampled subset of LLaVA-Video-178K, which contains diverse task types including QA and captioning.\nThe results in Table. 1 show that our token selector (FlexSelect-Lite) can generalize well to different question types.\n\nRegarding model scaling (Table 4), larger token selectors do achieve marginally better performance (67.2→67.4), suggesting improved generalization capacity. However, this comes at significant computational cost (3× response time for 3B vs 0.5B models). Given the minimal performance gains, the 0.5B model offers the optimal efficiency-performance trade-off for practical deployment.\n\n**Q6: Missing baseline: Majority Voting**\n\nWe compare majority voting with FlexSelect on LLaVA-Video-7B and will include it in our revision. Here are our results:\n\n||Sampled Frames In Total|Time Cost|VideoMME|\n|-|-|-|-|\n|LLaVA-Video-7B|64| 2.37s|64.4|\n|Majority Voting|64*8|2.37*8=18.96s|64.9|\n|FlexSelect|512|13.32s|68.9|\n\nFor majority voting, we divided each video into 64 temporal bins, randomly sampled one frame per bin, and repeated this process 8 times. The final answer was determined by the most frequent prediction.\n\nFlexSelect significantly outperforms majority voting while being more efficient. It enables the model to reason over semantically relevant tokens from the entire video simultaneously, rather than combining isolated predictions.\n\n**Q7: Dynamic token selection for tasks**\n\nThanks for your insightful advice. Actually, the number of necessary visual tokens varies by task, with some tasks relying on a few key tokens and others requiring a thorough visual analysis. As shown in table below, for caption tasks (VideoDC), performance improves as more tokens are selected, but this trend does not apply to VideoMME. While adaptively choosing the number of tokens based on user queries is a promising research direction, it poses challenges due to the complexity of the task and the difficulty of the video content. We leave this as the future work.\n\n||Retain Ratio (%)|VideoDC|VideoMME|\n|-|-|-|-|\n|FlexSelect|14.25|3.11|60.4|\n|FlexSelect|18.75|3.20|61.0|\n|FlexSelect|23.25|3.29|60.4|\n|FlexSelect|50.00|3.30|59.5|\n|LLaVA-OV-7B|100.00|3.30|58.5|\n\n**Q8: How Eq. 1 averaging scores across question tokens**\n\nWe average the attention scores between each visual token and all user query tokens across head dimension:\n\n $r_i = \\frac{1}{H \\cdot L_t} \\sum_{h=1}^{H} \\sum_{j=1}^{L_t} attn[h,j,i]$\n\nwhere H is the number of attention heads; $L_t$ is the number of query tokens; attn is the attention score between the j-th query token and the i-th visual token.\n\n**Q9: Temporal selection only**\n\nThanks for this discussion. We explored attention based frame selection by averaging the scores (derived from reference layers) across the frame dimension and inference only with the top-k highest-scoring frames. Here are our results on VideoMME using LLaVA-Video-7B.\n\n|Methods|Sample Frames|Selective Frames|Visual Tokens|VideoMME|\n|-|-|-|-|-|\n|Original|128|-|128*210|64.3|\n|Frame Selection|128|64|64*210 (13440)|65.4|\n|Token Selection|128|-|13440|66.1|\n\nThe results show that frame selection slightly improves performance but is less effective than token selection when using the same number of visual tokens. We find that this is because most attention scores are nearly uniform and minimal, and some top-scoring visual tokens are irrelevant and act as noise. As a result, the average attention scores at the frame level become smooth, making it hard to identify key frames.\n\n**Q10: \"No training\" in Table 3**\n\nThanks for the question. In the \"No training\" configuration, we directly utilize attention scores from the final layer of the tiny LLM to rank visual tokens without training. Owing to the pre-existing visual knowledge, the tiny LLM could produce a reasonable rank.\n\n**Q11: Were larger models undertrained?**\n\nThanks for this question. The larger token selector in Table 4 was well-trained. We attempted to train the larger token selector on a larger-scale dataset and found that the loss had already converged, and the performance remained basically unchanged when training with more data."}, {"review_id": "J64NZ5gwCP", "reviewer": "Reviewer_7tpr", "summary": "The authors propose FlexSelect, a token selection strategy designed to enhance the efficiency and scalability of VideoLLMs in processing long-form videos. The approach leverages cross-modal attention patterns from a reference transformer layer to identify and retain semantically relevant content, while filtering out redundant information. This is achieved through two main components: (1) a training-free token ranking pipeline that uses cross-modal attention weights to estimate token importance, and (2) a lightweight rank-supervised selector trained to replicate these rankings. The proposed method is designed to be modular and can be readily integrated into existing VideoLLM architectures, serving as a plug-and-play solution to extend temporal context handling capabilities. Empirical evaluations demonstrate consistent performance improvements across multiple long-video benchmarks, including VideoMME, MLVU, LongVB, and LVBench. Notably, the approach also yields significant computational speed-ups, highlighting its potential for practical deployment in resource-constrained setting.", "questions": "See weakness，my main concerns are as follows:\n\n1. Is there a necessity to adopt an LLM refer layer instead of traditional video-language models for token selection? Currently, there is a lack of convincing experimental evidence to support this approach.\n\n2. In what aspects do the key innovations and insights provided in this paper differ from previous studies?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strength\n\n1. The approach of partitioning long videos and adopting token selection proposed in this paper has achieved remarkable effectiveness. It significantly reduces the computational complexity while improving the performance of short-video models in processing long videos.\n2. The motivation and analysis of this paper are both reasonable and sufficient. The proposed solution also effectively addresses the corresponding issues, featuring a straightforward and easy-to-follow implementation.\n\n### Weakness\n\n1. Whether using an LLM layer as the refer layer outperforms traditional video language models (e.g., Siglip[1], InternVideo2[2]) remains unaddressed, especially when training a weaker student model for ranking—many studies [3, 4] have shown that traditional video embedding models can achieve similar results. The paper lacks relevant performance comparisons.\n2. The motivations for segmenting long videos for token reduction and using attention scores from deep LLM layers have been previously proposed in VideoChat-Flash[5[, but this paper omits comparisons and discussions with that work.\n3. The core approach of visual token reduction prior to LLM processing has been explored in prior studies (e.g., VisionZip[6]). The paper requires more detailed comparative analyses or experimental result comparisons with these works.\n4. The proposed compression method only enables local compression, failing to consider global compression. For instance, if only the final segment of a long video is relevant, the method still retains preceding content unnecessarily.\n\n\n\n[1] Sigmoid Loss for Language Image Pre-Training\n\n[2] InternVideo2: Scaling Foundation Models for Multimodal Video Understanding\n\n[3] LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agents\n\n[4] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding\n\n[5] VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling\n\n[6] VisionZip: Longer is Better but Not Necessary in Vision Language Models", "was_revised": true, "final_justification": "After careful consideration, I believe the paper deserves a rating above 4, though it does not yet meet the threshold for a 5. If possible, I would assign a 4.5. Overall, I maintain a positive opinion of this work.", "rebuttal": "We are truly appreciated for your valuable comments. In the following, we provide responses to the concerns.\n\n**Q1: The proposed compression method only enables local compression, failing to consider global compression**\n\nRegarding the concern about local vs. global compression, FlexSelect performs global token selection through our uniform frame set partition strategy (Sec 3.2). Rather than simple chunk-by-chunk segmentation, we partition frames uniformly across the entire video:\n\nFor example, if we have 512 frames and want to split them into 8 segments, segment 1 contains frames [0, 8, 16,..., 504]; segment 2 contains frames [1,9, 17,..., 505]; segment 8 contains frames [7, 15, 31,..., 511].\n\nThis ensures each segment has a global view of the entire video, allowing relevant tokens from any temporal position (including only the end of the video) to be selected.\n\nWe compare our uniform sampling method with the chunk-by-chunk sampling using LLaVA-Video-7B. The results are below:\n\n|Method|Frame Num|Segments Num|VideoMME|\n|-|-|-|-|\n|Chunk-by-chunk|512|8|67.4|\n|Uniform (Ours)|512|8|68.9|\n\nThis demonstrates our method's effectiveness in global compression while maintaining computational efficiency.\n\n**Q2: The motivations for segmenting long videos for token reduction and using attention scores from deep LLM layers have been previously proposed in VideoChat-Flash**\n\nWe acknowledge VideoChat-Flash as excellent related work and will add proper citations. Here we clarify our key differences:\n\n1. The segmentation strategy is different. VideoChat-Flash segments videos into **4-frame clips chunk by chunk** and extracts visual features using a video encoder, then employs an MLP to further compress each clip's features into a more condensed representation. This reduces each frame to just 16 tokens, dramatically decreasing the total token count for long videos and improving efficiency. The motivation behind the segmentation is that **the substantial redundant and repetitive information, such as backgrounds and objects present between adjacent frames in natural videos**. \n\n    FlexSelect, however, segments videos to ensure **the context length of each segment within the model’s maximum context limit**, enabling existing VideoLLMs to select relevant visual tokens from hundreds of sampled frames. Meanwhile, as clarified in Q1, FlexSelect employs a **uniform segmentation** to preserve a global view of the entire video within each video clip, which is different from the chunk-by-chunk segmentation in VideoChat-Flash.\n\n2. The token pruning strategy is different. VideoChat-Flash employs a progressive visual token dropout strategy based on the observation that when an LLM processes long video contexts, it **attends to the entire video in shallow layers while focusing on local details in deeper layers**. It uniformly drops video tokens in shallow layers and selectively drops them based on attention scores in deeper layers, further improving efficiency and slightly enhancing performance.\n\n    However, FlexSelect analyzes the semantic alignment between text and visual tokens across different layers, identifying **an intermediate layer that aligns these two modalities the best**. FlexSelect simply averages attention scores from this layer and drops low-scoring tokens, achieving significant performance improvement.\n\n**Q3: Traditional video model OR LLM layers for token selection? And comparison with VisionZip**\n\nThank you for the insightful discussion. Both LVAgent and LongVU employ traditional visual encoders to extract frame features and select relevant video regions. LVAgent introduces a novel multi-round framework that enables collaboration between multiple VideoLLM agents, achieving excellent performance in long video understanding tasks. In its perception stage, LVAgent finetunes a CLIP-ASP visual encoder to extract frame features and calculate clip scores to selecte important video segments. Similarly, LongVU uses DINOv2 to extract visual features from sampled frames and reduces temporal redundancy by computing cosine similarity between features to select relevant frames.\n\nLongVU’s core contribution lies in its efficient training on video data—its five-stage training strategy balances efficiency and robust video understanding. LVAgent’s strength comes from its Selection-Perception-Action-Feedback mechanism, where multi-round reasoning and multi-agent collaboration enable effective long video processing. In contrast, FlexSelect focuses on training-freely selecting query-related tokens. We will add these discussions in the revision.\n\nTo compare whether LLM layers or traditional visual encoders are more suitable for token selection, we evaluate VisionZip, which also training-freely selects top-k tokens based on attention scores from the visual encoder as dominant tokens and merges the remaining tokens as contextual tokens. While VisionZip did not report results on any video benchmarks, its official GitHub repository provides an implementation for Qwen2.5VL. We compare VisionZip and FlexSelect using Qwen2.5VL-7B under 64 sampled frames.\n\n|Method|Retain Ratio|VideoMME|LongVideoBench|MLVU|\n|-|-|-|-|-|\n| Qwen2.5VL-7B| 100%| 63.6| 59.3| 61.8 |\n| VisionZip| 50% (45% dominant tokens and 5% contextual tokens) | 62.4| 57.9 | 61.5 |\n| FlexSelect| 50%| 62.6| 58.8| 62.4 |\n\nWith retaining the same number of visual tokens, FlexSelect continues achieve better performance than VisionZip, indicating that for token-level visual token selection, attention scores from LLM layers is more effective than traditional video models.\n\n**Q4: In what aspects do the key innovations and insights provided in this paper differ from previous studies?**\n\nThanks for the question. We have discussed the difference between FlexSelect and VideoChat-Flash, LongVU and LVAgent in Q2 and Q3.  Here we clarify our innovations and insights compared with the FastV and VisionZip.\n\nFastV identifies and analyzes the inefficient visual attention phenomena in MLLMs from the perspective of visual tokens' attention proportion relative to all tokens. They find that after the second layer, image tokens garner an average attention score that amounts to only 0.21% of the score attributed to system prompts, while in the initial two layers, this figure reaches 50%. Thus, they drop visual tokens based on the second layer's attention scores, despite efficiency gains, leading to performance degradation. In contrast, FlexSelect analyzes attention scores from the perspective of semantic alignment between visual tokens and user query tokens through our proposed Recall@K experiment, locating the optimal intermediate layer in the LLM. Using the attention scores from this layer to select query-related visual regions achieves better performance.\n\nVisionZip finds that in visual encoder(CLIP or SigCLIP) of MLLMs, only few visual tokens hold higher attention weights while most visual tokens receive very low attention and add significant redundancy. So they locate the dominate visual tokens with high attention scores in visual encoder and merge the other visual tokens. Different from this, FlexSelect analysis the attention scores from LLM layers.\n\nIn summary, FlexSelect investigates the cross-modal interaction patterns between text and visual tokens across different layers in Video-LLMs. The results demonstrate that semantic information in visual tokens is not fully resolved in shallow layers but gradually emerges as depth increases, with this phenomenon being most prominent at an intermediate layer. Using the attention scores from this specific layer enables effective selection of query-relevant visual tokens, significantly improving both the efficiency of long-video processing and overall model performance in Video-LLMs."}, {"review_id": "UtUv4LuLJe", "reviewer": "Reviewer_qmfj", "summary": "This paper introduces FlexSelect, a token selection framework designed to enhance long-form video understanding for Video Large Language Models (VideoLLMs). The method leverages cross-modal attention scores from a reference layer within a pretrained VideoLLM to rank visual tokens by semantic relevance to a given query. It includes two strategies: (1) a training-free selection pipeline based on cross-modal attention scores, and (2) a rank-supervised lightweight token selector trained to mimic the ranking behavior of the full model. Experiments across several VideoLLMs and benchmarks show that FlexSelect achieves consistent accuracy improvements.", "questions": "I list my questions and limitations in the weaknesses. My questions are mainly about the conceptual inconsistency in the introduction and the layer selection strategy.", "limitations": "I list my questions and limitations in the weaknesses. The limitations of this paper lie mainly in the lack of comparison with the referenced attention-based token importance methods [12] and [29], and the potential overfitting to a single layer.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The key insight that semantic alignment peaks at an intermediate layer (identified empirically via Recall@K) is well-validated and intuitive. This forms the foundation for an interpretable and data-efficient ranking mechanism. \n\n2. The approach is architecture-agnostic and plug-and-play. It can be used for various VideoLLMs. The approach design is clear and the paper is well written.\n\n3. Results are presented across multiple models (7B to 72B), datasets, and evaluation protocols, with comprehensive ablations including token limits, layer selection, and training data scale.\n\nWeaknesses:\n\n\n1. There appears to be a conceptual inconsistency in the introduction. In the second paragraph, the authors state that cross-modal attention scores from pre-trained VideoLLMs often suffer from performance degradation due to inconsistent relevance patterns across transformer layers, making them unreliable indicators of semantic importance. However, in the fourth paragraph, they claim that cross-attention weights between textual queries and visual tokens progressively reflect semantic alignment and typically peak at an intermediate depth—which serves as a key motivation for their method. While it is understandable that the goal is to identify the most semantically informative layer, the narrative should more clearly distinguish between the general limitations of using raw attention and the motivation to selectively leverage it. Moreover, the paper does not include a direct comparison with prior attention-based token importance methods, such as [12] and [29], even though they also experimented on the VideoMME dataset. \n\n\n2. In Figure 2, several intermediate layers yield similarly high Recall@K scores, suggesting that semantic alignment is not restricted to a single layer. However, the authors use only the layer with the highest score to generate token-level ground truth for supervised training. A more robust strategy could be to aggregate attention scores across multiple top-performing layers to better capture token importance. This could reduce potential overfitting to a single layer and improve the generalizability of the learned token selector", "was_revised": true, "final_justification": "The authors have provided satisfactory empirical evidence and clarifications for most of my concerns in their response. I thus keep the acceptance recommendation.\n\nAs a further suggestion, I recommend that the authors provide visualizations of the selected tokens from different layers. This would help qualitatively explore why multi-layer aggregation does not lead to performance improvements, by showing whether the selected tokens across high-performing layers are indeed redundant or highly similar.", "rebuttal": "We appreciate your valuable comments. We address your concerns below:\n\n**Q1: Narrative should be more clear**\n\nWe acknowledge the narrative could be clearer. Our analysis reveals that specific intermediate layers provide faithful semantic alignment, while early and late layers show poor alignment. However, existing methods fail to identify these optimal layers: DyCoke [29] prunes tokens at layer 3, while FrameFusion [12] operates at the initial layers (0, 1, 2). Based on our analysis, attention scores in these early layers are highly unreliable for semantic relevance estimation. As discussed in the fourth paragraph, we observe that semantic alignment between textual queries and visual tokens gradually strengthens with deeper layers, peaking at an intermediate layer (e.g., layer 19 for LLaVA-Video-7B). This finding serves as the key motivation for FlexSelect and directly explains why prior attention-based methods suffer from performance degradation.\n\n**Q2:  Lack of comparison with the referenced attention-based token importance methods  Framefusion [12] and Dycoke [29]**\n\nWe appreciate this suggestion and will include direct comparisons in the revision. We compared FlexSelect with FrameFusion [12] on LLaVA-Video-7B and DyCoke [29] on LLaVA-OV-7B, using the same number of sampled frames and pruning ratio. Here are our results.\n\n| Method                      | Sample Frames | Retain Ratio (%)| VideoMME |\n|-----------------------------|--------------|--------------|----------|\n| LLaVA-Video-7B              | 64           | 100.00        | 64.4     |\n| LLaVA-Video-7B+FrameFusion  | 64           | 30.00          | 61.3     |\n| LLaVA-Video-7B+FlexSelect   | 64           | 30.00          | 64.9     |\n| LLaVA-Video-7B+FrameFusion  | 512          | OOM          | OOM      |\n| LLaVA-Video-7B+FlexSelect   | 512          | 6.25        | 68.9     |\n\n| Method               | Sample Frames | Retain Ratio (%)| VideoMME |\n|----------------------|--------------|--------------|----------|\n| LLaVA-OV-7B          | 32           | 100.00        | 58.5     |\n| LLaVA-OV-7B+DyCoke   | 32           | 14.25       | 58.3     |\n| LLaVA-OV-7B+FlexSelect| 32          | 14.25       | 60.4     |\n| LLaVA-OV-7B+DyCoke   | 512          | OOM          | OOM      |\n| LLaVA-OV-7B+FlexSelect| 512         | 6.25        | 63.7     |\n\nThe results show that FlexSelect achieves better performance than DyCoKe and FrameFusion under the same number of input frames and retain ratio. Additionally, FlexSelect supports 512 input frames due to its Frame Set Partition mechanism, obtaining more performance improvements.\n\n**Q3: Potential overfitting to a single layer.**\n\nThanks for the insightful question. We followed your suggestion and conducted additional experiments to investigate potential overfitting to a single layer. We identify layers with high Recall@K scores from Fig. 2 (layers 14, 16, 19, and 21 for LLaVA-Video-7B). We conduct training-free token selection using averaged attention scores and evaluate on the VideoMME benchmark. The results are below:\n\n| Layers       | VideoMME |\n|--------------|----------|\n| 19           | 64.5     |\n| 14+19        | 64.5     |\n| 16+19        | 64.6     |\n| 19+21        | 64.3     |\n| 14+16+19+21  | 64.5     |\n\nThe results show that multi-layer aggregation yields comparable performance to single-layer selection, with only marginal differences.\nThis suggests two key insights: (1) layers with high Recall@K tend to focus on similar visual tokens with comparable rankings, making aggregation effects minimal, and (2) our single-layer approach is not overfitting but rather efficiently capturing the semantic alignment already present across multiple high-performing layers.\n\nWe will add this multi-layer analysis to demonstrate the robustness of our approach."}]

Author rebuttal:

Meta-review: This paper received four borderline accepts as final ratings. Initially, reviewers praised the paper for its motivation, strong performance of the method, and plug and play nature of the model. However, reviewers had concerns with comparison to other state of the art methods and how this paper builds upon existing work. The rebuttal gave a great deal of information and answered many of the reviewers' questions leading to all positive final ratings. Because of this, the AC sees no reason to overturn the majority decision of the reviewers and recommends acceptance for this paper.

The AC reminds the authors to make the changes that the reviewers proposed and they promised as part of the rebuttal in the camera ready version of the paper.
Decision: Accept (poster)