VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models
Authors: Haichao Zhang
Yun Fu
Venue: NeurIPS 2025 poster ()
Keywords: Video LLMs; Token Reduction; Discrete Token Representation; Extreme Token Reduction; Token Merge; Token Clustering
Primary area: deep_learning
TL;DR: We introduce an extreme token-reduction task and a discrete representation (VQToken) that adaptively compresses video token sequences by 99.93% of their original length with only a 0.66% accuracy drop.

Abstract: Token-based video representation has emerged as a promising approach for enabling large language models (LLMs) to interpret video content. However, existing token reduction techniques, such as pruning and merging, often disrupt essential positional embeddings and rely on continuous visual tokens sampled from nearby pixels with similar spatial–temporal locations. By removing only a small fraction of tokens, these methods still produce relatively lengthy continuous sequences, which falls short of the extreme compression required to balance computational efficiency and token count in video LLMs.
In this paper, we introduce the novel task of **Extreme Short Token Reduction**, which aims to represent entire videos using a minimal set of discrete tokens.  We propose **VQToken**, a neural discrete token representation framework that
(i) applies adaptive vector quantization to continuous ViT embeddings to learn a compact codebook and 
(ii) preserves spatial–temporal positions via a token hash function by assigning each grid-level token to its nearest codebook entry.
On the Extreme Short Token Reduction task, our VQToken compresses sequences to just **0.07\%** of their original length while incurring only a **0.66\%** drop in accuracy on NextQA-MC benchmark. It also achieves comparable performance on ActNet-QA, Long Video Bench, and VideoMME. 
We further introduce the **Token Information Density** (**TokDense**) metric and formalize fixed-length and adaptive-length subtasks, achieving state-of-the-art results in both settings. Our approach dramatically lowers theoretical complexity, increases information density, way fewer tokens counts, and enables efficient video large language models in resource-constrained environments.

VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models
Haichao Zhang*
Yun Fu Northeastern University Boston, MA yunfuβece neu edu
Northeastern University
Boston, MA zhang.haichhoortteeee hww.hhhhhootttttsttrr...dddd
Abstract
Token-based video representation has emerged as promising approach for en- abling large language models (LLMs) to interpret video content. However, existing token reduction techniques, such as pruning and merging, often disrupt essential positional embeddings and rely on continuous visual tokens sampled from nearby pixels with similar spatial-temporal locations. By removing only small fraction of tokens, these methods still produce relatively lengthy continuous sequences, which falls short of the extreme compression required to balance computational efficiency and token count in video LLMs. In this paper, we introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens. We propose VQToken, a neural discrete token representation framework that (i) applies adaptive vector quantization to continuous ViT embeddings to learn a compact codebook and (ii) preserves opatial-tempon positions via token hash function by assigning each grid-level token to its near- est codebook entry. On the Extreme Short Token Reduction task, our VQToken compresses sequences just 0.07 % of their original length while incurring only 0.66 % drop accuracy on NexIQA-MC henchmark. also achieves comparable performance on ActNet-QA, Long Video Bench, and VideOMME. We further intro- duce the Token Information Density (TokDense) metric and formalize fixed-length and adaptive-length subtasks, achieving state-of-the-ar results in both settings. Our approach dramatically lowers theoretical complexity, increases information density, way fewer tokens counts, and enables efficient video large language models in aesource-constringg environments.
Introduction
Recent advances in Vision Language Models (VLMs) have enabled unified zero-shot capabilities across diverse tasks, including visual question answering Xiao et al. (202T): Li et al. (2024a), video-to-text generation Alayrac al. 2022), video segmentation Xue et al. (2022), and video under- standing Zellers et al. (002). Although VLMs excel at aligning visual and linguistic information, their substantial computational cost remains critical bottleneck-espeedee
for video large language models (yLLMs). Video inputs contain spatial-tempor information distributed across numerous frames, resulting in lengthy token sequences that significantly burden computational resources Doso vitskly et al. (2021): Yang et al. (2024). Consequently, as vLLMS seale in size Li et al. (004a)):
Zellers al. (2022). improving computational efficiency becomes imperative.
*Corresponding author. Prgjett/hhaghaachaoo.yyvvvooenn Code: github.com/Haai-chao /Vooenn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Comparison of model efficiency in terms of token number, throughput, FLOPs, run- time, accuracy, token information density, and complexity analysis. Note: n is the original token count; k is the number of tokens reduced by traditional methods; m is the compressed token count after our extreme reduction approach, with the relationship n > k > m² > m; denotes token dimensionality; and L represents transformer layer count. Given m < n. our token reduction module has a complexity of O(nn + md)d) as ((dd), significantly reducing LLM complexity to dm)) Module Complexity quantifies the computational cost of the token reduction method itself, while LLM Complexity reflects the computational reduction within the LLM, benefiting from the token reduction. "TokDense" is Token Information Density (accuracy contributed from per token).
Table 1:
Method
Token Num.
Tokee Num.
Throrzigus?
FLUIS (13)
Run- lime
Modale Complesits
LLM Comploxts
Accuracy?
VoiDDraee
11664
21.91
ClaTat)
53.58
01011
Token Franing IF 01-9c)
17%
16.10
Chinn P)P.)
20112
0102
ToMil 0.8ia
15%
11.55
Ohin +7..))
01511
Valtsbile 0.90)
1352
15%
11.49
Ofa' c)
Ohin 27.
29.6
01054
Ircepeinatnn 0.73el
3136
21%
13.59
11.34
((ind)
Otin -)7
52.20
Ourr Dysanns (alagive))
1388
6077%
1030
7.56
OT(n me [d]
59.72
4.122
Osn- Fiaed (m
0,149
10.47
4.5
(((n m4)
O(m)dd))
$7.46
1.796
Unlike textual inputs, video data require tokenizing pixel batches from each frame and concatenating them into extensive sequences. Transformers process these sequences through attention mechanisms at each layer, incurring computational complexity of OnnDL), As demonstrated in Table [ the token sequence length (ri) is the primary contributor to computational overhead, increasing exponentially as the token count grows. This overhead surpasses the influence of model parameters, layers (L), and embedding dimensions (D). Reducing token sequence length emerges as a promising solution, broadly applicable to most LLMs in plug-and-pla manner.
Despite extensive efforts to reduce redundancy in video token sequences, existing methods face three main challenges. First, token pruning approaches Kim et al. (2022): Liu et al. (2024b) remove seemingly redundant tokens but often discard critical information, degrading representation quality. Second, token merging techniques- such as ToMe Bolya et al. (2023), Vid-ToMe Lee et al. (2024), and Token Bilinear Interpolating Li et al. (2024a)- group similar tokens without explicit removal; however, they rely on fixed reduction ratios, which limits flexibility and leaves sequences excessively long for lcrge video data. Third, even after pruning or merging, the remaining tokens remain highly contiguous and similar, resulting in low information density and persistent redundancy that impede further compression.
We attribute these challenges to three key limitations. First, existing methods rely on fixed-count or fixed- perceeeeeet reduction strategies, which either leave sequences overly long, with redundant tokens, or prune so aggressively that critical information is lost. Second, they lack adaptive, context-sensitive mechanisms for selecting the most informative tokens in the frames. Third, none leverage vector quantization to cluster tokens into discrete categories, hindering substantial gains in information density through thorough compression. To address these limitations, we propose VQToken, vector- quantized token representation framework that dynamically clusters continuous ViT embeddings into a compact, discrete codebook. By mapping each token to its nearest codebook entry, VQToken produces minimal set of discrete tokens while preserving spatial-temporal relationships. Accurately capturing spatial- dynamics within this discrete clustering, however, remains a critical challenge.
The second major challenge is preserving ppatial-tempore coherence during token reduction. Tradi- tional pruning methods Kim et al. (2022) Liu et al. (2024b) often discard positional cues that are vital for tracking object motion accurately. Likewise, similarity-bast -based merging techniques Bolya et al. (2023): [ee et al. (2024); [i et al. (2024a) tend to ignore spatial-temporal encodings or reapply them inconsistently. which undermines dynamic context modeling. To overcome this, we introduce token-hashing mechanism based on vector quantization. First, each grid-level token is mapped to its nearest codebook centroid; then, we record its original (f, h, w) index in a three-dimensional hash table. This table integrates seamlessly into the VQToken architecture, preserving positional information in a compact discrete form. By leveraging the inherent redundancy of video data. In- spired by computationally expensive motion tracking techniques like optical flow, token- hashing mechanism offers a lightweight alternative that preserves essential opatial-temport context with minimal computational overhead.
The third challenge is devising an evaluation framework for highly compressed token sequences. Existing methods neither achieve substantial reduction nor measure information density, making it

---

difficult to compare token performance trade-offs or assess adaptability, To address this, we define the Extreme Token Reduction task with two subtasks: fixed-length compression, which measures LLM accuracy under predetermined token budget; and adaptive-length compression, which assesses performance when the token count dynamically determined by video content. We introduce Token Information Density (TokDense), defined as accuracy per retained token, to quantify each token's s contribution to task performance. Additionally, we propose separate complexity metrics-one for the token-reduction module itself and another for its impact on downstream LLM inference forming comprehensive evaluation suite for extreme token reduction methods.
Our contributions are summarized as follows,
1. We present a neural discrete token representation framework, VQToken, that applies adaptive vector quantization to continuous ViT embeddings to learn compact codebook, and preserves spatial-temporal positions via a hash token function. To the best of our knowledge, this is the first work to leverage vector quantization for token reduction in video large language models.
2. A formal definition of the Extreme Token Reduction task, together with the Token Information Density (TokDense) metric and separate complexity measures for the reduction module and downstream LLM inference, covering both ixxe--lenth and adaptive-length settings.
3. Empirical evidence that VQToken compresses video token sequences to just 0.07% of their original length with only 0.66% drop in NexIQA-MC accuracy, achieving leading effi- ciency and information density while maintaining competitive performance across multiple benchmarks.
2 Related Works
2.1 Video Large Language Models
Video Large Language Models (vLLMs) have emerged as powerful tools for bridging video un- derstanding and natural language processing, enabling complex interpretations of video content through language- based interactions. Recent advancements have demonstrated remarkable capabili- ties in aligning visual and linguistic modalities, exemplified by frameworks such as LLaVA [Lu et al.) (2023b) Li et al. (2024a): [Lu et al.] (2023a] (2024a), Flamingo Caamminng/Alaraaccc.coo et al. (2022). AuroraCap Chai et al. (2024), and MERLOT Reserve Zellers al. (2022). These methods typically rely on extensive pre-training using large-scale datasets like HD-VILA Xue et al. (2022), Intern Vid Wang et al. (2023), and NexiQA Xiao et al. (202T), generating lengthy token sequences to represent videos effectively.
2.2 Token Reduction
Token-reduction methods have become a central route to improving the efficiency of Vision Trans- formers Dosoviiskyy et al. (2021). Early approaches such as Token Pruning Kim et al. (2022) and Token Merging (ToMe) Bolya al. (2023) reduce compute by discarding redundant tokens or merging similar ones. More recently, Vid-ToMe Lee et al. (2024) extends merging to video by leveraging temporal redundancy across frames, and GRT et al. (2025) further adapts merging to high-FPS settings for dense video understanding. Despite these advancements, existing token reduction strategies generally adopt fixed token reduction ratios (c.g.. 50%). limiting flexibility and adaptability. Such fixed strategies can either inadequately reduce redundant tokens, resulting in lingering inefficiencies, or inadvertently merge tokens representing distinct objects, thus losing dynamics necessary for precise video interpretation. To overcome these limitations, our proposed VQToken framework introduces dynamic token clustering to generate compact token representation while explicitly preserving a dedicated token indices. Through our novel VQ-Attention mechanism, our approach effectively integrates spatial-temporal coherence into concise token sequences without compromising accu- racy, outperforming the token reduction methods, even in scenarios of extreme token compression.
crucial
motion information via
3

---

Veter-quniiali Tokra Walile
Quar Qurda
LLM
Segar Vidao
Reslege Buctio
Merpla
Codekask
Tihe Badisn
Figure Overview neural discrete token representation learning. First, the input video is tokenized into a continuous sequence of visual tokens. An adaptive discrete process then clusters and vector-quantizes these tokens into compact codebook. A token hash function records each token's original spatial-temporal location and maps it to its nearest codebook entry. The VQ-Attention module integrates the codebook with the index map to produce a compressed token sequence that preserves positional information. Finally, the compressed tokens and tokenized query are passed to the large language model for zero-shot inference.
3 Methods
3.1 Problem Definition: Extreme Token Reduction
The Extreme Token Reduction task aims to compress a long video-derived token sequence into minimal set of tokens without sacrificing downstream performance.
Formally, given video and query q. video-language model (vLLM) first tokenizes the video into Tokenize(v), then uses and to predict an answer a. A token reduction function maps
R t', with |t'
t
such that the vLLM's ' accuracy on predicting a remains comparable.
We assess token reduction methods via two subtasks and two complementary complexity metrics: Fixed-Length Reduction. Each method evaluated under predefined token budget m reduction ratio allowing fair comparisons among approaches that require explicit reduction rates.
Length Reduction. Methods dynamically select the optimal |t'| based the video's 's content complexity, enabling a per-instance trade-off between token count and predictive performance.
Additionally, we introduce two complexity metrics to isolate (i) the computational cost of the reduction module R, and (ii) the resulting impact on downstream LLM inference.
3.1.1 Module Complexity and LLM Complexity.
Token reduction modules introduce additional computation while reducing the downstream workload of the LLM. To disentangle these effects, we define two complementary metrics: Module Complexity measures the computational cost of the token reduction operations alone. LLM Complexity quantifies the reduced computational burden on the LLM, reflecting the shorter token sequence length after reduction.
3.1.2 Token Information Density (TokDensse.
As token sequences become extremely compact, it is crucial evaluate how much performance each retained token count contributes. We define TokDense as
Accuracy
TokDense
Token Count
where Accuracy is measured on the target benchmark and Token Count is the number of tokens fed into the LLM after reduction.
3.2 Neural Discrete Token Representation Learning
We introduce Neural Discrete Token Representation Learning, a vector-quantization architecture that dynamically minimizes token sequence length while preserving complete spatial-temporal motion information.

---

3.2.1 Adaptive Discrete Process
Tokens produced by Vision Transformers (ViTs) Dosovitskiy al. (021) often exhibit temporal continuity and redundancy: continuous visual patterns evolve over time but correspond to discrete semantic entities. Slight variations among tokens can obstruct effective grouping. To address this, we apply vector quantization to cluster similar token embeddings actoss frames into representative discrete tokens.
Unlike fixed-ratio merging methods such as ToMe Bolya al. (203),, which risk under-merging or spurious groupings, adaptive discrete process selects the number of clusters either statically or dynamically, For fixed-length reduction, we use classical K-Means Vassilvitskii and Arthur (2006); for adaptive-length reduction, we employ adaptive K-Means variant Bhatia et al. (2004). While video-segmentation approaches (e.g., SAM-based Ravi et al. (2024) can yield fine- tine-grained clusters, their computational overhead makes them less practical for this stage.
Token similarity is measured via cosine similarity. Formally, let ti...., tv denote the original token embeddings and K the chosen number of clusters. The discrete assignment function Fdis:: produces:
($1, .......: sk), (CI....,,CN)
Faise (t1, ................................ (N),
where e; E [l....,
K} is the cluster index for token tú, and sh { c, k} is the set of token indices assigned to cluster k. This clustering yields a compact discrete codebook of K representative
tokens for subsequent processing.
3.2.2 Vector-Quantization Architecture
To transform the discrete clusters into a compact token sequence while preserving spatial-temporal information, we design three components: a concise codebook, a token hash function, and a VQ-based reduction module.
Concise Token Codebook. Given the original token embeddings .1,.... In E TRD and their cluster assignments -1,--.. 8x from Eq. we build discrete codebook B E TR: k Each codebook entry by is computed as the centroid of the embeddings in cluster sy:
by
Lis EE
k= K.
8A
Here, by serves compact representative for all tokens in cluster k. This codebook captures representative visual patterns and object parts with minimal redundancy.
Token-Hash Fuction Mapping. To retain each token's 's original build 3D index map M E {1...., K) XXTxxXW For frame f and spatial coordinates (h, w), let == f x W) + h x W + w. Then
location, we
Mf
(5)
where T, H, W are frame count, height, and width of the VIT grid, and is the cluster index of token 4, This mapping preserves positional encodings by recording, for each grid cell, which codebook entry it belongs to.
VQ-Based Reduction Module.
We integrate the codebook B and index map M via lightweight Vt-Attention mechanism using a lightweight VQ-Attention block that enriches each centroid with motion context without increasing token count,:
M = MLP (Flatttttc)(MM) KDD
(6)
B' = MulttHHad -BW. K = BWK. V = MWv),
(7)
where Wo, WK E DxD and Wy E DxD are learnable projections. The output B' E R enriches each codebook vector with motion context, yielding the final compressed token set. These tokens are then fed into the downstream vLLM for inference.
5

---

Experiments
4.1 Implementation Details
4.1.1
Training Dataset.
We follow the LLaVA-One Vision Li ct al. (0024)) setu and fine-tune on LLaVA-Video-178K Zhang
al. (224). The corpus pairs videos with 1.3M instruction-following samples-178k captions,
960K open-ended questions, and 196K multiple-choiee questions -paanning diverse video scenarios.
4.1.2 Evaluation Benchmarks.
We evaluate on six diverse benchmarks: ActivityNet-QAA Yu (2019) (up 120 s) for spatio- temporal reasoning on short videos; VideoMME Fu et al. (2024) (avg. 17 min) for long-video comprehension; NEXT-QA Xiao et al.) (202T) for descriptive, causal, and temporal reasoning; LongVideo Wu et al. 2025) (up to h) for extended narrative understanding; and MVBench[L] ctt al. (024b) (35 s) comprising 20 reasoning-intenss
tasks. These benchmarks collectively test our approach across varied durations, resolutions, and reasoning challenges.
4.1.3 Training Setup.
Starting from the 0.5B Sa--paraeeter LLaVA-OL -eee--OOeiiiio model et al. (2024a) (QWen2 backbone Yang al. 20244), integrate our VQToken framework and fine-tune for zero-shot evaluation. Training uses four NVIDIA A100 GPUs for 85,000 iterations with AdamW and a cosine decay schedule (initial learning rates of x 10-5 for VQ-Attention and x 10-6 6 for the ViT backbone). We employ Zero2 optimization Rajbhandari et al. (2000) with batch size and gradient accumulation over steps.
4.1.4 Metries.
We report Accuracy (Acc.), the percentage of correct responses on multiple-choice and open-ended QA tasks; Token Count, the number of tokens processed per example; Throughput, measured in frames per second; FLOPs (T), the total tera-FLOPs required for inference; and Run- Time, the end-to-end inference latency. To disentangle the cost of token reduction from its downstream benefit, we also measure Module Complexity, the time complexity of the reduction module alone, and I.M Complexity, the complexity of the large language model given the reduced token sequence. Finally, Token Information Density --aeDeennee-eiieie in Eq. 2 quantifies accuracy per retained token.
4.1.5 Video Large Language Model Baselines.
Due to limited computational resources, evaluate the ..SB-parrmettr track, using LLaVA-OV- SI and LLaVA--Oe Vision as baselines. For efficiency, we integrate our VQToken framework with LLaVA-One Vision minimize GPU usage. We also compare against 7B-parameter versions; despite having 14x more parameters and greater compute, our 0.5B model-using only 0.14% of the original token count-out some 7B models, highlighting our method's efficiency.
4.1.6 Token Reduction Baselines.
For token reduction, we compare against several baselines: Token Pruning Kim et al. (2022) : A widely recognized method for reducing token numbers and increasing throughput in LLMs. Token Merging (ToMe) [Boly et al. (2023): A popular baseline for token reduction, known for its efficiency improvements. Video Token Merging Lee et al. (2024): The current state-of-the-art method for token reduction in video large language models, extending the capabilities of ToMe to video data, Interpolation: Introduced by Li al. (2024a), the use of bilinear interpolation reduce the number of tokens in visual representations, particularly for video frames. This approach allows the model to handle a larger number of frames by reducing
tokens per frame, achieving balance between performance and computational cost. (v) DyCoke Tao et al. (2024). the current SOTA method that employs temporal compression to merge redundant tokens across frames and dynamic KV-acche reduction to selectively prune spatial redundancy.

---

Table 2: Fixed-Length Token Reduction. We Table 3: Adaptive-Length Token Reduction. Mod- evaluate different token reduction approaches by els select token lengths dynamically based on in- retaining fixed number of tokens. Each method put sequences. For baselines, is adjusted to the same token budgets for fair settings optimized for most cases to ensure a fair comparison.
their default
comparison.
Preset Token Num...
32
64
Baseline
Tokens,
TokDensef
Token Pruning
29.12
34.50
31.31
Interpolating Lictal 2024a
3136
57.20
0.018
ToMe
35.72
38.50
40.10
Dycoke Tho
1662.12
57.70
0.035
VidToMe
39.64
45.10
46.20
Ours (Fixed)
32
57.46
1.796
Ours (Fixed)
57.03
57.46
$7.10
Ours (Dynamle)
13,08
57.72
4,413
4.2 Quantitative Comparison with vLLLM Baselines
Soble@compares our VQ-Token model against re- cent video-language models introduced between 2022 and 2024. To ensure a fair comparison, group baselines by model size-0.5B and 7B parameters accounting for neural scaling ef- fects Kaplan et al. (2020). Although VQ-Token is trained and evaluated strictly in zero-shot regime, also report non-zero-shot baselines fine-tuned on the evaluation datasets for complete- ness. We evaluate all models using two metrics: accuracy and token count. Our VQToken slightly outperforms the LLaVA-One Visiion baseline in accuracy while reducing the token count from 23.328 (100 %) to just 32 (0.14 %). dramatically lowering computational cost. Despite its smaller size (0.5B parameters), VQToken also surpasses several 7B vL..Ms in zero-shot accuracy, demon- strating that extreme token reduction can preserve or even improve performance. These results un- derscore the effectiveness our framework removing redundancy while maintaining essential spatial-temporal and semantic information. The extreme compression achieved by VQ-Token high- lights its potential to make large-scale video aaleo-language understanding significantly more computation-
MVllench Subtinies Pvalustion
Woving periii
Figure 2: MVBench subtask performance. (Normalized for visualization.) VQToken shows robust performance across reasoning- and interaction-centric subtasks, with the strongest improvements in action recognition and object interaction.
ally feasible.
Extreme Token Reduction Task
4.3.1 Fixed-Length Subtask.
To evaluate efficiency under extreme compression, compare VQ-Token against classical and state-of-the-art reduction methods-Tooken Pruning, ToMe, and VidToMe-using fixed token budgets. We configure model with predetermined cluster size K and set each baseline to retain exactly the same number of tokens (e.g., 12, or 64). This controlled setting isolates the effect of each reduction strategy on accuracy. As Table 2 shows. VQ-Token consistently outperforms frame-level merging (ToMe). sequence-level merging (VidToMe). and pruning across all extreme budgets, demonstrating superior preservation of semantic content under severe token constraints.
4.3.2 Adaptive-Length Subtask.
Here, each method dynamically selects the optimal token count based on video content complexity. report both accuracy and the average tokens used per sample ("Avg Tokens"). Table illustrates that, compared to interpolation-basee downsampling and our own fixed-length variant, the adaptive Vo-Token model achieves higher accuracy while consuming significantly fewer tokens on average. This result underscores its ability to balance efficiency and performance in content-awas manner.
7

---

Table 4: Performance across benchmarks. Despite compressing tokens by 99.86%, VQ-Token maintains competitive accuracy on diverse video understanding tasks.
Benchmark 100%
Token %
NextQA-MC
ActNet-QA
Long Long VideoBench
VideoMME
LLaVA-OV-SI
53.6
49.0
41.9
40.4
100%
57.2
50.5
45.8
43.5
VQ-Token (Ours)
0.14%
57.4
46.3
39.3
38.2
Table 5: Zero-shot performance of video language models. We report accuracy and token reduction relative to the 0.5B LLaVA-Oms Vision baseline (23,328 tokens 100%). Our VQ-Token achieves competitive accuracy with only 1.6% drop while using 0. 1.14% of the original tokens, outperforming other 0.5B models and several 7B models.
Model
#Parameters
Year
Zero-Shot
Token Num.
Mistral
Jiang 2025
7B
2023
51.1
100%
P3D Cheran el
21122
7B
2022
X
51.3
100%
VFC Momeni et al
7B
2023
$1.5
100%
LLoVi
2023
54.3
MVU
Kea et A134
7B
2024
/
55.2
100%
ATP Buch et al. PURRI
7B
2022
X
54.3
100%
LLaVA One Vision Li 2024a
0.58
2024
57.2
107%
OV-Si et (0140
$3.6
VQ-Token (Ours)
0.5B
2024
57.5
0.14%
4.4 Efficiency Comparison and Analysis
To quantify practical efficiency gains, we compare VQ-Token against existing token reduction methods under standardized settings. For Token Pruning, ToMe, and VidToMe, we retain 10% of the original tokens; for Interpolation, we use the default setting that retains 27%. For our approach, Ours-Fixed uses the optimal fixed token count from Table and Ours-Dynamic selects token counts adaptively via K-Means Bhatii et al. (2004). We evaluate seven metries: Token Count, Token Ratio (%), Throughput (clips/sec), FLOPs (T), Run-Tinn Time, Module Complexity (reduction module overhead), and LLM Complexity (downstream cost). Using vanilla LLaVA-OV as the backbone, each method is applied and measured under identical conditions.
As Table T shows, both Ours-Fixed and Ours-Dynamie achieve superior trade-offs between pression and accuracy, reducing theoretical complexity and run-time more than all baselines without sacrificing performance.
4.5 Performance on Multiple Benchmarks
4.5.1
Evaluation Across Diverse Settings.
To test robustness in real-world scenarios- spanning high resolution, long duration, and multi-step reasoning- evaluate on all benchmarks listed Sec. 4.1.2] Table accuracy alongside token reduction relative to the original sequence. Despite compressing tokens by 99.86 %, VQ- Token maintains competitive accuracy across tasks, demonstrating its effectiveness and robustness in preserving essential spatial-temporal and semantic information under extreme compression.
4.5.2 Evaluation Across Multiple Subtasks.
We further evaluate VQ-Token on 20 subtasks from MVBench, covering pose estimation, navigation, multi-step reasoning. and object interactions in dynamic video scenarios. As illustrated in Fig. our model achieves competitive results across most subtasks and excels in action recognition and object-interaction tasks, demonstrating its ability focus on critical motion and relational cues.
4.6
Ablation Study
4.6.1
Quantitative Ablation.
We evaluate each component's contribution by incrementally adding the discrete codebook, token hash function (Indices), and VQ attention to LLAVA-OV baseline. As shown Table [ the Base alone compresses tokens to 0.14% but incurs 22.0% accuracy drop, highlighting the loss of

---

Table 6: Ablation study. We incrementally add each component of VQToken to the LLaVA-OV baseline: discrete codebook, token hash function, and VQ-Attn. "rand" indicates randomized parameters. Each module contributes to accuracy, compression rate, and token information density,
VLM Codebook
Hash Fu.
VQ-Attn
Acc.
Tokens J
Reduction |
TokDense
57.2
23,328
100%
0.002
35.2
32
0.14%
1.100
rand
38.9
134
0.57%
0.290
rand
V
46.9
32
0.14%
1.466
57.5
32
0.14%
1.797
spatial-temporal cues. Incorporating the Indices marginally improves accuracy, although the LLM cannot yet leverage motion information effectively. Introducing Art restores accuracy substantially, demonstrating that VQ attention is essential to integrate positional context into the compressed tokens. Randomizing each module's ' parameters (denoted "rand"" leads to significant performance degradation, confirming the necessity of properly learned codebook, mapping, and attention for extreme token reduction.
4.6.2 Visualization of Adaptive Discrete Process.
Figure illlstrates clustering behaviors sample frames. We compare adaptive K-Means on token embeddings with the state-of-the-ard mask-based segmentation model Segment Anything (SAM) Ravi et al. (2024). Both methods group semantically similar regions and maintain cluster consistency across frames. While SAM yields finer-grate sner-grained regions, adaptive K-Means offers a more computationally efficient alternative that sufficiently captures object trajectories for the token hash function. This efficiency makes adaptive K-Means the preferred choice for vLLMs requiring extreme compression with minimal overhead.
4.6.3 Matched training schedule comparison
The original LLaVA-One Vision checkpoint was not trained on VadVA--ide--188 whereas our model fine-tuned for one epoch that corpus. To isolate schedule and data effects, fine-tune both the LLaVA-Ons Oaaa-OOneiiioon Vision baseline and VQToken for one epoch on the same LLaVA- Video- 178Ko-178KBA Zhang et al. (2024) dataset and compare under this matched setting (Table 7 Under the same schedule, VQToken retains 99.5% %6/8.770 7% of baseline accuracy on NexiiA/ActivityNer QA/VideoMME while using only 0.14% of tokens, yielding 300x gains in TokDense across all benchmarks.
Limitations and Future Directions
5.1 Task comparison with long-video understanding
In extremely long-video settings, performance can degrade as duration grows. VQToken is designed for the extreme token reduction regime- compressing each clip to very small, adaptive token budget
Ingurt
Segment
Augiling
Figure 3: Adaptive discrete visualization. Objects that exhibit similar visual appearance across frames should map to consistent clusters. We compare adaptive K-means variant (visualized with reduced number of clusters for clarity) and Segment Anything (SAM) as adaptive clustering front-ends.

---

Table 7: Matched training schedule comparison. Both models are fine-tuned for one epoch on MVV--Viido-1788 Zhang et al. (2024). VQToken retains most of the baseline's s accuracy while using only 0.14% of tokens, yielding x0 improvements in TokDense across benchmarks.
NexIQA
ActNet-QQ
VideoMME
Model
Fine-tuning
Token
Acc 1
TokDense
Acc
TokDense 1
Acc
TokDense
LLaVA-OneVision
L.aV OneVision (haseline)
epoch epoch
57.71
0.0049
47.16 46.25
0.0040
44.37
0.0038
100%
VQToken (ours)
57.44
1.7950
14453
38.22
1.2294
0.14%
Trade-off
99.53%
366x
98.07%
361x
88.66%
323x
99.86%
for efficiency-rattt than the long-video understanding setting that requires explicit modeling of hour-long footage and long-range temporal structure. Consequently, the current VQToken pipeline does not include mechanisms for stitching many clips into a coherent narrative or explicitly modeling very long dependencies.
Extreme token reduction and long-video understanding optimize for different goals. Extreme token reduction asks: "How can we represent clip with as few tokens as possible while preserving downstream utility?" The scope emphasizes aggressive, per-clip compression (e.g., 332 adaptive tokens) for -soutrc-consttrained scenarios such edge devices smart glasses. In contrast, long- video understanding asks: "How can we model and reason over long, continuous footage while maintaining temporal coherence and long-range dependencies?" The scope emphasizes ordered, informative representations across many clips or hours of content. Compressing a 30s clip to 32 tokens is fundamentally easier than compressing h stream to the same budget; longer content necessarily packs more diverse events per token, making fine- tne-grained reasoning harder.
5.2 Future directions toward long-video understanding with extreme token reduction
To bridge extreme token reduction and long-video reasoning, we see several promising directions: (i) Segmented windowing: apply VQToken to 1 1-2 min windows and fuse window embeddings with lightweight temporal transformer; (ii) Hierarchical VQ: first quantize at the segment level, then apply second-stage VQ over segment embeddings to capture inter-segment dependencies; and (iii) Adaptive budgeting: dynamically allocate more tokens to high-motion or semantically rich segments using small importance predictor.
6 Conclusion
We have introduced VQToken, the first neural discrete token representation framework to leverage adaptive vector quantization for extreme token reduction in video large language models. VQToken constructs a compact codebook from continuous ViT embeddings and preserves spatial-temporal positions via hash-based token mapping, enabling plug-and-play integration with existing architec- tures. To benchmark extreme compression, we formalized the Extreme Token Reduction task and proposed the Token Information Density ( (TokDense) metric, along with separate complexity measures for the reduction module and downstream I.IM inference. These contributions provide a compre- hensive evaluation suite for both fxxeh-length and adaptive-length reduction settings. Empirically, VQToken compresses token sequences to just 0.07 % of their original length, achieving over 99.9% reduction, with only 0.66 % drop in NexIQA-Me accuracy. matches comparable performance on ActivityNet-QUA Long Video Bench, and VideoMME, while delivering state-of-the art efficiency and information density. Ablation studies confirm that the codebook, token hash function, and VQ- attention are all critical to preserving semantic and motion information under extreme compression. Efficiency analyses demonstrate substantial reductions in FLOPs, latency, token information density, and overall computational complexity compared to prior methods. In future work, we will explore hierarchical clustering and learned cluster-size schedules to further optimize compression, as well as extend the VQToken framework to downstream tasks such vidco generation and motion prediction.
10

---

References
Ican-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35.23716-23736, 2022.
Sanjiv K Bhatia al. Adaptive k-means clustering. In FLAIRS, pages 195-699 2004.
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, 2023.
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" vidco" in Aidco-language understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 291 2917-2927, 2022.
Wenhao Chai, Eaxin Song. Yilun Du, Chenlin Meng. Vashisht Madhavan, Omer Bar- Tal, Jenq-Neng Hwang. Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark. arXiv preprint arXiv:2410.0305,, 2024.
Anoop Cherian, Chiori Hori, Tim K Marks, and Jonathan Le Roux. (2.5+ 1) apatio-temporte scene graphs for video question answering. In Proceedings of the AAAI Conference Artificial Intelligence, pages 453, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations (ICLR), 2021.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075 2024.
Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:23 10.223100.0830 06825, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodci. Scaling laws for neural language models. arXiv preprint arXiv:2001 08361, 2020.
Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 184-794,22 2022.
Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, and Xinyu Li. Video token merging for long-form video understanding. In NIPS, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint ar arXiv:2408. 03326, 2024a.
Kunchang Li. Yali Wang, Yinan He. Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu. Guo Chen, Ping Luo, al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Compuder Vision and Pattern Recognition, pages 22195-2211 221952 2024b.
Haotian I Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved haselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Yifei Liu, Mathias Gehrig, Nico Messikommer, Marco Cannici, and Davide Scaramuzza. Revisiting token pruning for object detection and instance segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2658-2666 2024b.
Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15579-15591, 2023.

---

Samyam Rajbhandari. Jeff Rasley. Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing. Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et Sam Segment anything in images and videos. arXiv preprint arXiv:2408. 14,112222 2024.
Shuhuai Ren, Linli Yao, Shicheng Li, Sun, and Lu Hou. Timechat: time- sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313 14323, 2024.
Keda Tao, Can Qin. Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models, arXiv preprint arXiv:2411.1500 2024.
Sergei Vassilvitskii and David Arthur. k-means+++: The advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM SIAM symposium on Discrete algorithms, pages 1027-1035, 2006.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang. et al. Intervvid: A large-scale video-tex text dataset for multimodal understanding and generation. In The Twelfth International Conference on Learning Representations, 2023.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828-28838 2025.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of explaining temporal actions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9777-9788 2021.
to
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang. Jianlong Fu, and Baining Guo. Advancing high-resolution video- language representation with large-scale video transcriptions. In International Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwci Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng Hc, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv preprint 2rXiv:2407.10671 2024.
Zhou Yu. Dejing Xu, Jun Yu, Ting Yu. Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web vidcos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1127-9134, 2019.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao. Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, pages 16354-16366. IEEE Computer Society, 2022.
Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang. Shoubin Yu, Mohit Bansal, and Gedas Bertasius. simple Ilm framework for long-range video
2023.
Haichao Zhang, Wenhao Chai, Shwai Hc, Ang Li, and Yun Fu. Dense video undcrstanding with gated residual tokenization, arXiv preprint 2509. 14799, 2025.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li. Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arviv preprint 140.0271,, 2024.
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The contribution has been listed at the end of introduction and appear in the
abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
13

---

Justification: We have discussed the limitations in the end of conclusion section as potential future direction for future researcher to explore.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The theoretical result and calculation as well as problem definition has been stated the main paper, while some complexity is based on adding the time complexity of each components.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
14

---

Training details and implemented details have been provided, all evaluation benchmarks are public available.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes] Please find code at htttts://ggthiii http:///gggtuubu...aaaaaaiii-eie chao Zhang Zhang/VQToken VQToken
Justification: Since all the details are given and all the data are public available, the read- ers can easily reproduce the experiment results. We are still waiting the administrative apporvement to open access the code. The code will be public available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/nips.cc/pppii wttp:ccpppppccceeet.ct.. otu.uddeee/ddeebbbisssiooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
15

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Almost all details are stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: With many experiments and metrics to prove the significance. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No negative impact on this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
17

---

Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All code, data, and models are public available and suitable for research.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com ddtasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes] We release pretrained VQToken checkpoints and code with usage documen-
tation. Model: Hugging Face; Code: GitHub
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
18

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
The LLMs is part of the architecture.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy /tttps://neurab wttprrippppcccoooeeeeeeeeeeeee /c/eeonneeeeencs//222////11 for what should or should be described.
19


Reviews:
[{"review_id": "ZVA7h1EQzO", "reviewer": "Reviewer_Gyx4", "summary": "The paper proposes Neural Discrete Token Representation Learning. It first uses k-means to cluster the tokens and uses the centroid as the token codebook. Then it constructs a hash function that encodes each patch's position and spatial information. It finally integrates the codebook and the hash representation with a VQ-based reduction module. Evaluations on several benchmarks demonstrate with reduced number of tokens the performance can match the original model.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Viewing the grouping of tokens as discrete codebook is a new angle.\n2. Integrating spatial information and token features using VQ-attention is an interesting approach.\n3. Experiments and ablation are relatively extensive.\n\nWeaknesses:\n1. The motivation for the VQ-based attention module is not very clear. Why does such an attention module \"enrich each centroid with\nmotion context without increasing token count\"? Why is hashing the patches to an integer an effective way to retain position and temporal information?\n\n2. The evaluation only compares against a few video token pruning baselines, and most comparisons are done with models that take full tokens, or . Token pruning in video LLM has a rich literature. The paper should compare with more recent token pruning methods under similar token numbers. Being able to reduce the number of tokens drastically is impressive, but such observation has been made before. To establish the value of this work, the comparison should focus on recent competitive methods.\n\n3. The overall clarity needs improvement. The motivation part is organized (three weaknesses of existing methods), but they are not very convincing. The method section does not talk about how each design address these weaknesses.  The method section overall is too short, lacking explanation of what motivated the designs.", "was_revised": true, "final_justification": "Some clarity issues were resolved, but the paper still lacks sufficient baseline comparisons on multiple benchmarks.", "rebuttal": "## Response to Reviewer Gyx4\n\n**We thank Reviewer Gyx4 for recognizing our contributions, especially the novel application of vector quantization for token reduction in video large language models.**\n \n**Unlike the other three reviewers, Reviewer Gyx4’s comments center on paper organization, writing style, and overall clarity, while slightly having some confusion on baseline selection.**\n\n**We appreciate the detailed feedback and will carefully revise the camera-ready version to improve clarity and organization.**\n\n---\n\n### 1. Motivation of the VQ-Attention Module  \n**Q1:** “Why does such an attention module ‘enrich each centroid with motion context without increasing token count’? Why is hashing the patches to an integer an effective way to retain position and temporal information?”  \n**A:** Sorry for the misunderstanding. We will expand the current motivation in the Methods (§ 3,  rows 137–141;143-144; 148-158; 169-171) to explain that our VQ-Attention module serves two complementary purposes:\n\n- **Preserve Motion Context:**  \n  After Adaptive Discretization, we reduce a full video to just 0.14 % of its original tokens. A naïve clustering of these centroids discards all temporal cues—leading to a 22.3 % drop in accuracy (Table 6). VQ-Attention re-introduces motion context by using each centroid’s original patch index (hashed from its 2D + time coordinates) as additional key/query input, allowing the transformer to recover temporal patterns without increasing the centroid count.\n\n- **Hashing for Positional Fidelity:**  \n  Unlike prior merging methods that average away positional encodings or pruning approaches that drop tokens entirely, we map each patch’s spatio-temporal coordinates into a simple integer hash. This hash serves as a “positional supplement” in VQ-Attention, ensuring each centroid remains linked to its exact location in time and space. In our ablation (Table 6), restoring hashed indices nearly fully recovers full-token accuracy (57.2 → 57.5) at only 0.14 % of tokens.\n\n**Update in Camera-Ready:**  \n- Add a dedicated paragraph in § 3.2.  \n- Include a schematic illustrating hashed-index injection.  \n- Clarify the module’s dual objectives.\n\n---\n\n### 2. Comparison to Recent Token-Pruning Methods  \n**Q2:** “The paper should compare with more recent token pruning methods. To establish the value of this work, the comparison should focus on recent competitive methods.”  \n**A:** Sorry for the misunderstanding and thank you for highlighting the importance of foregrounding our empirical comparisons. In the current draft, we evaluate and outperform more than three very recent state-of-the-art methods, all published within the last six months, under comparable (their original) or more generous (10 %) token budgets, for example:\n\n- **VidToMe** (NeurIPS 2024, Dec 24) – a very recent video token-merging approach (Table 1–2)[1].  \n- **DyCoke** (CVPR 2025, June 25) – the current SOTA in dynamic token-pruning for video LLMs (Table 3)[2].  \n- **Interpolative Merging from LLaVA-OneVision** (TMLR, Feb 7, 2025) – an alternate token-merging strategy[3].\n\nIn contrast, **VQToken** achieves higher top-1 accuracy and denser representations (TokDense) while using only 0.07 %–0.14 % of the original tokens—an order of magnitude fewer than these baselines.\n\n**Revision & Highlight Plan:**  \n1. Add a “Baselines” subsection in Section 4 detailing selection criteria (recency, token-budget, task alignment).  \n2. Enhance table captions with each method’s mechanism, venue, and token budget.  \n3. Provide an explicit performance breakdown in the main text, highlighting VQToken’s accuracy gains at matched token counts.  \n4. Include any (if have, currently we don't find any) newly published extreme token reduction methods (≥ 99 % compression) prior to camera-ready submission.\n\n---\n\n### 3. Novelty of Extreme Token Reduction  \n**Q3:** “Being able to reduce the number of tokens drastically is impressive, but such observation has been made before.”  \n**A:** To our knowledge, no prior work has:\n1. Formally defined extreme token reduction with minimal token counts as a standalone task, and compressing token inputs by up to 99.84%.  \n2. Introduced evaluation metrics that quantify information preservation at such scales.  \n3. Demonstrated robust performance on standard video understanding benchmarks under these constraints.  \n\nMost existing methods cap out at 30 %–80 % reduction with some accuracy loss. We welcome references to any methods exceeding 99 % reduction and will include direct comparisons in our revision.\n\n---\n\n### 4. Clarity and Method Structure  \n**Q4:** “Overall clarity needs improvement. The Method section does not explain how each design addresses the stated weaknesses.”  \n**A:** Thank you for pointing out the need for a clearer organization. In the current draft, we present high-level ideas immediately after each of the three challenges  (rows 49–59, 60–71, and 72–81), then dive into mathematical details. To improve readability and explicitly link each design choice to its motivating challenge, we will:\n- Insert section markers in the Introduction (e.g. § 3.1 “Adaptive Discretization,” § 3.2 “VQ-Attention,” etc.) so readers can instantly locate each component.  \n- Precede every mathematical derivation with a brief “Motivation” paragraph stating which challenge it solves and why it matters under extreme token reduction.\n\nThese changes will make the logical flow crystal clear. If there are still parts that feel disconnected, we would greatly appreciate pointers to specific sentences or transitions so we can further tighten the narrative.\n\n---\n\n## Conclusion\n\nWe appreciate these observations and will integrate the above revisions to clarify our motivations, strengthen empirical comparisons, and improve readability. We are confident these changes will fully address Reviewer Gyx4’s concerns and underscore VQToken’s novelty and impact.\n\n---\n\n### References\n1. Lee, Seon-Ho, et al. “Video token merging for long-form video understanding.” *NeurIPS* (Dec 24, 2024).  \n2. Tao, Keda, et al. “DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models.” *CVPR* (June 25, 2025).  \n3. Li, Bo, et al. “LLaVA-OneVision: Easy Visual Task Transfer.” *Transactions on Machine Learning Research* (Feb 7, 2025)."}, {"review_id": "2Ha2ZVbfTC", "reviewer": "Reviewer_BtRV", "summary": "The paper analyzes the shortcomings of existing token reduction techniques for video LLMs, and introduces a discrete token representation method to achieve extreme short token reduction. They use VQ to learn a codebook, and use a token hash function that records each token’s original position to preserve spatial-temporal position. Experiments show good performance on NextQA-MC, but lose fairly on long video benchmarks.", "questions": "1. According to table 5, it seems longer videos will lead to more severe performance drop. Can authors discuss and give deeper analysis about this?\n\n2. As VQToken only shows strong performance on NextQA and all ablation studies (Table2, 3, 6) and comparison with SOTAs (Table 5) also only report NextQA, the results may be not reliable enough as VQToken decreases heavily on others. Authors should add ablation studies and comparison on other benchmarks.\n\n3. Training details. Authors seem train from LLaVA-OV checkpoint with LLaVA-Video[1] training data? LLaVA-Video is not refered and cited in the paper, neither compared. Direct comparison with LLaVA-Video may not be fair, as VQToken introduces newly initialized parameters. A similar trained baseline with same LLaVA-Video data is required to compare.\n\n[1] Yuanhan Zhang, et al.Video Instruction Tuning With Synthetic Data. https://arxiv.org/pdf/2410.02713", "limitations": "See questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strength:\n● This paper introduces a discrete codebook for video token compression in llms. Though discrete VQ is widely used in unified models (unifying understanding and generation), and is proved somewhat decreasing understanding ability, the potential of extreme short token reduction remains unexplored.\n● VQToken achieves promising results on NextQA with quite high compressing ratio.\n\nweakness:\n● VQToken shows fairly large decreasing on other three benchmarks. The only promising result on NextQA is not strong enough to prove the effectivenss and needs more evidences. See questions.", "was_revised": true, "final_justification": "The author has addressed part of my concerns. I will keep my score at Borderline Accept.", "rebuttal": "## Response to Reviewer BtRV\n\nWe thank Reviewer BtRV for raising these concerns. We have added more fair baselines than those used in current benchmarks, provided additional ablation studies across multiple datasets, and demonstrated why our approach qualifies as state-of-the-art in extreme token reduction efficiency. We also highlight the academic value of extreme token reduction task and how future work can leverage it to advance related but distinct tasks in broader research areas.\n\n---\n\n### Q1. Support on long Video Understanding Task  \n> “According to Table 5, it seems longer videos lead to more severe performance drop. Can authors discuss and give deeper analysis about this?”\n\n#### A1  \nWe truly thank your valuable questions and suggestion for improving our work. We add more discussion and experiments hoping this can solve your questions. Please feel free to tell us if this solve your concerns during discussion period.\n\n**Meaning of these different tasks**  \nWe thank the reviewer for highlighting this important trend. The long-video understanding task is another important, yet **completely different, task** in video understanding. Our proposed extreme token reduction task is also crucial and supports unique application scenarios.  \nIndeed, compressing a 30 s clip into 32 tokens is very different from compressing a 3 hour clip into the same budget—longer clips necessarily bundle more diverse content per token, making fine-grained reasoning harder.\n\n**Different Task Objectives**  \n- **Extreme Token Reduction**  \n  - **Goal:** “How can we represent a video clip with as few tokens as possible, while still supporting downstream tasks?”  \n  - **Scope:** Aggressively compress each clip (e.g., adaptively less than 32 tokens) for **resource-constrained settings** (edge devices, smart glasses, autonomous vehicles).\n\n- **Long-Video Understanding**  \n  - **Goal:** “How can we model and reason over hours of continuous footage, preserving detailed temporal coherence and long-range dependencies?”  \n  - **Scope:** Retain an informative, ordered representation across many clips.\n\nOur VQToken pipeline is specifically optimized for the extreme token reduction task and does **not** include mechanisms for stitching together multiple clips or explicitly modeling very long-range event structure for the long-video understanding task.\n\n**Empirical Length Analysis**  \nIn Table 5, despite achieving more than a **300× improvement** over TokDense while using only **0.07 %–0.14 %** of the tokens and and outperform on many effiency metrics , we observe NextQA (≈ 60 s clips): 0.45 % accuracy drop, ActivityNet-QA (≈ 2 minutes clips)1.93 % accuracy drop, VideoMME (≈  17 minutes): 11.54 % accuracy drop, clearly illustrates the accuracy–efficiency trade-off.\n\nThis confirms that extreme compression incurs larger costs as clip duration increases.\n\n**Future Directions to Extend Extreme Token Reduction into Long-Video Understanding**  \nWe regard extending our extreme token reduction task to long-video understanding as a very promising future direction, which can benefit both tasks of research topics. Some ideas include:  \n1. **Segmented Windowing**  \n   - Split long videos into shorter (e.g., 1–2 minute) windows, apply VQToken per window, then fuse window embeddings with a lightweight transformer.  \n2. **Hierarchical VQ**  \n   - First quantize at the segment level, then perform a second-stage VQ on segment embeddings to capture inter-segment dependencies.  \n3. **Adaptive Budgeting**  \n   - Dynamically allocate more tokens to high-motion or semantically rich segments using a small importance predictor.\n\n> _We will add this analysis and outline these future directions in the camera-ready version._\n\n---\n\n### Q2. Ablation Studies Across Multiple Benchmarks  \n> “All ablation studies (Table 2, 3, 6) and comparisons report only NextQA. Please add ablations on other benchmarks.”\n\n#### A2  \nWe appreciate this suggestion. In the submitted draft, we focused our ablations on NextQA—both to control costs (avoiding commercial API calls) and to iterate rapidly over many settings. To demonstrate that each component’s contribution generalizes beyond NextQA, we have now run the three-stage ablation (None → +Codebook → +VQ-Attn) on additional benchmarks like ActivityNet-QA.\n\n| Benchmark      | Variant              | Accuracy ↑ | TokDense ↑  |\n|----------------|----------------------|------------|-------------|\n| **NextQA**     | None        | 57.2       | 0.0025    |\n|                | +Codebook               | 35.2       | 1.1000       |\n|                | + VQ-Attn (full)     | 57.5       | 1.7969    |\n| **ActivityNet-QA** | None   | 49.0          | 0.0021           |\n|                | +Codebook         | 36.8          | 1.1500           |\n|                | + VQ-Attn (full)     | 46.3       | 1.4469    |\n\n---\n\n### Q3. Fair Comparison via Matched Fine-Tuning  \n> “Please ensure fair comparison by aligning training regimes across baselines.”\n\n#### A3  \nWe appreciate the reviewer’s suggestion to ensure a fair comparison by aligning training regimes. \n\n1. Many prior video‐LLM works compare models trained under different schedules or data regimes (because there are many closed-source methods trained on private or internet-scale dataset and resources) on almost every benchmark. \n\n2. To address this common issue, we fine-tuned the LLaVA-OneVision baseline for one epoch on the same LLaVA-Video-178K[1] dataset, matching our VQToken training,and report:\n\n| Model                        | Fine-Tuning                     | NextQA Acc ↑ | NextQA TokDense ↑ | ActivityNet-QA Acc ↑ | ActivityNet-QA TokDense ↑ | VideoMME Acc ↑ | VideoMME TokDense ↑ | Token % ↓ |\n|------------------------------|---------------------------------|--------------|-------------------|-----------------------|----------------------------|----------------|----------------------|-----------|\n| **LLaVA-OneVision (baseline)** | 1 epoch on LLaVA-Video-178K[1]   | 57.71        | 0.0049            | 47.16                 | 0.0040                     | 44.37          | 0.0038               | 100 %     |\n| **VQToken**                  | 1 epoch on LLaVA-Video-178K[1]    | 57.44        | 1.7950            | 46.25                 | 1.4453                     | 39.34          | 1.2294               | 0.14 %    |\n| **Trade-off**                | –                               | 99.53 %      | ×366              | 98.07 %               | ×361                       | 88.66 %        | ×323                 | 99.86 %   |\n\n- **Accuracy retention:** VQToken retains 99.5 % of NextQA’s accuracy, 98.1 % of ActivityNet-QA’s, and 88.7 % of VideoMME’s under matched fine-tuning.  \n- **Token efficiency:** Uses 800×–360× fewer tokens(0.14\\%), yielding a 300×+ improvement in TokDense across all benchmarks.\n\n> _We will include this table, discussion, and citation in the camera-ready version._\n\n---\n\n### W1. Concerns on Performance Drops  \n> “VQToken shows fairly large decreases on other three benchmarks. The only promising result on NextQA is not strong enough to prove effectiveness and needs more evidence.”\n\n#### A4  \nWe respectfully disagree that VQToken suffers “large” performance drops in the context of extreme token reduction. As shown in above table, under a 99.86 % reduction in token count, VQToken still retains (achieves an excellent efficiency–performance trade-off):  \n- **NextQA:** 99.5 % of baseline accuracy (57.4 vs. 57.7 pp)  \n- **ActivityNet-QA:** 98.1 % of baseline accuracy (46.3 vs. 47.2 pp)  \n- **VideoMME:** 88.7 % of baseline accuracy (39.3 vs. 44.4 pp)\n\nThis corresponds to a **> 300× increase** in token efficiency (TokDense) across all benchmarks. No prior method simultaneously:  \n1. **Removes > 99 % of tokens**,  \n2. **Retains > 88 % accuracy** on diverse video-QA tasks, and  \n3. **Reduces inference throughput, FLOPs, memory footprint, and latency by orders of magnitude**.\n\n> _We will further clarify this argument in the camera-ready and highlight VQToken’s unprecedented balance of compression ratio, accuracy retention, and computational efficiency._\n\n\n---\nReference\n[1] Yuanhan Zhang, et al.Video Instruction Tuning With Synthetic Data. https://arxiv.org/pdf/2410.02713"}, {"review_id": "WrWYmAx7q8", "reviewer": "Reviewer_LjpF", "summary": "In this paper the authors propose VQToken, a video token reduction framework for Vision Transformers to represent entire videos using a minimal set of discrete tokens. It uses adaptive K-Means to cluster the video patch embeddings and use the cluster center as the codebook entry. It also use a token-hashing mechanism to embed the spacial and temporal information into the compressed tokens. It achieves promising results in LLaVA-OneVision model, dramatically decrease the video token amount while preserving the performance in downstream VLM tasks.", "questions": "1. In the adaptive-length reduction mode, is the reduction ratio fixed for one give video clip? Is it possible to dynamically adjust the reduction ration in the video clip?\n2. How long does one video clip last typically in the NExT-QA evaluation benchmark, where you claim a 0.07% reduction ratio? What is NextQA-MC? Is it a subset of NExT-QA?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed token reduction framework can do adaptive-length reduction, dynamically select the compression rate based on video's content complexity\n2. The proposed token reduction framework is simple and practical, potentially easy to apply.\n3. The experiment results seems promising, demonstrating non-trivial improvements compared to the baseline reduction methods.\n\nWeakness:\n\nThe base model is LLaVA-OneVision (0.5B). It is a small model after all and it is doubtful whether the proposed token reduction framework will become the performance bottleneck when applied to larger base models.", "was_revised": true, "final_justification": "The proposed method seems novel, and the experiment results demonstrate its value in real-world applications.", "rebuttal": "## Response to Reviewer LjpF\n\nWe are deeply grateful to Reviewer LjpF for the exceptionally thorough and professional review. Your insightful comments and clear understanding of the challenges in efficient video‐LLM research are invaluable. We appreciate your time and expertise in helping us strengthen our work, and we will gladly incorporate your suggestions and questions about details and will release our full code and implementation details.\n\n---\n\n### Q1: Adaptive‐length Reduction Mode  \n> “Is the reduction ratio fixed for one given video clip? Is it possible to dynamically adjust the reduction ratio within a clip?”\n\n**A1:**  \nThank you for this excellent question. The reduction ratio in our adaptive‐length mode is **not** fixed per clip. We use **adaptive clustering** (e.g., adaptive K-Means) at test time, allowing different clips, and even different segments within a clip, to yield varying token counts. Because this clustering is decoupled from the core VQToken parameters, one can dynamically adjust the number of centroids (for example, by tuning the _K_ in K-Means or substituting alternative segmentation models like SAM) **without retraining the model**. We appreciate your suggestion and will clarify this flexibility in the revised manuscript.\n\n---\n\n### Q2: NextQA‐MC vs. NExT‐QA and Clip Length  \n> “How long does one video clip last typically in the NExT-QA evaluation benchmark, where you claim a 0.07% reduction ratio? What is NextQA-MC? Is it a subset of NExT-QA?”\n\n**A2:**  \nWe sincerely thank you for asking for these details. We evaluate on the **mc test-set split** from the HuggingFace repo (lmms-lab/NExTQA), commonly referred to as **NextQA-MC**. This split contains specially curated multiple‐choice QA pairs and is widely used by recent works (e.g., LLaVA-OneVision, DyCoke).\n\nIn our **Ours-Dynamic** setting (Table 1, last two rows):\n- **Average tokens per clip:** 13.08  \n- **Uncompressed baseline tokens:** 23,328 (LLaVA-OV without reduction, row 243)  \n- **Reduction ratio:**  \n$\\displaystyle \\frac{13.08}{23{,}328} \\approx 0.07 \\\\%$\n\nWe will add this detailed calculation and a brief description of NextQA-MC in Section 4.2 to ensure full clarity.\n\n---\n\n### W1: LLM Scaling Up  \n> “The base model is LLaVA-OneVision (0.5 B). It is doubtful whether token reduction becomes a bottleneck on larger models.”\n\n**A3:**  \nWe greatly appreciate this important perspective. While we selected the 0.5 B backbone to highlight practical efficiency gains, we agree that larger models may better recover compressed information. To validate, we scaled our experiments to the **7 B variant**:\n\n| Model           | Accuracy | TokDense  |\n|-----------------|----------|-----------|\n| 7 B LLaVA-OV    | 81.40    | 0.007   |\n| 7 B VQToken     | 78.59    | 2.456   |\n\nThese results confirm that VQToken not only maintains but improves performance when paired with a larger backbone, supporting our theoretical expectation that increased model capacity aids recovery from extreme compression. We will include this table and discussion in the revised paper."}, {"review_id": "snWCn492bf", "reviewer": "Reviewer_RRqQ", "summary": "The authors propose VQ-Token to achieve extreme token compression for video large language models. It utilizes vector quantization to cluster continuous visual feature tokens from a video into a very small, discrete \"codebook,\" and compress the number of tokens to just 0.07% of the original. Simultaneously, it uses a token hash function to preserve key spatiotemporal information, preventing the loss of dynamic cues during compression.", "questions": "1. In Table 4, by retaining only 0.14% of the tokens, the model can achieve performance close to that of the full-token model, but a performance gap still exists. Will the performance on video understanding tasks be improved by moderately increasing the number of retained tokens? A scaling experiment showing the performance trend with respect to the token count would be very helpful to illustrate this point.\n2. For the results on benchmarks like LongVideoBench, would it be possible to provide the specific numerical scores for its various subsets? This would be very helpful for understanding the specific effect of the method on each individual task.", "limitations": "See weakness and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces VQ-Token, a highly novel and effective framework that uses vector quantization to create discrete representations, enabling extreme and efficient video token compression.\n2. The definition of the 'Extreme Token Reduction' task and the 'TokDense' metric provides a strong foundation for future research.\n3. The proposed method is supported by extensive and rigorous experiments, demonstrating state-of-the-art compression ratios (down to 0.07%) with a negligible drop in accuracy (0.66%).\n\nWeakness:\n1. The computational overhead may be introduced by the dynamic codebook generation. Although the k-Means algorithm is relatively efficient, it still needs to perform a full clustering computation on thousands of high-dimensional tokens at every inference step. This will add latency to the overall computation, but it is unclear what the specific FLOPs are for this process or how much latency it adds when compared to other methods.", "was_revised": true, "final_justification": "4: Borderline accept", "rebuttal": "## Response to Reviewer RRqQ\n\nWe are deeply grateful to Reviewer RRqQ for their exceptional thoroughness, insightful feedback, and thoughtful suggestions. Your careful analysis has enabled us to sharpen our presentation, fortify our arguments, and address nuances we might have otherwise overlooked. We sincerely appreciate the time, expertise, and genuine engagement you’ve invested in our manuscript, your guidance has been instrumental in enhancing its clarity and impact.\n\nBelow, we present our revised responses, remaining faithful to our original method and refraining from introducing any new architectures. We trust these clarifications will fully address your concerns.\n\n---\n\n### Q1. Performance Trend w.r.t. Token Count  \n> “A scaling experiment showing the performance trend with respect to the token count would be very helpful.”\n\n**A1:**  \nThank you for this suggestion. We performed a fixed-cluster scaling study with our pretrained parameters at three different token budgets and compared against three recent methods. All models were evaluated on NextQA under identical conditions:\n\n| Method         | Tokens | Accuracy ↑ | TokDense ↑ |\n|----------------|-------:|-----------:|-----------:|\n| Token Pruning  |     12 |      29.12 |      2.427 |\n| Token Pruning  |     32 |      34.50 |      1.078 |\n| Token Pruning  |     64 |      31.31 |      0.489 |\n| ToMe           |     12 |      35.72 |      2.977 |\n| ToMe           |     32 |      38.50 |      1.203 |\n| ToMe           |     64 |      40.10 |      0.627 |\n| VidToMe        |     12 |      39.64 |      3.303 |\n| VidToMe        |     32 |      45.10 |      1.409 |\n| VidToMe        |     64 |      46.20 |      0.722 |\n| **Ours (Fixed)** |    12 |      57.03 |      4.753 |\n| **Ours (Fixed)** |    32 |      57.46 |      1.796 |\n| **Ours (Fixed)** |    64 |      57.10 |      0.892 |\n\n**Key insight:**  \nEven at just 12 tokens, our fixed-cluster variant achieves 57.0 % accuracy—far above other methods—while maintaining the highest token efficiency.\n\n> **Note on fewer clusters:**  \n> Our model was pretrained with a 32-cluster default; extending to higher _K_ (e.g. 64) without additional architecture or fine-tuning yields smaller gains. From experiments, it seems the performance peak is at the token number used during finetuning. It might be shown if we use adaptive token counts during training rather than testing to have the performance trend grow with length. We view scaling to arbitrary token budgets as a valuable future direction with additional modules needed, as discussed below.\n\n**Future Direction (Long-Form Videos):**  \nSplit each video into multiple mid-length clips and apply our reduction to each segment—capping each at less than 32 tokens. Concatenating these segment-level tokens produces a compact representation of the full video: still far shorter than conventional methods, yet rich enough to capture long-form content. This segmentation strategy will enable our approach to scale to long-video understanding while preserving extreme token efficiency.\n\n> _We will include this scaling table and discussion in the camera-ready version._\n\n---\n\n### Q2. LongVideoBench Subset Scores  \n> “Would it be possible to provide the specific numerical scores for its various subsets?”\n\n**A2:**  \nWe evaluated our method on each LongVideoBench subset—defined by maximum frame counts of 16, 32, and 64—using the lmms-eval toolkit. Preliminary results:\n\n| max_frames |   Acc   | TokDense |\n|-----------:|--------:|--------:|\n|         16 | 26.03  | 1.63 |\n|         32 | 34.11  |1.07 |\n|         64 | 20.79  | 0.32 |\n\n> Since the LongVideoBench repository does not provide separate evaluation scripts for each subset, we followed the official LongVideoBench and lmms-eval tutorials to generate these numbers. We will add this table as an appendix and reference it alongside Figure 3 in the main text.\n\n> We commit to extending more detailed subset analysis to more additional benchmarks  and including all corresponding results in the camera-ready version.\n---\n\n### Q3. Latency & FLOPs Breakdown  \n> “Add latency to the overall computation, but it is unclear what the specific FLOPs are or how much latency.”\n\n**A3:**  \nWe agree that a clear complexity breakdown is valuable. The overall latency and time complexity has two parts:  \n1. **Token reduction module** (e.g., K-Means clustering)  \n2. **Subsequent LLM inference** on the shorter token sequence  \n\nIn addition to our measured results (see Table 1, p. 2), we will provide a theoretical analysis of time complexity and FLOPs for the K-Means step, which dominates our pipeline.\n\n#### Theoretical Time Complexity of K-Means  \n- **Notation:**  \n  - $M$: number of data points (tokens)  \n  - $D$: embedding dimension  \n  - $K$: number of clusters  \n  - $N$: number of iterations  \n\n- **Per-iteration cost:**  \n  1. **Assignment:** $O(M\\,K\\,D)$ distance computations  \n  2. **Update:** $O\\bigl((M+K)\\,D\\bigr)$ centroid re-estimation  \n\n- **Total time complexity:**  \n  $$\n    \\sum_{i=1}^{N} \\bigl[O(M\\,K\\,D) + O((M+K)\\,D)\\bigr]\n    = O(N\\,M\\,K\\,D)\n  $$\n\n#### FLOPs Analysis for K-Means\n\n- **Notation:**  \n  - $M$: number of tokens  \n  - $D$: embedding dimension  \n  - $K$: number of clusters  \n  - $N$: number of iterations  \n\n- **Assignment step (per iteration):**  \n  We can compute a $D$-dimensional distance for each of the $M$ tokens against each of the $K$ centroids. Each distance costs $2D$ subtractions, $D$ multiplications, and $D-1$ additions, for a total of $2D + (D-1) \\approx 3D$ flops, so  \n  $\\mathrm{FLOPs}_{\\mathrm{assign}} = M \\times K \\times \\bigl(2D + (D-1)\\bigr) \\approx 3\\,M\\,K\\,D.$\n\n- **Centroid-update step (per iteration):**  \n  Summing assigned token embeddings ($M \\times D$ additions) and normalizing each of the $K$ centroids ($K \\times D$ divisions) gives  \n  $\\mathrm{FLOPs}_{\\mathrm{update}} = M\\,D + K\\,D = (M + K)\\,D.$\n\n- **Total per iteration:**  \n  $\\mathrm{FLOPs}_{\\mathrm{iter}} = 3\\,M\\,K\\,D + (M + K)\\,D = D \\times (3\\,M\\,K + M + K).$\n\n- **Total over $N$ iterations:**  \n  $\\mathrm{FLOPs}_{\\mathrm{total}} = N \\times \\bigl[3\\,M\\,K\\,D + (M + K)\\,D\\bigr] = N\\,D \\times (3\\,M\\,K + M + K).$\n\n\n\n> We will refer to Table 1 for measured module and end-to-end LLM latency, and include this theoretical breakdown in the revised manuscript. \n\nIn addition to the theoretical FLOPs breakdown, we will include actual latency measurements (in milliseconds) for each component in Table 1 of the camera-ready version, specifically:\n1. K-Means clustering latency\n2. LLM inference latency\n3. End-to-end total pipeline latency\nThis will enable readers to directly compare both the theoretical complexity and the practical runtime overhead of each step.\n\n---\n\nWe hope these clarifications and the added tables will fully address the reviewer’s points. Thank you again for guiding us toward a stronger, more transparent presentation."}]

Author rebuttal:

Meta-review: a) The paper introduces VQToken, a framework for extreme token reduction in video large language models (LLMs). The method clusters patch embeddings via k-means into a discrete codebook and augments tokens with a hash-based positional encoding to preserve spatiotemporal cues. A VQ-based reduction module integrates these discrete tokens with attention. The approach achieves dramatic token compression (down to 0.07% of original tokens) while retaining competitive accuracy on benchmarks such as NextQA, with reduced performance on long-video tasks.

b) Using discrete VQ codebooks for token reduction in video LLMs is novel, and combining them with positional hashing is innovative.
One of the most important aspects of this method is the extreme compression ratio (while the accuracy is kept fixed). This is certainly a very useful property for scaling video LLMs. Moreover, the method is simple, adaptable, and potentially impactful for real-world deployment. There are also new ideas for the evaluation (e.g., TokDense) and the presentation is found clear.

(c) The evaluation lacks comparisons to recent strong token pruning/compression methods, limiting the claim of state-of-the-art. Performance drops significantly on long-video tasks, raising concerns about generalization. The method section is relatively short, with unclear motivation for some design choices (e.g., VQ-attention, hashing). The k-means clustering step may add inference-time overhead, but this is not quantified. Finally, results are shown only on a relatively small base model, leaving scalability uncertain.

(d) The work is technically sound and offers a practical, impactful framework for extreme video token reduction. While novelty is incremental and evaluation incomplete, the ability to achieve significant compression with competitive accuracy is meaningful. The contribution does not reach the level of a spotlight/oral, but it justifies acceptance.

(e) One reviewer rejected the paper citing weak baselines and unclear motivation, while others found the approach promising and novel in practice. The rebuttal improved clarity but left baseline and overhead issues unresolved. Overall, consensus leaned toward borderline accept
and the AC agrees with them.
Decision: Accept (poster)