Title: VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video LLMs; Token Reduction; Discrete Token Representation; Extreme Token Reduction; Token Merge; Token Clustering
Abstract: Token-based video representation has emerged as a promising approach for enabling large language models (LLMs) to interpret video content. However, existing token reduction techniques, such as pruning and merging, often disrupt essential positional embeddings and rely on continuous visual tokens sampled from nearby pixels with similar spatial–temporal locations. By removing only a small fraction of tokens, these methods still produce relatively lengthy continuous sequences, which falls short of the extreme compression required to balance computational efficiency and token count in video LLMs.
In this paper, we introduce the novel task of **Extreme Short Token Reduction**, which aims to represent entire videos using a minimal set of discrete tokens.  We propose **VQToken**, a neural discrete token representation framework that
(i) applies adaptive vector quantization to continuous ViT embeddings to learn a compact codebook and 
(ii) preserves spatial–temporal positions via a token hash function by assigning each grid-level token to its nearest codebook entry.
On the Extreme Short Token Reduction task, our VQToken compresses sequences to just **0.07\%** of their original length while incurring only a **0.66\%** drop in accuracy on NextQA-MC benchmark. It also achieves comparable performance on ActNet-QA, Long Video Bench, and VideoMME. 
We further introduce the **Token Information Density** (**TokDense**) metric and formalize fixed-length and adaptive-length subtasks, achieving state-of-the-art results in both settings. Our approach dramatically lowers theoretical complexity, increases information density, way fewer tokens counts, and enables efficient video large language models in resource-constrained environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose VQ-Token to achieve extreme token compression for video large language models. It utilizes vector quantization to cluster continuous visual feature tokens from a video into a very small, discrete \"codebook,\" and compress the number of tokens to just 0.07% of the original. Simultaneously, it uses a token hash function to preserve key spatiotemporal information, preventing the loss of dynamic cues during compression.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces VQ-Token, a highly novel and effective framework that uses vector quantization to create discrete representations, enabling extreme and efficient video token compression.\n2. The definition of the 'Extreme Token Reduction' task and the 'TokDense' metric provides a strong foundation for future research.\n3. The proposed method is supported by extensive and rigorous experiments, demonstrating state-of-the-art compression ratios (down to 0.07%) with a negligible drop in accuracy (0.66%).\n\nWeakness:\n1. The computational overhead may be introduced by the dynamic codebook generation. Although the k-Means algorithm is relatively efficient, it still needs to perform a full clustering computation on thousands of high-dimensional tokens at every inference step. This will add latency to the overall computation, but it is unclear what the specific FLOPs are for this process or how much latency it adds when compared to other methods.\n\nQUESTIONS:\n1. In Table 4, by retaining only 0.14% of the tokens, the model can achieve performance close to that of the full-token model, but a performance gap still exists. Will the performance on video understanding tasks be improved by moderately increasing the number of retained tokens? A scaling experiment showing the performance trend with respect to the token count would be very helpful to illustrate this point.\n2. For the results on benchmarks like LongVideoBench, would it be possible to provide the specific numerical scores for its various subsets? This would be very helpful for understanding the specific effect of the method on each individual task.\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

comments: SUMMARY:
In this paper the authors propose VQToken, a video token reduction framework for Vision Transformers to represent entire videos using a minimal set of discrete tokens. It uses adaptive K-Means to cluster the video patch embeddings and use the cluster center as the codebook entry. It also use a token-hashing mechanism to embed the spacial and temporal information into the compressed tokens. It achieves promising results in LLaVA-OneVision model, dramatically decrease the video token amount while preserving the performance in downstream VLM tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The proposed token reduction framework can do adaptive-length reduction, dynamically select the compression rate based on video's content complexity
2. The proposed token reduction framework is simple and practical, potentially easy to apply.
3. The experiment results seems promising, demonstrating non-trivial improvements compared to the baseline reduction methods.

Weakness:

The base model is LLaVA-OneVision (0.5B). It is a small model after all and it is doubtful whether the proposed token reduction framework will become the performance bottleneck when applied to larger base models.

QUESTIONS:
1. In the adaptive-length reduction mode, is the reduction ratio fixed for one give video clip? Is it possible to dynamically adjust the reduction ration in the video clip?
2. How long does one video clip last typically in the NExT-QA evaluation benchmark, where you claim a 0.07% reduction ratio? What is NextQA-MC? Is it a subset of NExT-QA?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the shortcomings of existing token reduction techniques for video LLMs, and introduces a discrete token representation method to achieve extreme short token reduction. They use VQ to learn a codebook, and use a token hash function that records each token’s original position to preserve spatial-temporal position. Experiments show good performance on NextQA-MC, but lose fairly on long video benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nstrength:\n● This paper introduces a discrete codebook for video token compression in llms. Though discrete VQ is widely used in unified models (unifying understanding and generation), and is proved somewhat decreasing understanding ability, the potential of extreme short token reduction remains unexplored.\n● VQToken achieves promising results on NextQA with quite high compressing ratio.\n\nweakness:\n● VQToken shows fairly large decreasing on other three benchmarks. The only promising result on NextQA is not strong enough to prove the effectivenss and needs more evidences. See questions.\n\nQUESTIONS:\n1. According to table 5, it seems longer videos will lead to more severe performance drop. Can authors discuss and give deeper analysis about this?\n\n2. As VQToken only shows strong performance on NextQA and all ablation studies (Table2, 3, 6) and comparison with SOTAs (Table 5) also only report NextQA, the results may be not reliable enough as VQToken decreases heavily on others. Authors should add ablation studies and comparison on other benchmarks.\n\n3. Training details. Authors seem train from LLaVA-OV checkpoint with LLaVA-Video[1] training data? LLaVA-Video is not refered and cited in the paper, neither compared. Direct comparison with LLaVA-Video may not be fair, as VQToken introduces newly initialized parameters. A similar trained baseline with same LLaVA-Video data is required to compare.\n\n[1] Yuanhan Zhang, et al.Video Instruction Tuning With Synthetic Data. https://arxiv.org/pdf/2410.02713\n\nLIMITATIONS:\nSee questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Neural Discrete Token Representation Learning. It first uses k-means to cluster the tokens and uses the centroid as the token codebook. Then it constructs a hash function that encodes each patch's position and spatial information. It finally integrates the codebook and the hash representation with a VQ-based reduction module. Evaluations on several benchmarks demonstrate with reduced number of tokens the performance can match the original model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Viewing the grouping of tokens as discrete codebook is a new angle.\n2. Integrating spatial information and token features using VQ-attention is an interesting approach.\n3. Experiments and ablation are relatively extensive.\n\nWeaknesses:\n1. The motivation for the VQ-based attention module is not very clear. Why does such an attention module \"enrich each centroid with\nmotion context without increasing token count\"? Why is hashing the patches to an integer an effective way to retain position and temporal information?\n\n2. The evaluation only compares against a few video token pruning baselines, and most comparisons are done with models that take full tokens, or . Token pruning in video LLM has a rich literature. The paper should compare with more recent token pruning methods under similar token numbers. Being able to reduce the number of tokens drastically is impressive, but such observation has been made before. To establish the value of this work, the comparison should focus on recent competitive methods.\n\n3. The overall clarity needs improvement. The motivation part is organized (three weaknesses of existing methods), but they are not very convincing. The method section does not talk about how each design address these weaknesses.  The method section overall is too short, lacking explanation of what motivated the designs.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}