Title: One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Computer Vision

Long Video Understanding
Abstract: Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards *one token per frame* at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into *learnable* and *progressive* modules for *token-level compression* (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate *frame-level compression*, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named *question-conditioned compression* (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, *i.e.*, the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined *token-level* and *frame-level* leads to an e**x**treme compression model for long video understanding, named **XComp**, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient *supervised compression tuning* stage that only requires 2.5\% of the supervised fine-tuning data, yet boosts the accuracy from 42.9\% to 46.2\% on LVBench and enhances multiple other long video benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new video compression method that rely on associated each video frame to a single token (using LLM layers) and then choose the most relevant tokens associated to a given question. The methods reduces the numbers of tokens progressively by adding a stack of LLM layers that reduce each time a bit more the number of visual tokens until reaching a single token, then an attention mechanism coined as Segmented Local Attention select the most relevant tokens to answer a question. The X-Comp model is based on fine-tuning VideoChat-Flash-2B model. The authors run their experiments on LongVideoBench, MLVU, VideoMME (Long) and LVBench. They show that their fine-tuned method improve performances on those benchmarks with respect to the base VideoChat-Flash model. They also argue that their methods have less attention biases (focusing on either the beginning or the end of a token sequence) since it can discards easily the frames that are not relevants.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and easy to read. The figures are great to better understand the method. The motivation is also well defined and the improvements over the base VideoChat-Flash-2B model seems good on long-video understanding. From the experiments, it's clear that longer a video is, more gain in performances we can expect with this method. The ablation study is also an important contribution to this paper by exploring the gain that are added by each of the component of the methods (such as which attention strategy or compression method is used).\n\nThe main weakness of this work which is highlighted by the authors in the limitation part is the reliance on only VideoChat-Flash which is not sufficient to see wether this method can also work with other models or not. Also the ablation studies are not consistently performed across the same benchmarks so it's harder to parse the importance of each of the components.\n\nQUESTIONS:\nWhy didn't you performed the ablation of Table 4 and 5 on LVBench?\n\nLIMITATIONS:\nYes, the limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper “One Token per Highly Selective Feature” proposes a lightweight and general token reduction method called OTPF, which significantly improves inference efficiency for vision and multimodal models. By selecting and compressing only the most informative features from early layers—based on activation strength or selectivity—OTPF generates a small number of tokens that preserve essential semantic information. This approach can be used without training or integrated into end-to-end learning. Experiments on image classification, video understanding, and multimodal QA demonstrate that OTPF achieves substantial token reduction (up to 90%) with minimal performance drop, offering a simple, modular, and effective solution for efficient large-model deployment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. OTPF achieves substantial token reduction (up to 90%) while preserving accuracy across multiple tasks, demonstrating strong practical value in resource-constrained settings.\n2. The method is compatible with a wide range of backbones (ViT, CLIP, ResNet) and tasks (image classification, video understanding, multimodal QA), showing strong versatility.\n3. OTPF can be used without any additional training, yet also supports end-to-end learning, making it easy to integrate into existing pipelines.\n4. Outperforms or matches state-of-the-art baselines (e.g., uniform sampling, VideoMAE) in low-token regimes, validated across diverse benchmarks like ImageNet-1K and Kinetics-400.\nWeaknesses:\n1. OTPF’s effectiveness depends heavily on the quality of early-layer representations. If the backbone lacks strong discriminative power, selected tokens may be suboptimal.\n2. OTPF compresses every input into a fixed number of tokens, regardless of the semantic richness or spatial/temporal complexity. This static compression may underrepresent informative regions in complex scenes and over-represent irrelevant parts in simple ones.\n3. The token selection relies on hand-crafted heuristics (e.g., activation magnitude, channel max), which may not always correlate with semantic importance—especially in domain-shifted or adversarial settings.\n\nQUESTIONS:\nHave the authors tried OTPF in such settings, or do they envision ways to extend it to support dense output tasks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper proposes a token compression framework termed XComp for long video understanding, addressing the challenge of token inefficiency in video-language models. The key idea lies in its hierarchical compression strategy: LP-Comp leverages LLM layers to learn progressive token compression (reducing frames to single tokens), improving density while avoiding heuristic failures of prior work; QC-Comp introduces query-conditioned frame selection via attention scores, mitigating positional bias through video segmentation. The combined approach enables extreme compression of 1 token per frame while preserving query-relevant information. Experiments demonstrate significant gains over VideoChat-Flash on long-video benchmarks.

STRENGTHS AND WEAKNESSES:
- S1: The idea of letting LLM layers learn compression (LP-Comp) is well-motivated, moving beyond handcrafted heuristics.
- S2: QC-Comp’s attention-based segmentation elegantly handles position bias—a known limitation in long-context LLMs.
- S3: The paper is with promising empirical validation. Multi-benchmark improvements suggest generalizability.

- W1: While the proposed method could achieve comparable or better performance on several benchmarks, the efficiency comparison with baseline VideoChatFlash. I would like the authors to give more discussion and quantative results on time efficiency and FLOPs.
- W2: Compression works well on general videoQA. However, the performance on fine-grained video tasks like video OCR (like MME-VideoOCR) and reasoning benchmarks (like CLEVRER) should be considered to verifiy its generalization ability.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03