Title: InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: KV-cache

VLM

Long context

Streaming Video Understanding

Memory-Efficient Inference
Abstract: Modern multimodal large language models (MLLMs) can reason over hour-long video, yet their key–value (KV) cache grows linearly with time—quickly exceeding the fixed memory of phones, AR glasses, and edge robots. Prior compression schemes either assume the whole video and user query are available offline or must first build the full cache, so memory still scales with stream length. InfiniPot-V is the first training-free, query-agnostic framework that enforces a hard, length-independent memory cap for \textit{streaming} video understanding. During video encoding it monitors the cache and, once a user-set threshold is reached, runs a lightweight compression pass that (i) removes temporally redundant tokens via Temporal-axis Redundancy (TaR) metric and (ii) keeps semantically significant tokens via Value-Norm (VaN) ranking. Across four open-source MLLMs and four long-video and streaming-video benchmarks, InfiniPot-V cuts peak GPU memory by up to 94\%, sustains real-time generation, and matches or surpasses full-cache accuracy—even in multi-turn dialogues. By dissolving the KV cache bottleneck without retraining or query knowledge, InfiniPot-V closes the gap for on-device streaming video assistants.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces InfiniPot-V, a novel framework designed to address memory constraints in streaming video understanding (SVU) by compressing the key-value (KV) cache in a training-free, query-agnostic manner. Unlike traditional offline video understanding methods that assume access to the entire video, InfiniPot-V processes arbitrarily long video streams in real-time, maintaining a fixed memory footprint. The framework employs two complementary metrics: Temporal-axis Redundancy (TaR), which prunes redundant tokens based on cosine similarity across time, and Value-Norm Importance (VaN), which prioritizes tokens based on the L2 norm of their Value vectors for semantic salience. InfiniPot-V achieves significant memory reduction (down to 6K tokens for 6K token contexts) while maintaining accuracy comparable to or better than existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper presents a robust evaluation across diverse benchmarks (VideoMME, EgoSchema, MLVU, LongVideoBench, RVS Ego, RVS Movie), covering both offline and streaming tasks.\n- InfiniPot-V addresses a critical bottleneck in deploying SVU on memory-constrained devices, a pressing issue as video-based multimodal assistants become more prevalent.\n- The introduction of TaR and VaN as complementary compression metrics is novel, particularly in the context of streaming video. \n\nWeaknesses:\n- More comparisons with previous works on reducing the number of vision tokens in LMMs should be included to enhance the completeness of the experiments.\n- While the framework’s focus on on-device deployment is significant, the paper does not discuss compatibility with specific hardware (e.g., mobile GPUs or TPUs) or provide real-world deployment examples.\n\nQUESTIONS:\n- How does InfiniPot-V perform in edge cases, such as videos with minimal temporal redundancy (e.g., highly dynamic scenes) or low semantic variance (e.g., static backgrounds)? Are there scenarios where TaR or VaN might fail to preserve critical information?\n-  The paper mentions a “layer-adaptive pooling strategy” in the VaN metric but provides limited details on its implementation. How is the pooling intensity determined for each layer, and what factors (e.g., model architecture, token distribution) influence this adaptation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, InfiniPot-V algorithm, which is the training-free KV cache compression method, is proposed for streaming video understanding. The proposed algorithm employs two techniques, TaR and VaN, to keep the KV cache within a certain memory limit. Specifically, TaR measures the temporal redundancy score of each patch in past frames and remove the redundant patches. VaN is the metric for token-level semantic importance, which is actually L2norm of value embedding. The proposed algorithm keeps the tokens with high VaN scores (after pooling) in the memory. The experimental results including extensive analysis show that the proposed algorithm achieves comparable or better performances than conventional algorithms.\n\nSTRENGTHS AND WEAKNESSES:\n> **Strength**\n* This paper is well-written and easy to follow.\n* This paper demonstrates the validity of the design of the proposed algorithm through various empirical analyses. Furthermore, by applying the proposed method to different MLLMs and evaluating their performances on multiple benchmark datasets, it shows the effectiveness of the proposed InfiniPot-V.\n* The proposed algorithm is a practical solution that can be easily applied to existing MLLMs to enable memory-efficient inference. It achieves a better computation-performance trade-off compared to existing approaches. Moreover, the paper provides a detailed explanation of the method to ensure reproducibility.\n\n> **Weakness**\n* While the proposed method seems simple and effective, the proposed scoring methods are not particularly novel.\n* The mathematical notation could be improved for better clarity.\n\nQUESTIONS:\nThe proposed algorithm may have the potential to support memory-efficient inference across various MLLMs, depending on their intended use and deployment environment. However, it would be even more compelling if the following aspects can be further improved. Of course, I would also be interested in hearing the authors' thoughts on my questions.\n\n---\n\n* L159, the definition of $H$ seems to be missing, although its meaning can be inferred from the context.\n\n* Figure 2 (b) compares the static patch similarities of keys and values. It is useful, but it would also be beneficial to compare the key similarity of non-static patches. This could demonstrate that static and non-static patches can be effectively distinguished based on $s^{TaR}$.\n\n* Table 4 shows the ablation study results. There seems to be many alternative designs worth comparing in addition to VaN Reverse. For example, instead of using TopK selection, it is possible to select tokens at fixed intervals after sorting them by scores. It may also be possible to identify important frames by computing the average VaN score for each frame, and then select a different number of tokens from each frame based on its importance. While Top-K selection is simple and effective, it would be interesting to see additional comparative experiments with other alternative methods.\n\n* It would be helpful to visualize the selection results based on the actual scores. This would provide an intuitive understanding of how effectively the proposed method identifies important tokens.\n\n\n* Typo (minor)\n1. There is a mixed use of $K_{recent}, V_{recent}$  and $K_\\textrm{recent}, V_\\textrm{recent}$ (e.g. Algorithm 1 and Eq.4)\n2. Similarly, $s^{VaN}$ and $s^{TaR}$ are also used with inconsistent notation. (e.g. Algorithm 1, Eq. 4, and L181)\n3. Figure 2 (c) VaN (spatial) -> VaN (Spatial), for consistency in capitalization.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces InfiniPot-V, a novel framework designed to address the critical problem of unbounded key-value (KV) cache growth in Multimodal Large Language Models (MLLMs) during streaming video understanding. The authors propose the first training-free, query-agnostic compression framework that enforces a hard, length-independent memory budget, making it suitable for resource-constrained edge devices. The core method, termed Continual KV cache Compression (CKV), monitors the cache size during video stream processing and, upon reaching a user-defined threshold, performs an efficient in-place compression. This compression is guided by two lightweight, complementary metrics: Temporal-axis Redundancy (TaR) to prune temporally redundant tokens and Value-Norm Importance (VaN) to preserve semantically salient tokens. The authors demonstrate that InfiniPot-V can be applied as a plug-and-play module to various MLLMs, significantly reducing peak memory usage while maintaining or even surpassing the accuracy of full-cache baselines across multiple long-video and streaming benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* **Significance and Originality:** The paper tackles a crucial and practical bottleneck for deploying MLLMs on edge devices for real-world applications like real-time video assistants. It introduces what appears to be the first framework to enforce a truly fixed memory budget for streaming video understanding in a query-agnostic and training-free manner. The proposed TaR and VaN metrics provide an elegant and lightweight solution that contrasts sharply with existing methods that are often query-dependent, require full cache materialization, or necessitate costly retraining.\n* **Quality and Performance:** The work is technically strong and validated with impressive empirical results. InfiniPot-V demonstrates remarkable efficiency, achieving up to 94% peak GPU memory reduction with only a 0.5% latency overhead. Across four open-source MLLMs and six benchmarks, it consistently matches or exceeds the accuracy of full-cache baselines, showcasing that significant compression can be achieved with minimal performance degradation. The thorough ablation studies effectively validate the synergy between the TaR and VaN components.\n* **Practicality and Impact:** The plug-and-play and training-free nature of InfiniPot-V makes it highly practical and likely to have a significant impact on the field. By eliminating the need for retraining and CPU offloading, it provides a viable path toward deploying powerful video understanding models on consumer devices like smartphones and AR glasses, bridging a critical gap between large-scale research models and real-world applications.\n\n**Weaknesses:**\n\n* **Quality/Robustness of Compression Heuristics:** The proposed compression criteria have potential weaknesses in certain scenarios. The TaR metric, which relies on identifying temporal redundancy, may be less effective for videos with constant, rapid scene changes or frequent edits where static content is minimal. Similarly, the VaN metric, which uses the $\\ell_2$ norm of Value vectors as a proxy for semantic importance, could potentially discard low-norm but contextually critical visual cues, such as small text overlays or subtle but important objects. The robustness of these heuristics across more diverse and challenging video styles is not fully explored.\n* **Scope of Evaluation and Generalization:** The evaluation, while extensive, has some gaps. The benchmarks used are primarily in English, leaving the framework's performance on multilingual content unevaluated. Furthermore, the experiments are conducted under controlled conditions (e.g., fixed frame rates), and the system's robustness in more challenging, real-world scenarios like low-light conditions, occlusions, or dynamic frame rates remains unvalidated. The experiments are also conducted on high-end A100 GPUs, with no deployment validation on the target edge devices (e.g., Jetson Orin), leaving questions about real-world latency and power consumption unanswered.\n* **Limited Multimodal Scope:** The current framework focuses exclusively on compressing the visual modality's KV cache. In many real-world videos, audio and transcribed speech are critical parallel streams of information. The paper does not address how to integrate or co-compress audio or text-based KV caches, which would be essential for a truly holistic understanding of multimodal streams.\n\nQUESTIONS:\n1.  **Robustness Under Challenging Conditions:** How does InfiniPot-V perform on videos that challenge its core assumptions? For example, how does TaR handle high-motion sports videos with minimal static background, and can VaN reliably preserve small but crucial textual information on screen? Have you considered adaptive thresholds for TaR and VaN to mitigate potential failures in such adverse conditions?\n2.  **Hardware-Specific Performance:** The paper demonstrates impressive efficiency on A100 GPUs. To fully validate its practicality for edge computing, could you provide latency and power consumption metrics on representative ARM-based devices like a smartphone SoC or NVIDIA Jetson?\n3.  **Extending to Multimodal Streams:** A key area for future work is multimodal compression. How do you envision extending InfiniPot-V to jointly handle visual, audio, and synchronized text (e.g., subtitles) streams? Would this require new cross-modal redundancy metrics to decide which modality's tokens to preserve, or could each stream be compressed independently?\n4.  **Long-Term Semantic Retention:** For ultra-long videos, such as hour-long surveillance footage, repeatedly compressing the KV cache might lead to the gradual decay of important semantic information from early in the stream (\"catastrophic forgetting\" in the cache). Have you analyzed this effect? Would a hierarchical caching mechanism (e.g., storing summaries of older, heavily compressed blocks) be a potential solution to ensure long-term memory retention?\n\nLIMITATIONS:\nSuggestions are listed in the questions and weaknesses parts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInfiniPot-V introduces a continual KV-cache compression framework that keeps GPU memory usage constant regardless of video length. When the cache reaches a user-set budget, the system compresses it by (i) pruning temporally redundant patches via Temporal-axis Redundancy (TaR) and (ii) preserving semantically salient regions via Value-Norm (VaN). The method is training-free and query-agnostic; across four open-source MLLMs and six benchmarks, it slashes peak memory by up to 94 % while sustaining real-time 14 fps generation and matching or surpassing full-cache accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Straightforward, training-free cache compression that can be plugged into existing MLLMs without retraining.\n 2. Consistent memory reduction (≈ 94 %) across four backbone models with only marginal speed overhead, proving the engineering practicality.\n 3. Paper is clearly written; algorithmic steps and ablation studies are easy to follow, which aids reproducibility.\n\nWeaknesses\n1. Claimed novelty is thin. The Temporal-axis Redundancy module repeats the “drop static frames” idea already seen in recent work such as LongVU; VaN likewise relies on a well-known “high-entropy tokens carry more information” intuition, and the layer-wise pooling tweak is incremental.\n2. Comparative study on streaming video understanding is shallow: only ReKV is benchmarked, limited to a single dataset (RVS), and the method does not reach the current SOTA there. Broader baselines and benchmarks in SVU are missing.\n3. Method improves Qwen2-VL on EgoSchema yet hurts Qwen-2.5-VL noticeably, raising doubts about robustness; the paper offers no analysis for this drop.\n4. The ablation study is not strong enough. More simple strategies should be compared. For example, add an ablation that replaces TaR with LongVU’s key-similarity filter and replaces VaN with a simple top-k entropy selection. Show accuracy and memory for each variant.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}