InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding
Authors: Minsoo Kim
Kyuhong Shim
Jungwook Choi
Simyung Chang
Venue: NeurIPS 2025 poster ()
Keywords: KV-cache
VLM
Long context
Streaming Video Understanding
Memory-Efficient Inference
Primary area: deep_learning
TL;DR: InfiniPot-V presents KV-cache control framework for streaming input video processing with fixed memory usage

Abstract: Modern multimodal large language models (MLLMs) can reason over hour-long video, yet their key–value (KV) cache grows linearly with time—quickly exceeding the fixed memory of phones, AR glasses, and edge robots. Prior compression schemes either assume the whole video and user query are available offline or must first build the full cache, so memory still scales with stream length. InfiniPot-V is the first training-free, query-agnostic framework that enforces a hard, length-independent memory cap for \textit{streaming} video understanding. During video encoding it monitors the cache and, once a user-set threshold is reached, runs a lightweight compression pass that (i) removes temporally redundant tokens via Temporal-axis Redundancy (TaR) metric and (ii) keeps semantically significant tokens via Value-Norm (VaN) ranking. Across four open-source MLLMs and four long-video and streaming-video benchmarks, InfiniPot-V cuts peak GPU memory by up to 94\%, sustains real-time generation, and matches or surpasses full-cache accuracy—even in multi-turn dialogues. By dissolving the KV cache bottleneck without retraining or query knowledge, InfiniPot-V closes the gap for on-device streaming video assistants.

InfiniPot-V. V: Memory-Constrained KV Cache Compression for Streaming Video Understanding
Minsoo Kim It Kyuhong Shim² Jungwook Choilt Simyung Chang
Hanyang University "Sungkyunkwan University
Qualcomm AI Research, Qualcomm Korea YH
(minso02333, (hoii)ehanyyan ac.kr khshim@skku. edu simychan@qti. qualcomm.com
Abstract
Modern multimodal large language models (MLLMs) can reason over hour-long video, yet their key-value (KV) cache grows linearly with time-quickly exceed- ing the fixed memory of phones, AR glasses, and edge robots. Prior compression schemes either assume the whole video and user query are available offline or must first build the full cache, memory still scales with stream length. InfiniPot-V is the first training- ffree, query-agnostic framework that enforces a hard, length- independent memory cap for streaming video understanding. During video coding it monitors the cache and, once a user-set threshold is reached, runs a lightweight compression pass that (i) removes temporally redundant tokens via Temporal-axis Redundancy (TaR) metric and (ii) keeps semantically significant tokens via Value-Norm (VaN) ranking. Across four open-source MILMS and four long-video and streaming-video benchmarks, InfiniPot-V cuts peak GPU memory by up 94%, sustains real-time generation, and matches surpasses full-cache accuracy-even in multi-turn dialogues. By dissolving the KV cache bottleneck without retraining or query knowledge, InfiniPot-V closes the gap for on-device streaming video assistants.
Introduction
Recent advances in multimodal large language models (MLLMs) have dramatically expanded the scope of visual reasoning. Vision lssion-language instruction tuming now allows single backbone answer open-ended questions over long video sequences (27 30 48). while context-extensioe techniques such FlashAttention-2 and RingAttention push the effective window into the million-token regime These breakthroughs underpin a new generation of streaming video assistants and humanoid robots that promise continuous, real-time scene understanding on mobile phones, AR
7 26 33
glasses and edge robots [14] 31 42 37).
Streaming video understanding (SVU) diverges from conventional offline video understanding (OVU). Offline models see the entire clip and user query before inference, so they can tailor every compression retrieval step. In streaming, frames arrive incrementally and future queries unknown, forcing all pre-query processing to be anery-agnottic. In addition, device memory is fixed, yet the transformer emits hundreds of tokens per frame, the key-value (KV) cache grows linearly. For example, 1 15-min, 10 fps clip processed by LLLVA--ext- LlaVA-NextVVide--77 already needs demands ~ 100 GB of KV
storage, far beyond the tens of gigabytes available mobile robotic platforms 52 19
Work done during an internship at Qualcomm Technologies, Inc 'Corresponding authors. 'Qualcomm Al Research, an initiative of Qualcomm Technologies, Inc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prior work tackles long-video memory constraints at three stages (Fig D. Frame Sampling [10] drops frames before encoding, reducing m but severely degrading temporal coverage and accuracy. Input-Viss Vistiin Compression (/VC) 36 40 prunes redundant vision tokens after encoding, lowering Prefill load but still requiring the full vision token set to be stored in memory. KV cache Compression (KVC) [24 12 selects query-relevant tokens after the Prefill step, offering the highest accuracy but only after materializing the full KV cache. The challenge intensifies in streaming scenarios: memory usage for Frame Sampling, IVC, and KVC grows almost linearly with video length, eventually exceeding device limits. KV cache offloading (e.g., ReKV (351 ) expands memory space yet incurs costly data transfer, repeated for each query. Thus, existing approach delivers the key property SVU needs: a Lengthhindeepee inngennindeeendent and query-agnostic streaming video compression.
A natural approach to address memory constraints in streaming video is to exploit the strong spa- tiotemporal redundancy of video streams. We introduce InfiniPot-V, the first framework specifically designed for memory-constrained SVU. InfiniPot-V is fraining-free, query-agrostic, and operates continuously during inference. When the KV cache reaches a user-defined memory threshold M, it performs an in-place compression that frees space for new frames while preserving the semantic essence of prior context. This compression is guided by two lightweight and complementary metrics. Temporal-axts Redundancy (TaR) models Key embeddings as a 3D tensor over time and removes tokens with high cosine similarity to recent frames, effectively pruning static or repetitive content. Value-Norm Importance ( (VaN) ranks the remaining tokens by the l norm of their Value vectors strong, model-agnostic proxy for semantic salience and applies a layer-adaptive pooling strategy. This compression is highly efficient, adding negligible lateney while strictly enforcing memory limits.
Extensive evaluation confirms the effectiveness of this design. Across four open-source MILLMs (3B and 7B) and six long-video benchmarks covering both offline (VideoMME, EgoSchema, MLVU, Long VenhoBench) and streaming (RVv-Ego/Movie, OVO-Bench, StreamingBench) tasks--nfimiPott V reduces input context length usage to as low as 6K for 50K-token contexts, with accuracy matching or exceeding full-cache baselines. It maintains real-time performance at 14 frames per second with only 0.5% compression overhead. Additionally, its query-agnostic nature offers clear benefits in multi- turn dialogue settings (Appendix.) C.. By eliminating the KV cache bottleneck without retraining or query dependency, InfiniPot-V paves the way for practical, on-device multimodal assistants.
2 Background
We aim to deploy streaming video understanding (SVU) applications 49 35 in memory-constrained environments. Unlike offline video understanding (OVU) (54] II 45 which assumes access to the entire video, SVU must process arbitrarily long video streams and answer questions at any time step using only the frames observed up to that point. Given a video stream Vr := [U1, ..., er] with T' frames and set of questions = 91, 92. qN. SVU answers each question q, at time (1 < < T') using only the observed frames V42 == [V1, U2 v.).
As SVU deals with unbounded video streams, memory-efficiem processing is essential. In this section, we describe how multimodal large language models (MLLMs) handle long videos, review prior approaches to memory reduction in OVU, and analyze their limitations when applied to SVU.
(See Appendix. F for a detailed discussion of related work.)
2.1 Preliminary: Offline Long Video Understanding
Video Processing in MLLMs. Multimodal Large Language Models (MLLMs) 122 48 43 process
offline videos through a structured pipeline (Fig. [a)). Given a video Vr := [U1, U2 ....... UT] of
uniformly sampled frames, a vision encoder fvar transforms each frame into visual tokens:
(I) where - x T denotes the total number of sampled tokens, where is the number of tokens per frame (determined by input resolution and ViT patch size), and D is the token embedding dimension.
X = fvr((Vr= = [21, ,i. NN] E RND
The token sequence X is then passed to the LLM in two phases: Prefill and Decoding. During the Prefill phase (Fig. aa), step 2), all tokens are processed at once construct the initial KV cache. The attention operation computes:
QKT VD
Q = X
K XWW,
V XW.,
atto Softmax
+ M
+M| AM), V.
(2)

---

Full
50.0%
Offline Video
Offline Video
Offline Video
06
1. Vision Encoding
Long Recuracc
Vision Tokens
Vision Takens
64
Vision Tokens
2. Prefill (LLM)
Cache
62
Sampling
25.0%
KV Cache
60
KVC
12.5%
3. Decoding (LLM)
(Ours)
6.25%
Offtine
Input
(e)
Memory Compression (GB)
Understanding
Compression (IVC)
Compression (KVC)
(e) OVU Accuracy/ Memory with Video Compression
Pooc Meerery Compuccee (at1)
Stream Video
KV Cache-1
KV Cache-2
Constrained
KV Cache-3
Samphng
Memory
100
200
300
400
(d) Continual KV Cache Compression (CKV)
Number of frou Viduo Sinwar
Streaming Understanding
(f) SVU Memory Consumption
Figure 1: MLLMs Video Understanding and Compression. (a) OVU pipeline; (b) IVC: compresses
vision tokens after encoding: (c) KVC: compresses KV cache after prefill; (d) CKV: iteratively processes and compresses KV caches to constrain memory usage; (e) Accuracy vs. GPU memory consumption for compression across four token reduction ratios (50%. 25%. 12.5%, 6.25%) on MLVU using Qwen-2-VL-78. LongVU[36 is used for IVC, SnapKV(24) for KVC; (f) GPU memory usage as input video stream length increases. IVC/K VC/KVC/CKV target 6K cache; Sampling uses of input frames. Measured on A.00 80GB single GPU.
where W, WA, W, E DxD are projection matrices and M is causal mask enforcing autoregressive decoding.
In the Decoding phase (Fig. a), step 3), the model generates tokens one at time using cached keys and values from the prefill phase. To avoid redundant computation, the KV cache C (K, V) is updated incrementally:
C++1 (KKA++1)) {V, 0++1)),
where k141, #++1 correspond to the KV embeddings of the newly processed token.
Offline Long- Video Compression Strategies
Long-Video
Long videos produce extremely long token sequences X, leading prohibitive GPU memory and latency during decoding. Prior works tackle this bottleneck in the offline setting through three classes of methods (Fig.
T-c): c):
(1) Frame Sampling [10]. Uniformly sampling a shorter clip Vf. C reduces the input length and, hence, memory usage is also reduced proportional to compression rate.
(2) Input- Vision Compression (IVC) (36. 40). After vision encoding. IVC aggressively prunes redundant vision tokens, keeping only a salient subset X' y'c (Fig. to shrink the context fed into the language decoder for memory- comprssed Prefill.
Conduct compression after prefill: KVC computes importance scores 14 = -E= Attn [z; +x,) over the last w tokens and retain top-M entries for the memory budget M by applying eviction policy म, yielding a compressed cache = TC) for memory memory-comprest compressed Decoding. (Fig. (T)) Note that the и eviction policy is highly dependent the content of the last w tokens, reflecting the user query, and is thus referred to as query-dependent cache compression method (see Appendix. for further analysis).
(3) KV cache Compression (KVC) 24
These techniques are effective when the entire video available upfront, they implicitly assume (i) unconstrained memory for compression and (ii) known easily approximated query.

---

2.3 Challenges in Streaming Video Understanding
Fig. Te) compares three offline compression methods on a fixed 50 K-token video at four compression ratios (darker shades indicating higher ratios: 50%, 25% 12.5%, 6.25%), revealing fundamental trade-off between memory usage and accuracy. Frame sampling skips frames to save memory, but severely degrades recognition accuracy. Increasing the sample ratio improves accuracy but quickly inflates memory usage. IVC starts with a large memory footprint for all vision tokens before selecting which to discard. KVC, which operates on more expressive key-value features, achieves the highest accuracy but requires the largest Prefill cache. Notably, even under a favorable offline setting-with full video access and offline query-none of the methods achieve both high accuracy and low memory usage.
This trade-off becomes more severe in the streaming video understanding (SVU) setting. As shown in Fig ft), peak GPU memory usage increases with stream length. KVC exhibits near-linear memory growth, as it must materialize all vision tokens and build the full KV cache before compression. Furthermore, due to query-dependent nature, KVC must re-execute the memory-intensive prefill stage whenever the user query changes. Frame sampling and IVC also grow linearly, albeit more slowly, eventually exceeding the memory capacity of practical edge devices (e.g., 32GB [[9] as the stream continues. ReKV [35], a recent KVC method, addresses this by offloading the KV cache to CPU memory, but this introduces substantial offloading overhead and compression latency.
These findings highlight two core requirements for SVU: (1) fixed memory budget that does not grow with stream length, and (2) query-agnostic token retention strategies. Existing methods fail to meet at least one of these, limiting their suitability for SVU. To overcome this, we propose Continual KV cache compression (CKV), illustrated Fig. [d). CKV processes frames in small blocks and compresses the cache whenever the fixed memory limit is reached, ensuring constant memory usage throughout streaming. Additionally, for query-agnostic token retention, our approach employs lightweight spatiotemporal metrics to identify and preserve semantically significant tokens without relying on future queries. As a result, despite operating under strict memory constraints, CKV achieves accuracy on par with or better than KVC (Fig e)), while consuming far less memory than IVC or frame sampling (Fig. TT). The algorithmic details are described in Sec
Algorithm 1 Continual KV cache Compression (CKV) with InfiniPot-V
Require: Memory budget [M], target cache size [C]. TaR ratio a Initialize K, < 9
b Empty KV cache
while video stream continues do (1) Process: Know; Vien f- Process new frame; K if len () 2 [Md] then
[K: Knew), V +- [V; Vaen
b Append new tokens
Memory budget exceeded
(2) Extract: Kreeent, Viewen f- recent v frames from K, V
b Iee(K) = len (V)
M
(3) TaR: s Tal <- ComputeTaRRcoreee
K); InR <- TopK Talk ca len Keson ))
B Sec. 3.1
(4) VaN: 8 VaN +- Compute VadaptiiveVNSSores((((
TvaN + TopK VaN
(1 - a) CC)
b Soc. 3.2
(5) Combine: I + Ina U Iva U Indices( Indicos(Knecm)) ); K + KLT, V + V Compress to |C]
size
end if
user query arrives then Generate response using current K, V
end if
end while
3 InfiniPot-ye V: Memory-Constrained Streaming Video Understanding
We present InfiniPot-V, V, a CKV framework designed for memory-constrained SVU. As shown in Fig. [d) and Algorithm [ InfiniPot-V processes video streams by applying continual KV cache compression within a fixed memory budget. In this framework, KV embeddings from incoming frames are stored until the memory limit [M] is reached. At that point, compression reduces the cache to smaller target size |C] ([M)
> [C'), retaining only the most essential vision tokens based on two criteria. The freed space (MM - CD accommodates new frames. This process repeats continuously, enabling efficient stream processing under strict memory constraints. When user query is issued, the model answers using the compressed cache that summarizes visual context from all prior frames. Notably, compression adds only 0.5% overhead relative to input frames processing time.

---

frames
Pest Frame
Frame
Recent
Constrained KV Cache (MM)
Vision Token
3D Reshaped
KV Cache
patches
Compression
Statio Petch (Exict)
Moving Patch (Retain)
Time fraimes)
(a) Temporal-axis Redundancy Reduction (TaR)
1.0
Key Value
TeR (temporal) selected
Similaity
0.5
Holgr
VaN (spatial)
selected
Recent (r]
0.0
15
Layer Index
(b) Static Patch Similarity Comparison
(c) InfiniPot-V (Ours): Temporal-Spatial Scoring
Figure 2: Sratio-Temporal KV cache Compression (TaR and VaN), (a) Temporal redundancy across adjacent frames, showing static patches that can be evicted from past frames; (b) Layer- wise cosine similarity of Key/Value embeddings for static patches between consecutive frames in -VVi---ee---ido------ (c) InfiniPot-V performs query-agnostic spatiotemporal compression, reducing temporal redundancy with TaR and selecting tokens via VaN spatial scoring.
InfiniPot-V V leverages two token eviction criteria: Temporal-axis Redundancy (TaR) and Value Norm (VaN) for identifying erucial tokens for compressing KV cache. In the following subsections, detail each criterion, and finally describe how to effectively combine them.
3.1 Temporal-axis Redundancy (TaR) Reduction via Patch-wise Similarity
Video streams exhibit inherent spatiotemporal redundancy across frames 36 40
In this section,
we focus on exploiting temporal redundancy, as illustrated in Fig. 2 a) where static patche: background) persist across frames. For MILMS processing videos with fixed memory usage, identi- rying this redundancy in KV caches is crucial. Our analysis in Fig. 2 b) reveals that Key embeddings effectively capture temporal redundancy, exhibiting higher cosine similarity for static patches between adjacent frames compared to Value embeddings, across all layers.
Building on this insight, we propose TaR, a technique that performs a patch-wise comparison of Key embeddings along the temporal axis to detect and reduce redundant tokens. As shown Fig. Cc). we introduce a 3D reshaping of Key embeddings to enable direct comparison of corresponding patches across frames. Based on this structured KV cache, the TaR implementation starts with a memory constraint of MM tokens, processing f consecutive video frames, each containing p |M vision tokens. maintain temporal continuity, we designate the r latest frames as recent frames and retain them in full. The older past frames (f frames) are selectively compressed based on their patch-wise similarity to recent frames.
To measure the patch-wise similarity between frames, we divide the current Key embeddings K E H x(fxp)) x into Krocem E xrxpx and Knus E #x representing the recent and past frames respectively. For each spatial coordinate (i,j), compute the fz-normalized cosine similarity between recent and past frames of the same patch coordinate:
s TaR (t, 1.j)
Σ cos (((bah.)) k((('
(4)
Here, 3 (t, i, is the importance score of the patch in 1-th frame at (i. coordinate. The negative sign is applied that higher computed score indicates lower redundancy (i.e., the token is more distinctive). This ensures that tokens with less temporal similarity to recent frames are prioritized.
In MLLMs, each vision patch corresponds single token, these terms interchangeably.
5

---

We then select the least redundant tokens (i.e., higher score) in past frames using the Top-K operator:
TAR TopK |C| - reemm)) ))
(5)
where |C] is the target cache compression size and recent rp accounts for the recent frame tokens that are always retained. The compressed key-value pairs are formed by concatenating the selected key frame tokens with all recent frame tokens:
TaR = Concat [[[ ITaR::] (eceet)
V = Concat (oocaat([e [V|:, V Viocent.
By fully preserving the most recent frames, we maintain complete information on rapidly changing or newly introduced content, while selectively retaining distinctive visual elements from the past.
3.2 Spatial Semantic Importance Preserving with Value Norm (VaN)
While TaR focuses on reducing temporal redun- dancy, VaN serves a complementary role: iden- tifying and preserving semantically salient re- gions within each video frame, independent of
15-
Entropy
the query. To achieve this, we employ Value embeddings (V), which inherently capture se- mantic information transformer attention [4]].
High-vaN Low-VaN
Specifically, we introduce Value Norm (VaN) as a metric for token-level semantic importance:
10
15
20
25
Layer Index
(a) Entropy Analysis of Vision Tokens
VaN (1..)))
Analysis of Value Norm. We hypothesize that tokens with higher VaN contain richer seman- tic information, making them more valuable for video understanding. To quantify seman- tic importance, we project vision token rep- resentations from each layer into the vocabu- lary space [29] and compute the entropy of the resulting word probability distribution, where the higher entropy implies greater informative- ness 3. As shown Fig. 3 (a), tokens with higher VaN consistently exhibit higher entropy, confirming their semantic significance. This advantage translates to improved performance: F (b) shows that retaining high-Val VaN tokens achieves substantially higher video understand- ing accuracy across various compression ratios compared to low- WaN (VaN Reverse) tokens.
Accuracy (%)
Localty (4)
5.0
7.5
CV
Denter-Dist.
0.2
55
VaN VaN Reverse
2.5
0.1
0.0
1/2
1/4
1/8
10
20
Compression
Layer
(b) VideoMME Accuracy
Spatial Locality Analysis
Figure 3: Value Norm (VaN) Analysis.
(a)
Entropy analysis of vision token representations grouped by their VaN scores. (b) VideoMME, per- formance under varying cache compression ratios using either VaN or reverse-Var VaN for token selec- tion. (c) Layer-wise locality of VaN, measured by center distance and coefficient of variation (CV); lower values indicate stronger spatial consistency. LLaVA-Next-74 with Video-MMr used.
Layer-wise Adaptive Pooling. An analysis of VaN distributions reveals strong spatial locality patterns in early to middle layers, which gradually diminish in deeper layers as shown in Fig, Bic). To measure spatial locality patterns across layers, we employ two methods: (1) compute the average distance between the center point and surrounding points within a 3 x 3 window spanning the VaN values of each frame (center-dist)) and (2) measure the Coefficient of Variance (CV) to quantify dispersion of VaN distributions. Lower values in both that VaN scores are closely clustered, implying high spatial locality, whereas higher values reflect greater dispersion and lower locality.
center-dist. and CV-innicate
As shown in Fig. (c), both metrics consistently indicate strong locality in early to middle layers, while gradually diminishing in deeper layers. Based on this observation, we design an adaptive spatial pooling mechanism that adjusts the average pooling kernel size per layer. To implement this, we design mapping function g that assigns kernel sizes in inverse relation to each layer's CV:
PoolSize (CV) = ,(VV) where 9: R+ 7,7,577
This approach assigns larger pooling kernels (e.g., 7) to lower layers with smaller CV values (higher spatial locality), and smaller kernels (e.g., 1, implying no pooling) to upper layers with larger CV values, thus preserving fine-graine fine-grained details where needed.

---

Method Max Duration
Size
Frames
Budget M
EgoSchema min
MLVU 120 min
VideoMME
LVB
60 min
60 min
GPT4-V*
lfps 1fps
55.6
60.7
GPT4-o
72.2
66.2
77.2
66.7
LLaVA-OVe
60.1
64.7
VU
7B
1fps
67.6
65.4
60.6
VU
3B
1fps
8K
59.1
55,9
51.5
Qwen-2-VL
7B
768
50K
65.2
65.8
63.9
58.8
Qwen-2-VL+ Ours
7B
768
6K
65.6
65.8
62.8
58.4
LLaVA-Next
7B
128
25K
67.6
68.7
62.8
63.5
LLaVA-NexI Ours
128
65.8
65.2
60.9
Qwen-2.5-VL VL
3B
768
50K
64.4
63.3
60.3
59.9
Qwen-2.5-VL 2.5-VL Ours
3B
768
6K
61.8
62.1
59.3
56.5
Table 1: Comparison of various MILMS accuracy four Offlin Video Understanding (OVU)
benchmarks. denotes the numbers from official paper.
Compression
Budget
Video MME
MLVU
Method
M
Short
Med
Long
Holistic
Single
Multi
Avg.
FullKY
50K
74.7
62.1
55.0
76.3
73.9
43.3
TTC
3K
66.8
51.2
47.9
72.1
58.8
33.2
54.8
(IVC)
6K
72.6
55.0
51.7
76.3
60.9
36.7
58.4
67.9
51.0
49.3
71.5
58.6
(IVC)
6K
72.6
56.2
51.6
74.3
61.1
35.9
57.9
InfiniPot-V
3K
73.9
57.8
51.8
77.7
70.4
43.2
63.1
(CKV)
6K
74.1
60.8
53.4
77.2
72.3
44.8
64.3
Table 2: Comparison under memory-constraineg settings (3, 6K memory-budget) with Input Video Compression (IVC) methods: TTC from DyCoke [40] and STC from LongVU [36]. Qwen-2-VL- used across VIdeoMME and MLVU benchmarks. Results comparing memory-unconstrained IVC methods (without KV cache compression) with InfiniPot-V V are provided in Tab, A6
For KV cache compression, we select tokens using VaN scores processed through our adaptive pooling mechanism, retaining the Top-/C| tokens with highest pooled VaN values as deseribed in Fig. c): IvaN = TopK VaN (Cl)
K VaN = K|: IvaNsil, VvaN = V(:, :..
3.3 Design Space Exploration
Combining TaR and VaN for Token Selection. TaR and VaN capture complementary aspects of spatio-temporal redundancy in streaming video. To integrate them, we prioritize
selection by first allocating a/C' tokens to TaR, then filling the remaining (1 - a)) C| with VaN-selected tokens. This two-stage selection strategy effectively balances temporal and feature importance. A detailed hyperparameter exploration, including sweeps over a and the size of the recent frame window r, is provided in Appendix. A.2
Comparison with Nemory-Constraines Alternatives. natural question is whether IVC KVC can be adapted for SVU under memory constraints. To explore this, we apply query-agnostic methods such as spatial token compression (STC) and token temporal merging (TTC) from Lo (36) and DyCoke [40] InfiniPot-V outperforms all these baselines by a notable accuracy margin, demon- strating the strength of continual compression over expressive key-value embeddings (details in Tab. 2)

---

LLaWWNextt-7 VideoMME
LLaVi-Next-7B- MLVU dev
LLaVA-Next-78 LVB dev
60
67
65
62
Recurscy
62
83-
Uniform Select
58
SnapKY
InfiniPot
InfiniPot-V (Ours)
55
54
-47
52
52-
3/261/
1/4
1/2
1/26
1/
1/2
1/101/8
1/4
3/2
Qwcn 2-V--7- VL-7B VidcoMME
Qwan 2 VB MLVU dev
Qwin-2-VL- 2-VL- LVVB dev
5.-
64
58
Accuracy
60-
62
57
Unitorm Select
60
56
InfiniPot
58
55
InfiniPot-V (Ours)
56
3/16
1/8
1/4
1/2
1/16
1.4
1/2
1/161/6
1/4
3/2
Compression Ratio
Cumpression Ratio
Compression Ratio
Figure 4: KV cache Compression (KVC) methods evaluation results with offline long video under- standing tasks under Continual KV Cache Compression (CKV) framework. Performance across four compression ratios (1/16, 1/8, 1/4, 1/2) for LLaVA-Next-78 (top row) and Qwen-2-VL-78 (bottom row) on VideoMME, MLVUdev, and Long VideoBenchav (LVBaev) tasks. The full evaluation results shown Table A5
4 Experiments
4.1 Experimental Setup
Benchmarks. We evaluate our InfiniPot-V on both offline video understanding (OVU) and stream- ing video understanding (SVU) tasks. For OVU, we utilize representative multiple-choice based long video understanding benchmarks (ranging from 3 minutes to over hours); VideoMME (Н].
MLVU (5), Long VideoBench (LVB) [5), and EgoSchema (28).
For SVU, we employ RVS-Ego/Movie streaming video QA benchmark 49 featuring open-ended questions paired with timestamps, and evaluate the answers using 5-tturb- follow-
ing 49 355. We further extend multiple-choiee based SVU evaluation, OVO-Bench [23 and
StreamingBench [25]. OVO-Bench evaluates temporal reasoning under three scenarios-bacee tracing, real-time visual perception, and forward active responding-wiir StreamingBench evaluates real-time visual comprehension of streaming videos.
Models. We apply our method on four state-of-the-ar MILMs capable of long-vides understanding: DL [43]. Qwen-2.5-VL VL [48] LLLVA-OV-7B [22), and LLaVA-Next- Video [52]. Details on input video sampling settings and benchmark details are provided in Appendix,
4.2 Evaluatal Results
Offline Video Understanding. To assess the absolute compression capability of our method, we evaluate InfiniPot-V with both commercial MLLMs (GPT-4V [30], GPT-4o (3I) and state-of- the-art public models designed for offline video understanding, including LLaVA-OV (22), and LongVU [36]. Unlike these specialized, fully trained models, InfiniPot-V V a training-free free, plug- in framework compatible with MLLMs of various scales, enabling high performance under fixed memory budgets. As shown Tab. InfiniPot-V reduces memory usage to just 25% (6K tokens) for LLaVA-Next (originally 25K tokens) and 12.5% for Qwen-VI. series (6K vs. 50K), with minimal performance loss. Notably, achieves comparable or better accuracy than Long VU the 7B scale and significantly outperforms it at 3B, demonstrating both efficiency and scalability.
Comparison with IVC under Memory Constraints. To evaluate recent query-agnostic IVC
methods under memory-constrained CKV, we adopt unified setup VideoMME, and MLVU: token temporal merging (TTC) from DyCoke 40) and spatial token compression (STC) from Long VU (36) applied to compress vision tokens to fit the target memory budget [M], while KV cache is

---

RVS-Ego
RVS-Movie
Execution Time
Total Memory Usage
LLaVA-OV-7B
Ace
Score
Acc
Score
Video Enc. (msec/Frame)
GPU
CPU
ReKV
60.1
3.9
53.4
3.8
285.7
37,5 GB
+ 18.8GB/h
ReKV w/o off.
55.8
3.3
50.8
3,4
74.6
27.2 GB
0
InfiniPot-V
57.9
3.5
51.4
3,5
76.3
27.8 GB
0
Table Streaming benchmark comparison to offloading- based KV cache control method. (ReKV) Video Enc. shows execution time per frame, GPU indicates peak memory usage, and CPU denotes the size of video KV-Cache offloaded to CPU per hour. Results based on an I-hour video processed with 0.5 fps sampling rate in streaming mode. LLaVA-OV-78 is used.
Qwen-2.5-VL
Scale
OVO-BW
OVO-Real
OVO-FW
OVO-Avg
StreamingBench
Uniform Select
7B
44.5
61.1
47.8
51.7
75.2
InfiniPot-V
47.6
65.9
47.9
53.6
76.4
Table 4: Comparison on OVO-Bench (BW: Backward, Real: Realtime, FW: Forward) and Streaming- Bench (Real-time visual understanding) using Qwen-2. 2wen-2.5-VL-78 under 4K memory budget.
managed using sliding window attention (SWA) (2 When operated under such constraints, these IVC methods suffer from notable accuracy degradation. In contrast, InfiniPot-V V performs KV cache compression using TaR and VaN, leveraging expressive key-value representations to achieve superior average accuracy under a 6K memory budget-ccorespondiig to an 88% lossless compression rate.
Comparison with KVC under Memory Constraints. Fig evaluates KVC methods within our CKV framework under constrained memory across offline video understanding tasks. Compression ratios (1/16, 1/8, 1/4, 1/2) are defined based on each model's maximum frame capacity (e.g., 128 frames for VALVA-Neet, 768 for Qwen-2-VL), Our InfiniPot-V consistently outperforms all baseline methods (Uniform Seleet, SnapKV, InfiniPot) across all tasks for both LLaVA-Next-22 and Qwen-2- VL-7B, demonstrating superior video understanding performance. Under CKV constraints-where actual query access is not available- qecelll--uuryeeeeeeee qeuery-deeendett methods like SnapKV degrade significantly. In contrast, InfiniPot-V V maintains strong accuracy even at high compression ratios (e.g., 1/16), thanks to its query-agnostic selection via TaR and VaN.
Streaming Video Understanding. We first evaluate InfiniPot-V V on streaming video understanding (SVU) using two Streaming VQA benchmarks, RVS-Ego and RVS-Movie, with LLaVA-OV-7B. As baseline, we compare against ReKV (35), a state-of-the-ar SVU method, under two configurations: (1) CPU-GPU with CPU offloading, which allows spilling KV cache to CPU memory, and (2) CPU-GPU without offloading, simulating shared-memory devices where CPU memory is unavailable pre-occupied [19]. Tab. reports SVU accuracy, compression time, and memory usage. While CPU offloading enables ReKV to retain the full KV cache, it incurs substantial transfer overhead; without offloading. ReKV suffers from severe accuracy degradation due to limited local memory. In contrast, InfiniPot-V operates entirely within GPU memory, eliminating offloading latency and achieving higher accuracy, making it practical solution for memory- constrained systems.
To further evaluate accuracy in streaming scenarios, we extend the analysis to two additional bench- marks: OVO-Bende [23 and StreamingBench (25). as summarized in Tab. Compared to the uniform token selection baseline, InfiniPot-V demonstrates consistent gains across all metrics, no- tably improving recall of past visual information on the OVO-BW (backward) task (44.5 + 47.6) and achieving higher real-time understanding accuracy on the OVO-Real (Realtime) (61.1 - 65.9) and StreamingBench score (75.2 > 76.4), demonstrating the effectiveness of our TaR-VaN compres- sion in eliminating temporal redundancy while preserving spatially salient semantics, showing its capability under challenging SVU settings.
4.3 Ablation Study
Tab. validates our design decisions for TaR and VaN. Reversed strategies (TaR Reverse and VaN Reverse) significantly degrade performance by discarding distinctive or semantically important tokens. Within TaR, aatch-wise similarity proves more effective than frame-level similarity (64.5 vs. 62.9).

---

MLVU
Holistic Reasoning
Single Detail
Multi Detail
Ablation Study
Topic
Anomaly
Plot
Needle
Ego
AO
AC
Avg
Full KV
85.2
67.5
72.7
83,9
65.1
54.1
32.5
65.9
Uniform Select
83.7
66.5
67.9
76.1
58.5
51.0
27.2
61.5
TaR Reverse
79.0
64.5
56.9
65.6
55.1
45.2
21.8
55.5
TaR Frame
82,9
66.0
67.0
78.9
63.6
51.0
31.1
62.9
TaR
85.9
66.5
71.8
78.0
62.2
51.7
35.4
64.5
VaN Reverse
78.3
66.5
56.2
66.8
53,4
46.3
17.5
55.0
VaN
84.4
68.0
68.6
76.6
61.9
52.5
29.1
63.0
VaN + Pool
85.2
68.0
71.4
77.5
63.1
52.1
31.5
64.1
TaR VaN + Pool 86.3
68.0
72.7
63.9
54.1
35.4
65.8
Table 5: Ablation study of TaR, VaN, and their combination, Experiments conducted on MLVU using Qwen-2-VL-78 with 6K memory budget.
Method
Length (sec)
Context Length
VRAM (GB) 4
FPS (frame/sec) f
Throughput (tok/sec) t
Power (J) L
Full InfiniPot-V
100
18K
6.2 6,4 (1.0x)
1.6 1.2 (1.4x)
9.2 (1.5x)
9.2 (1.8x)
Full KV
300
54K
26.6
5.0
2.0 9.2 (4.6x)
4.7 2.4 (2.0x)
InfiniPot-V
10.1 (2.6x)
6.4 (1.3x)
Full KV
500
90K
39.0
4.1
1.2 9.1 (7.3x)
7.5
InfiniPot-V
10.7 (3.6x)
6.3 (1.5x)
3.7 (2.0x)
Full KV
600
108K
OOM
OOM
OOM
OOM 4.3
InfiniPot-V
11.3
6.4
9.2
Table Streaming video processing performance results on NVIDIA Jetson AGX Orin. Streaming 10-minute video samples (FPS 0.2-1) processed with Qwen-2.5-VL-33. measuring memory, FPS (frames processed per second), throughput, and power consumption.
VaN alone surpasses the baseline, and its performance improves further with adaptive pooling (64.1 vs. 63.0). Combining TaR and VaN yields the highest accuracy, significantly outperforming the baseline. Additional integration explorations are discussed in Appendix, A.2
4.4 Edge Device Deployment Results
continual KV compression with TaR and VaN scor- ing-on the NVIDIA Jetson AGX Orin 19 using Qwen-2. 5-VL-3B and 10-minute Streaming- Bench 25 videos (0.2-1 1 FPS). As shown in Tab our method achieves consistent efficiency across all metrics. Peak memory remains nearly constant (9.2-10.7 GB) while Full KV grows linearly (13.8-+39.0 GB). yielding 3.6x reduction at 500 seconds streaming video. Prefill speed, measured FPS (frames processed per second during prefill), stays stable 6.3-6.4 FPS, whereas Full K drops to 4.1 FPS. Generation throughput also increases up to 7.3x (1.2-9.1 tok/sec) under fixed memory budget. Power usage scales proportionally with compute, improving energy efficiency by nearly 2x (7.5+3.77)) J). Notably, CKV enables continuous inference on 600-second streams where Full KV fails with OOM, confirming its practicality for real-time multimodal reasoning on memory-constrained edge devices.
We evaluate our CKV framework-combingg
5 Conclusion
In this paper, we proposed InfiniPot-V, a training-free KV cache control framework for streaming video processing in memory-constrained environments. Built around practical con- straints-uun unavalle- queries and strict memory budgets during compression employs two novel token eviction criteria, TaR and VaN, achieving significant improvements in long video understanding under streaming scenarios.
10

---

References
[I] Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: inlention-guides token dropping for efficient inference of high-resolution vision-language models, 2024.
[2] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long- mon-meetmentt transformer. arXiv:200.005150 2020.
[3] David Chan, Rodolfo Corona, Joonyong Park, Cheol Jun Cho, Yutong Bai, and Trevor Darrell. Analyzing the language of visual tokens, 2025.
[4] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024.
[5] Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Hao- tian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. LongVILA Scaling long-context visual language mod- els for long videos. In The Thirteenth International Conference on Learning Representations, 2025.
[6] Giulio Corallo, Orion Weller, Fabio Petroni, and Paolo Papotti. Beyond rag: Task-aware ky cache compression for comprehensive knowledge reasoning, 2025.
[7] Tri Dao. FlashAttentionn-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICIR), 2024.
[8] Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective
1_2 norm-based strategy for KV cache compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18476-18499, November 2024.
[9] Sebastian Farquhar, Jannik Kossen, Livia Kuhn, al. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625-6300 630, 2024.
[10] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition, 2019.
[11] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint ar51v:2405.21075. 2024.
[12] Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning, In The Thirteenth International Conference on Learning Representations, 2025.
[13] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang. Jiawei Han, and Jianfeng Gao. Model tells you what discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024.
[14] Google DeepMind. Project ASTRA.
project-astra/ 2024.
[15] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Ro- hit Girdhar, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane. Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jáchym Kolář, Satwik Kotur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano,
11

---

Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995-19012, June 2022.
[16] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length Ilm inference. arXiv preprint arXiv:2411.09688 2024.
[17] Qingqiu Huang. Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 709-727. Springer, 2020.
(18) Yuxiang Huang, Binhang Yuan, Xu Han, Chaojun Xiao, and Zhiyuan Liu. Locret: Enhancing eviction in long-context Ilm inference with trained retaining heads on consumer-grade devices, 2025.
[19] Leela S. Karumbunathan. NVIDIA Jetson AGX Orin Series Technical Brief. Technical Report TB 15__10749-001_v11 11.2, NVIDIA Corporation, July 2022. Version 1.2.
[20] Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, and Hyun Oh Song. Kvzip: Query-agnostic kv cache compression with context reconstruction, 2025.
[21] Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. InfiniPot: Infinite context processing on memory-constrained LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16046-16060, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[22] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03336, 2024.
[23] Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge. Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. Ovo-bench: How far is your video-Ilms from real-world online video understanding?, 2025.
[24] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[25] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for milms to achieve streaming video understanding. arXiv preprini arXiv:2411.038 1.03628, 2024.
[26] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise transformers for near-infinite context. In The Twelfth International Conference Learning Representations, 2024.
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 18992-49166.. Curran Associates, Inc., 2023.
[28] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023,
12

---

[29] Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing languaee models. The Thirteenth Inter- national Conference on Learning Representations, 2025.
[30] OpenAl. Gpt-4v(ision) system card, 2023.
[31] OpenAl. Hello gpt-4o. https://openss com/Indexx/ellloggtt 2024.
[32] Junyoung Park, Dalton Jones, Matthew Morse, Raghavv Goel, Mingu Lee, and Chris Lott. Keydiff: Key similarity- based kv cache eviction for long-context Ilm inference in resource- constrained environments, 2025.
[33] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024.
[34] Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. Introducing gemini 2.0: our new ai model for the agentic era. petpss//blogg http////eeccoooooo/oooeeeeeeeeee google-gemt gemini ai pepdatte-deceberr 2024, 2024.
[35] Zhelun Yu Shangzhe Di. Streaming video question answering with in-context video KV-cache retrieval. In The Thirteenth International Conference on Learning Representations, 2025.
[36] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. Longvu: Spatiotemporal adaptive compression for long video-language understanding. 2024.
[37] Morgan Stanley, Humanoids: Investment implications of embodied ai. Tech- nical report, Morgan Stanley, June 2024. Accessed via Future Management Group: https:/www..uturmmaanse Sogoots-ooraa-Sttaney.ooo
Humanodd-
[38] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
[39] Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Danning Ke, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient KV cache compression through retrieval heads. In The Thirteenth International Conference on Learning Representations, 2025.
[40] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models, 2024.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. Guyon, Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[42] Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, Nasif Zaman, Prithul Sarker, Andrew G Lee, and Alireza Tavakkoli. Meta smart glasses large language models and the future for assistive glasses for individuals with vision impairments. Eye, 1836))0036-100 2024.
[43] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint 121111, 2024.
[44] T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the h.264/ave video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7):560- 576, 2003.
13

---

[45] Haoning Wu. Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[46] Mingze Xu, Mingfei Gao, Shiyu Li, Jiasen Lu, Zhe Gan, Zhengfeng Lai, Meng Cao, Kai Kang, Yinfei Yang, and Afshin Dehghan. Slowfast-llava- 1.5: A family of loken-efficien video large language models for long-form video understanding, 2025.
[47] Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Think: Thinner key cache by query-driven pruning. In The Thirteenth International Conference on Learning Representations, 2025.
[48] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[49] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Fltrham Memory-basee real-time understanding for long video streams, 2024.
[50] Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Mingi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster, 2024.
[51] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Sparsevim: Visual token sparsification for efficient vision-language model inference, 2024.
[52] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024.
[53] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[54] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-tas long video understanding. arXiv preprint arXiv:2406. 2024.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: The code and data involve proprietary components that are subject to company policy restrictions, and thus cannot be publicly released. However, detailed implementation settings and reproduction instructions are thoroughly documented in the main text and Appendix A and B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
Beaiii/guiies//odeeehieeee Sws.Suummisoooliiyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
wwwpommssssiiii cessonnoolcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:s///eennpss..coooo///ttttad
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |ppper swi thcode. com/ conasdatsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer our LLM policy htteop////eerrrooocccoooforreees for what should or should not be described.
21

---

Algorithm 2 InfiniPot-V Algorithm
Require: Video stream V, memory constraint |M] target KV cache size [C), recent frame count r, CV
thresholds [TI, 72, 73). (.).T.. TaR ratio E. [0, 1|, frames corresponding to [M] tokens, vision token number
per single frame p [M
Ensure: Compressed KV cache {K1, Vi
1: Let Cuæ a|C| be the TaR selection budget
2: Let CvaN (1 [C][C be the VaN selection budget
Initialize empty cache for each layer E (1....,44 L}
4: while processing video stream V do
5: Accumulate KV embeddings until reaching MM]
6: for each layer 1 do
Temporal-axis Redundancy (TaR)
8:
Reshape K; into n E IR X and Kpnut RRXxs-xXXXDD for each patch (t, i,j) past frames do
,),, - Ei== (os(K(t)).) Kit' (.55) pust." recent.
11:
end for
12:
(St, CraR)
Select least redundant tokens
#/ Value Norm (VaN) with Adaptive Pooling
14:
VaN, <- ||Vil|2
15:
// Compute CV for adaptive pooling
16:
MII mean( (VaN:)
std((aN:)
18:
CV₁ + oi/pi
19:
I/ Determine pooling size using mapping function pool_sine size,ize < g(CVi)
20:
D Using thresholds (71,72, m
7, ifCV <m
5, in S CV < 72
21:
where (CC))
3. if S CV < 73
1. ifCV T3
22:
VaNpoot.1 + Averrageooll2d mool_size, If Combine TaR and VaN by prioritizing TaR selecede tokens
23:
24:
VaNpoot.1 [Ti] <- max(VaNpoot])
> Prioritize TaR tokens
25:
Tt < OVpK(VaNpoot.. CT)
Final token selection
26:
Ki +- K.[:, Tr. Vi < Vil:
b Update layer KV cache with compressed KV cache
27:
end for
28:
end while
A InfiniPot-V Algorithm and Configuration
A.1 Algorithm Description
Algorithm 2 presents the complete process of InfiniPot-Vi V's cache control framework along with its compression formulation. InfiniPot-V processes video streams by continuously pre-filling and compressing the KV cache using two token selection strategies: Temporal-ate Redundancy (TaR) and Value Norm (VaN). For TaR, the algorithm splits video frames into recent frames (the latest frames) and past frames, then computes cosine similarities between corresponding patches to identify and remove redundant visual tokens. (Line 10)
For spatial semantic importance token selection, a layer-wise adaptive pooling mechanism based on VaN is employed. The pooling size is dynamically determined by the Coefficient of Variation (CV) of the VaN, (Line 18) where higher CV indicates a sparser or more distinct feature distribution. Pre- computed model-specific CV thresholds {TI, T2, }3} determine pooling sizes from the set (1,,5,77) selecting larger windows for uniform (low CV) VaN distributions and smaller ones for sparse (high CV) VaN distributions (Line 21).
To integrate both criteria, TaR-selected tokens are prioritized by assigning them the maximum VaN score before the final token selection. Specifically, by setting VaN [Z]] max (aaN poot,). (Line 24) and then applying a TopK selection, the algorithm ensures that temporally distinctive tokens are preserved while allowing VaN to select additional tokens based on semantic importance.
22

---

VideoMME
M
TaR
+ (1
a))
VaN
M
6K
Qwen-2- --V--22VV--BB
0 (VaN)
0.2
0.4
0.6
0.8
1 (TaR)
Short (-3 min)
74.4
74.3
74.1
74.9
74.4
73.8
Medium (3 min)
59.9
59.3
61.3
61.4
61.2
58.2
Long (30 120 min)
51.9
51.0
52.4
53.1
53.4
53.2
Average
62.1
61.6
62.6
63.1
63.0
61.7
0 (VaN)
0.2
0.4
0.6
0.8
1 (TaR)
Short (-3 min)
69.8
69.8
72.0
71.1
71.9
68.8
Medium (3 min)
59.3
59.3
59.2
58.7
57.7
57.3
Long (30 120 min)
52.1
52.1
52.7
52.0
51.4
51.0
Average
60.4
60.4
61.3
60.6
60.3
59.0
Table A1: TaR and VaN Combination Ratio: Sweep over combination ratio in TaR and VaN combination under a 6K memory budget (lM) on VideoMME. a=0 and a=1 correspond to VaN- only and TaR-only, respectively. The best-perg configurations are shown in bold, while the second-bes best results are underlined.
Qwen-2-VL-7B
MLVU
VideoMME
[M| 6K
Holistic
Single
Multi
Avg.
Short
Medium
Long
Avg.
y= f x 0.125
77.6
66.2
43.9
63.1
68.7
57.3
51.0
59.0
r f x 0.25
77.8
67.1
43.5
63,4
68.0
57.2
51.1
58.8
f 0.50
78.6
64.6
39.0
61.3
65.1
56.7
49.7
57.2
IMI/IC| = 0.75
78.2
71.7
44.6
65.8
74.4
59,9
51.9
62.1
M|/[C 0.50
76.8
68.2
42.3
63.3
74.1
60.3
52.9
62.4
M|/IC| = 0.25
73.2
65.9
39.1
60.3
73.6
56.9
50.4
60.3
Table A2: Recent Frame and Compression Ratio Exploration: Top: Sweep over recent frame
numbers determined by multiplying various ratios (0.125, 0.25, 0.5) to (frame number corre- sponding to memory budget [M in TaR. Bottom: Performance under varying compression ratios |M]/[C|| across MLVU and VideoMME with -wen-2-VL-7B. TaR performs best with r S 0.25 and compression ratio 2 0.5. The highest values are shown in bold, with the second-highest underlined.
A.2 Hyper-Parameter Exploration
InfiniPot-V involves three main hyper-parameters: the TaR and VaN budget allocation ratio a, the number of recent frames used in TaR, and the target compression size C applied at each continual KV cache compression step. This section presents comparative experiments exploring each hyper-parameter.
TaR and VaN Budget Ratio (a: We compare the accuracy of offline video understanding (OVU) task across different values a, which determines the budget allocation between TaR and VaN under a fixed memory budget (IM) = 6K), for both Qwen-2-VLL VL-BB and LLaVA-Next-7B models. As shown Tab. AT performance peaks when a between 0.4 and 0.6, outperformig the use of either VaN-only (a = 0) or TaR-only (a = 1). This confirms the effectiveness of our approach, which jointly considers both spatial and temporal dimensions for KV cache compression.
Recent Frames (r) and Compression Ratio (MM//CC Tab. A2 presents exploration experiments for two key hyperparameters: the recent frames number r, which determines the proportion recent frames within the memory budget in TaR, and the compression ratio [M]/[C), which defines what proportion of the compression size |C| to maintain relative to the memory budget [M] in continual KV cache compression (CKV).
For the recent frame number r (Tab. A2 Top), we observe optimal performance on both MIVU and VideoMME benchmarks when < 0.25/. Setting = 0.5ʃ results in an excessive number of frames being designated as the latest frames for temporal redundancy measurement, which limits the effectiveness of redundancy reduction. This limitation is reflected in the decreased performance
23

---

metrics. (61.3 vs 63.4 4 in MLVU and 57.2 vs 59.0 in VideoMME) Note that the sweep experiments are conducted using TaRl settings (a 1).
For the compression ratio (MM)/C /C) we conduct comparative experiments across three ratios (0.75, 0.50, and 0.25). As shown in Tab. A2 Bottom, excessive compression ratio such 0.25 in CKV results in noticeable performance degradation. These findings confirm that a ratio of 0.5 or higher represents an appropriate configuration for CKV.
Based on these explorations, we standardize the hyperparameter values at a = 0.5, r = 0.125, and |M|/|C|| 0.75 for all main experimental results when evaluating InfiniPot-V. V.
B
Experimental Setting Details
B.1 MLLMs Video Sampling Details.
For all benchmarks, we employ consistent uniform frame sampling strategy to ensure maxi- mized long video understanding performance across all settings. For Qwen-2-VL [43), which supports dynamic image resizing based on the number of frames, we use the hyper-parameter con- figuration reported to yield the best performance in their original work:
= 768, 768 28 28. Although theoreti-
128 x 28 x 28 and @OREO_MAX_PPO cally larger token budgets could be set, we adopt this configuration to match the optimal context length of 50K reported in the original paper [43], top of which applied KV cache com- pression. For LLaVA, we set the number of sampled frames to 128 to ensure it remained within the model's 's trained context length (<32K). With this video sampling configuration, Qwen-2-VL. [43) uses 384 frames with 130 tokens per frame, resulting in a total context length of 49,920 tokens, while LLatA-Next [52) and LLa VA-OV 22 use 128 frames with 196 tokens per frame, yielding total of 25,088 for offline long video inputs.
VIDEO_MIN, SIDEO_MIN_PIXEL
B.2 Long Video Understanding Benchmark Details
Offline Video Understanding(OVU) We evaluate our method on four multiple-choice based offline video question answering benchmarks: Video-MME (1]. MLVU (54), EgoSchema (28], and LongVideoBende 45
For MLVU and EgoSchema, we use the development sets for evaluation.
For Video-MME, we report results without subtitles version. This is because prepending subtitles for all video frames as a single context block directly before the question represents an unrealistic setting that is incompatible with streaming scenarios, where subtitles are typically unavailable during real-time video processing and would not be accessible as complete context in advance.
Streaming Video Understanding (SVU) For SVU evaluation, we use two benchmarks: RVS- Ego and EVS-Movie [49]. RVS-Ego is constructed from 10 videos from the Ego4D [15) dataset, while RVS-Movie uses 22 long videos from MovieNet [7.. Each benchmark consists of a QA set containing open-ended generation questions and their corresponding timestamps indicating when each question should be presented during video streaming.
The evaluation process works as follows: during CKV processing, when the video stream reaches the timestamp of given question sample, we present the question and generate an answer based on the compressed KV cache accumulated up to that point. The generated answers are then compared against ground -rrth answers using GPT-3.5 S-turbo--tttt 01 to produce accuracy and score metrics.
B.3 Baseline Settings
Input Video Compression (IVC) Details. For the comparison with Input Video Compression (IVC) methods in Tab. and A6 we implement Lu (36] and DyCoke (40 as follows: For LongVU, we apply Spatial Token Compression (STC) every 8 frames as specified in the original paper. STC compresses vision token embeddings by identifying temporally redundant patches using cosine similarity between patches. We adjust the similarity threshold to control the compression rate while maintaining the original methodology. For DyCoke, we implement Token Temporal Merging (TTM) which, similar to LongVU, compresses vision encoder output features. TTM calculates cosine similarity between patches in adjacent frames to eliminate redundant patch embeddings. Following
24

---

the original paper, we process compression every 4 frames and adjust the similarity threshold to control compression rates.
For fair comparison in the Continual KV cache compression (CKV) framework in Tab. 2 we adapt both methods work within memory constraint JM]. Specifically, we compress each input video stream to size [C] and implement sliding window attention [2] to evict older KV cache entries once the cache size reaches the predefined memory limit (lM)) D. This adaptation ensures all methods operate under identical memory constraints for a fair comparison with InfiniPot-V. For benchmark comparing InfiniPot-Y V with IVC methods that use full vision encoding without cache compression,
see Tab. A6
KV Cache Compression (KVC) Details. In Fig.1 we compare three KV cache compression methods within Continual KV cache compression (CKV), First, Uniform Select, inspired by uniform video sampling approaches, selects frames at regular intervals and retains all KV cache tokens corresponding those frames. For SnapKV (4), follow the original method configuration under the CKV process, using the last 32 tokens of the given budget M tokens as an observation window (w) to calculate attention scores for token selection (see Eq. T in Appendix. D.Ip. Additionally, we apply ID pooling with kernel size of 7 to these scores, as done in the original implementation"" For InfiniPot 2T] we design a proxy prompt for video compression: "Provide a detailed description of this video." This prompt is utilized in the CaP method to generate attention scores and apply KV cache compression. Detailed experimental results are provided in Tab. A5
V Slyper-Parametr Settings. To provide additional performance comparison with compres- sion methods specialized for MLLMs, we also include performance comparisons with Fast 44] in L and R. which specify the layer where token pruning
Tab[A5] FastV requires two hyper--prameete begins and the percentage of tokens to prune. For a fair comparison, we adjust the R of FastV to ensure that the total number of KV cache entries across layers matches the total entry count of other baselines that maintain the same number of K KV-cache entries across each layer. Specifically, for Qwen-2-VLL the (L, R) pairs corresponding to memory budgets of 3K, and 6K are set (2, 2.8%) and (2, 5.8%) respectively.
B.4 Positional Encoding Details.
MLLM backbone LLMs utilize positional encoding to differentiate vision token positions. LLaVA- Next [52 and VLV [22] use standard ID RoPE 338. while Qwen-2-Ve 2-VL 43 employs 3D RoPE for multimodal encoding. For Offline Video Understanding(OVU), we apply KV cache compression after positional encoding (i.e., Post RoPE). However, Streaming Video Understanding(SVU presents challenge: continuous video stream processing can exceed the model's 's maximum positional range. For example, in LLaVA models with 196 tokens per frame, streaming more than 6 minutes of video at 0.5 FPS exceeds the 32K context window (note that RVS-Ego and RVS-Movie average over 60 minutes).
To address this, we adopt strategies from InfiniPot (1 and ReKV 35 re-ordering positional indices to fit within the memory budget |M at each CKV step. Specifically, we cache the pre-positional encoded KV's hidden states and re-assign positional indices during decoding, ensuring they never exceed |M position regardless of video length. While this enables SVU for arbitrarily long videos, it discards the original positional information of vision tokens. In particular, additional handling required for Qwen-2-VL's 3D RoPE. Developing methods that preserve the original spatial and temporal position encoding while supporting streaming video lengths beyond the model's 's positional capacity remains an open direction for future work.
C Multi-Turn Video Understanding Analysis
Fig.AA Al presents a qualitative comparison between query- dery-deepnndnnt (SnapKV) 24 and aggro-agnostic (InfiniPot-V) KV cache compression approaches in multi-turn conversations with streaming video input. When SnapKV performs compression based on Q1, it generates answers almost identical to the Full KV cache for that specific query (Q1), answering that the butter was placed in the refrigerator.
25

---

Q1: Where did put the butter after eating? [Single-Detail]
Full-VV After extingg the person put the butter the refrigerator.
The parson
butter the tridge cating O
InfiniPal-V
fridge eelling
Q2: When tuminn off the stove, what kind of kitchen utensil were on the stove? [Single Detail]
Ful-KVV: There
stowe. ?
SnapKy: There were two spoons on the stove. InfiniPot-V. There was kettle the stove
many pieces
appear
during cooking process? [Multi-Detail]
FuilKV: The video shows that there are two pieces of bread the pan and two pieces of bread on the plato, total 4 pieces of bread.
?
Snapky: The video shows that there two pieces of bread in total during cooking process. X the cooking
InfiniPPi-V The
total
broad being
Figure A1: Qualitative Results of Multi-Turn Conversation: Full-KV uses 16K cache while InfiniPot-V V and SnapKV employ 3K compressed KV cache. SnapKV performs query-guu guery-guided cache compression based on QI before proceeding with multi-turn conversation. The video sample is from the MLVU ego reasoning subtask, using the Qwen-2-VL-78 model. 128 frame sampling is used.
However, this query-guided compression strategy reveals significant limitations when handling different types of queries (Q2, Q3) about the same video content. Specifically, SnapKV makes critical errors in subsequent queries misidentifying a kettle as "two spoons" in Q2 and incorrectly counting the total number of bread pieces in Q3.
In contrast, InfiniPot-V maintains accurate answers consistently across all three queries using the same 3K compressed KV cache. correctly identifies that the butter was placed in the fridge (Q1), recognizes the kettle on the stove (Q2), and counts all 4 pieces of bread throughout the cooking process (Q3), demonstrating the effectiveness of auery-agnostic compression for multi-turn streaming video scenarios.
D Why Query-Agnostic KV Cache Compression Matters for SVU?
In this section, we provide a detailed analysis of why query-agnostic compression essential for Streaming Video Understanding (SVU), building upon the requirements discussed in Sec. [. To demonstrate how these SVU-specific constraints impact existing KV cache compression methods, we present case study across three representative scenarios.
D.1 Preliminary: Attention-based KV Cache Compression
Eviction-basee based KV cache compression reduces eache size by removing tokens with the lowest impor- tance scores. Employing attention scores for computing token importance scores is the predominant
approach in previous methods 24 12 16).
In methods such as SnapKV V (24), the importance scores u, of a token Ty are computed by aggregating attention scores from the last w tokens (i.e., observation window) which contain the user instruction:
N
Attn(a; - 2z),
N=N-
where N is the current sequence length. Using these scores, the KV cache is compressed by retaining the top-M tokens with the highest aggregated attention scores. Here, M defines the memory budget: I = TopK(u, (u, .M) and u [21. _uN] indicates the importance scores of all tokens. The compressed Key and Value caches are then formed by extracting tokens indices I:
K KT. I.:) V Vi..I.:]
where K. V E H are the uncompressed Key and Value caches with H heads, N tokens, and per-head dimension D. This approach has two characteristics: (1) requires computing the full KV cache for all tokens before compression, and (2) it requires the user query to be present at the end
26

---

many people watching
Vision
Cache
Whogg! inin..
Compressed
Miss GT Framel
Memony NEII
LLM Backbone
KV Compression
KV Compression
KY Comprwoion
Constrained
Frofil
Memory (IM))
GT Evicted!
Finge
SYS Token
Viston Token
INST Taker
SYS Token
Vilio Token
Syatem
GT Frame
Close- ended Video
Close ended Video
Streering Video
(a) Case 1. Non-Streaming Scanario
(b) Case Non-Streaming Scenario
[c] Casa Streaming Scanario /
Memory Constrained Query Agnozlic?
Memory Constrained? X Query Agnastic? of
Quary Agnostic?
Figure A2: KV Cache Compression Case Study with SVU: Illustration of cache control strategies under three conditions, differing in the presence of two core requirements for Streaming Video Understanding (SVU): memory constrained (MC) and query agnostic (QA). (a) Case 1: Query- guided compression retains relevant (GT) frames for accurate responses. (b) Case Without query guidance, compression fails to preserve critical frames, resulting in inaccurate responses. (c) Case (Streaming scenario): streaming video processing, where frames arrive continuously, continual KV cache compression (CKV) is necessary, but queries are unavailable during compression.
of the context. We refer to these approaches as query-guided or attention-based cache compression methods
D.2
Case Study: Towards Streaming Video Understanding with CKV
To investigate the applicability of attention-based KV cache compression methods to streaming video understanding, we examine three cache control strategies (Fig. A2
Case 1. Recent KV cache compression meth- ods [24] [2] assume full access to context and queries at compression time, shown in
Attention Scoring
Prefill [M]
Gen.
Gen.
Case
[M] = 3K
|M] - 6K
Fig. A2]a). In this memory-unconstrrainee set- ting, the model observes the full input before compression. Previous works (24 have demon- strated that attention scores effectively identify query-relevant tokens KV cache (orange box corresponding GT Frame in Fig. A2 a)), en- abling compression that retains critical infor- mation while discarding less important tokens. As shown in Tab, A33 this approach maintains performance comparable the uncompressed cache setup (68.01 vs 68.75) at the cost of large memory usage at compression, detailed in Fig.
Full KV
n/a
25K
68.75 (t)
Case
Attn(g + mx)
68.01
68.40
Antn[a + 2x)
60.35
63.42
Case
Attn[q" my
25K
60.60
63.50
Aun(go xx]
60.32
62.28
Case 3 Attn(q- xx)
3K/6K
57.55
59.98
Table A3: Case study of Attention Scor- ing: conducted on MIVU benchmark with -LaVA-Next-Vided
Note that memory- constrained setting (Case 3) shares the same budget during prefill and generation stages.
Case Fig. A2 (b) illustrates how attention-based cache compression fails when user queries unavailable during compression. Under this scenario, although the memory budget is assumed unconstrained, the KV cache is compressed without consideration of (future) queries, causing important visual tokens (orange tokens cache corresponding to the GT Frame) lost during compression. To quantify this degradation and explore alternatives, we test compression with generic queries (q': "What is happening in this video?". q" :"What are the key events in this video?") and the last vision tokens (q,) for importance scoring:
Throughout this paper, "query" refers the user's instruction question related to the given video.
27

---

0.9
01. 9
Context Longth
Case 1.2
Case 3 (Ours)
01 91
Mem (GB)
TIFT (8)
Mom (GB)
TTFT (s)
jaccard Simllanty
0.8
42. 93
25K
5K
21.29
33.76
0.98
1.21
21.60
20.93
1.08
1.12
50K
58.55
2.12
22.16
1.17
0.7
100K
79.38
3.27
22.85
1.20
Table A4: Peak GPU Memory and TTFT:
0.6
Comparison of peak memory usage and Time- across different con-
0.5
text lengths for cennory-uncoostraineed (Case 1, 2) and memory-constrained (Case 3) ap- proaches.
0.4
5
10 15
20
25
Layer Index
Jaccard Similarity between KV Compare KV cache sets selected by dif- ferent queries (q1, 42, q3) across layers.
Figure A3:
Caches:
Ug Attn (gatt + xt), Qah E "d.'"".q0
Tab. A3 and Fig. A2 a) show that these alternatives significantly degrade performance (60.32 vs 68.75), even with unconstrained memory.
Case 3: Streaming Scenario. Beyond the query-agnostic challenge Case deploying streaming video understanding on sesource-constrinnee devices requires fixed memory usage for the KV cache. For input video streams, these constraints necessitate continual compression when new frames arrive and memory capacity is reached, as shown in Fig. A2 c). To evaluate this scenario, we use the query-agnostic approach from Case 2 with vision tokens (go) for importance scoring, while compressing the KV cache whenever memory limits are reached. As shown in Tab. A3 this combined constraint further degrades performance (57.55 vs 60.32), highlighting the challenge of preserving key information under both query-agnostic and memory-constraings settings.
This case study reveals two key challenges for KV cache compression in streaming video: 1) the need for query-agnostic compression due to continuous incoming video, and 2) the requirement to maintain fixed memory constraints. These challenges cause significant performance drops previous methods (24 4] motivating Continual KV cache compression (CKV) specifically designed for memory-constrained streaming video.
Attention Scoring Analysis We further analyze the query-dependent nature of attention-based KV cache compression using the VideoMME benchmark. To investigate why performance varies with different queries, we compute the Jaccard similarity between token sets selected for different queries across layers using attention scores each layer. For this analysis, 91, 92, 43 represent three distinct questions associated with the same video sample in the VideoMME benchmark. As shown in Fig. A3 the similarity between token sets decreases significantly in the middle-to-late layers, dropping to around 0.4. This indicates that each query selects a different set of tokens, particularly in deeper layers. This analysis highlights that attention-basee scoring methods inherently select query- speciiccc tokens, explaining the performance degradation when query information is unavailable or changes during streaming video scenarios.
E
Memory and Latency Measurement Results
Table presents measurements of peak memory consumption and Time-To- To-First-Tokee (TTFT) during the prefill stage, conducted on single NVIDIA A100-80GB GPU using PyTorch. The experiments averaged over five runs with three warmup iterations, compare the performance of memory-unconstrained (Case 1, 2) and memory-constrained (Case 3) approaches across various context lengths. For memory-unconstrainee quirements, escalating from 21.29 GB at 5K tokens to 79.38 GB at 100K tokens, accompanied by proportional increase in TTFT from 0.98 to 3.27 seconds.
methods, we observe linear growth in memory re-
28

---

Our memory- constrrinnnttttnneed continual KV cache compression (Case 3) exhibits remarkably different behavior. Despite the increasing context length, the peak memory usage shows only minimal growth, rising modestly from 20.93 GB at 5K tokens to 22.85 GB at 100K tokens. Similarly, the TTFT remains relatively stable, increasing from 1.08 to 1.20 seconds across the same range. These detailed measurements demonstrate that our approach effectively maintains near-constant resource utilization while processing extended video frames.
F
Related Work
F.1
MLLMs for Long Video Understanding
Recent advances in long-context MLLMs have attracted significant attention. Notable examples include Gemini-2.0. 2.0 [34], supporting streaming video: LongVILA [5]. capable of handling up to 6,000 video frames; LLaVA-Next-Vinn Video [52], which leverages high-quality synthetic instruction data; and Qwen-2-VL. 43
enabling hour-long video analysis via multimodal RoPE.
F.2
Input- Vision Compression (IVC)
To address the computational demands of long-form video processing, several approaches have been proposed compress redundant visual information before enters the backbone LLM.
LongVU [36] adopts query-dependent input frame sampling and redundant pixel removal for fine- grained video understanding, but the two-tower vision encoding results in high latency during input sampling, making it impractical for streaming scenarios. Additionally, this approach requires training specialized models to operate in the proposed manner, limiting its applicability to existing pre-trained models.
DyCoke 40 reduce redundancies between adjacent frames at the input video level and dynamically updates query-related tokens the KV cache from external storage. Slow-Fas-LLLVVA-.35 [46] proposes dividing input video processing into separate slow and fast pathways, using different projection methods to reduce input vision tokens. However, this approach still suffers from limitation of requiring all input vision tokens to be processed simultaneously and necessitates additional model training.
F.3 KV Cache Compression (KVC)
Understanding the long context in MLLMs demands efficient KV cache control to manage memory growth and latency overhead. KV cache compression methods be broadly categorized into query-dependent and query-agnostic approaches.
Query-Dependent KV cache Compression. Methods like SnapKV
24
H2O
53
HeadKV 12
and ThinK [47 leverage query-to-conted attention scores to identify crucial KV entries but require the full context to be prefilled before compression, making them impractical under memory constraints. In the multimodal domain, FastV 44) accelerates prefill by pruning vision tokens certain layers based their attention scores from the final query token. Sparse [51) selects visual tokens relevant to user queries via cross-attention. Overall, query-dependent methods effectively compress context but struggle to handle diverse queries for the given context after compression [39]. ReKV 35 addresses streaming video scenarios by offloading video-related KV cache to CPU memory and retrieving query- -eeendeeennd cache entries on demand. This approach relies external storage and suffers from data transfer overhead, making it unsuitable for understanding.
streaming video
-Agro-ig KV cache Compression. Recent works pursue query-agnostic KV cache compres- sion to eliminate reliance on future queries [13 g
o 32 In particular, SqueezedAtteeniie [16] uses key-based clustering but requires full-context encoding, limiting its applicability to memory- constrained settings. InfiniPot [2]] compresses context by approximating potential user queries through a task-specific proxy prompt, but it's fixed prompt restricts flexibility. In the vision domain, HiRED 1 and FasterVL.M [50] utilize [CLS] token attention scores for compression decisions. However, their reliance on special tokens restricts their application to recent MI.LMs that lack such tokens 48 (2), limiting their broader applicability.
16
20
29

---

MC
Strcaming
QA
Compression Method
Budget
Prefill
Decoding Budget
Sbort
Medium
VideoMME
Long
Aug.
Holistic
Single
Mulci
Avgg
I.B
Avg.
Qwen-2-V1--78
Full
50K
50K
74.68
62.11
63.93
76.34
73.91
58.77
62.85
FastV [
48/5K (R = 2.8)
54,11
50.11
48.67
50.96
69.59
59.40
35.84
55.01
47.94
51.30
Case
= 2)
48/6K (R 5.8)
59.67
54.55
50.78
55.00
72.00
64.08
33.47
57.60
50.53
54.38
SnapKV 24
50K
74.00
63.07
22.08
39,07
59.06
61
50K
6K
74.22
60.55
54.33
63.03
77.59
73.91
42,90
66.10
58.80
62.64
Cioc
Uniform
50K
50K
3K
72.00
70.33
54.67
58.78
49.55
52.11
60.96
58.18
72.29
77.08
59.06
39.07
33.51
55.54
59.11
59.80
57.84
60.73
SnapKV1
50K
3K
69.00
54.00
50.66
57.89
75.
63.48
35.35
58.99
56,70
57.86
50K
6K
72.11
57.56
52.22
60.63
76.46
66.43
36.22
60,66
56.72
59.34
3K
66.00
33.51
55.45
Uniform
6K
6K
72.33
53.
48.67 67
58.11
72.55
62.19
3.5.67
57.00
55.82
56.99
12K
12K
74.00
55.33
51.44
60.26
75.94
65.53
37.01
60.36
57.91
59.51
24K
24K
74.22
59.22
77.22
71.10
40.78
58.60
61.67
3K
3K
66.67
52.22
49.89
56.26
75.88
63.488
35.35
58.99
54,91
56.72
Clse 3 (CKV)
SuapKV1
12K
6K
12K
6K
74.44
72.00
55.33
51.33
59.55
76.
75.71
66.43
36,
60.66
55.15
56.89
58.45
24K
24K
74.22
61.000
53.78
63.00
77.66
71.82
39.90
64.37
59.09
62.15
3K
3K
67.11
54.55
51.00
57.55
74,94
61 80
36.60
58.36
$4,00
56.64
InfiniPot
6K
57.33
60.52
75.02
63.18
37.09
54.64
58.
12K
12K
74,00
57.78
53,22
61.67
74,46
66.46
38.30
60.70
56.94
59.77
24K
24K
74.22
60.55
53.56
62.78
76.03
71.11
40.29
63.71
57.85
61.44
3K
57.78
27.73
57.66 64
65
InfiniPut-V
6K
74.11
60.78
77.16
72.31
44.75
12K
12K
74.22
62.68
$3.89
63.59
76.90
73:41
43.97
65.99
59.18
62.92
24K
24K
74.22
63.22
53.11
63.52
76.91
73:97
42.18
65.73
58.94
62.73
L.LaVA-Next-V Video-78
Full KV
25K
25K
74.33
60.11
54,11
62.85
80.60
73.73
49.44
68.75
63.55
65:05
Uniform
25K
74.33
62.33
72.38
62.35
64.75
Cave
x
25K
6K
73.89
62.00
51.78
63.56
80.66
72.25
49.62
68.19
62.55
64.76
SnapKV
20
25K
25K
3K
74.44
74.44
59.89
53.78
62.70
62.78
80.60
80.41
73.01
49.67
68.46
62.34
64.59
64,50
Cine 2
Uniform
25K
25K
6K
3K
71.00
66.33
54.00
56.33
51.55
49.67
59.63
56.67
75.12
27.84
59.65
60
38.55
43.92
62.90
58.04
61.09
59.14
61.41
57.95
SnapKy
25K
56.55
56.19
57.56
25K
6K
69.55
58.44
52.78
60.26
80.86
63.65
45.07
63,26
59.90
61.14
1,5K
1.5K
56.22
46.89
44,00
49.04
69.72
52.53
36.53
$2.87
$4.92
52.
Uniform
3K
52.74
74.
54. 44
6K
6K
64.89
$5.67
-1.77
56.78
76.71
61.14
34.55
57.99
57.72
57,50
12K
12K
72.67
59.89
53.00
61.85
80.03
67.33
44.31
64.38
61.04
62.42
1,5K
1.5K
58.00
53.65
52. $2.22
Clse (CKV)
SnapK
3K
3K
62.11
54.55
48.55
55.07
76.94
59.18
35.71
57.55
54,71
55.78
6K
6K
66.33
56.
51.11
$7.85
79.00
62.15
37.12
59.98
57.81
58.55
12K
12K
72.11
58.00
53,11
61.07
79.71
67.99
44.89
64.74
58,83
61.55
1.5K
1.5K
53.22
51.11
47.55
53.11
699 89
30.54
$2.88
52.14
52:71
InfiniPot A1
6K
3K
6K
3K
58.22
62.44
53.89
$1.78
51.11
19.33
55,81
$1.22
76.45
72.42
55.88
57.97
34.45
37.07
57.28
54.48
52.43
55.58
53.71
56.22
12K
12K
70.55
59.22
52.55
60.77
79.84
67.81
45.57
59.23
1.5K
1.5K
63.89
52.55
47.11
54.52
77.08
57.32
34.64
56.49
56.48
55.85
InfiniPot-V
6K
3K
3K
72.44
67.78
59.55
56.22
50.35
58.11
61.11
77.88
65.74
40.31
61.94
60.86
58.37
59.47
12K
12K
73.89
58.67
52.11
61.55
80.91
71.16
51.57
68.35
61.84
63.91
Table A5: InfiniPot-V vs KVC Offline long video understanding evaluation results under memory- constrained scenario (case 3), with MC (Memory-Constern Constraidd)) and QA (Query-Agnostic conditions marked. Results are reported on (1) Video-MME: Short: -3min, Medium: 3- 3-30min, Long: 30min-2h, (2) MIVU - Holistic, Sengle-Detail, Multi-Detail LVU, and (3) LVB (Long VideoBechmaakkk
G
Experimental Results Data
G.1 Comparison between InfiniPot-V and KVC
Tab. A5 provides a detailed performance comparison between KV cache compression (KVC) methods and InfiniPot-V across offline video understanding (OVU) benchmarks under various compression ratios for two models: Qwen-2-Ve LL and LLaVA-Next.
In Case 1, where the full prefill is conducted and the final query is accessible at compression time, FastV demonstrates significantly inferior performance at similar compression ratios due to its aggressive token-pruning strategy. In contrast, SnapKV shows robust performance at high compression ratios across both models by utilizing the full context KV cache and retaining vision tokens that are highly correlated with the given query.
30

---

Qwen-2-VL.
Vision
Decoding Budget
MLVU
VideoMME
IVC Methods
Budget
Holistic
Single
Multi
Avg.
Short
Med
Long
Avg.
Avg.
Full KV
50K
50K
76.3
73.9
43.3
65.9
74.7
62.1
55.0
63.9
64.2
Uniform
50K
6K
77.7
69.8
41.6
64.0
74,9
58.0
52.8
61.9
62.5
TTM 40
50K
6K
78.2
70.0
42.7
64.5
74.9
59.2
52.7
62.3
62.9
STC
36
50K
6K
77.9
71.5
44.7
65.7
74.3
59.6
54.6
62.8
63.8
InfiniPot-V
6K
77.2
72.3
44.7
65.8
74.1
53.4
62.8
63.8
Uniform
50K
3K
75.7
66.5
38.6
61.1
72.2
53.4
50.0
58.6
59.4
TTM 40
50K
3K
77.3
67.8
39.5
62.4
72.7
56.2
52.2
60.4
61.0
STC
36
50K
3K
76.9
68.2
41.7
63.1
71.2
55.9
53.7
60.3
61.3
InfiniPot-V
3K
3K
77.7
70.4
43.2
64.7
73.9
57.8
51.8
61.1
62.5
Table A6: InfiniPot-V vs IVC: Performance comparison between Input-Vision Vision Compression (IVC) methodology and InfiniPot-V. Vision budget denotes the vision token length before IVC, while decoding budget refers to the input token length used during decoding. Evaluated using Qwen-2-VL with MLVU and VideoMME datasets.
Case 2 examines the query-agnostic setting. where, as explored in our earlier case study in Ap- pendix. D.2 SnapKV exhibits notable performance degradation across both models when applied in agnostic manner, showing performance comparable to uniform selection baseline.
In Case 3, which represents the CKV framework scenario where the constrained memory budget is used for both prefill and decoding stages, InfiniPot-V significantly outperforms all three baselines across various compression ratios on both models, showcased in Fig. 2
G.2 Comparison between InfiniPot-V and IVC
Table A6 presents a performance comparison between Input- Viston Compression (IVC) methods and InfiniPot-V the MLVU and Video-MM benchmarks using the Qwen-2-VL. model. Under 6K decoding budget, the IVC methods demonstrate robust overall performance by utilizing the full vision encoding budget (50K tokens). InfiniPot-V V achieves comparable slightly superior performance to these methods while operating under constrained memory budgets for both vision encoding and decoding stages (6K tokens).
When the decoding budget is compressed to 3K tokens, the IVC methods exhibit performance degradation, with V'U's STC methodology achieving the highest performance among the IVC approaches. Notably, InfiniPot-V demonstrates both efficiency and effectivenees by achieving higher accuracy than IVC methods that utilize the full vision encoding budget, while operating under constrained budgets (3K) for both vision encoding and decoding stages.
H
Limitation and Future Work
InfiniPot-V introduces the first training-frees query-agnostic framework for memory-constraiin streaming video understanding, enabling length-indeepmnenn KV cache compression with minimal accuracy loss across long-form, real-time scenarios. However, several avenues exist for further advancement. Current approaches focus primarily on vision tokens, yet real-world streaming applica- tions involve multiple modalities including speech, text, and video simultaneously. Future work could extend our framework to unified multimodal compression, enabling more realistic and comprehensive streaming understanding systems that efficiently manage diverse input types within fixed memory constraints.
Additionally, current fixed budget allocation between TaR and VaN components could benefit from adaptive mechanisms that dynamically adjust compression ratios based on input characteris- tics alls-allo more resources to temporal redundancy reduction for static scenes prioritizing spatial importance for content-rich frames. Furthermore, while InfiniPot-V's training-free nature ensures broad applicability, end-to-end learning approaches could optimize models specifically for continual compression scenarios, potentially enabling more aggressive compression ratios through learned token importance estimation tailored streaming video understanding tasks.
31


Reviews:
[{"review_id": "iDxFN6E0dC", "reviewer": "Reviewer_otEA", "summary": "InfiniPot-V introduces a continual KV-cache compression framework that keeps GPU memory usage constant regardless of video length. When the cache reaches a user-set budget, the system compresses it by (i) pruning temporally redundant patches via Temporal-axis Redundancy (TaR) and (ii) preserving semantically salient regions via Value-Norm (VaN). The method is training-free and query-agnostic; across four open-source MLLMs and six benchmarks, it slashes peak memory by up to 94 % while sustaining real-time 14 fps generation and matching or surpassing full-cache accuracy.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Straightforward, training-free cache compression that can be plugged into existing MLLMs without retraining.\n 2. Consistent memory reduction (≈ 94 %) across four backbone models with only marginal speed overhead, proving the engineering practicality.\n 3. Paper is clearly written; algorithmic steps and ablation studies are easy to follow, which aids reproducibility.\n\nWeaknesses\n1. Claimed novelty is thin. The Temporal-axis Redundancy module repeats the “drop static frames” idea already seen in recent work such as LongVU; VaN likewise relies on a well-known “high-entropy tokens carry more information” intuition, and the layer-wise pooling tweak is incremental.\n2. Comparative study on streaming video understanding is shallow: only ReKV is benchmarked, limited to a single dataset (RVS), and the method does not reach the current SOTA there. Broader baselines and benchmarks in SVU are missing.\n3. Method improves Qwen2-VL on EgoSchema yet hurts Qwen-2.5-VL noticeably, raising doubts about robustness; the paper offers no analysis for this drop.\n4. The ablation study is not strong enough. More simple strategies should be compared. For example, add an ablation that replaces TaR with LongVU’s key-similarity filter and replaces VaN with a simple top-k entropy selection. Show accuracy and memory for each variant.", "was_revised": true, "final_justification": "Overall, the paper is well-structured and clearly written, and the proposed SVU approach shows a certain degree of innovation within the field. However, the current version only compares against a single SVU baseline, and more comprehensive benchmarking and baseline comparisons are still needed. Given the preliminary results on an additional benchmark presented in the rebuttal, and the authors’ commitment to include further discussion and comparisons with more SVU methods in the revision, I am inclined to give a borderline accept score. However, I cannot justify a full accept, as there is no guarantee that the authors will fully follow through on their commitments or that the additional results will show significant improvements over other SOTA methods in the SVU domain.", "rebuttal": "We appreciate the reviewer's thoughtful and constructive feedback. We are encouraged that the reviewer found our training-free method straightforward and practically valuable for achieving significant memory reduction with minimal overhead. Here are clarifications and answers to the concerns raised by the reviewer.\n\n---\n\n**1. TaR and VaN’s novelty over existing compression metrics, supported by ablation study**  \n> **Weakness 1.** Claimed novelty is thin. The Temporal-axis Redundancy module repeats the “drop static frames” idea already seen in recent work such as LongVU; VaN likewise relies on a well-known “high-entropy tokens carry more information” intuition, and the layer-wise pooling tweak is incremental.\n\n>**Weakness 4.** The ablation study is not strong enough. More simple strategies should be compared. For example, add an ablation that replaces TaR with LongVU’s key-similarity filter and replaces VaN with a simple top-k entropy selection. Show accuracy and memory for each variant.\n\nTaR and VaN fundamentally differ from the reviewer's suggested approaches: while *LongVU’s key-similarity filter* operates on input vision tokens’ embedding (token-axis) and *Top-K entropy selection method* uses final logit projections (also per-token axis), our methods work directly on KV cache representations at each attention head (per-head axis). This enables more diverse and expressive identification of spatiotemporal redundancy.\n\nWe address the reviewer's concerns by first explaining how TaR and VaN fundamentally differ from LongVU's Key Filter and Top-K Entropy selection, then demonstrating these distinctions through ablation studies where we replace our methods with the suggested alternatives on video understanding evaluation task.\n\n| Method | Topic | Anomaly | Plot | Needle | Ego | AO | AC | Avg ↑ |\n|--------|-------|---------|------|--------|-----|----|----|---------|\n| Full KV | 85.2 | 67.5 | 72.7 | 83.9 | 65.1 | 54.1 | 32.5 | 65.9 |\n| Key Filter | 82.1 | 67.0 | 64.8 | 70.5 | 59.1 | 44.8 | 23.8 | 60.8 |\n| Entropy | 79.9 | 64.5 | 61.4 | 71.3 | 59.4 | 49.4 | 23.8 | 60.2 |\n| TaR | 85.9 | 66.5 | 71.8 | 78.0 | 62.2 | 51.7 | 35.4 | 64.5 |\n| VaN | 84.4 | 68.0 | 68.6 | 76.6 | 61.9 | 52.5 | 29.1 | 63.0 |\n| TaR + VaN (Ours) | 86.3 | 68.0 | 72.7 | 80.3 | 63.9 | 54.1 | 35.4 | **65.8** |\n\n*Table R1. Offline Video Understanding (MLVU) Performance - Qwen-2-VL-7B (6K Cache budget)*\n\n**TaR leverages Key embedding similarity for effective temporal redundancy reduction.** We demonstrate for the first time how similarity between Key embeddings at each layer's KV cache effectively identifies video temporal redundancy. Our analysis shows that Key embedding similarity successfully detects static patches (see `Fig2.b`), with ablation studies confirming its effectiveness for long video understanding accuracy (see `Tab.4`). This approach differs from LongVU's input vision token compression that forces *identical token eviction across all KV cache layers.*\n\n**VaN establishes Value norm as a semantic importance proxy through intermediate layer analysis.** We conduct entropy analysis of vision tokens at each intermediate layer by projecting their hidden states, demonstrating that tokens with higher Value norms consistently exhibit higher entropy and better preserve long video accuracy when retained (see `Fig.3`). This provides concrete evidence that L2 norms of Value embeddings serve as effective semantic importance scores, complemented by our layer-wise adaptive pooling technique. Unlike Top-K Entropy selection that operates at the token level using final vocabulary projections, VaN compresses based on L2 norms of Value embeddings at each attention layer.\n\n**Ablation study with suggested alternatives confirms the superiority of our approach.** `Table R1` shows that replacing TaR with LongVU's Key Filter results in nearly 4% performance degradation (64.5→60.8), while replacing VaN with Top-K Entropy selection leads to nearly 3% performance drop (63.0→60.2). These results demonstrate that our KV cache compression enables more temporally meaningful token retention and semantically important token preservation across attention heads compared to the suggested alternatives.\n\n\n---\n\n**2. New Streaming Benchmark Evaluation Results Demonstrating Our Method’s Robustness**  \n> **Weakness 2.** Comparative study on streaming video understanding is shallow: only ReKV is benchmarked, limited to a single dataset (RVS), and the method does not reach the current SOTA there. Broader baselines and benchmarks in SVU are missing.  \n\n> **Weakness 3.** Method improves Qwen2-VL on EgoSchema yet hurts Qwen-2.5-VL noticeably, raising doubts about robustness; the paper offers no analysis for this drop.\n\nTo address the reviewer's concerns about the shallow streaming video evaluation and robustness issues with Qwen2.5-VL, we conducted additional evaluation with Qwen2.5VL-3/7B across StreamingBench [1], a recent streaming video understanding benchmark for comprehensive real-time video understanding assessment with real memory usage measurement on NVIDIA Jetson AGX Orin hardware. \n\n| Qwen2.5-VL | Method | OP | CR | CS | ATP | EU | TR | PR | SU | ACP | Avg ↑ | Mem ↓ (GB) |\n|-------|--------|----|----|----|----|----|----|----|----|-----|--------|----------------|\n| 3B | Full KV | 78.1 | 77.7 | 75.1 | 83.4 | 78.1 | 76.7 | 70.2 | 70.2 | 68.2 | 73.4 | 26.6 |\n|  | Key Filter | 76.0 | 75.8 | 74.5 | 78.8 | 72.7 | 75.7 | 67.6 | 63.8 | 64.3 | 69.7 | 10.2 |\n|  | Entropy | 77.9 | 77.3 | 74.5 | 81.1 | 72.7 | 74.8 | 67.6 | 64.2 | 64.0 | 70.4 | 12.1 |\n|  | TaR + VaN (Ours) | 77.4 | 75.8 | 81.1 | 83.0 | 77.0 | 76.6 | 69.4 | 66.7 | 66.9 | **72.2** | **10.1** |\n| 7B | Full KV | 82.3 | 80.3 | 85.9 | 83.7 | 79.3 | 79.8 | 85.3 | 69.6 | 71.9 | 78.7 | 34.8 |\n|  | Key Filter | 81.7 | 77.3 | 82.9 | 82.7 | 73.3 | 79.8 | 83.3 | 64.6 | 67.9 | 74.7 | 18.7 |\n|  | Entropy | 79.6 | 78.1 | 81.4 | 80.1 | 74.5 | 78.8 | 82.4 | 66.3 | 68.0 | 74.2 | 20.7 |\n|  | TaR + VaN (Ours) | 81.7 | 79.7 | 84.9 | 83.3 | 78.9 | 79.4 | 86.1 | 68.3 | 70.5 | **76.4** | **18.6** |\n\n*Table R2. StreamingBench Evaluation Results (Realtime video understanding) - Qwen2.5-VL-3B and 7B used with 6K cache budget*\n\nOur method demonstrates robust performance on this comprehensive streaming video understanding benchmark. `Table R2` shows that InfiniPot-V achieves lowest memory consumption which is 2-2.6x compression over full KV-cache (10.1GB vs 26.6GB for 3B, 18.6GB vs 34.8GB for 7B) on edge devices while consistently outperforming the suggested filter and entropy-based alternatives by ~2% in both model sizes. These compelling and consistent trends confirm that our methods robustly achieve superior accuracy-efficiency trade-off to existing compression metrics.\n\n[1] Lin et al, “StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding” Arxiv 24.11"}, {"review_id": "Fur5M6ieNx", "reviewer": "Reviewer_dgbM", "summary": "This paper introduces InfiniPot-V, a novel framework designed to address the critical problem of unbounded key-value (KV) cache growth in Multimodal Large Language Models (MLLMs) during streaming video understanding. The authors propose the first training-free, query-agnostic compression framework that enforces a hard, length-independent memory budget, making it suitable for resource-constrained edge devices. The core method, termed Continual KV cache Compression (CKV), monitors the cache size during video stream processing and, upon reaching a user-defined threshold, performs an efficient in-place compression. This compression is guided by two lightweight, complementary metrics: Temporal-axis Redundancy (TaR) to prune temporally redundant tokens and Value-Norm Importance (VaN) to preserve semantically salient tokens. The authors demonstrate that InfiniPot-V can be applied as a plug-and-play module to various MLLMs, significantly reducing peak memory usage while maintaining or even surpassing the accuracy of full-cache baselines across multiple long-video and streaming benchmarks.", "questions": "1.  **Robustness Under Challenging Conditions:** How does InfiniPot-V perform on videos that challenge its core assumptions? For example, how does TaR handle high-motion sports videos with minimal static background, and can VaN reliably preserve small but crucial textual information on screen? Have you considered adaptive thresholds for TaR and VaN to mitigate potential failures in such adverse conditions?\n2.  **Hardware-Specific Performance:** The paper demonstrates impressive efficiency on A100 GPUs. To fully validate its practicality for edge computing, could you provide latency and power consumption metrics on representative ARM-based devices like a smartphone SoC or NVIDIA Jetson?\n3.  **Extending to Multimodal Streams:** A key area for future work is multimodal compression. How do you envision extending InfiniPot-V to jointly handle visual, audio, and synchronized text (e.g., subtitles) streams? Would this require new cross-modal redundancy metrics to decide which modality's tokens to preserve, or could each stream be compressed independently?\n4.  **Long-Term Semantic Retention:** For ultra-long videos, such as hour-long surveillance footage, repeatedly compressing the KV cache might lead to the gradual decay of important semantic information from early in the stream (\"catastrophic forgetting\" in the cache). Have you analyzed this effect? Would a hierarchical caching mechanism (e.g., storing summaries of older, heavily compressed blocks) be a potential solution to ensure long-term memory retention?", "limitations": "Suggestions are listed in the questions and weaknesses parts.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* **Significance and Originality:** The paper tackles a crucial and practical bottleneck for deploying MLLMs on edge devices for real-world applications like real-time video assistants. It introduces what appears to be the first framework to enforce a truly fixed memory budget for streaming video understanding in a query-agnostic and training-free manner. The proposed TaR and VaN metrics provide an elegant and lightweight solution that contrasts sharply with existing methods that are often query-dependent, require full cache materialization, or necessitate costly retraining.\n* **Quality and Performance:** The work is technically strong and validated with impressive empirical results. InfiniPot-V demonstrates remarkable efficiency, achieving up to 94% peak GPU memory reduction with only a 0.5% latency overhead. Across four open-source MLLMs and six benchmarks, it consistently matches or exceeds the accuracy of full-cache baselines, showcasing that significant compression can be achieved with minimal performance degradation. The thorough ablation studies effectively validate the synergy between the TaR and VaN components.\n* **Practicality and Impact:** The plug-and-play and training-free nature of InfiniPot-V makes it highly practical and likely to have a significant impact on the field. By eliminating the need for retraining and CPU offloading, it provides a viable path toward deploying powerful video understanding models on consumer devices like smartphones and AR glasses, bridging a critical gap between large-scale research models and real-world applications.\n\n**Weaknesses:**\n\n* **Quality/Robustness of Compression Heuristics:** The proposed compression criteria have potential weaknesses in certain scenarios. The TaR metric, which relies on identifying temporal redundancy, may be less effective for videos with constant, rapid scene changes or frequent edits where static content is minimal. Similarly, the VaN metric, which uses the $\\ell_2$ norm of Value vectors as a proxy for semantic importance, could potentially discard low-norm but contextually critical visual cues, such as small text overlays or subtle but important objects. The robustness of these heuristics across more diverse and challenging video styles is not fully explored.\n* **Scope of Evaluation and Generalization:** The evaluation, while extensive, has some gaps. The benchmarks used are primarily in English, leaving the framework's performance on multilingual content unevaluated. Furthermore, the experiments are conducted under controlled conditions (e.g., fixed frame rates), and the system's robustness in more challenging, real-world scenarios like low-light conditions, occlusions, or dynamic frame rates remains unvalidated. The experiments are also conducted on high-end A100 GPUs, with no deployment validation on the target edge devices (e.g., Jetson Orin), leaving questions about real-world latency and power consumption unanswered.\n* **Limited Multimodal Scope:** The current framework focuses exclusively on compressing the visual modality's KV cache. In many real-world videos, audio and transcribed speech are critical parallel streams of information. The paper does not address how to integrate or co-compress audio or text-based KV caches, which would be essential for a truly holistic understanding of multimodal streams.", "was_revised": true, "final_justification": "Thank you for your response. I have also read the other reviews and the authors' rebuttal, which provided detailed explanations on several aspects, including the comparative study on streaming video understanding, novelty, robustness, and on-device deployment. I currently have no further concerns, and therefore decide to maintain my original rating.", "rebuttal": "**1. Exploration of TaR/VaN Robustness under Challenging Conditions**\n> **Weakness 1.** The TaR metric may be less effective for videos with rapid scene changes where static content is minimal. The VaN metric could potentially discard low-norm but contextually critical visual cues. \n\n> **Question 1.** How does InfiniPot-V perform on videos that challenge its core assumptions? How does TaR handle high-motion \nvideos, and can VaN preserve crucial textual information? Have you considered adaptive thresholds?\n\nWe appreciate the reviewer's insightful observation about potential trade-offs between TaR and VaN under different video conditions. Your suggestion about adaptive TaR/VaN combination represents a particularly valuable direction for addressing challenging conditions beyond our current scope.\n\nTo explore how our metrics behave under diverse video conditions, we conducted stress tests examining TaR and VaN performance by analyzing subtasks of StreamingBench [1] with distinct visual requirements: Attribute Recognition (requiring fine-grained object detail retention), Event Understanding (demanding temporal sequence preservation), and Causal Reasoning (needing complex visual information integration). Using Qwen-2.5-VL-3B on streaming video understanding [1] tasks, we swept the TaR ratio (α) from 0.0 (VaN only) to 1.0 (TaR only):\n\n| Method           | Attribute Recognition | Event Understanding | Causal Reasoning | Average |\n|------------------|----------------------|-------------------|------------------|---------|\n| Full KV             | 83.4                       | 78.1                   | 78.7                 | 73.4     |\n| Uniform Selection| 82.0                 | 74.2              | 74.2             | 70.8    |\n| VaN only (α=0.0) | 83.3             | 73.9              | 75.8             | 71.4    |\n| α=0.2            | **83.7**                 | 72.1              | 77.3         | 71.9    |\n| α=0.4            | 82.7                 | 74.5              | **78.1**             | 72.0    |\n| α=0.6            | 83.0                 | **77.0**          | 75.8             | **72.2**|\n| α=0.8            | 82.7                 | 76.4              | 76.6             | 72.1    |\n| TaR only (α=1.0) | 82.0                 | 76.8              | 75.8             | 71.2    |\n\n*Table R1. StreamingBench evaluation with TaR ratio (α) exploration. Qwen2.5-VL-3B used with 6K KV cache budget (Full KV: 25K-50K, 0.2 - 1 fps)*\n\n`Table R1` reveals how task characteristics influence the relative effectiveness of each metric: For Attribute Recognition, TaR-heavy approaches underperform compared to VaN-focused selection (82.0 vs 83.3), suggesting that when fine-grained object details matter, temporal redundancy reduction alone is insufficient. Conversely, for Event Understanding, VaN-only approaches show clear limitations compared to TaR-weighted combinations (73.9 vs 77.0), indicating that semantic importance scoring can miss crucial temporal sequence information.\n\nOn the other hand, for complex tasks like Causal Reasoning that require understanding diverse visual cues, our balanced combination achieves superior performance (78.1 with α=0.4), and overall our approach consistently outperforms baselines (72.2 vs 70.8 average). This demonstrates that while each metric has task-specific dynamics, our framework maintains robust effectiveness across scenarios.\n\nThese results reveal the potential for adaptive compression strategies in challenging video conditions. While our current fixed combination works well across diverse tasks, developing dynamic α adjustment based on video characteristics (motion intensity, scene complexity) represents a promising direction for extreme scenarios like high-motion sports or videos with subtle visual elements. We view this as an exciting avenue for future work to enhance robustness in demanding real-world conditions.\n\n---\n\n**2. On-Device Deployment Results (NVIDIA Jetson Orin)**\n> **Weakness 2, Question 2.** Could you provide latency and power consumption metrics on representative ARM-based devices like a smartphone SoC or NVIDIA Jetson to validate practicality for edge computing?\n\nWe appreciate you highlighting the experiments on edge-devices. We deployed our method on NVIDIA Jetson Orin for streaming video understanding inference and measured actual latency, memory usage, and power consumption.\n\n| Length (s) | Context # | Method  | Memory (GB)      | FPS (frame/sec)     | TP (tok/sec)        | Power (J)         |\n|:----------:|:----------------:|:--------|:-----------------:|:-------------------:|:-------------------:|:-----------------:|\n| 100        | 18K              | Full KV | 13.8              | 6.2                 | 5.0                 | 1.6               |\n|            |                  | Ours    | 9.2 (1.5x ↓)      | 6.4 (1.0x ↑)        | 9.2 (1.8x ↑)        | 1.2 (1.4x ↓)      |\n| 300        | 54K              | Full KV | 26.6              | 5.0                 | 2.0                 | 4.7               |\n|            |                  | Ours    | 10.1 (2.6x ↓)     | 6.4 (1.3x ↑)        | 9.2 (4.6x ↑)        | 2.4 (2.0x ↓)      |\n| 500        | 90K              | Full KV | 39.0              | 4.1                 | 1.2                 | 7.5               |\n|            |                  | Ours    | 10.7 (3.6x ↓)     | 6.3 (1.5x ↑)        | 9.1 (7.3x ↑)        | 3.7 (2.0x ↓)      |\n| 600        | 108K             | Full KV | OOM               | OOM                 | OOM                 | OOM               |\n|            |                  | Ours    | 11.3          | 6.4             | 9.2             | 4.3           |\n\n*Table R2. Edge device performance on NVIDIA Jetson AGX Orin. StreamingBench 10-minute video samples (FPS 0.2-1) processed with Qwen-2.5-VL-3B, measuring memory, FPS, throughput, and power consumption.*\n\n`Table R2` demonstrates clear advantages of our Continual KV cache compression (CKV) approach: \n1. Peak memory usage remains nearly constant while Full KV grows linearly (13.8→39.0 GB), achieving 3.6x memory reduction (39.0GB vs 10.7GB) at 500 seconds. \n2. Prefill processing maintains stable performance (6.3-6.4 FPS) while Full KV degrades significantly (6.2→4.1 FPS). \n3. Generation throughput dramatically improves due to our fixed cache size, achieving up to 7.3x speedup (1.2 → 9.1) over Full KV.   \n4. Power consumption scales efficiently, providing nearly 2x energy (7.5 → 3.7) savings thanks to consistent computational load regardless of video length. \n\nMost critically, our method enables *processing of 600-second videos and beyond* where Full KV encounters OOM, demonstrating practical deployment capability on resource-constrained edge devices.\n\n---\n\n**3. Limited Multilingual/Multimodal Scope and Long-Term Semantic Retention**\n\n> **Weakness 2-3. Question 3** Multilingual performance unevaluated.  A key area for future work is multimodal compression. How do you envision extending InfiniPot-V to jointly handle visual, audio, and synchronized text (e.g., subtitles) streams?\n\n> **Question 4.** For ultra-long videos, repeatedly compressing the KV cache might lead to gradual semantic information decay. Have you analyzed this effect? Would a hierarchical caching mechanism be a solution?\n\n**Multilingual Extension:** We plan multilingual evaluation as crucial future work. SVBench [2] offers bilingual annotations (English, Chinese) for our next cross-lingual validation step.\n\n**Multimodal Extension:** Extending beyond visual compression represents a crucial direction for truly holistic multimodal understanding. For the audio modality, speech streams face similar linear context growth challenges—modern neural audio codecs adopt fixed ratios (40-160ms per token), creating the same scalability issues we address for vision. Fortunately, speech tokens exhibit high inter-token redundancy [3] while maintaining semantic similarity to language tokens, making them amenable to KV cache-level reduction using our approach.\n\nFor synchronized text streams like subtitles, we anticipate relatively fewer tokens compared to vision or speech (typically ~5x fewer text tokens than single-frame vision tokens). Additionally, synchronized text carries critical contextual information. Therefore, these tokens should be preserved with minimal compression.\n\nThe most intriguing opportunity lies in cross-modal compression at the KV cache level. While previous input-level approaches (LongVU) used cross-modal attention for compression, they required dedicated encoders and heavy computational overhead. Given that modern multimodal models like Qwen2.5-Omni accept interleaved vision/text/speech tokens as input, exploring how each modality's tokens influence interleaved KV cache representations could be a compelling next step for developing cross-modal redundancy metrics in KV cache compression.\n\n**Long-Term Semantic Retention:** This is a great suggestion for ultra-long videos requiring domain-specific exploration. For real-time user-AI video interactions [4], if users ask about current and recent content, a hierarchical approach with reduced compression for recent frames and increased compression for older content seems reasonable. However, for critical applications like surveillance footage where information decay could be problematic, a more selective strategy appears required. Using TaR-like metrics to detect sudden scene changes, we could selectively preserve transition frames with minimal compression or strategically offload them when necessary. This maintains critical temporal landmarks while operating within memory constraints.\n\n[1] Lin et al, StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding” Arxiv 24.11  \n[2] Yang et al, SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding, ICLR 25  \n[3] Zheng et al, FreeCodec: A disentangled neural speech codec with fewer tokens, interspeech 25  \n[4] Google, Project Astra"}, {"review_id": "1ABj97kZNh", "reviewer": "Reviewer_tw4N", "summary": "In this paper, InfiniPot-V algorithm, which is the training-free KV cache compression method, is proposed for streaming video understanding. The proposed algorithm employs two techniques, TaR and VaN, to keep the KV cache within a certain memory limit. Specifically, TaR measures the temporal redundancy score of each patch in past frames and remove the redundant patches. VaN is the metric for token-level semantic importance, which is actually L2norm of value embedding. The proposed algorithm keeps the tokens with high VaN scores (after pooling) in the memory. The experimental results including extensive analysis show that the proposed algorithm achieves comparable or better performances than conventional algorithms.", "questions": "The proposed algorithm may have the potential to support memory-efficient inference across various MLLMs, depending on their intended use and deployment environment. However, it would be even more compelling if the following aspects can be further improved. Of course, I would also be interested in hearing the authors' thoughts on my questions.\n\n---\n\n* L159, the definition of $H$ seems to be missing, although its meaning can be inferred from the context.\n\n* Figure 2 (b) compares the static patch similarities of keys and values. It is useful, but it would also be beneficial to compare the key similarity of non-static patches. This could demonstrate that static and non-static patches can be effectively distinguished based on $s^{TaR}$.\n\n* Table 4 shows the ablation study results. There seems to be many alternative designs worth comparing in addition to VaN Reverse. For example, instead of using TopK selection, it is possible to select tokens at fixed intervals after sorting them by scores. It may also be possible to identify important frames by computing the average VaN score for each frame, and then select a different number of tokens from each frame based on its importance. While Top-K selection is simple and effective, it would be interesting to see additional comparative experiments with other alternative methods.\n\n* It would be helpful to visualize the selection results based on the actual scores. This would provide an intuitive understanding of how effectively the proposed method identifies important tokens.\n\n\n* Typo (minor)\n1. There is a mixed use of $K_{recent}, V_{recent}$  and $K_\\textrm{recent}, V_\\textrm{recent}$ (e.g. Algorithm 1 and Eq.4)\n2. Similarly, $s^{VaN}$ and $s^{TaR}$ are also used with inconsistent notation. (e.g. Algorithm 1, Eq. 4, and L181)\n3. Figure 2 (c) VaN (spatial) -> VaN (Spatial), for consistency in capitalization.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "> **Strength**\n* This paper is well-written and easy to follow.\n* This paper demonstrates the validity of the design of the proposed algorithm through various empirical analyses. Furthermore, by applying the proposed method to different MLLMs and evaluating their performances on multiple benchmark datasets, it shows the effectiveness of the proposed InfiniPot-V.\n* The proposed algorithm is a practical solution that can be easily applied to existing MLLMs to enable memory-efficient inference. It achieves a better computation-performance trade-off compared to existing approaches. Moreover, the paper provides a detailed explanation of the method to ensure reproducibility.\n\n> **Weakness**\n* While the proposed method seems simple and effective, the proposed scoring methods are not particularly novel.\n* The mathematical notation could be improved for better clarity.", "was_revised": true, "final_justification": "During the rebuttal, the authors addressed my concerns well, and I appreciate all their efforts. I have not identified any additional weaknesses in the proposed algorithm, so I have decided to maintain my original rating of “accept.”", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive feedback. We particularly appreciate your recognition of our paper's clarity, comprehensive empirical analysis across multiple MLLMs and benchmarks, and the practical applicability of InfiniPot-V as a training-free solution for memory-efficient inference.\n\n---\n**1. Alternative Designs Exploration for VaN**\n> **Question 3.** There seems to be many alternative designs worth comparing in addition to VaN Reverse. For example, instead of using TopK selection, it is possible to select tokens at fixed intervals after sorting them by scores. It may also be possible to identify important frames by computing the average VaN score for each frame, and then select a different number of tokens from each frame based on its importance. While Top-K selection is simple and effective, it would be interesting to see additional comparative experiments with other alternative methods.\n\nThank you for suggesting several alternative design candidates for VaN token selection. Based on your suggestions, we implemented and evaluated a VaN alternative (per-frame) using Qwen2-VL-7B on the MLVU benchmark with a 6K memory budget. The experimental results are shown in `Table R1` below:\n\n| Method             | Topic | Anomaly | Plot | Needle | Ego  | AO   | AC   | Avg  |\n|--------------------|:-----:|:-------:|:----:|:------:|:----:|:----:|:----:|:----:|\n| Full KV              | 85.2  | 67.5    | 72.7 | 83.9   | 65.1 | 54.1 | 32.5 | 65.9 |\n| VaN Reverse        | 79.0  | 64.5    | 56.9 | 65.6   | 55.1 | 45.2 | 21.8 | 55.5 |\n| VaN                | 84.4  | 68.0    | 68.6 | 76.6   | 61.9 | 52.5 | 29.1 | 63.0 |\n| VaN Per Frame      | 85.2  | 68.0    | 62.7 | 77.8   | 60.5 | 51.4 | 30.5 | 63.2 |\n| VaN + Pooling      | 85.2  | 68.0    | 71.4 | 77.5   | 63.1 | 52.1 | 31.5 | **64.1** |\n\n*Table R1. VaN alternative designs comparison with Offline Video Understanding (MLVU) benchmark.  \nQwen2-VL-7B used with a 6K memory budget. (AO: Action Order, AC: Action Count)*\n\nThe results demonstrate that while VaN Per Frame (computing average VaN scores per frame) shows marginal improvement over standard VaN on average (63.2 vs 63.0), it exhibits notable weaknesses in detail-oriented tasks. For instance, in Plot QA, VaN Per Frame drops significantly compared to standard VaN (62.7 vs 68.6), likely because frame-level averaging loses fine-grained spatial information crucial for detailed visual reasoning. In contrast, our proposed adaptive pooling method consistently improves performance (68.6 → 71.4 in Plot QA), confirming the effectiveness of our layer-adaptive spatial pooling strategy over alternative designs.\n\n---\n**2. Analysis Suggestions and Notation Clarity**\n> **Question 2**. Figure 2 (b) compares the static patch similarities of keys and values. It is useful, but it would also be beneficial to compare the key similarity of non-static patches. This could demonstrate that static and non-static patches can be effectively distinguished based on s^TaR.\n\n> **Question 4.** It would be helpful to visualize the selection results based on the actual scores. This would provide an intuitive understanding of how effectively the proposed method identifies important tokens.\n\nThank you for the detailed suggestions regarding static patch analysis and visualization. \n\n**Analysis for Non-Static Patch:** We appreciate your recommendation to compare key similarity for non-static patches, which would indeed strengthen our argument about the effectiveness of TaR in distinguishing static and non-static content. In our observations, non-static content patches exhibit lower key embedding similarity compared to value embedding similarity, further validating TaR's discriminative capability. Due to constraints in this rebuttal, we cannot include the additional analysis figures in rebuttal, but we commit to including this comparison in the revised version.\n\n**Visualization of Selection Results:** This is a great suggestion that would provide a valuable intuitive understanding of our method's behavior. We observe that different attention heads at each layer exhibit diverse selection patterns, which would enable analysis of the specific regions each metric prioritizes per head (e.g., some heads focus on moving objects while others attend to specific background elements). We will include this analysis in the revised version.\n\n**Mathematical Notation:** Thank you for the meticulous notation corrections. We will address all the inconsistencies you identified, including the mixed use of s^TaR/s^VaN notations, K_recent formatting, and capitalization consistency in figures. These corrections will be incorporated throughout the revised manuscript."}, {"review_id": "nbbc3V61Yr", "reviewer": "Reviewer_QVy1", "summary": "The paper introduces InfiniPot-V, a novel framework designed to address memory constraints in streaming video understanding (SVU) by compressing the key-value (KV) cache in a training-free, query-agnostic manner. Unlike traditional offline video understanding methods that assume access to the entire video, InfiniPot-V processes arbitrarily long video streams in real-time, maintaining a fixed memory footprint. The framework employs two complementary metrics: Temporal-axis Redundancy (TaR), which prunes redundant tokens based on cosine similarity across time, and Value-Norm Importance (VaN), which prioritizes tokens based on the L2 norm of their Value vectors for semantic salience. InfiniPot-V achieves significant memory reduction (down to 6K tokens for 6K token contexts) while maintaining accuracy comparable to or better than existing methods.", "questions": "- How does InfiniPot-V perform in edge cases, such as videos with minimal temporal redundancy (e.g., highly dynamic scenes) or low semantic variance (e.g., static backgrounds)? Are there scenarios where TaR or VaN might fail to preserve critical information?\n-  The paper mentions a “layer-adaptive pooling strategy” in the VaN metric but provides limited details on its implementation. How is the pooling intensity determined for each layer, and what factors (e.g., model architecture, token distribution) influence this adaptation?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper presents a robust evaluation across diverse benchmarks (VideoMME, EgoSchema, MLVU, LongVideoBench, RVS Ego, RVS Movie), covering both offline and streaming tasks.\n- InfiniPot-V addresses a critical bottleneck in deploying SVU on memory-constrained devices, a pressing issue as video-based multimodal assistants become more prevalent.\n- The introduction of TaR and VaN as complementary compression metrics is novel, particularly in the context of streaming video. \n\nWeaknesses:\n- More comparisons with previous works on reducing the number of vision tokens in LMMs should be included to enhance the completeness of the experiments.\n- While the framework’s focus on on-device deployment is significant, the paper does not discuss compatibility with specific hardware (e.g., mobile GPUs or TPUs) or provide real-world deployment examples.", "was_revised": true, "final_justification": "Thank you for your response. After carefully considering both the content of the paper and the rebuttal, I have decided to keep my score unchanged.", "rebuttal": "Thank you for the thorough and constructive review. We particularly appreciate your recognition of our robust evaluation across diverse benchmarks, the critical importance of addressing memory constraints for on-device deployment, and the novelty of our TaR and VaN complementary compression metrics for streaming video understanding.\n\n---\n**1. Comparison with Vision Token Reduction Methods**\n\n> **Weakness 1**. More comparisons with previous works on reducing the number of vision tokens in LMMs should be included to enhance the completeness of the experiments.\n\n| Qwen2-VL-7B   | Topic | Anomaly | Plot | Needle | Ego  | AO   | AC   | Avg  |\n|:--------------------------|:-----:|:-------:|:----:|:------:|:----:|:----:|:----:|:----:|\n| Full KV                  | 85.2  | 67.5    | 72.7 | 83.9   | 65.1 | 54.1 | 32.5 | 65.9 |\n| FaskV [1]                | 82.9  | 63.5    | 55.5 | 72.1   | 56.8 | 47.5 | 21.4 | 57.1 |\n| LongVU [2] | 82.1  | 67.0    | 64.8 | 70.5   | 59.1 | 44.8 | 23.8 | 60.8 |\n| InfiniPot-V              | 86.3  | 68.0    | 72.7 | 80.3   | 63.9 | 54.1 | 35.4 | 65.8 |\n\n*Table R1. Comparison with vision token reduction methods (FastV, LongVU) in CKV framework across offline video understanding benchmark (MLVU). Qwen2-VL-7B with 6K memory budget used.*\n\nWe compared InfiniPot-V against two representative vision token reduction approaches (FastV and LongVU) under the Continual KV cache compression (CKV) framework across offline video understanding benchmark. FastV [1] prunes vision tokens at specific layers based on attention scores, permanently removing redundant vision tokens from subsequent layers forward pass. LongVU's STC [2] reduces redundant vision tokens at the input level using vision embedding similarity. Both methods were adapted to our CKV setting for fair comparison.\n\nAs shown in `Table R1`, both vision token reduction methods exhibit significant performance degradation when compressed to 12% of the original context length (50K → 6K tokens), with FastV dropping to 57.1% average accuracy and LongVU to 60.8%. In contrast, InfiniPot-V maintains performance nearly identical to Full KV (65.8% vs 65.9%) through its TaR and VaN methodology. These results demonstrate that our KV cache-level approach to reducing spatiotemporal redundancy is significantly effective compared to existing vision token compression methods for long video understanding.\n\n[1] Chen et al, An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, ECCV 24  \n[2] Shen et al, LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding, ICML 25\n\n---\n**2. On-Device Deployment Results (NVIDIA Jetson Orin)**\n> **Weakness 2.** While the framework’s focus on on-device deployment is significant, the paper does not discuss compatibility with specific hardware (e.g., mobile GPUs or TPUs) or provide real-world deployment examples.\n\nWe conducted deployment experiments on NVIDIA Jetson AGX Orin, a representative edge device platform, and measured actual latency, memory usage, and power consumption.\n\n| Length (s) | Context # | Method  | Memory (GB)      | FPS (frame/sec)     | TP (tok/sec)        | Power (J)         |\n|----------|----------------|--------|-----------------|-------------------|-------------------|-----------------|\n| 100        | 18K              | Full KV | 13.8              | 6.2                 | 5.0                 | 1.6               |\n|            |                  | Ours    | 9.2 (1.5x ↓)      | 6.4 (1.0x ↑)        | 9.2 (1.8x ↑)        | 1.2 (1.4x ↓)      |\n| 300        | 54K              | Full KV | 26.6              | 5.0                 | 2.0                 | 4.7               |\n|            |                  | Ours    | 10.1 (2.6x ↓)     | 6.4 (1.3x ↑)        | 9.2 (4.6x ↑)        | 2.4 (2.0x ↓)      |\n| 500        | 90K              | Full KV | 39.0              | 4.1                 | 1.2                 | 7.5               |\n|            |                  | Ours    | 10.7 (3.6x ↓)     | 6.3 (1.5x ↑)        | 9.1 (7.3x ↑)        | 3.7 (2.0x ↓)      |\n| 600        | 108K             | Full KV | OOM               | OOM                 | OOM                 | OOM               |\n|            |                  | Ours    | 11.3          | 6.4             | 9.2             | 4.3           |\n\n*Table R2. Edge device performance on NVIDIA Jetson AGX Orin. StreamingBench 10-minute video samples (FPS 0.2 - 1) processed with Qwen-2.5-VL-3B, measuring memory, FPS, throughput, and power consumption.*\n\n`Table R2` demonstrates clear advantages of our Continual KV cache compression (CKV) approach: \n1. Peak memory usage remains nearly constant while Full KV grows linearly (13.8→39.0 GB), achieving 3.6x memory reduction (39.0GB vs 10.7GB) at 500 seconds. \n2. Prefill processing maintains stable performance (6.3-6.4 FPS) while Full KV degrades significantly (6.2→4.1 FPS). \n3. Generation throughput dramatically improves due to our fixed cache size, achieving up to 7.3x speedup (1.2 → 9.1) over Full KV.   \n4. Power consumption scales efficiently, providing nearly 2x energy (7.5 → 3.7) savings thanks to consistent computational load regardless of video length. \n\nMost critically, our method enables *processing of 600-second videos and beyond* where Full KV encounters OOM, demonstrating practical deployment capability on resource-constrained edge devices.\n\n---\n\n**3. InfiniPot-V Edge Cases Exploration**\n> **Question 1.** How does InfiniPot-V perform in edge cases, such as videos with minimal temporal redundancy (e.g., highly dynamic scenes) or low semantic variance (e.g., static backgrounds)? Are there scenarios where TaR or VaN might fail to preserve critical information?\n\nWe agree that different video scenarios present varying challenges for TaR and VaN, and understanding these trade-offs is crucial for real-world deployment. To systematically explore edge case performance, we conducted experiments across StreamingBench [1] subtasks with distinct visual requirements: Attribute Recognition (fine-grained object detail retention), Event Understanding (temporal sequence preservation), and Causal Reasoning (complex visual information integration). Using Qwen-2.5-VL-3B, we swept the TaR ratio (α) from 0.0 (VaN only) to 1.0 (TaR only):\n\n| Method           | Attribute Recognition | Event Understanding | Causal Reasoning | Average |\n|------------------|----------------------|-------------------|------------------|---------|\n| Full KV             | 83.4                       | 78.1                   | 78.7                 | 73.4     |\n| Uniform Selection| 82.0                 | 74.2              | 74.2             | 70.8    |\n| VaN only (α=0.0) | 83.3             | 73.9              | 75.8             | 71.4    |\n| α=0.2            | **83.7**                 | 72.1              | 77.3         | 71.9    |\n| α=0.4            | 82.7                 | 74.5              | **78.1**             | 72.0    |\n| α=0.6            | 83.0                 | **77.0**          | 75.8             | **72.2**|\n| α=0.8            | 82.7                 | 76.4              | 76.6             | 72.1    |\n| TaR only (α=1.0) | 82.0                 | 76.8              | 75.8             | 71.2    |\n\n*Table R3. StreamingBench evaluation with TaR ratio (α) exploration. Qwen2.5-VL-3B used with 6K KV cache budget (Full KV: 25K-50K, 0.2 - 1 fps)*\n\n`Table R3` reveals how task requirements influence metric effectiveness: For Attribute Recognition, TaR-heavy approaches underperform compared to VaN-focused selection (82.0 vs 83.3), suggesting that when fine-grained object details matter, temporal redundancy reduction alone is insufficient. Conversely, for Event Understanding, VaN-only approaches show clear limitations compared to TaR-weighted combinations (73.9 vs 77.0), indicating that semantic importance scoring can miss crucial temporal sequence information.\n\nOn the other hand, for complex Causal Reasoning requiring diverse visual cues, our balanced combination achieves superior performance (78.1 with α=0.4), and overall our approach consistently outperforms baselines (72.2 vs 70.8). This demonstrates that while edge cases occur based on task characteristics, TaR and VaN together show effective performance for complex tasks. Developing adaptive combination strategies based on task-specific requirements represents an intriguing direction for future work.\n\n---\n\n**4. Layer-Adaptive Pooling Strategy Clarification**\n> **Question 2.** The paper mentions a “layer-adaptive pooling strategy” in the VaN metric but provides limited details on its implementation. How is the pooling intensity determined for each layer, and what factors (e.g., model architecture, token distribution) influence this adaptation?\n\nThe pooling intensity is determined by analyzing the Coefficient of Variation (CV) of VaN distributions at each layer, as detailed in `Algorithm 2` (lines 16-21).\nSpecifically, we compute CV = σ/μ for each layer's VaN values and apply a mapping function g that assigns pooling kernel sizes based on pre-computed thresholds $\\{\\tau_1, \\tau_2, \\tau_3\\}$\n> **Layer-Adaptive PoolSize assignment based on CV:**\n>\n> ```python\n> # Based on spatial variance (CV), assign pooling size\n> if CV < τ1:\n>     PoolSize = 7   # high spatial locality\n> elif τ1 ≤ CV < τ2:\n>     PoolSize = 5\n> elif τ2 ≤ CV < τ3:\n>     PoolSize = 3\n> else:  # CV ≥ τ3\n>     PoolSize = 1   # low spatial locality\n> ```\n\nAs shown in `Fig. 3 c`, early layers exhibit strong spatial locality (low CV), requiring larger pooling kernels to preserve local spatial patterns, while deeper layers show more dispersed distributions (high CV), requiring minimal pooling to maintain fine-grained semantic distinctions. This adaptive approach determines pooling intensity based on the distribution characteristics of Value embedding norms (VaN) at each layer.\n\n[1] Lin et al, StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding Arxiv 24.11"}]

Author rebuttal:

Meta-review: Final rating: 4. Borderline Accept/ 4: Borderline Accept/ 5:  Accept/ 4: Borderline Accept. The paper introduces InfiniPot-V, a training-free, query-agnostic KV-cache framework for streaming video that enforces a fixed memory cap by pruning temporally redundant tokens (TaR) and retaining semantically important ones (VaN). It reduces peak memory by up to 94%, sustains real-time generation, and matches or exceeds full-cache accuracy across multiple MLLMs and long/streaming benchmarks, enabling on-device video assistants.

Reviewers agreed the paper tackles a key practical bottleneck for on-device MLLMs (e.g., real-time video assistants) and demonstrates strong engineering pragmatism. However, they questioned thin novelty (TaR parallels LongVU’s “drop static frames”; VaN echoes high-entropy/importance heuristics), shallow comparisons (only ReKV on RVS; not SOTA), and weak ablations lacking simple baselines. They also flagged robustness/scope gaps—improving Qwen2-VL but hurting Qwen2.5-VL; heuristics brittle under rapid cuts or subtle cues; English-only, controlled settings; no edge-device validation; no audio/text KV handling—plus missing broader token-reduction baselines, unclear notation, and absent hardware compatibility/deployment examples.

After the rebuttal, reviewers indicated their concerns were satisfactorily addressed and leaned toward acceptance. The ACs concur, citing the work’s practical impact and the technical soundness of a training-free, query-agnostic framework with complementary TaR and VaN. Therefore, the area chairs agree to accept the paper and recommend the authors refine it according to the reviewers’ suggestions for the camera-ready version.
Decision: Accept (poster)