QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design
Authors: Benjamin Schneider
Dongfu Jiang
Chao Du
Tianyu Pang
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: Long video understanding has emerged as a crucial capability in real-world applications
such as meeting summarization, video surveillance, educational lecture analysis, and content
moderation. However, it remains computationally prohibitive for VideoLLMs, primarily due
to two bottlenecks: 1) sequential video decoding, the process of converting the raw bit stream
to RGB frames can take up to a minute for hour-long video inputs, and 2) costly prefilling
of up to several million tokens for LLM inference, resulting in high latency and memory
use. To address these challenges, we propose QuickVideo, a system-algorithm co-design
that substantially accelerates long video understanding to support real-time downstream
applications. It comprises three key innovations: QuickCodec, a parallelized CPU-based
video decoder that achieves 2–3× speedup by splitting videos into keyframe-aligned intervals
processed concurrently. QuickPrefill, a memory-efficient prefilling method using KV-cache
pruning to support more frames with less GPU memory; and an overlapping scheme
that overlaps CPU video decoding with GPU inference. Together, these components reduce
the time required to process a long video input by a minute, enabling fast, efficient video
understanding even on limited hardware. Experiments show that QuickVideo generalizes
across durations and sampling rates, making long video processing feasible in practice.

Published in Transactions on Machine Learning Research (01/2026)
QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design
Benjamin Schneider* University of Waterloo
Dongfu Jiang* University of Waterioo
Dongjh.Honn@Sunaaeerlo.oco
Chao Du SeaAT Lab
duchao@sea.com
Tianyu Pang SeaAT Lab
tianyyuang@Beea.com
Wenhu Chen University of Waterloo
Wenhu. Chen @ueaaerloooo
Reviewed on OpenReview: https: openreview. net/ forum? id- Rpcagzesue
Abstract
Long video understanding has emerged as crucial capability in real-world applications such as meeting summarization, video surveillance, educational lecture analysis, and content moderation. However, it remains computationally prohibitive for VideoLLMs, primarily due to two bottlenccks: 1) sequential video decoding, the process of converting the raw bit stream to RGB frames can take up to a minute for hour-long video inputs, and 2) costly prefilling of up several million tokens for LLM inference, resulting in high latency and memory use. To address these challenges, we propose QuickVideo, a system-algorithm co-design that substantially accelerates long video understanding support real-time downstream applications. It comprises three key innovations: QuickCodec, a parallelized CPU-based video decoder that achieves 2 3x speedup by splitting videos into seyframe-aligned intervals processed concurrently. QuickPrefill, a memory--ffiiient prefilling method using KV-cache pruning to support more frames with less GPU memory; and an overlapping scheme that overlaps CPU vidco decoding with GPU inference. Together, these components reduce the time required to process long video input by minute, enabling fast, efficient video understanding even limited hardware. Experiments show that QuickVideo generalizes across durations and sampling rates, making long vidco processing feasible in practice.
1 Introduction
Video data has become the dominant modality for conveying information online.
As of 2023, video data accounts for two thirds of all data transmitted over the Internet (Su et al., 2024). Much of this data is "long video" ranging from minutes to hours in duration, from online conferencing, gaming, social networking, and movic streaming. This torrent of online vidco data demands efficient and automated understanding for problems such as content moderation (Akyon & Temizel, 2022), real-time surveillance (Yuan al., 2023), and accessibility (Liu al,, 2021). Video Large Language Models (VideoLLMs) (Bai al., 2025; Zhu al., 2025a; Chen al., 2025a) have emerged as powerful tools to support these downstream
* "The first two authors have equal contribution

---

Published in Transactions on Machine Learning Research (01/2020)
Qurck (OPU) Omnckthccona
Compressed video stream
GPU Prefill
Pruned KV Cache for LLM Decoding
Figure An overview how QUICKVIDEO overlaps video decoding on CPU (QUICKCODEC) and prefill on GPU (QUICKPREFILLL). QUICKCODEC concurrently processes intervals of the compressod video bit stream. QUICKPREFILL uses independent groups of frames, therefore it can begin prefill once the first frames are decoded, outputting carefully selected KV vectors. As QUICKCODEC loads frames synchronously, QUICKPREFILL process the next prefill group immediately. This results in video decoding and prefill being almost enircly overlapped.
tasks. By natively processing entire video inputs, VideoLLMs exhibit phenomenal potential to understand and reason about vidco content, offering practical solution for managing and extracting information from the exponentially growing flood of video data across the Internet (Zhou et al., 2024b).
However, using VideoLLMs for long video understanding suffers from several efficiency challenges. First, the entire vidco must be decoded from raw bitstreams into RGB frames before the model can begin processing. Current frameworks require up minute to decode the frames from hour-long video input, introducing high latency before any context prefill start. Second, the prefilling step itself is both computationally and consume
memory intensive (Weng et al., 2024). Each frame representing an instantaneous moment hundreds of tokens in the model context (Zhu et al., 2025b; Chen et al., 2025b). As result, even a modest frame rate (e.g., 2 FPS) for an hour-long video can lead to millions of tokens, far exceeding the memory budget of standard GPUs. Qwen2.5-VL. (Bai et al., 2025) introduced several architecture modification to accelerate video processing. However, using Qwen2.5- VL--7, prefilling an hour-long HD video sampled at its native 2 fps still requires more than the 80 GB of memory offered by an A100/H100 SXM4 GPU, Even after roducing the frame sampling rate by 4X, prefilling still takes over 25 seconds datacenter-gee These inefficiencies result in a frustrating user experience, characterized by long delays and prohibitively high hardware requirements. Users with limited computational resources are effectively excluded from accessing the long video understanding capabilities of VideoLLMs.
hardware.
mitigate the computational overhead long video VideoLLMs use extremely low frame sampling rales when processing long video inputs, instead of their native FPS Bai al. (2025); Chen et. al. (2025a). Frames are sampled as much as minute apart during hour-long video understanding (Zhu et al., 2025b; Chen et al., 2025b). A minute gap between sampled frames can result in missing crucial video segments required for an understanding task. Low frame sampling rates also make fine-grained temporal and motion understanding impossible, as intervening frames are mostly removed (Nie et al., 2024). Effective long video understanding thus requires loading and prefilling thousands of frames while preserving temporal continuity. Developing faster, more efficient VideoLLMs is critical for enabling comprehension of videos that span hours.
Currently, vidco decoding and context prefilling are treated as disjoint and sequential stages the VideoLLM pipeline. Moreover, video decoding is largely overlooked, despite contributing substantially to end-to-end latency. To remedy this, we introduce QUICKVIDEO understanding. QUICK VIDBO reduces the latency and resource requirements of these key bottlenecks in long video understanding, Our framework empowers fast video understanding on video inputs consisting of hundreds of thousands of frames, while maintaining the sampling rates required for fine-grained understanding. QUICKV VIDEO introduces three core contributions for accelerating long video understanding in VideoLLMs:
framework for faster, memory-efficient long video
(1) System-Level + QUICKCODEC: drop-in replacement video decoder designed for VideoLLMs. By redesigning video decoding for VideoLLMM frame sampling, achieve 2-3x speedup compared to existing
2

---

Published in Transactions on Machine Learning Research (01/2026)
libraries when loading hour-long video inputs.
(2) Aegeri - QUICKPREFILI: a group-based prefilling strategy combined with key-value (KV) cache pruning, which significantly reduces both computation time and memory usage during the prefilling stage, while incurring less than 3% accuracy degradation in most benchmarks.
(3) Co-Design - Overlapped Execution Scheme: the strategy lightly couples CPU-based QUICK and GPU-based QUICK
enabling near-complete overlap to maximize efficiency. QUiCKVIDRY reduces the time to infer 30 minute video input by more than 3x, from 69.7 seconds to only 20.0 seconds. The results demonstrate the effectiveness of our system-algorithm co-design.
2 Background
We provide an overview of VideoLLM inference and key concepts in video processing. Although details vary, this background is broadly applicable to standard VideoLLL architectures and video standards. For clarity, we use "video decoding" to describe the process of decoding the compressed video into a tensor of video frames, and use "LLM decoding" to denote the process of auto-regressive decoding of a large language model.
2.1 VideoLLM Inference
VideoLLMs must first decode a compressed video into packed frame tensor before lokenization. The resulting raw frames are then passed through a visual encoder, which converts them into video tokens suitable for input to the LLM. Unlike text preprocessing, which relies on lightweight tokenizers, video decoding is inherently slow on both CPU and GPU due to its sequential nature (Wiegand et al., 2003; Sullivan et al., 2012). Despite this, prior work L.M video understanding has largely overlooked the lateney incurred by this stage. Following preprocessing, the generation process of a VideoLLM consists of two stages: (1) Prefill, where both video and text tokens are processed to compute key-value (KV) caches for each transformer layer; and (2) LLM decoding, where tokens are generated autoregressively using the stored KV representations. The prefill stage is computationally expensive due to the quadratic complexity O(n²) of self-attention over long sequences, while the decoding stage is memory- iemooryinteesive as it requires storing and repeatedly accessing the full KV cache.
and x = (xi,.... xxx1) represent the video and lext tokens, respectively, with video tokens preceding the text. For each transformer layer I E {1,..., L), the KV cache comprises lensors k(()) v(O) ((X*++|XX* [=XX|||[[Xxxxxx" where n, is the number of attention heads and d, is the per-head dimensionality. For example, let SB InternVL-2.5 (Chen et al., 2025a) model process a one-hour video at frame per second, the total required memory is around 400GB (see subsection D.2). This memory footprint makes KV cache storage critical bottleneck in VideoLLM inference, significantly limiting the maximum processable video length and constraining the feasible batch size.
Let X" = (x""....,
2.2 Long Video Processing
Multimedia container formats like MP4 or MKV bundle all the elements required for media playback, including video streams, audio streams, subtitles, and metadata (Koenen, 1999). In these containers, videos are stored compressed bit streams (Koenen, 1999; Wiegand et al., 2003). Tn multimedia processing libraries like FFmpog (Tomar, 2006), vidco decoding is described by queue D that enqueues fixed-sized blocks of the bit stream, called packels, as inpul and dequeues video frames. We denote bit stream S = (Po. ,I....,PP-- and a video V = (fo. fı...., ) as ordered lists of packets and frames, respectively. Each frame fi is tensor containing 8-bit integers of shape (3 x h x w), where h is the pixel height and w is the pixel width. In general, packels are not frame aligned, enqueueing a single packet to the decoder can cause the decoder to output zero, one or polentially multiple frames (Wiegand et al., 2003). This is because frames require varying amounts of information encode, and therefore cannot be aligned to fixed-sized packets. Furthermore, video frames are not encoded independently in bit stream, surrounding frames contain redundant information. Therefore, the video encoder encodes the residual of the frame in the bitstream, instead of the frame itsell (Wiegand et al., 2003; Sullivan et al,, 2012). For this reason, video decoding is a largely sequential process, where
The encoded residual of frame may require information from previous or future frames decode (Wiegand et al., 2003).
3

---

Published in Transactions on Machine Learning Research (01/2026)
previous frames must decoded first and then the residual information encoded in the bit stream can be used to decode the next frame (Wiegand et al., 2003). Although the video encoder may also reorder frames in the bit stream for efficiency, the decoder always outputs frames in the order that they should be displayed during playback (Tomar, 2006).
Packet and Frame Metadata. Although metadata not. directly encoded the bit stream or frame itself, for simplicity, we denote metadata corresponding to packets or frames as if they are fields. The packet and frame metadata is stored in the container, not the bit stream (Koenen, 1999). The presentation timestamp (pts) of frame is a 64-bit unsigned integer that represents when the frame should be displayed to user (Tomar, 2006). Most formats do not include global frame positioning information in metadata. We instead use Equation (1) to rescale the presentation timestamp for frame f to obtain f's index i in V.
(m (m-1): f.pts ptsmar ptsmin
(1)
pla and minn the minimum and maximum presentation timestamp for the video stream. Each packet has keyframe flag that marks that video decoding can begin from its position (Koenen, 1999; Tomar, 2006).
2.3 Keyframes and Seeking
As video decoding relies on surrounding frames, it is a sequential process. However, during playback, users may want to navigate and skip through the video. To support this, the bit stream contains keyframes, which act reset points from which video decoding can begin. Keyframes are encoded at semi-regular intervals in S, usually few seconds apart. To use keyframes to navigate in S, we use the SEEK subroutine. SEEK(S, pts) finds the keyframe packet p; E S such that decoding from p: yiclds all f such that f-pts 2 pts. However, seeking introduces overhead, as it requires flushing decoder buffers and reinitializing state (Tomar, 2006).
Algorithm Seek-based video decoding
Require: Bit stream S, Ordered set I, Video Decoder D, h, Allocate memory block F of size [I] x 3 x
2: for i E I do
3: Estimate pts of fs
4: Pt - SEEK(S,, pls)
Seek to the keyframe before St in S
5: Decode Pis Pitl..... until D outputs f;
Write fr to F
7: return F
Algorithm 1 is standard approach when decoding video for machine learning (Distributed (Deep) Machine Learning Community, 2019; PyTorch Team, 2025). For each desired frame fü given by selected indices in I S (1,2...., m 1}, the algorithm does the following: It seeks for the keyframe closest to fa in S, and then it decodes packets until D outputs /t- Ss is saved in the buffer F. This algorithm performs well for sparse access patterns, as there are large gaps between desired frames, seoking before dccoding each frame is idcal.
3 Method
In this section, we introduce QUICKVIDEO, which consists of three main components:
3.1
QuickCodec: Long Video Decoding for VideoLLMs
Given a bitstream for a video V f2...,,mm. fm); where each frame has height h, width w, and the desired degree of concurrency is c, our goal to compute F such that for all j E 0, 1,...,1Z] - we have F, = Sz() where I S 1,2, m In other words, F is a packed tensor containing all the frames selected by We assume m is known from containcr metadata or can be estimatod using ptsmaz and pf8min.
The efficiency of our algorithm relies two key observations:
4

---

Published in Transactions on Machine Learning Research (01/2026)
(1) It is faster to use cores to decode short videos than to cores to sequentially decode single long video. Video decoding for human playback focuses on the latter case, as humans watch earlier frames while later frames are decoded. However, due to inter-frame dependencies, sequential video decoding is difficult to parallelize (Wiegand et al., 2003). In contrast, VidcoLLMs require the entire vidco input to be loaded upfront. Therefore, we can decompose the loading of a long video V into loading short. videos that, collectively span V. However, decoding cannot start at arbitrary frames-it it must begin at keyframes. The KEYFRAME INTERVALS subroutine (Appendix A) parses the metadata of and computes e approximately equal-length intervals, starting and ending at keyframes, that cover the entire video. We parallelize over these intervals in Algorithm 2.
(2) VideoLLMs typically sample frames at a short, regular interval, usually 2 FPS (Bai et al., 2025), which is often smaller than the interval between keyframes in standard codecs. Consequently, bask- decoding must still decode from all keyframes, leading to redundant seeks. Our algorithm requires only one seek operation per core, instead of a number of seeks proportional the number of frames.
Algorithm QUICKCODEC
Require: Bit stream S, Ordered set I, Video Decoder D, h, W, C, m intervals that start and end on keyframe
J KEYFRAME INTERVALS(S.0)
Allocate shared memory F of size |Z] x3xhx
Initialize memory offsct map M
4: for & {0, 1.......,I22 - 1} do
5: M[[[k]] - k
Maps frame index to memory offset in F D Parallelize over intervals Seek to the packet at the start of the keyframe interval
>
for all (ptsstart: ptsend) E J in parallel do
Pt SEEK(S, plo
repeat
9:
while D not empty do f + D.dequeue()
10:
11:
if f-pts ptsend then
12:
break
13:
Compute with equation if i in M then
14:
15:
- M[+]
Get the memory offsct for fi in F Write frame into shared memory tensor
16:
F. -
17:
Deenqueue(ps)
18:
P; + Pi+1
Get next packet in bit stream S
19:
until f-pts plsend
20:
return F
Algorithm 2 presents the core of our video decoding algorithm. The algorithm begins by using metadata to compute keyframe-aligned intervals that span the video (line 1). Lines 2 5 initialize a shared memory block F and compute a dictionary M that maps indices of selected frames to unique memory offsets in F. We then decode the long video in e parallel intervals (lines 6-19). Video decoding starts by seeking to the start of each interval plsstart, which is guaranteed to be keyframe (line 7). Packets are enqueued for decoding (lines 17-18) until the decoder yields frames for processing (line 9). If the timestamp of dequeued frame is greater than equal to the interval endpoint ptsends parallel processing ends (lines 11-12, 19). As the intervals in J span S, ptsmin and ptsmaz are given by the smallest and largest values in J, respectively. We use Equation (1) to compute the index i of f (line 13). Finally, we save I to F if f is a selected frame (lines 14 14-16). Because decoding from a keyframe yields all frames with greater pts values, and D outputs frames in pts order, when the parallelized loop exits (line 19), all selected frames with pts in the interval [pts ptsend) will have been output by D and saved to F. Thus, as J spans S, when the algorithm returns, F will contain all selected frames.
5

---

Published in Transactions on Machine Learning Research (01/2026)
3.2 QuickPrefill: Efficient Group- Based Prefilling for VideoLLMs
After decoding the video bitstream into packed tensors, they are fed into the VideoLLM for inference. However, LLM generation with long contexts a well-known challenge due to high memory usage and computational cost. To address this, we introduce QUICKPREFIL, a grouped prefilling and KV cache pruning method that accelerates processing and significantly reduces memory requirements.
denote the sequence of video tokens, where |X"| is the total number of tokens, and each token x E F² is a d-dimensional vector. To reduce memory overhead during prefilling, we partition the video token sequence into G disjoint groups: X" = X"...., X2, where cach group X; contains approximately Ng- G once, we sequentially prefill each group and store the corresponding key-value (KV) cache K vl for each transformer layer I. This strategy significantly reduces peak activation memory usage by a factor of G and remains effective even when combined with efficient attention mechanisms such as FltshAttention. Empirically, it enables hour-long video understanding while keeping GPU memory usage within practical limits (c.g., reducing memory overhead by over 100 GB; see subsection D.2).
Group-Based Prefill Let X" =
X tokens. Instead of processing the entire sequence at
Group-Based KV Cache Pruning While group-based prefill reduces peak activation memory, the KV cache memory remains a major bottleneck. To address this, we prune unimportant KV cache vectors when processing each group, maintaining retention ratio p (0, 1]. This reduces KV cache memory usage by factor of ]
The pruning decision is based on an importance score function s, which produces an ordered list of KV entries. We select the top-k KV entries until the retention ratio is satisfied:
K 1
- where 1 = TopK (s(K( v(9)
= -...kppNNN. N.)N)
where TopK returns the indices of the top-k entries. We consider several heuristic importance functions s from prior works (Devoto et al., 2024; Guo et al., 2024; Zhang et al., 2023). In this paper, we primarily use the following three: 1) Key Norms (small):
-L2(K!") 2) Value Norms: L2(V!"); 3) Attention Scores: s matmul(K))) ) Here, L2 denotes the L2-norm function, and Qt)) E R (x(gxx)d is the query vector of text tokens in layer I. We adopt Key Norms (small) as the default importance function in QUICKPREFILL due to its strong performance. Using key norms also allows to use efficient fused attention implementations, such as FlashAllention (Dao, 2023).
3.3 Overlapping QuickCodec and QuickPrefill
The preceding sections introduced two complementary components: QUICKCODEC for CPU-based video decoding and QUICKPREFILL for GPU-basod group-wise prefilling. However, running these components sequentially underutilizes resources- -Grus remain idle during video decoding, and CPUs are underutilized during prefilling. To address this inefficiency, we propose an overlapped execution scheme that enables concurrent processing across CPU and GPU resources.
To achieve this, we slightly adapt frame loading: Instead of using e cores to load e intervals, we divide V into intervals, where >> c, using KEYFRAME INTERVALS(V, s). We then load the frames from the s intervals using e cores, prioritizing earlier intervals so that frames corresponding are available sooner. This allows us to exploit QUICKCODEC's fast video decoding while ensuring that early frames in V are prioritized for QUICKPREFILL. Once the frames required for the first group are loaded, QUICKPREFILL begins processing immediately, while QUICKCODEC contimues decoding subsequent frames in the background. After QUICKPREFILL finishes processing à group, it stores the resulting KV cache and checks whether QUICKCODEC has loaded the frames required for the next group. If so, QUICKPREFILL starts processing the next group immodiately. This design forms a producer-consumer pipeline between CPU decoding and GPU prefilling, ensuring the GPU is only idle if it is waiting for the CPU to finish decoding the next set of frames.
the first blocks of video
6

---

Published in Transactions on Machine Learning Research (01/2026)
The performance improvement of this overlap strategy can be formalized. Let tdee and frrefat denote the time to decode and prefill a group of frames, respectively. With our overlap strategy, the execution time is modeled by Equation (3).
total Σ max (prefitt + A
A is small latency introduced by QUICKCODEC's metadala parsing and startup. Since each group contains small number of frames and tokens, this strategy achieves near-optimal overlap between CPU and GPU resources, resulting in substantial speedup for hour-long video processing. Note that some VideoLLMs include additional preprocessing steps (e.g., position embedding calculation or normalization), which we do not include in this analysis (Bai et. al., 2025).
4 Experiments
We evaluate QuickVidoo's performance on practical long video understanding tasks. section 4.1, we benchmark QUICKCODEC against existing Irameworks. We also examine the limitations of QUICKCODEC, identifying cases where seek-based frameworks (Algorithm 1) outperform we evaluate the performance of QUICKPREFILL across four long video understanding benchmarks, analyzing the trade-off between accuracy and efficiency. Finally, in section 4.3, we demonstrate that the prefill and video decoding stages can be almost entirely overlapped, effectively reducing end-to-end inference time by nearly a minute for long video imputs.
method. Next, in section 4.2,
4.1 QuickCodec Results
Video Loading Speed. We benchmark the time required load an hour-long 24 FPS 1920x1080p HD video, sampled at 1 FPS and resized to 448x448 pixels. The video is one-hour segment of popular movic encoded with default FFmpog settings using H.264, the most widely used codec (Kerdranvat et al., 2020). Sampling frames at 1-2 FPS is standard practice in VideoLLMs, balancing computational efficiency with task performance (Bai et al., 2025). We resize frames to 448x448 pixels, which matches the maximum per-frame resolution used most VideoLLMs (Zhu et al., 2025a; Chen et al., 2025a). All experiments are
Time Load Hour-long Video
160
155.99
Decord
+9.61
TorchCodec
140
TorchVision Resize
27.39
QuickCodec (ours)
120
Realistic Configurations
Time (seconds)
100
96.27
19.62
80
14:.43
60
+9.64
43.88
41.36
40
+9.66
20
29.75
02.27
8.83
Number of Cores
Figure 2: Speed comparison of Decord, TorchCodec (with Resize), and QUICKCODEC when loading hour-long videos. We ablate across different levels of parallelization (core counts).

---

Published in Transactions on Machine Learning Research (01/2026)
conducted an AWS m7a. 16xlarge instance. Each timing result is averaged over five runs, with 95% confidence interval no greater than 0.5 seconds.
We compare QuickVideo against two widely used video decoding frameworks:
Decord (Distributed (Deep) Machine Learning Community; 2019): multimedia loading framework designed for machine learning applications. While no longer actively maintained, Decord remains integrated into popular libraries like Hugging Face's Transformers (Wolf et al., 2020) and, by extension, inference frameworks such vLLM (Kwon al., 2023).
TorchCodec (PyTorch Team, 2025): A work-in-progress library from the PyTorch team designed to offer faster multimedia processing than TorchVision (maintainers & contributors, 2016). TorchCodec lacks some features of mature frameworks, such as built-in support for frame resizing. Thus, we report timings that combine TorchCodec loading with a resizing step via TorchVision. TorchCodec not optimized for decoding with more than 16 cores; we observe that increasing core count beyond 16 can even degrade performance. In collaboration with the TorchCodec leam, algorithm 2 has been implemented into TorchCodee, we do not use this setting in our timings.
As shown in Figure 2, QUICKCODEC outperforms other libraries across varying core counts. While other frameworks plateau at 16 cores, QUICKCODEC: scales up to 64 cores. We highlight the 16- and 32-core cases as these are the most common configurations in practical deployments; most compute providers allocate between 16 and 32 CPU cores per GPU (Google, 2025; Amazon, 2025; Microsoft, 2025). At 16 32 cores, QUICKCODEC is 2-3x faster than other libraries when loading an hour-long video, reducing video loading time by over 20 seconds.
Speed Across Video Durations. Our framework relies pre-computing intervals and sufficient keyframes for paralleliza- tion. Therefore, we expect reduced benefits for shorter videos. We benchmark QUICKCODEC on videos of varying lengths, from minute hour, using the same source video (an hour-long movie) cut to different durations. All tests use FPS sam- pling and 16 cores for video decoding. Results are averaged over five runs on an AWS m17a.16xlart instance, with 95% confidence interval of at most 0.2 seconds. We find that QUICK- CODEC is consistently faster than other frameworks for videos longer than 1 minute (Figure 3). Its advantage grows with video Bonth-QUicKCon
Time to Load across Video Durations
Decord
TorchCodec
40
TorchCodec+Resie
QuickCodec (ours)
30
Time
10-
is 1.7x faster than Decord for 10-minute video and 2.1x faster for 1-hour video. We fur- ther discuss scenarios where reek-based decoders outperform QUICKCODEC in Appendix B.
0-
min
5 min
10 min
30 min
60 min
Video Duration
Figure 3: Video decoding performance
across different video durations (1 FPS sam- pling).
4.2 QuickPrefill Results
We evaluate QUICKPREFILL on four long video understanding benchmarks, with videos ranging from minutes to hours: VidcoMME (Fu et al., 2024), LongVideoBench (Wu et al., 2024), LVBench (Wang et al., 2024), and MLVU (Zhou et al., 2024a). All generations use greedy sampling, and results are reported via the lmms-eval model (Bai et al., 2025)
framework (Zhang et al., 2024). Experiments are run on the Qwen2.5-VL- using a single A100 (40GB) GPU with 8 replicas.
Effectiveness of Different KV Cache Pruning Methods. We evaluate the impact of various KV cache pruming stratogics on model accuracy, as summarized in Table 1. We compare several pruning techniques against a no-pruning baseline (p = 1), fixing the relention ratio p at 0.5 and the group size at 16 frames. The Key Norms (small) method achieves the best balance between efficiency and accuracy, retaining over 95% of the model's original performance while halving the KV cache size and computation. In the 1024-framo sctting, it retains over 98% of the original performance. Notably, this method outperforms muery-attention- token selection strategies. While prior work (Devoto al., 2024) has shown that negative T.2 norms of keys
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Efffectiveness of different KV cache pruning methods the group-based prefilling scenario. We use the Key Norms (small) as the default KV cache pruning method for QUICKPREFILL due its superior performance and query- query-agonistic nature.
Group Size # #Frames
VideoMME w/o subtitle
LongVideoBench val
LVBench test
MLVU dev
KV Pruning
Avg
Performance
Frames
62.41
59.69
40.09
63.86
56.51
100.00%
16
Value Norms
0.5
47.63
35.98
30.92
31.38
36.48
64.55%
16
Attention Scores
0.5
58.63
52.95
37.83
59.87
52.32
92.58%
Key Norms (smalt)
60.56
56.17
37.70
62.34
54.19
95.90%
128 Frames
66. 46
60.96
42.87
66.86
59.27
100.00%
Value Norms
48.56
37.32
30.73
38.51
38.78
65.42%
16
Attention Scores
0.5
60.96
55.20
39.70
64.36
55.06
92.89%
16
Key Norms (smalt)
0.5
63.41
58.19
39.57
64.99
56.54
95.39%
256 Frames
65.78
61.56
43.90
68.65
59.97
100.00%
16
Value Norms
0.5
48.33
38.89
31.38
37.74
39.08
65.17%
16
Attention Scores
0.5
62.52
57.22
41.96
67.27
57. 24
95.45%
16
Key Norms (small)
0.5
64.04
60.21
41.90
66.73
58.22
97.08%
1024 Frames
62.00
60.43
42.29
63.48
57.05
000.00%
16
Value Norms
0.5
47.37
33.66
29.18
32.65
35.71
62.60%
Attention Scores
62.22
58.49
42.03
64.45
56.80
99.56%
16
Key Norms (small)
0.5
59.99
61.59
40.80
64.76
56.78
99.53%
Effect of Group Size on Model Performance (p=0.5)
Effect of Retention Ratio on Model Performance (Group Size=16)
61
101.0%
Deatret grosping Traed p=3.0)
100.0%
100.0%
97.5%
Average Accuraty (%6
99.0%
Average Accuracy (%)
95.0%
59
Performance (%)
Performance (%)
92.5%
98.0%
58
90.0%
97.0%
87.5%
57
Average Accuracy Average (p=1.0)
85.0%
56
Relative Performance
Average Accuracy
82.5%
Performance (p=1.0)
Relative Performance
94.0%
80.0%
4
8
16
32
64
128
0.1 0.2
0.6 Retention Ratio (p)
0.4
0.8 0.9
1
Group Size
Figure 4: Ablation study on group size and retention ratio. Data from Table 2.
correlate strongly with attention scores in text-only LLMs, our results extend this finding to VideoLLM prefilling, highlighting the generalizability and practical utility of key norm-based pruning.
We also conduct ablation studics on group size and retention ratio p (sce Appendix E). As shown Table and Figure 4, group size has minimal impact on model performance, while increasing p consistently improves accuracy, approaching the no-pruning baseline. Smaller group sizes reduce activation memory, while lower p

---

Published in Transactions on Machine Learning Research (01/2026)
QuickVideo with Qwen2.5-VL-88
Baseline
QuickVideo
Vidde Loading
Prefill
QuickVideo (overlapped)
LLM decoding
Video loading
10
20
30
40
Time (seconds)
Figure 5: Latency breakdown for video loading, prefill, and LLM decoding in end-to-end inference. We compare a baseline Qwen2.5-VT. Bai et al. (2025) implementation, the same model with QUICKPRRFILL, PRefILL. and QUICKCODEC, and our block-overlapped design.
values reduce KV cache memory. These findings provide practical guidance for balancing memory efficiency and model accuracy based on system constraints.
4.3 Latency in End-to-End QuickVideo Inference
We integrate QUICKCODEC and QUICKPREFILL into a (reen.5-VL-7B-nnstteee (Bai et al., 2025) inference pipeline. We evaluate two configurations: (1) loading the entire video with QUICKCODEC followed by QUICKPREFILL, and (2) our group-overlapped design. Latency for video loading, prefill, and LLM decoding is benchmarked an end-to-end pipeline. For QUICKPREFILI, we use the Key Norms (small) pruning method with p 0.2 and set the group size to 32 frames. We use a 30-minute video (sampled at FPS) as the bascline implementation runs out of memory with longer videos. Experiments SXM GPU and AMC Ryzen 9 7950x CPU, allocating 16 cores for video processing. For the overlapped implementation, we use 64 intervals (s in section 3.3) for parallelized loading. All timings are averaged over 5 runs. Figure 5 presents latency breakdowns for all three implementations. After applying QUICKCODEC, we significantly reduce video loading time from 25.0 seconds to 15.4 seconds. By overlapping video loading and prefill, we reduce latency by 39.45% when compared to the baseline. We find that once prefill and video decoding are overlapped, the combined stage becomes completely vidco-docoding bound.
conducted on an H100
5 Discussion and Related Work
GPU support for video decoding. Video decoding can be accelerated by GPU computing. However, due to interframe dependencies, the speedup is not nearly as large as GPU acceleration for Al computations PyTorch Team (2025). Furthermore, especially in the case of long video, GPU-based video decoding can result. in device memory problems; the hour-long video we use for benchmarking (Section 4.1) is x600 1920 x020x800x byte = 16.6 GB| before being resized. This results a significant, portion of GPU resources being allocated to video tensors, and can cause CUDA out-of-memory errors if not handled delicately. For simplicity, most existing inference librarics dcfault to using CPU for vidco decoding (Kwon et al., 2023; Wolf et al., 2020). More sophisticated pipelines, such as NVIDIA's 's Cosmos training, use dedicated hardware for handling the video processing (NVIDIA et al., 2025).
Efficient VideoLLMs Inference. Recent VideoLLMs (Lin et al., 2023; Li et al., 2024; Chen et al., 2025a)
have demonstrated strong video understanding capabilities. Early models like Video-LLaVA (Lin et al., 2023) and VideoLLama-2 (Cheng et al., 2024) were limited around input frames due to constrained training data and unoptimized architectures. More advanced models such as Qwen2.5-VL (Bai et al., 2025) and InternVideo2.15 (Wang et al., 2025) now handle hundreds of frames by adopting architectural innovations including Group Query Attention (GQA) (Ainslie et al., 2023), MRoPE (Bai et al., 2025), and Special Token Merging (Chen et al., 2025a), which reduce KV cache size and enhance temporal reasoning. Nonetheless, the KV cache and activation memory still grow linearly with context length, creating bottlenecks in hour-long video inference. Meanwhile, existing token pruning techniques either address only image-level contexts (Wen et al., 2025; Chen al., 2024; Shang et al., 2024; Xing et al., 2024), or optimize for short prefill and long decoding scenarios (Devoto et al., 2024; Zhang et al., 2023; Xiao et al., 2023). In contrast, we target efficient. prefill for millions of video tokens, introducing a method that achieves substantial memory savings and speedup
10

---

Published in Transactions on Machine Learning Research (01/2026)
with minimal accuracy loss, thereby enabling scalable long video understanding resource-constraneed hardware.
6 Conclusion
We introduced QuickVideo, a framework to accelerate long video understanding. Our framework has three core contributions: QuickCodec: systems framework for fast video loading, designed for VideoLLM frame sampling. QuickPrefill: An efficient algorithm for prefilling video tokens. Co-design: Lastly, we show that our video loading and prefill algorithm can be almost entirely overlapped, drastically reducing the time latency of these stages during inference. Overall, QuickVideo reduces time to infer long video input by more than 3x. Our work advances the capabilities for real-time video understanding applications, addressing key efficiency challenges in long video inference.
Broader Impact Statement
As video has become the dominant modality of data, efficient long video understanding has extremely broad implications, both positive and negalive. On the posilive side, better long video understanding allows to better interpret our digital landscape. In 2022, 30,000 hours of video were uploaded to YouTube every hour (Ceci, 2024). That number is absolutely much higher today. Without efficient long video understanding systems, we cannot understand our own digital artifacts, due to the scale at which we create them. Furthermore, long video understanding also has extremely compelling use-cases for information accessibility. A video-first. internet is difficult to navigate for visually impaired people, with important information potentially only accessible in video format (Liu et al., 2021). Efficient, robust long video understanding presents can serve backbone for tools for assisting video understanding for the visually impaired. However, efficient long video understanding also has potentially negative effects. As people's lives are increasingly documented as video and uploaded to the internet, long video understanding models could become a tool for privacy intrusion (Feldstein, 2022).
References
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr'on, and Sumit K. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. ArXiv, abs/2305, 13245, 2023. URL https///ppp.seeattcscccooo.....r org./oorr (Dt.orppssI.280811177771
Fatih Cagatay Akyon and Alptekin Temizel. Deep architectures for content moderation and movie content rating, 2022. URL https://arx/e httpp://aaxxv..rr//bbb//// 2t1.b///222.....33.
Amazon. Amazon ec2 p5 instances. https://aws http:///wwaaaazz...oooamma amazon. http://e/iintancettyyeeee 2025-05-10.
2025. Accessed:
Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: www.ivorggabss/50222 13923.
Laura Ceci. YouTube: hours of vidco uploaded every minute 2022| Statista - statista.com. https: //www.stee www.wsttastta...mmm stratisstics22447//oorrre vidoo--pllolddddddeeee wttt-oyuuubeeeer--yeeeee 2024. [Accessed 16-05-2025).
Keshigeyan Chandrasegaran, Agrim Gupla, Lea M. Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video-language understanding, htgg/bbs/2411 02411.04999.
2024. URL https://arx/:
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer Plug-and-play inference acceleration for large vision-language models. European Conference on Computer Vision, 2024. URL https: pttps://apt.. //ap1. semanticschol www.olaarrrrrrrrpoooeeeee org/CorpusID: 268358224.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong Hc, Zhiyong Wu, Huipeng Deng, Jiaye Gc, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang- Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025a. URL htttpp://arxiv.orge wtt..o///aa//22.022.......
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025b. URL https: http://axxvvvv.oo hwtpr/iii.rgg/bb//22222 2b1.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in vidco-llms. aiv preprint ar arxi::206.007 07476, 2024. URL https postps://abe //arxiv...iv///// orgo //////////// /206. 07476.
Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. abs/2307. 00001, 2023. URL. https://api. www.ttticccooooooroooaaoa I 2599936673
ArXiv,
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective 1_ norm- based strategy for KV cache compression. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 18476-18499, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10/5533/11222..em// www.eemainnleeeeeemmm URL https://ee http:///aaaanhoooy.ooooooo
Distributed (Deop) Machine Learning Community. Decord. https://github. cww/ ht//dmllclae con//mlc//eccrdd 2019. Accessed: 2025-05-10.
Steven Feldstein. Global Erpansion of AT Surveillance. Carnegie Endowment for International Peace, 2022.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms in video analysis. ArXie, 0ts/2405.21077 2024. URL https:
//api. wwwanticscholle.......... org/ Io:
Google. Gpu machine types. https://clout
wttelle..cccccooo com/coomtte/docs//////ooooooo/ gpus, 2025. Accessod: 2025-05-10.
Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. In Conference on Empirical Methods in Natural Language Processing, 2024. URL httpp:://pp..eenn..cccclola.... wtt.aaattt.cccoal....ooooooraaaa wwtp//Corpss..:2005502222.
Michel Kerdranvat, Ya Chen, Rémi Jullian, Franck Galpin, and Edouard François. The video codec landscape in 2020. ITU Journal: ICT Discoveries, 3(1):33 83, 2020.
Rob Koenen. Mpeg-4 overview. Technical report, International Organization for Standardization, 1999. URL https://sound media. mitt........... aedu/resources//pee///uuu httpa/////eeeea///........ pdf.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficiont memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Tlava-onevision: Easy visual task transfer. ArXiu, abs//4408.03326, 2024. URL
https://api... sen...siamaaniccolaa... http://cccccaaaooo//rrorrrr CTtp:CCorpuSSI:27711199444
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Conference on Empirical Methods in Natural Language Processing, 2023. URL httpss///pp..smmatiiccccholll org/CorpustD 2652881444
Xingyu Liu, Patrick Carrington, Xiang 'Anthony' Chen, and Amy Pavel. What makes videos accessible to blind and visually impaired people? In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450030966. doi: 10. 00.1145/341176444442333 URL https://doi.
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https: //github.
con/pytorchhvision 2016.
Mierosoft. Nd-h100-v5 sizes series. https://learn. microsoft.....mmommmmmmm httt:://llaammmmccoootttt.ooo http:///////////ee////ttaaaaaaaa
2025. Accessed: 2025-05-10.
Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, and Li Zhang. Slowfocus: Enhancing fine-grained temporal understanding in video Ilm. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 81808 88885. Curran Associates, Inc., 2024. URL https://proceedding www.rrreeedinn....ii........... newrips....... cc/ htt.:///......11111.666666611020
saper_files/ppe
Conference .df.
NVIDIA, :, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung- Yi Lin, Huan Ling, Ming- Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Roda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheug Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, and Artur Zolkowski. Cosmos world foundation model platform for physical ai, 2025. URL https:// /aapss///axivv.e/e http//aaxvvvooogabaeaa/ ptrg/abs/2501... 2550.03575.
PyTorch Team. lorchcodec. https: / https:///ithbb wwtp:pptorrcc//orccccoeeeecoeeee 2025. Accessed: 2025-05-10.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge Adaptive token reduction for efficient large multinodal models. ArXie, abs/2403.15382 2024. URL https://aee //api semanticscholar. org/CorpusID 268667281
Noam M. Shazcer. Glu variants improve transformer. ArXiu, abs/2002.05202, 2020. URL https://apt
semanticacholar.c org/ Corpus wrw.aarpssooCCCCC.I....... 211096588.
Hairong Su, Shibo Wang, Shusen Yang, Tianchi Huang, and Xuebin Ren. Reducing traffic wastage in video streaming via bandwidth-effiienn bitrate adaptation. IEEE Transactions on Mobile Computing, 23(11):10361 10377, November 2024. ISSN 2161-9875. doi: 10. 10.1109 11119/99tc.20000 20c..0004.3333449. d/dx..oi..org www.oo..//...... /0/./11...0 1109/TMC. 2024 2074.333349..
URL http:
Gary J. Sullivan, Jens-Raine Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions on Circuits and Systems for Video Technology, 22(12): 1649-1668, 2012. doi: 10. N0.1109/TCSVT.20 2TT.STC001..2211111.
Suramya Tomar. Converting video formats with fimpeg. Linax Journal, 2006(146):10, 2006.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lybench: An extreme long video understanding benchmark. ArXin, abs abs/2446.088335, 2024. URL attps://ap..emmaiccccli.i.. www.stiiscooolaa.........c
wrg. DrggCorpusII::210 270391667.
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyun Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kaiming Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2.5: Empowering video milms with long and rich context modeling, ArXiv, abs/2501.12286, 2025. URT. httpps://appt /tapp:////a..mmmatiiiiiia coemanttccccholan wtw.ccaaa...o/oorrrreeeeeeeee httg/oorpsID:2257877777
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplication matters more.
ArXie, abs/2502.1 11494, 2025. URL https://api. wwt.p.sssaatiiccohlaaaee : 1 276408079.
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models, 2024. URL https://aaaat org/abs/2404 2/55/bbb/2404..33844..
T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the h.264/ave video coding standard. IEEE Transactions on Circuils and Systems for Video Technology, 13(7):560 576, 2003. doi:
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Authony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Hugginglace's transformers: State-of-the-art natural language processing, 2020. URL https:// //axxiv.///iiv.///a arg/abs//aa 19tps/1990.077.1....
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. ArXiv, abss/2407.15755 2024. URL https://api. http:://appp.eeannncccccoo.o.eee org/ I 271329366.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ArXiu, abs/2309.1744 17453, 2023. URL httpss///a....smmmttcccoooooooo org CCrppusI:: 263310483.
Long Xing, Qidong Huang, Xiao wen Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui Hc, Jiaqi Wang, Feng Wu, and Dahua Lin. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. ArXiv, abs/2410 22110.124777 2024. URL attps://apt semanticscholar. erg/CorpusID:211 273507888.
Tongtong Yuan, Xuange Zhang, Kun Liu, Bo Liu, Chen Chen, Jian Jin, and Zhenzhen Jiao. Towards surveillance rideo-and-language understanding: New dataset, baselines, and challenges, 2023. URL
https://arxiv org/abs//3/tt 13925.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Ti, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URL https://arxiv http:///axxvv.oo.oaaae perg/abs/244 12772.
Zhenyu (Allen) Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for elficient generative inference of large language models. ArXiv, abs/2306.1B 14048, 2023. URL attps://api. semanticscholar. Dit.aaarrrrrr/rrrrs1:::22522
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXie preprint ar arXiv:2406.04266 04264, 2024a.
Pengyuan Zhou, Lin Wang, Zhi Liu, Yanbin Hao, Pan Hui, Sasu Tarkoma, and Jussi Kangasharju. A survey on generative and Ilm for video generation, understanding, and streaming, 2024b. URL https: waarxiv.org/abs//// 16038.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun Hc, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang. Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multmmdal models, 2025a. URL https: / /arxiv. 20g//bb//5000....
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Yc, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025b. URL https: //axxiv..vvvv///////////a//aiav org/ hb///ba///2/ ab2/ 10479.
15

---

Published in Transactions on Machine Learning Research (01/2026)
A Parallelized Interval Algorithm
Additional video decoding background. The container contains various metadata about packets that use during our interval parsing algorithm. For locality purposes, modalities such as audio and video are often interleaved in the bit stream S. Therefore, it is important to filter out audio packets when parsing the metadata stream. As packets are not frame-aligned, the pts field does not exactly represent the display time of frame. Also, packets can be reordered by the decoder, the first or last, packets may not correspond to the first and last frames.
Algorithm Calculate Parallelized Intervals
1: procedure KEYFRAME INTERVALS(S, c)
2: K,, ptsmin: ptsmax SCAN PACKETS(S) {ptsmin ptsman)
D Scan packet metadata. D Ordered list of keyframe intervals. Evenly spaced intervals in the video.
p (pts ptssmmn) for E ,......, do
plsestimate - (e x) + plomin + FINDINSERTIONIDDXXX.. ptsestimate)
7:
8: if [K attemmee < [K - -1.. then J=
9.
10: else
11:
J= J (K2) return J
12:
procedure SCAN PACKETS(S)
4 Scan bit stream to get timestamps.
13:
ptsmin +
14:
plsmar + 00 K-O
15:
Sorted set of keyframe timestamps.
16:
for pi E S do
17:
if p..type "video" then continue
Skip packets not used to decode video.
18:
19:
if Pr-pls NULL then continue if pr.pts ptsmin then
Skip packets do not have pts meladata.
20:
21:
22:
ptsmin P.-ps
23:
if pt.pts > ptsmaz then
24:
plomax + Pi-pls
25:
if ke..eyframe True then
26:
K-KU (pr.pts}
27:
return K., pt8ming ptsmax
Algorithum 3 computes intervals that we can parallelize video decoding over. For effective parallelization, it is essential that these intervals are roughly length and keyframe- the start of each interval. SCAN PACKETS parses the metadata of the packet stream to find the location of all keyframes in S, as well as the minimum and maximum pts in S. If the packet does not belong to the video stream or the timestamp is NULL, the packet skipped.
such that Algorithm 2 can seek to
After finding the locations of keyframes on line 2, KEYPRAME INTERVALS computes e intervals as follows: We calculate the length of 1 of the stream, in pts units (line 4). On lines 5-10, we search for the keyframes с closest to being ith through the vidco, given by ptsestimate- FINDINSERTIONINDEX uses binary scarch to find where in the list of keyframes ploestimate would be inserted. After finding the insertion point j, the algorithm checks whether the keyframe before or after j is closer to ptsestimate- The closest keyframe location added to J, the list of intervals. J[O] ptsmin and J[c- 1]] ptsmaz: to ensure that the intervals span the vidco. [[1]. J[2.... 2] are keyframe-aligned and equally spaced. Therefore, J, a list containing c values, can be interpreted intervals: I' HH[i, 1]) i 0,1.... c).
16

---

Published in Transactions on Machine Learning Research (01/2026)
B Effect of sampling rates on QuickCodec's efficiency
As QUICKCODEC does not seek between loading frames, all video frames are decoded during video loading. Con- versely, seek- based frameworks skip decoding segments of vidco if there are large gaps between sampled frames. In Figure 6, we find that our framework has faster video loading when there is a second or less gap between sampled frames. Our library performs best when using imple-
45
Decord
TorchCodec
40
TorchCodec+ Resize
QuickCodec (ours)
30
Time
VidcoLLM sampling rates (1-2 FPS). Currently, ment ation always loads the whole video, and therefore does not benefit significantly from sparse sampling patterns. Our implementation could be adapted to leverage sceking when delects that the user has sampled with a large gap between frames, closing the performance gap with seek-based libraries PyTorcc Team (2025); Distributed (Deep) Machine Learning Community (2019). This would make our library more flexible, and eliminate potential performance sharp edge, where users accidentally use our QUICK for sparse sampling. We leave this as a direction for future library improvements.
25
15-
10-
5
25
4s
16s
Sampling Interval
Figure 6: Video decoding performance for differ- ent video durations with FPS sampling.
C Containers and Video Decoding
A multimedia container file format, like MP4 or MKV, bundles together all the elements required for media playback, including video streams, audio streams, subtitles, images, and meladata (Koenen, 1999). Video streams are compressed into bit streams by codecs The bit streams are formatted in standards like H.264 (Wiegand et al., 2003) and H.265 (Sullivan et al., 2012). A codec consists of two algorithms: a video encoding algorithm that takes in a sequence of frames and outputs a compressed bit stream and video decoding algorithm that takes the bit stream input and outputs video frames. We focus video decoding, it is the required operation before the video can be used VideoLLM input.
D QuickPrefill Efficiency Analysis Details
D.1 Activation Memory Analysis
The activation memory modern LLM architecture mainly comes from two components of each transformer block: 1) Attention Block and 2) MLP Block. We analyze the potential activation memory usage in formulas in the followings and show that group-based prefilling can effectively reduce the activation memory by G times, where G is the number of groups.
Attention Block Modern LLMs commonly adopt FlashAttention (Dao, 2023), memory-effliiet attention algorithm that computes exact attention ion with reduced memory usage by fusing multiple steps and processing attention in blocks. While the naive attention implementation would instantiate the full attention matrix A EED R SxS FlashAtlention avoids this by computing attention block by block. Let Q, K, V E R BxSxdhind denote the query, key, and value tensors respectively, with na heads and dhead dmodal
FlashAttention
ni
divides the input sequence into blocks of size B, (for keys/values) and B, (for queries) process attention efficiently within GPU memory constraints. Following (Dao, 2023), the dominant activation memory in FlashAttention comes from storing Q, K, V. The block-based processing means that at any given time, only blocks of the attention matrix of size B, x B, are materialized in memory, Assume using float16 data type, the total activation memory can be expressed as:
Matia (3B . S n4p chead + B nhp B, B.) bytes
(4)
17

---

Published in Transactions on Machine Learning Research (01/2026)
The first term accounts for storing Q. K, and V tensors, while the second term accounts for the block of attention matrix being processed. With appropriate block sizes B, and Bg (typically set based on GPU memory constraints), the second term remains relatively small. Assuming B 1, S (|X" | 921600, dmodel 4096, n4 = 8, B, Be 1024, we compute:
Matin = (3 921600 8 512 1- 8 1024 1024) 2 bytes
(5)
= 60, 584, 722, 432 bytes
(6)
21.1 GB
(7)
While PlashAttention siguificantly reduces memory requirements compared to naive attention implementation, this analysis shows it still consumes substantial memory for very long sequences. With group-based prefilling using G 225 groups, we can reduce the sequence length by G times, reducing Mattn from 21.1 GB approximately 0.09 GB. This dramatic reduction enables the processing of extremely long sequences that would otherwise be infeasible.
The SwiGLU (Swish-Gated Linear Unit) (Shazeer, 2020) enhances transformer models through improved gating mechanisms and has been adopted as the default MLP architecture in many popular LLMs including InternVL2.5 and Qwen2.5 series (Bai et al., 2025; Chen et al., 2025a). For input representation x E R Amodel the SwiGLU operation is defined as:
MLP Block
SwiGLU((x: W down (S gate W W.p
(8)
where W gate: W upp
R daodmode
W E
I Aamoddadi Xdg and SiLU(x) x = with oxx) I+6-
For batch of sequences, activation memory analysis reveals requirements at each computational step. With batch size B, sequence length S, hidden dimension dmodel, intermediate dimension dg, and data type float16, the total activation memory for a single SwiGLU layer is:
Mact = (B - S (2dmodel + 4dg)) 2 bytes
(9)
For a one hour video sampled with FPS (3600 frames in total), parameters can be set B = 1, S (IX*I + +X*1)) 921600, dmodel 4096, and dg 14336:
Mact (1 921600 (2 4096 + 4 14336)) 22 bytes
(10)
241, 591,59199 400 bytes
(11)
112.5 GB
(12)
This substantial memory requirement highlights the computational challenges in deploying SwiGLU-based models for high-resolution inputs with extended sequence lengths. However, if we prefill the tokens group by
group, we can reduce the S by times, and thus reduce the activation memory Mact by times. Assuming
each group contains tokens of 16 frames, then G 3600 225 and we can reduce Maet from 112.5 GB to 0.5 GB, which is substantial improvement.
16
D.2 KV cache Memory Analysis
When using InternVT.2.5-8B (Chen et al., 2025a), with each frame encoded as 256 takens ([VI 3,600 256 921,600), and |Q| 256 text tokens, L 28 layers, ny 8 heads, and dy 512, the total memory required to store the KV cache in float16 precision is:
Memory 22 x L ([x") (x()) x nu x da x 2 bytes 393.9 GB
(13)
18

---

Published in Transactions on Machine Learning Research (01/2026)
E
Ablation Study on Group Size and Retention Ratio
Table Ablation study of different group sizes and retention ratio p. We use Key Norms (small) as the KV
pruning method here.
Group Size
Long VideoBench
MLVU (dev)
p
VideoMME
(val)
LVBench
Avg
Performance
Varying Group Size
1
65.78
61.56
43.90
68.65
59.97
100.00%
4
0.5
63.78
60.36
42.61
66.81
58.39
97.36%
0.5
64.00
60.88
42.35
66.94
58.54
97.62%
16
0.5
64.04
60.21
41.90
66.73
58.22
97.08%
32
0.5
63.59
59.46
41.51
66.78
57.84
96.44%
64
0.5
63.89
60.51
42.29
66.83
58.38
97.34%
128
0.5
63.56
59.24
42.61
66.97
58.09
96.87%
Varying Retention Ratio p
16
1
65.78
61.56
43.90
68.65
59.97
100.00%
16
0.1
55.89
53.40
36.02
59.02
51.08
85.18%
16
0.2
59.74
56.47
39.57
61.58
54.34
90.61%
16
0.4
63.22
58.94
41.19
65.75
57.27
95.51%
16
0.6
64.74
60.81
41.90
67.48
58.73
97.93%
16
0.8
65.70
61.41
43.51
68.37
59.75
99.63%
16
0.9
65.85
61.18
43.71
68.70
59.86
99.82%
F
HourVideo Results
VourVideo (Chandrasegarau et al., 2024) is a video-unnerrtannding benchmark designed to test the reasoning, perception, sunmarization and navigation abilities of VideoLLMs. In Table 3, we compare our best performing pruning method, Key Norms (small), to a no pruning baseline. We use 256 frames and group size frames, as we found that is the strongest setting for the bascline (Table 1). The overall score is computed by weighting each category by its respective number of samples. We note that the navigation calegory requires answering with image, which our model does not support.
G QuickVideo CPU and GPU Memory Profiles
We provide CPU memory profile corresponding to Figure 5. QuickVideo's CPU memory is made up of 3 components: the torch main process, QuickCodec workers, and the shared memory block into which video frames are loaded (line 2 in Algorithm 2). The torch main process has consistent memory utilization (RSS
Method
Reasoning
Perception
Summarization
Navigation
Overall
No Pruning
31.76
32.38
30.49
20.00
31.47
Key Norms (small)
33.09
28.98
31.71
17.50
31.13
Performance
104.2%
89.5%
104.0%
87.5%
98.9%
Table 3: Results Hour CaaideooChannnnn (Chandrasegaran et al., 2024).
19

---

Published in Transactions on Machine Learning Research (01/2026)
Group Size Peak Allocated (MB)
95th Percentile Allocated (MB)
Base Model
15816.09
16
16718.12 12
16670.19
32
17349.22
17240.74
64
18606.21
18494.28
Table 4: Memory allocation statistics by group size.
95-percentile and peak of 1557.86 MB), QuickCodec workers have peak RSS of 89.27 MB and 95th percentile of 88.68 MB per worker. The shared memory buffer is allocated with constant size (Algorithm 2).
Table shows GPU memory profile for processing 30 minute video with different group sizes. We report both the peak allocated memory and 95th percentile allocated memory. The majority of memory is allocated for the base model, processing groups of size 16 and 32 increases peak memory utilization by 854.1 MB and 1533.13 MB, respectively.
Implementation Details
QUICKCODEC, TorchCodec and Docord use FFmpeg to implement the decoder D. QUICKCODEC uses system shared memory directly, instead of a parallelization library such as python's malllproccessing, as we find that serializing video tensors for inter-process communication introduces significant latency. Shared memory is designed for fast inter-process communication and is not secure. QUICKCODEC should not be used hosts running untrusted processes.
Limitations
As it is slow and resource intensive, most VideoLLMs not trained to their 1-2 FPS short video sampling rates when using processing long video (Bai et al., 2025; Chen et al., 2025a; Zhu et al., 2025a). Instead, they use very low sampling rates over large time-spans, as we discussed in Section 1. Therefore, VideoLLMs do not (yet) gain a large performance advantage by processing a large number of frames. However, it is clear that a model that has seconds-long gaps between frames can never capture fine grained temporal and spatial details. Our hope is that making long vidco understanding (with rcalistic sampling rates) practical from a systems and algorithm perspective, we will empower the development of such models, Another limitation is that our QUICK CODEC timings only H.264 coded video for timings. Although H.264 is the dominant. standard, it is not universal.
20


Reviews:
[{"review_id": "LKdDfwUIHw", "reviewer": "Reviewer_Wskh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## **Summary**\n\nThis paper presents **QuickVideo**, a system–algorithm co-design framework for real-time long video understanding in **Video Large Language Models (Video LLMs)**. The work tackles two major bottlenecks in current long-video inference stack (1) **sequential video decoding** and (2) **memory-intensive prefilling**, and proposes three solutions:\n\n- **QuickCodec**, a parallelized, keyframe-aligned video decoder achieving **2–3× speedup for hour-long videos** over existing tools such as Decord and TorchCodec.  \n- **QuickPrefill**, a group-based prefilling approach with **KV cache pruning** that significantly reduces GPU memory usage with **less than 3% accuracy degradation**.  \n- An **overlapped CPU–GPU execution scheme** that overlaps decoding and inference, **reducing total inference time for a 30-minute video from 69.7 s to 20.0 s** (more than **3× speedup**).\n\n---\n\n## **Strengths**\n\n- Clear system–algorithm co-design that bridges efficiency and practicality for long video inference.  \n- Strong empirical validation with multiple long-video benchmarks.  \n- Effective demonstration of real-time capabilities with measurable latency reductions and memory savings.\n\n---\n\n## **Weaknesses**\n\n- **HourVideo benchmark results are missing.** Since HourVideo was designed specifically for hour-long video-language understanding, including it would help contextualize performance even if the numbers are less favorable.  \n- **Comparison against token pruning methods** for KV cache compression is missing. Including this would help illustrate the Pareto frontier of speed versus accuracy and benefit the community.  You may consider a recent work like this: https://research.nvidia.com/labs/lpr/storm/\n- In **Sec. 4.2 (Table 3)**, it is unclear whether the **attention matrix** is materialized when computing attention-score-based KV cache pruning. If so, this could be prohibitively expensive for GPU memory; clarification is needed.  \n- The evaluation uses only a **single model**. Adding one more model (e.g., from a different family such as **LLaVA-Video**) on 1–2 benchmarks would help confirm that performance trends in Table 1 generalize across models.  \n- **TorchCodec evaluation details.** TorchCodec supports both CPU and CUDA backends. Please clarify whether the reported comparisons use the CPU or GPU version. If only CPU results are reported, consider including **TorchCodec[CUDA]** benchmarks for completeness since QuickVideo also has meaningful implications for improving video LLM training pipelines. \n- **Conduct benchmarks on newer GPU hardware** such as **H100 or B200** to contextualize real-time performance claims on modern GPUs.\n\n**Overall:** This is a well-executed paper that advances the field of efficient long video understanding through thoughtful system–algorithm design.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The summary already covers the main evidence. The claims are generally well supported by experiments and benchmarks, but addressing the identified weaknesses such as including HourVideo results, comparisons to token pruning, and clarification on TorchCodec CPU vs. GPU results would make the evidence more convincing.", "audience_yn": "Yes", "audience_explanation": "Yes. The work will interest researchers in multimodal learning, efficient inference, and video understanding. It provides practical system-level improvements for VideoLLMs, a topic gaining traction in both academic and applied machine learning communities.", "requested_changes": "See weaknesses above. Addressing those points, particularly adding HourVideo results, token pruning comparisons, clarification on attention matrix computation, one additional model evaluation, clarification on TorchCodec CPU vs. GPU results (and including TorchCodec[CUDA] benchmarks if possible), and conducting benchmarks on newer GPU hardware such as H100 or B200 to contextualize real-time performance claims, would strengthen the submission.", "broader_impact_concerns": "The paper already includes a **Broader Impact Statement** that appropriately discusses both accessibility benefits and potential privacy risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z8GANOOQnm", "reviewer": "Reviewer_iAq4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper targets long-video VideoLLM latency by designing QuickCodec, keyframe-aligned interval parallel decoding on CPU; QuickPrefill, grouped prefill with KV-cache pruning on GPU. It is a CPU/GPU overlap scheme. QuickVideo reports 2–3× faster loading at 16–32 cores and an end-to-end reduction from 69.7s to 20.0s on a 30-min, 1 FPS input (overlapped pipeline). It shows small average accuracy drops at moderate pruning ratios across VideoMME, LongVideoBench, LVBench, and MLVU.\n\nStrengths:\n\n1. The proposed method makes the pruning happening during the Prefill stage itself, while traditional methods mostly apply pruning during the decoding stage, since for long videos, prefill itself is the bottleneck. \n\n2. The number of seek operations is reduced from being on the same order as the number of frames to just once per core, which is a targeted optimization for long video data-loading scenarios.\n\nWeakness:\n\n1. This paper of system-algorithm co-design is more like a known idea on multi-core decoding from an engineering perspective, but customized it as a decoder specialized for Video-LLMs. It would be helpful to have brief intuitions about model's choice, to make it more transparent to understand why the model is working, like the intuition why “key-norm (small)” works the best. \n\n2. A concise clarification of Eq. (3) stating whether it is a bound/approximation and under what producer–consumer assumptions would make the analysis more robust.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. QuickCodec scales to 64 cores and is consistently faster than common libraries at 16–32 cores; speed gains grow with video length. The number of seek operations is reduced from being on the same order as the number of frames to just once per core, eliminating much of the fixed overhead from seeking and resetting the decoder state. Within each segment decoding is still sequential, but across segments it can be parallelized over 16–64 cores. In practice, on common 16–32 core setups, this achieves a 2–3× speedup for a one-hour video, saving more than 20 seconds of loading time; the longer the video, the greater the advantage.\n\n2. Long videos explode in prefill, while decoding is short. So QuickVideo shifts KV pruning into the prefill stage, which makes sense. Using a single heuristic (key-norm) plus grouping retains ~95–99% accuracy at ρ=0.5 on multiple long-video benchmarks, with an ablation over group size/ρ.\n\n3. The method works better for longer videos. It requires a sufficient number of keyframes and longer videos to “sustain” parallelism. For very short videos or extremely sparse random access, the traditional “frame-by-frame seek” approach may actually be more suitable (the paper’s appendix discusses such cases).", "audience_yn": "Yes", "audience_explanation": "1. CPU decoding tailored to low-FPS VideoLLM sampling + grouped prefill + overlap is well-motivated and practically useful. \n\n2. The latency breakdown is informative. \n\n3. QuickVideo does group-based prefill + KV pruning right away, discarding less important KV entries before decoding even begins. It makes long-video prefill feasible on commodity GPUs, and reduces activation and KV memory simultaneously, while keeping >95% accuracy.", "requested_changes": "1. QuickPrefill adopts the “key-norm (small)” rule as the default importance function. While this choice performs well empirically, the underlying motivation is only briefly discussed. Because retaining low-norm keys may be non-obvious at first glance (one might expect low-norm keys to correlate with lower attention), it is exciting to see more justification and can make the choice more convincing.\n\n2. The paper provides a clear theoretical activation analysis and latency breakdown, but does not include synchronized CPU/GPU memory–time profiles. Since interval-parallel decoding may increase CPU RSS and pinned buffers as thread/interval counts grow (due to decoder state and temporarily decoded frames), it would be valuable to confirm whether CPU memory remains manageable in longer videos at higher parallelism. Consider including synchronized memory-time profiles (CPU RSS, pinned host memory, GPU allocated/KV/activations) for representative settings (e.g., 30–60 min videos at 1–2 FPS; $c\\in\\{16,32,64\\}$; $\\rho\\in\\{1.0,0.5,0.2\\}$; a couple of group sizes). Peak and 95-percentile summaries would suffice and would complement the latency breakdown. \n\n3.  Eq. (3) models total latency as $\\max(t_{\\text{dec}}+t^g_{\\text{prefill}},\\,t_{\\text{prefill}}+t^g_{\\text{dec}}+\\Delta)$, which seems to mainly reflect boundary groups. Since the time for prefilling and decoding in each group varies, the steady-state duration may be better approximated by $\\sum_{g=1}^G \\max(t^g_{\\text{dec}},t^g_{\\text{prefill}})$ plus startup/cleanup. A short clarification on whether Eq. (3) is intended as a bound, an approximation, or an empirical fit would strengthen the presentation.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kur0tpXR8d", "reviewer": "Reviewer_bSCb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates a timely and important problem: enabling efficient long-video understanding with VLMs, where both video decoding and prefilling constitute severe bottlenecks. The authors propose QuickVideo, a system–algorithm co-design that integrates: (1) QuickCodec, a parallelized CPU-based decoder aligned with keyframes; (2) QuickPrefill, a grouped prefill and KV-cache pruning strategy; and (3) an overlapping execution scheme that overlaps CPU decoding with GPU inference. The method is validated on four video understanding benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) using Qwen2.5-VL-7B, achieving up to 3× end-to-end speedup with only minor accuracy degradation, thereby making long-video understanding more practical for real-world deployment.\n\nStrengths:\n\n1. The paper addresses a meaningful problem, optimizing video understanding, which is both necessary and has clear real-world applications.\n2. The motivation is sound, as both prefilling and decoding present challenges; in particular, optimizing the codec process can significantly reduce the cost of transmitting large long-video files.\n\nWeaknesses:\n\n1. The paper lacks comparisons and discussion with related work on sparse attention for video LLMs[1,2].\n2. Can this method be applied to accelerate video-LLM API serving? Do you have any test results in such scenarios?\n\n[1] XAttention: Block Sparse Attention with Antidiagonal Scoring, ICML'25.\n\n[2] MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention, ICML'25.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The motivation of this work is well-founded, and it is validated on four video understanding benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) using Qwen2.5-VL-7B.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a meaningful problem, optimizing video understanding, which is both necessary and has clear real-world applications.", "requested_changes": "1. Could you add comparisons and a discussion of related work on sparse attention for video LLMs [1,2]?\n2. Can this method be applied to accelerate video-LLM API serving? Do you have any test results in such scenarios?\n\n[1] XAttention: Block Sparse Attention with Antidiagonal Scoring, ICML'25.\n\n[2] MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention, ICML'25.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)