QuickVideo: Real-Time Long Video Understanding with System Algorithm
  Co-Design
Authors: name: Benjamin Schneider; name: Dongfu Jiang; name: Chao Du; name: Tianyu Pang; name: Wenhu Chen
Published: 2025-05-22 03:26:50

Abstract: Long-video understanding has emerged as a crucial capability in real-world
applications such as video surveillance, meeting summarization, educational
lecture analysis, and sports broadcasting. However, it remains computationally
prohibitive for VideoLLMs, primarily due to two bottlenecks: 1) sequential
video decoding, the process of converting the raw bit stream to RGB frames can
take up to a minute for hour-long video inputs, and 2) costly prefilling of up
to several million tokens for LLM inference, resulting in high latency and
memory use. To address these challenges, we propose QuickVideo, a
system-algorithm co-design that substantially accelerates long-video
understanding to support real-time downstream applications. It comprises three
key innovations: QuickDecoder, a parallelized CPU-based video decoder that
achieves 2-3 times speedup by splitting videos into keyframe-aligned intervals
processed concurrently; QuickPrefill, a memory-efficient prefilling method
using KV-cache pruning to support more frames with less GPU memory; and an
overlapping scheme that overlaps CPU video decoding with GPU inference.
Together, these components infernece time reduce by a minute on long video
inputs, enabling scalable, high-quality video understanding even on limited
hardware. Experiments show that QuickVideo generalizes across durations and
sampling rates, making long video processing feasible in practice.