Title: LongVTG-R1: Reinforcement Learning for Robust Long-Video Temporal Grounding
Primary area: foundation or frontier models, including LLMs
Keywords: video understanding, temporal grounding, multimodal large language model (MLLM)

Abstract: We present the first Reinforcement Learning (RL)-based framework that equips Multimodal Large Language Models (MLLMs) with long-video temporal grounding skills, and demonstrate that this approach also generalizes to improve performance on general question-answering (QA) tasks. Unlike dominant supervised fine-tuning (SFT) methods, RL enables models to acquire temporal grounding abilities without risking catastrophic forgetting of their core understanding. However, adopting RL for long-video temporal grounding reveals a challenge in balancing exploitation of pre-trained knowledge with exploration of new localization skills. To address this, we propose Token-aware KL Regularization, which selectively relaxes the KL-divergence regularization on timestamp-related tokens to guides exploration. Moreover, effective exploration requires a learning signal that alleviates the sparsity of key events in long videos, for which we introduce a denser reward, the Center Distance Reward (CenDist). To further mitigate grounding ambiguity between language queries and visually similar content, and to facilitate effective RL training, we automatically construct SceneTG, a dataset with sharp boundaries and unambiguous queries. Our resulting model, QwenLongTG, achieves state-of-the-art results across a suite of temporal grounding benchmarks and, through its generalization to QA, provides the first evidence that precise temporal localization is a foundational skill for broader long-video understanding.

TL;DR: We present LongVTG-R1, the first RL-based framework for long-video temporal grounding, which leverages novel regularization, reward design, and dataset construction to achieve state-of-the-art performance and generalize to QA tasks.