MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware
  Multi-Segment Grounding
Authors: name: Fuwen Luo; name: Shengfeng Lou; name: Chi Chen; name: Ziyue Wang; name: Chenliang Li; name: Weizhou Shen; name: Jiyue Guo; name: Peng Li; name: Ming Yan; name: Ji Zhang; name: Fei Huang; name: Yang Liu
Published: 2025-05-27 04:50:07

Abstract: Video temporal understanding is crucial for multimodal large language models
(MLLMs) to reason over events in videos. Despite recent advances in general
video understanding, current MLLMs still struggle with fine-grained temporal
reasoning. While reinforcement learning (RL) has been explored to address this
issue recently, existing RL approaches remain limited in effectiveness. In this
work, we propose MUSEG, a novel RL-based method that enhances temporal
understanding by introducing timestamp-aware multi-segment grounding. MUSEG
enables MLLMs to align queries with multiple relevant video segments, promoting
more comprehensive temporal reasoning. To facilitate effective learning, we
design a customized RL training recipe with phased rewards that progressively
guides the model toward temporally grounded reasoning. Extensive experiments on
temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG
significantly outperforms existing methods and generalizes well across diverse
temporal understanding scenarios. View our project at
https://github.com/THUNLP-MT/MUSEG.