TRACE: Temporal Grounding Video LLM via Causal Event Modeling
Authors: name: Yongxin Guo; name: Jingyu Liu; name: Mingda Li; name: Xiaoying Tang; name: Qingbin Liu; name: Xi Chen
Published: 2024-10-08 02:46:30

Abstract: Video Temporal Grounding (VTG) is a crucial capability for video
understanding models and plays a vital role in downstream tasks such as video
browsing and editing. To effectively handle various tasks simultaneously and
enable zero-shot prediction, there is a growing trend in employing video LLMs
for VTG tasks. However, current video LLM-based methods rely exclusively on
natural language generation, lacking the ability to model the clear structure
inherent in videos, which restricts their effectiveness in tackling VTG tasks.
To address this issue, this paper first formally introduces causal event
modeling framework, which represents videos as sequences of events, and predict
the current event using previous events, video inputs, and textural
instructions. Each event consists of three components: timestamps, salient
scores, and textual captions. We then propose a novel task-interleaved video
LLM called TRACE to effectively implement the causal event modeling framework
in practice. The TRACE processes visual frames, timestamps, salient scores, and
text as distinct tasks, employing various encoders and decoding heads for each.
Task tokens are arranged in an interleaved sequence according to the causal
event modeling framework's formulation. Extensive experiments on various VTG
tasks and datasets demonstrate the superior performance of TRACE compared to
state-of-the-art video LLMs. Our model and code are available at
https://github.com/gyxxyg/TRACE.