Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video
  Large Language Models
Authors: name: Haibo Wang; name: Zhiyang Xu; name: Yu Cheng; name: Shizhe Diao; name: Yufan Zhou; name: Yixin Cao; name: Qifan Wang; name: Weifeng Ge; name: Lifu Huang
Published: 2024-10-04 10:04:37

Abstract: Video Large Language Models (Video-LLMs) have demonstrated remarkable
capabilities in coarse-grained video understanding, however, they struggle with
fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM,
a novel Video-LLM adept at perceiving and reasoning over specific video moments
in a fine-grained manner. We identify that current Video-LLMs have limitations
for fine-grained video understanding since they lack effective temporal
modeling and timestamp representation. In light of this, we sharpen our model
by incorporating (1) an additional temporal stream to encode the relationships
between frames and (2) discrete temporal tokens enriched with specific time
knowledge to represent timestamps. To optimize the training of
Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with
simple video-captioning tasks and progressively introducing video temporal
grounding tasks of increasing complexity. To further enhance
Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded
VideoQA dataset by an automatic annotation pipeline. Extensive experiments
demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding
tasks such as temporal sentence grounding, dense video captioning, and grounded
VideoQA, but also shows great potential as a versatile video assistant for
general video understanding.