Temporal Reasoning Transfer from Text to Video
Authors: name: Lei Li; name: Yuanxin Liu; name: Linli Yao; name: Peiyuan Zhang; name: Chenxin An; name: Lean Wang; name: Xu Sun; name: Lingpeng Kong; name: Qi Liu
Published: 2024-10-08 16:10:29

Abstract: Video Large Language Models (Video LLMs) have shown promising capabilities in
video comprehension, yet they struggle with tracking temporal changes and
reasoning about temporal relationships. While previous research attributed this
limitation to the ineffective temporal encoding of visual inputs, our
diagnostic study reveals that video representations contain sufficient
information for even small probing classifiers to achieve perfect accuracy.
Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning
capability stems from the underlying LLM's inherent difficulty with temporal
concepts, as evidenced by poor performance on textual temporal
question-answering tasks. Building on this discovery, we introduce the Textual
Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning
tasks in pure text format from existing image-text datasets, addressing the
scarcity of video samples with complex temporal scenarios. Remarkably, without
using any video data, T3 enhances LongVA-7B's temporal understanding, yielding
a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark,
which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video
samples. Additionally, the enhanced LongVA-7B model achieves competitive
performance on comprehensive video benchmarks. For example, it achieves a 49.7
accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful
large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further
analysis reveals a strong correlation between textual and video temporal task
performance, validating the efficacy of transferring temporal reasoning
abilities from text to video domains.