Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or
  True Temporal Understanding?
Authors: name: Bo Feng, name: Zhengfeng Lai, name: Shiyu Li, name: Zizhen Wang, name: Simon Wang, name: Ping Huang, name: Meng Cao

Existing video understanding benchmarks often conflate knowledge-based and
purely image-based questions, rather than clearly isolating a model's temporal
reasoning ability, which is the key aspect that distinguishes video
understanding from other modalities. We identify two major limitations that
obscure whether higher scores truly indicate stronger understanding of the
dynamic content in videos: (1) strong language priors, where models can answer
questions without watching the video; and (2) shuffling invariance, where
models maintain similar performance on certain questions even when video frames
are temporally shuffled. To alleviate these issues, we propose VBenchComp, an
automated pipeline that categorizes questions into different domains:
LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions
can be answered without viewing the video; Semantic questions remain answerable
even when the video frames are shuffled; and Temporal questions require
understanding the correct temporal order of frames. The rest of the questions
are labeled as Others. This can enable fine-grained evaluation of different
capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that
are hidden by traditional overall scores, and we offer insights and
recommendations for designing future benchmarks that more accurately assess
video LLMs.