Time Blindness: Why Video-Language Models Can't See What Humans Can?
Authors: name: Ujjwal Upadhyay; name: Mukul Ranjan; name: Zhiqiang Shen; name: Mohamed Elhoseiny
Published: 2025-05-30 17:59:12

Abstract: Recent advances in vision-language models (VLMs) have made impressive strides
in understanding spatio-temporal relationships in videos. However, when spatial
information is obscured, these models struggle to capture purely temporal
patterns. We introduce SpookyBench, a benchmark where information is
encoded solely in temporal sequences of noise-like frames, mirroring natural
phenomena from biological signaling to covert communication. Interestingly,
while humans can recognize shapes, text, and patterns in these sequences with
over 98% accuracy, state-of-the-art VLMs achieve 0% accuracy. This performance
gap highlights a critical limitation: an over-reliance on frame-level spatial
features and an inability to extract meaning from temporal cues. Furthermore,
when trained in data sets with low spatial signal-to-noise ratios (SNR),
temporal understanding of models degrades more rapidly than human perception,
especially in tasks requiring fine-grained temporal reasoning. Overcoming this
limitation will require novel architectures or training paradigms that decouple
spatial dependencies from temporal processing. Our systematic analysis shows
that this issue persists across model scales and architectures. We release
SpookyBench to catalyze research in temporal pattern recognition and bridge the
gap between human and machine video understanding. Dataset and code has been
made available on our project website: https://timeblindness.github.io/.