Title: Vinoground: Today’s LMMs Don’t Understand Short Counterfactual Videos
Primary area: datasets and benchmarks
Keywords: short videos, temporal understandings, large multimodal models

Abstract: There has been growing sentiment recently that modern large multimodal models (LMMs) have addressed most of the key challenges related to short video comprehension. As a result, both academia and industry are gradually shifting their attention towards the more complex challenges posed by understanding long-form videos. However, is this really the case? Our studies indicate that LMMs still lack many fundamental reasoning capabilities even when dealing with short videos. We introduce Vinoground, a temporal counterfactual LMM evaluation benchmark encompassing 1000 short and natural video-caption pairs. We demonstrate that existing LMMs severely struggle to distinguish temporal differences between different actions and object transformations. For example, the best model o3 only obtains ~50% on our group score metric, showing a large gap compared to the human baseline of ~90%. All open-source multimodal models and CLIP-based models perform much worse, producing mostly random chance performance. Through this work, we shed light onto the fact that temporal reasoning in short videos is a problem yet to be fully solved. We will publicly share our benchmark.

TL;DR: Modern SoTA LMMs still demonstrates subpar performance at temporal reasoning with our temporal counterfactual benchmark composed of natural videos.