Introduction
Constituting 65\% of all internet traffic in 2023, videos are an area of huge potential for the next chapter of leveraging artificial intelligence Fu et al. ( 2021 ); Zellers et al. ( 2021 ); Fu et al. ( 2023a ) . For example, Video Question Answering Lei et al. ( 2018 ) and Video Summarization Xu et al. ( 2016 ) are two existing datasets that empirically evaluate video models. Yet, they do not assess more challenging tasks, such as reasoning through specific relationships between multiple frames. Just like how humans understand videos by processing frames across time steps, AI’s ability to accomplish multi-frame reasoning is a core task for video understanding.

Multi-frame video reasoning is bottlenecked by the sheer computing resources needed to process videos, which typically contain 24 frames per second and can vary widely. However, intelligible videos tend to have little variation from frame to frame. Akin to selecting principle components on the axes containing the highest orthogonal variance, picking a small sample of keyframes can capture much of the video’s meaning.
Understanding a video via extracted keyframes poses the challenge of multi-hop reasoning, which requires stronger language model capabilities.

To evaluate this challenging multi-hop multi-frame video reasoning, we elicit video chain-of-thought ( VideoCOT ). We propose two tasks to test such capabilities in existing models – Video Infilling and Video Prediction . In the Video Infilling setting, the goal is to predict the masked keyframes’ descriptions when given the previous and next keyframes’ descriptions in sequence as context, following the spirit of masked language modeling. In the Video Prediction, the goal is to predict the descriptions of the next keyframes in the sequence when given a set of previous keyframes’ descriptions, similar to the next token prediction task. These tasks can help analyze video generation and whether video models truly understand the dynamic relations between subsequent video frames, given the variable context gap between keyframes.

To benchmark our proposed Video Infilling and Prediction tasks, we construct a dataset by proposing an automated method to extract keyframes. In addition to the frame itself, we propose two textual representations – unstructured dense captions and FAMOuS structured scene descriptions ( Figure 1 ). The unstructured captions are intended to extract more significant and visually descriptive information compared to existing captioning systems, a necessary component to provide enough context for more challenging tasks. In addition, we systematically create structured scene descriptions from these unstructured dense captions by extracting the frame’s focus, action, mood, objects, and setting using weak human supervision for quality assurance. The FAMOuS categories are inspired by play scripts, which maintain much of the visual detail of the unstructured captions while providing a clear, structured way to reason through visual tasks with high degrees of freedom.

We propose the following contributions:

We systematically collect an inference-time challenge dataset of keyframes for video reasoning augmented with two textual representations: unstructured dense captions for visually-descriptive information and FAMOuS scene descriptions for structured reasoning.

We propose the Video Infilling and Video Prediction tasks to benchmark the video chain-of-thought capabilities in existing models.

We empirically demonstrate that existing models have the potential for multi-hop multi-frame video reasoning tasks but have a significant area for improvement as future work.