RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
AI Reasoning.
Large language models (LLMs) demonstrate considerable gains on existing reasoning benchmarks with strategies such as chain-of-thought ( CoT ) Wei et al. ( 2022 ) and few-shot demonstrations Brown et al. ( 2020 ) . Vision-language models (VLM) Alayrac et al. ( 2022 ); Chowdhery et al. ( 2022 ); Driess et al. ( 2023 ) have furthered LLMs’ capabilities by adding the visual modality to perform tasks such as visually-guided text generation Rose et al. ( 2023 ); Zhu et al. ( 2023 ) , vision question-answering Wang et al. ( 2022a ); Kim et al. ( 2021 ) , and image captioning Li et al. ( 2022 ); Liu et al. ( 2023 ) . We believe the logical next step is to extend these existing models to the video domain. VIP’s annotated collection of extracted keyframes from real-world videos offers a resource to evaluate reasoning abilities within the video domain.
Datasets for Video Understanding.
Existing video datasets are often limited by domain specificity or require a supplementary representation (e.g., audio, text) Lei et al. ( 2018 , 2020a ); Tapaswi et al. ( 2016 ); Miech et al. ( 2019 ); Kim et al. ( 2017 ); Mun et al. ( 2017 ) .
These datasets also provide simplifications as textual summaries Xu et al. ( 2016 ); Guadarrama et al. ( 2013 ) or a single video frame Yu et al. ( 2019 ); Zeng et al. ( 2017 ); Maharaj et al. ( 2017 ) , which by themselves can be sufficient to complete the task. While some datasets consider the multi-frame component Jang et al. ( 2017 ); Yi et al. ( 2020 ); Mun et al. ( 2017 ); Fu et al. ( 2023b ) for higher order complexity, VIP differs in trying to reduce the computational intensity of video reasoning without reducing the task difficulty and generality. VIP is a dataset of real-life videos that spans a breadth of domains and assesses multi-hop, multi-frame video reasoning without requiring significant computation to train on videos.
Textual Representations of Videos.
Early video models are trained with visual keyframes and textual questions as input and return textual answers as output Sukhbaatar et al. ( 2015 ); Kim et al. ( 2017 ); Jang et al. ( 2017 ) . Then, researchers started to unify the video, keyframe, and text embedding spaces Miech et al. ( 2019 ); Kim et al. ( 2018 ); Zellers et al. ( 2021 ); Kim et al. ( 2017 ); Guadarrama et al. ( 2013 ); Bhattacharya et al. ( 2023 ) . VidIL leverages the contemporary in-context inference paradigm with few-shot demonstrations, including frames, captions, and visual tokens to prompt language models to solve VidL tasks Lei et al. ( 2020b ); Wang et al. ( 2022b ) .
In contrast to these existing works, VIP introduces textual representations at the keyframe level and then leverages them to reason about specific video segments using VideoCOT.
VIP Dataset Construction
To construct the dataset, we first outsource the video corpus to stem from the YouTube-8m dataset Abu-El-Haija et al. ( 2016 ) , whose diverse, realistic videos with human-labeled topics align well with our desiderata. To effectively enable multi-frame video reasoning, we downsample visually static categories such as weather, which may not contain much change throughout the video. The weights of each category 2 2 2 Videos may be in multiple categories. is described in Figure 2 .

Then, to reduce the computational complexity of video processing, we reduce a video into a set of keyframes that seek to capture the video’s meaning ( section 3.1 ). To accommodate the limitations of existing models, we also generate two forms of visually-descriptive, textual representations– unstructured dense captions and FAMOuS scene descriptions ( section 3.2 ). Figure 3 summarizes this automated pipeline, which reduces the cost ordinarily spent to manually construct such a dataset.
Selecting Video Frames.
The bottleneck to models in the video modality is the computational intensity. We select video frames that best capture the overall video content to mitigate this issue. Instead of training a model to choose a dynamic number of keyframes – which would be computationally expensive – we design an algorithm to prune semantically similar keyframes (Algorithm 1 ). However, selecting keyframes in this manner comes with the tradeoff that too many frames can introduce redundancy while too few can remove critical context. We choose a large set of candidate keyframes to balance these considerations, which we then dynamically prune. We employ an off-the-shelf keyframe extractor instead of learning a model ourselves. We choose to use Katna 3 3 3 https://github.com/keplerlab/katna as the baseline keyframe extraction tool as it is open-sourced and easy to onboard. Katna selects keyframes by leveraging the differences in LUV colorspace, brightness, contrast, blur, and k-means clustering of images.
Pruning Redundant Frames.
Once baseline candidate keyframes are selected, we prune them by removing low-quality, semantically similar frames. First, we remove blurry keyframes with low Laplacian scores, which indicate the absence of intensity changes. Then, we use object detection models Detic Zhou et al. ( 2022 ) and GRiT Wu et al. ( 2022 ) to filter keyframes that contain minimal objects, which indicates blurriness as these models are quite sensitive to all background objects.

After removing low-quality frames, we use CLIP Radford et al. ( 2021 ) to create embeddings for the keyframe image and its list of detected objects and positions from the previous step in the pipeline. This combination helps us compare frames using pixel similarity and object invariance. We take the average cosine similarity score for the keyframe’s image and object embeddings compared to the surrounding keyframes and prune the frames with the highest similarity. As people tend to be the primary subject of these videos, we add an additional check only to prune keyframes containing people if either of the surrounding frames also includes people.
Textual Representations of Keyframes
Next, to complement the keyframe images, we construct two textual representations of scenes: an unstructured, dense caption that provides visually descriptive insight into the scene; and second, a FAMOuS scene description that offers a structured approach to the reasoning process. We first generate the dense caption, which we then use to extract specific information for the structured scene description. These frame descriptions allow for leveraging existing LLM/VLM capabilities for video reasoning and generation.
Unstructured, Dense Captions.
To create visually descriptive frame descriptions, we first extract three things from each keyframe: a caption, an object list, and a dense caption list. Together, these outputs paint a visual description of the keyframe – the object list and the dense captions describe the focus, objects, and setting, while the caption details the focus, action, and mood. We specifically use Detic Zhou et al. ( 2022 ) , a tool that accurately detects objects without much detail, to simply list the frame’s objects. To extract more descriptive, object-level detail for high-quality scene descriptions, we use GRiT Wu et al. ( 2022 ) , which returns dense captions describing each object. Finally, we obtain the keyframe’s overall caption using LLaVA Liu et al. ( 2023 ) .

Because these individual models are prone to hallucination Dai et al. ( 2023 ) , we ensure the accuracy of our unstructured descriptions by engineering Detic and GRiT to return confidence scores for each of their outputs. Additionally, we utilize the Wiki descriptions of each video topic in the YouTube-8M dataset to extract a grounding list of baseline objects using GPT-4 . Finally, we feed in all of the extracted outputs into GPT-4 to generate the final dense caption.
FAMOuS Structured Scene Descriptions.
Structure can improve the reasoning ability of a model by providing concrete targets. To provide structure, we take inspiration from scene plays which clearly label and describe the scene. Specifically, we identify and extract the focus, action, mood, objects, and setting from the dense caption using GPT-4 , categories which should capture the most important visual information in a concise, structured manner.
Dataset Contributions
The VIP dataset is the first to evaluate multi-hop video reasoning via a video-chain of thought. This novel paradigm promotes efficient and robust video reasoning through automated keyframe extraction (Algorithm 1 ) over a breadth of domains ( Figure 2 ). Our two textual representations of keyframes ( Figure 1 ) add significantly granularity to videos (with an average caption length of 114 tokens) compared to traditional video caption datasets ( Table 1 ). This enables reasoning on more specific visual and semantic changes which occur between frames, more closely mimicking how humans process videos by thinking frame by frame.

To ensure the quality of our collected dataset, we verify correctness via crowdsourcing on Amazon Mechanical Turk (Appendix A ). Workers are paid to evaluate the quality of structured scene descriptions and edit those of low quality. Unstructured dense captions are corrected using the validated structured scene descriptions with GPT-4 and verified with another round of human evaluation.
Video Reasoning Tasks
Taking inspiration from existing natural language tasks, we propose two tasks for videos that explore a model’s multi-frame reasoning capabilities. The Video Infilling task requires models to predict a set of keyframes given the preceding and following frames, akin to masked language modeling for keyframes. Video Prediction tasks models to predict the most likely sequence of frames to follow a given series of frames - parallel to the text completion task. Video infilling and prediction of keyframes are two general tasks with several downstream contexts that can benefit from video understanding and completion.

To concretely define the tasks below, we represent the sequence of chronological keyframes as k_{1},\ldots,k_{n} , their respective unstructured dense captions as u_{1},\ldots,u_{n} , and FAMOuS structured scene descriptions as s_{1},\ldots,s_{n} .
Video Infilling Task
Suppose a subsequence of frames k_{i},\ldots,k_{j} is masked. In the video infilling task, the target is for a model to learn to reconstruct these masked frames using preceding context frames k_{i-n},\ldots,k_{i-1} and following context frames k_{j+1},\ldots,k_{j+n} where n is the number of frames provided as context. Without loss of generality, this task follows for both textual representations using u and s as inputs and outputs instead of k . In the multimodal setting, we can use pairs (k,u) or (k,s) as inputs and outputs.

This task requires models to capture a scene’s temporal variations and transitions, including changes in visual elements, object positions, and contextual factors. Furthermore, the task’s difficulty scales two-fold. First, decreasing the context window n will reduce the ability to leverage hints from surrounding keyframes to infill informatively; combined with the necessity to perform multi-hop reasoning between each pair of frames in sequence, insufficient context could result in training divergence. Second, increasing the number of frames to predict in sequence between i,j also raises similar challenges as too large a gap could add several degrees of freedom, resulting in significant infilling variability. Successfully predicting intermediate keyframes may illuminate models’ abilities to reason through the dynamic evolution of scenes and identify critical deltas in videos.
Video Prediction Task
Suppose we are given a sequence of context frames k_{i-n},\ldots,k_{i} . In the video prediction task, we aim to predict the f following frames k_{i+1},\ldots,k_{f} . Without loss of generality, this task follows for the unimodal text and multimodal representations. Much like the infilling task, the difficulty increases by decreasing the context window or increasing the prediction span. Since the prediction task only provides past context, predicting a longer sequence following may be harder as the possibilities increase exponentially.
Setup.
Although it would be ideal to benchmark multi-modal language models on our proposed tasks, the current pre-trained models (e.g., Open Flamingo and Otter Awadalla et al. ( 2023 ); Li et al. ( 2023 ) ) are not designed to accommodate multiple image inputs off-the-shelf. As a result, we chose to benchmark the video infilling and prediction tasks as a language task, generating keyframes as represented by dense captions or FAMOuS descriptions. We use GPT-3 , GPT-4 , and Vicuna 4 4 4 We use the pre-trained Vicuna -13B checkpoint. as leading models, with in-context inference using one demonstration in both our infilling and prediction tasks. To mitigate hallucination, we leverage greedy decoding. In each task, the goal is to infill or predict three intermediate or subsequent keyframes, respectively. Evaluation metrics are computed as the mean of these three generated keyframes compared to the ground truth. Results are reported using one prompt, but a follow-up analysis shows prompt stability through low-variation among other prompts ( Table 2 ).
Metrics.
We use three standard text comparison metrics: ROUGE L , BERTScore Zhang* et al. ( 2020 ) , and SentenceBERT Reimers and Gurevych ( 2019 ) . ROUGE L is best suited for tasks aimed to generate text that exactly matches the ground truth. BERTScore leverages BERT embeddings, which utilize the surrounding context. SentenceBERT is similar to BERTScore but computes the similarity of texts using sentence-level embeddings instead of word embeddings. These metrics combined provide initial scope into keyframe generation performance from both the semantic and contextual perspective.
Primary Results
We break down the primary results ( Table 3 ) into four key points.
Number of Context Frames.
Although the output size is fixed, we investigate how varying the input size affects the complexity of the VIP tasks. Consistent with intuition, we observe higher performance given additional context. However, the performance boost with each additional keyframe is marginal. With scores for all three metrics significantly lower than other tasks of similar spirit , it appears that our multi-hop, multi-frame prediction task is quite challenging using only textual representations for existing state-of-the art language models . This low baseline performance may overshadow the change in difficulty as a result of varying context frames.
Dense Captions vs FAMOuS Descriptions.
Dense captions consistently show stronger performance using our selected metrics than FAMOuS descriptions. As our evaluation metrics emphasize word similarity, they may favor dense captions which contain filler words used to form complete sentences. In the FAMOuS structure, descriptions are broken down by category, which reduces the verbosity, thereby increasing the difficulty for word comparison metrics.
Infilling vs Prediction Tasks.
We consistently observe that models have stronger performance on the infilling task compared to the prediction task . To most fairly compare the two tasks, compare the Infilling-1 task, which aims to predict three intermediate frames given one predecessor and one successor keyframe, with the Prediction-2 task, which aims to predict the three keyframes following the two context frames. Aside from a few non-significant outliers, these models perform better across all metrics and both textual representations. This is inline with intuition as bidirectional context reduces the complexity of the problem .
Individual Model Performance.
Across models, the performance does not follow any obvious trends, which is surprising considering the size difference in the open-source Vicuna compared to the human-reinforced GPT-3 and GPT-4 models. By metrics, we observe GPT-3 and Vicuna performs slightly better performances on ROGUE L compared to GPT-4 , suggesting exact word consistency may be better in these earlier models. GPT-4 ’s and GPT-3 ’s edge in SentenceBERT suggest their generations may align more semantically than an off-the-shelf Vicuna .
FAMOuS Component Analysis
We decompose model performance on FAMOuS structured scene descriptions on a component level ( Table 4 ) for the Prediction-3 task, which aims to predict the three keyframes following the three input keyframes. Through ROUGE L , we observe models perform significantly better identifying objects , compared to the other four components. Comparing BERTScore , models appear to semantically align with the ground truth on the focus component better and more poorly in understanding of the keyframe’s action . Finally, the SentenceBERT results suggest that models better maintain overall sentence similarity when considering components of the image’s environment, such as mood , objects , and setting . These trends highlight that reasoning through textual representations for basic components such as keyframe focus and objects is a strength of language models, while reasoning about more dynamic components such as the action necessitates a more intricate understanding of the keyframes and could benefit from a video representation.
Causal Aspect Analysis
We examine the difference in performance between physical and social causal reasoning ( Table 5 ). A task necessitates physical causal reasoning when the video changes stem from external, tangible forces, like a wave crashing on a beach. Conversely, a task involves social causal reasoning when video changes result from social cues, such as a character becoming joyful during a surprise party. Observation of the results show that social causal reasoning tasks scored higher on BERTScore while physical causal reasoning tasks scored higher on SentenceBERT .
These results may be an outgrowth of the FAMOuS Component Analysis section 5.2 , where a consistent character focus and objects present in many social scenarios yield higher token-level similarity with BERTScore . By contrast, the consistent environmental qualities like action or mood – present in many physical scenarios– result in a greater SentenceBERT score.
Domain Analysis
We also outline the overall results from all experiments corresponding to the different visual domains of our videos in Table 6 . Although we found several categories to be noisy due to low sample sizes, certain categories like Games perform well, while others like Jobs & Education fall behind. We hypothesize that the availability of domain-specific training data as well as intrinsic dimensionality needed to model interactions within these topics jointly contribute to such observations.
Qualitative Observations
Figure 5 provides a visual depiction of the outputs generated by GPT-4 and Vicuna for the prediction task. Inspecting the depicted outputs from both models, it’s evident that they lack some semantic congruence with the ground truth, underscoring the limitations that language model-based approaches face in in video reasoning. Figure 7 and Figure 5 further demonstrate impressive early performance using video-chain of thought, though the examples’ strikingly similar output suggests some overfitting to training data. Despite the observable limitations, it’s clear that the language models have a clear baseline video understanding. Still, both the quantitative and qualitative axes highlight that only using unimodal language doesn’t generalize their strong language task performance to VIP ’s video reasoning tasks, which naturally opens several areas for subsequent research threads.
Future Work
In this paper, we aim to lay the groundwork for exploring the challenging topic of multi-frame, multi-hop reasoning within the existing capabilities of deep learning models. Naturally, this opens several directions for exciting future work.

In the language model space, we benchmark the performance of several leading models using textual representations of keyframes for video-related reasoning following a standard in-context inference procedure with few-shot demonstrations. This invites the opportunity to discover more targeted inference-time techniques using language models or vision-language models to improve the performance of video reasoning tasks beyond a general paradigm. Similarly, additional training-time effort could be worthwhile through fine-tuning or a more traditional train-validate-test paradigm to learn skills beyond the general pre-trained learnset. In this vein, collecting additional data samples could improve the feasibility of these research threads.

Beyond the language modality, bridging the video reasoning task end-to-end with video is a longer-term research direction with immediate benefits in animation. Our paper reduces the video reasoning task into a language task with a textual output. Image synthesis would be an immediate step to reconstruct the keyframe image. Then, video synthesis from a set of images would naturally follow. Finally, unifying these disjoint tasks could benefit from error reduction and improved usability.

As video reasoning is a new space, developing robust evaluation metrics would be a valuable contribution. Some desirable but difficult properties to consider in this area include the ability to capture both the spatial and temporal invariance that could occur through videos, as multiple interchangeable actions are plausible within different areas of the frame and sequences.

Finally, our general video reasoning tasks pose the prospect of efficient transfer learning where improving on such a task could benefit several new applications, similar to the contemporary boom of language technologies.
Conclusion
We present the inference-time Video Infilling and Prediction dataset to evaluate models’ video reasoning abilities by performing a video chain-of-thought on video keyframes. To collect this dataset, we introduce a novel pipeline to systematically extract keyframes and generate corresponding textual representations – unstructured dense captions and structured FAMOuS scene descriptions. We benchmark state-of-the-art language models on VIP tasks and evaluate their ability to generate these textual representations of keyframes through a video chain-of-thought. These models display potential to perform video-related reasoning yet have significant potential to improve. By testing multi-hop, multi-frame reasoning abilities on sparse keyframes, we hope to promote research into developing models that support real-world video understanding and video generation while being resource efficient.
Model Selection for Benchmarking.
The primary limitation of our work is that current multimodal models do not support multiple image inputs per input query as a trivial use-case. As a result, despite our paper proposing a novel dataset intended for video-related reasoning, we currently only benchmark large language models. We encourage future research to explore the reasoning capabilities for video scene description generation.
Automation Process.
While our work aims to systematically generate samples to evaluate the video-related reasoning capabilities of existing AI systems, we acknowledge the potential for error when using other AI systems to generate such examples. As a result, we add a layer of human supervision where crowd workers are used to first to classify whether generated scene descriptions are sufficiently correct. Then, we must make use of expert annotators to manually correct the generated scene descriptions that were flagged as poor quality for quality assurances purposes of this dataset.
Potential for Bias.
We acknowledge the potential for bias in the data collection process and inference task design. We have taken a number of steps to mitigate bias, including ensuring diversity in video content selection, and regularly reviewing and refining the annotation process to minimize any unconscious bias. In addition, we are committed to addressing and correcting any biases that may arise during the evaluation and analysis of VIP model performance. Our dataset is restricted to English captions, thereby containing a bias toward culturally Western scenes. In a multilingual setting differential behaviors between language classes would probably be observed Saxon and Wang ( 2023 ) .