Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought
Authors: Vaishnavi Himakunthala
Andy Ouyang
Daniel Philip Rose
Ryan He
Alex Mei
Yujie Lu
Chinmay Sonar
Michael Saxon
William Yang Wang
Venue: EMNLP 2023 Main (2023)
Keywords: chain of thought
video reasoning
large language models
dataset
vision and language

Abstract: Despite exciting recent results showing vision-language systems’ capacity to reason about images using natural language, their capacity for video reasoning remains underexplored. We motivate framing video reasoning as the sequential understanding of a small number of keyframes, thereby leveraging the power and robustness of vision-language while alleviating the computational complexities of processing videos. To evaluate this novel application, we introduce VIP, an inference-time challenge dataset designed to explore models’ reasoning capabilities through video chain-of-thought. Inspired by visually descriptive scene plays, we propose two formats for keyframe description: unstructured dense captions and structured scene descriptions that identify the focus, action, mood, objects, and setting (FAMOuS) of the keyframe. To evaluate video reasoning, we propose two tasks: Video Infilling and Video Prediction, which test abilities to generate multiple intermediate keyframes and predict future keyframes, respectively. We benchmark GPT-4, GPT-3, and VICUNA on VIP, demonstrate the performance gap in these complex video reasoning tasks, and encourage future work to prioritize language models for efficient and generalized video reasoning.

Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Shain-of-Though
Vaishnavi Himakunthala*, Andy Ouyang*, Daniel Rose*, Ryan He*, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, William Yang Wang
University of California, Santa Barbara
(vaishnavi, andyouyang, danielrose, ryanhe, yujielu, saxon]@ucsb... edu {alexmei, csonar, wi william}@cs. www.uuussee..... edu
Abstract
like how humans understand videos by processing frames across time steps, Al's ability to accom- plish multi-frame reasoning a core task for video understanding.
Despite exciting recent results showing vision- language systems' capacity reason about im- ages using natural language, their capacity for video reasoning remains underexplored. We motivate framing video reasoning the se- quential understanding of a small number of keyframes, thereby leveraging power and robustness of rision-language while alleviat- ing the computational complexities of process- ing videos. To evaluate this novel application, we introduce VIP!, an inference-time challenge dataset designed to explore models' reason- ing capabilities through video thoouh. Inspired by visually descriptive scene plays, we propose two formats for keyframe descrip- tion: unstructured dense captions and struc- tured scene descriptions that identify the focus, action, mood, objects, and setting (FAMOuS) of the keyframe. To evaluate video reasoning. we propose two tasks: Video Infilling and Video Prediction, which test abilities generate mul- tiple intermediate keyframes and predict future keyframes, respectively. We benchmark GPT-4 GPT-3, and VICUNA on VIP, demonstrate the performance gap in these complex video rea- soning tasks, and encourage future work pri- oritize language models for efficient and gener- alized video reasoning.
Multi-frame video reasoning is bottlenecked by the sheer computing resources needed to process videos, which typically contain 24 frames per sec- ond and can vary widely. However, intelligible videos tend to have little variation from frame to frame. Akin to selecting principle components on the axes containing the highest orthogonal variance, picking small sample of keyframes can capture much of the video's meaning. Understanding a video via extracted keyframes poses the challenge of multi-hop reasoning, which requires stronger language model capabilities.
To evaluate this challenging multi-hop multi- frame video reasoning, we elicit video chain-of- thought (VIDEOCOT). We propose two tasks to test such capabilities in existing models - Video Infilling and Video Prediction. In the Video In- filling setting, the goal is to predict the masked keyframes' descriptions when given the previous and next keyframes' descriptions in sequence context, following the spirit of masked language modeling. In the Video Prediction, the goal is predict the descriptions of the next keyframes in the sequence when given a set of previous keyframes' descriptions, similar to the next token prediction task. These tasks can help analyze video generation and whether video models truly understand the dy- namic relations between subsequent video frames, given the variable context gap between keyframes.
1 Introduction
Constituting 65% of all internet traffic in 2023,
videos are an area of huge potential for the next chapter of leveraging artificial intelligence (Fu et al., 2021; Zellers et al., 2021; Fu et al., 2023a). For example, Video Question Answering (Lei et al., 2018) and Video Summarization (Xu et al., 2016) are two existing datasets that empirically evalu- ate video models. Yet, they do not assess more challenging tasks, such reasoning through spe- cific relationships between multiple frames. Just
To benchmark our proposed Video Infilling and Prediction tasks, we construct dataset by propos- ing automated method extract keyframes. In addition the frame itself, we propose two textual representations unstructured dense captions and FA MOuS structured scene descriptions (Figure 1). The unstructured captions are intended to extract more significant and visually descriptive informa-
*Denotes equal contribution.

---

Extracted Keyframes
Structured Scene Descriptions (FAMOuS)
Unstructured Dense Captions
Focus: Man holding a chainsaw preparing to cut
a dense forest, a man is preparing to cut down Christmas tree, creating mood of anticipation. The man, wearing appropriate protective gear, holds chainsaw, ready to act. The scene is marked by the contrast between the excitement of upcoming Christmas celebrations and the focus of cutting down the tree. Key elements in the scene, including the man, the chainsaw, the trees, and the brown dirt, contribute to this poignant snapshot the holiday season, highlighting the intersection of tradition and respect for nature.
down the Christmas tree
Action: Man getting ready to fell tree
Mood: Anticipation
Objects: Man, chainsaw, trees in the forest,
brown dirt
Setting: the forest among trees with brown dirt on the ground
Figure I: The Video Infilling and Prediction Dataset consists of two ways to describe keyframes: an unstructured dense caption and structured scene description with five components: Focus, Action, Mood, Objects, and Setting (FAMOUS). The unstructured dense captions are highly detailed dense captions that can promote visually descriptive reasoning tasks, while structured scene description provide concise, visual description the keyframe that aid in more focused reasoning tasks.
tion compared to existing captioning systems, a necessary component to provide enough context for more challenging tasks. In addition, we system- atically create structured scene descriptions from these unstructured dense captions by extracting the frame's focus, action, mood, objects, and setting using weak human supervision for quality assur- ance. The FAMOuS categories are inspired by play scripts, which maintain much the visual detail of the unstructured captions while providing a clear, structured way to reason through visual tasks with high degrees of freedom.
Related Work
AI Reasoning. Large language models (LLMs) demonstrate considerable gains on existing rea- soning benchmarks with strategies such as chain- of-thought (COT) (Wei et al., 2022) and few- shot demonstrations (Brown et al., 2020). Vision- language models (VLM) (Alayrac et al., 2022: Chowdhery al., 2022: Driess et al., 2023) have furthered LLMs' capabilities by adding the visual modality to perform tasks such as visually-guided text generation (Rose et al., 2023; Zhu et al., 2023), vision question-answering (Wang et al., 2022a; Kim et al., 2021), and image captioning (Li et al., 2022; Liu et al., 2023). We believe the logical next step is to extend these existing models to the video domain. VIP's annotated collection of extracted keyframes from real-world videos offers resource to evaluate reasoning abilities within the video do- main.
We propose the following contributions:
We systematically collect an inference-time chal- lenge dataset of keyframes for video reasoning augmented with two textual representations: un- structured dense captions for information and FAMOuS scene descriptions for structured reasoning.
Datasets for Video Understanding. Existing video datasets are often limited by domain speci- ficity or require supplementary representation (e.g., audio, text) (Lei et al., 2018, 2020a; Tapaswi et al., 2016; Miech al., 2019; Kim al., 2017:
We propose the Video Infilling and Video Pre- diction tasks to benchmark the video chain-of- thought capabilities in existing models.
We empirically demonstrate that existing mod- els have the potential for multi-hop multi-frame video reasoning tasks but have a significant area for improvement as future work.
Mun et al., 2017). These datasets also provide sim- plifications as textual summaries (Xu al., 2016; Guadarrama et al., 2013) or a single video frame (Yu et al., 2019; Zeng et al., 2017 77; Maharaj et al.,

---

2017), which by themselves can be sufficient to complete the task. While some datasets consider the multi-frame component (Jang et al., 2017: Yi et al., 2020; Mun et al., 2017; Fu et al., 2023b) for higher order complexity, VIP differs in trying to reduce the computational intensity of video reason- ing without reducing the task difficulty and general- ity. VIP is dataset of real-life videos that spans breadth of domains and assesses multi-hop, multi- frame video reasoning without requiring significant computation to train on videos.
Hobbles
Leisure
Food Drink
Business &
Industrial
Perts &
Animais
Travel
Autos & Vehicles
Home &
Video Categories
Jobs &
Education
Law &
Textual Representations of Videos. Early video models are trained with visual keyframes and tex- tual questions as input and return textual answers as output (Sukhbaatar et al., 2015; Kim et al., 2017; Jang ct al., 2017). Then, researchers started to unify the video, keyframe, and text embedding spaces (Miech al., 2019; Kim al., 2018; Zellers et al., 2021; Kim et al., 2017; Guadarrama et al., 2013; Bhattacharya al., 2023). VidIL leverages the contemporary in-context inference paradigm with few-shot demonstrations, including frames, cap- tions, and visual tokens to prompt language models to solve VidL tasks (Lei et al., 2020b; Wang et al., 2022b). In contrast to these existing works, VIP introduces textual representations at the keyframe level and then leverages them reason about spe- cific video segments using VideoCOT.
People & Society
Reference
Science
Games
Health
Real Estate
Proportion Videos Covered
Figure 2: Distribution of VIP's real- worldworld video do- mains, weighted to emphasize videos containing signifi- cant visual change.
3.1 Representative Keyframe Selection
3
VIP Dataset Construction
Selecting Video Frames. The bottleneck to mod- els in the video modality is the computational inten- sity. We select video frames that best capture the overall video content to mitigate this issue. Instead of training model to choose dynamic number of keyframes which would be computationally expensive we design an algorithm to prune seman- tically similar keyframes (Algorithm 1). However, selecting key framees in this manner comes with the tradeoff that too many frames can introduce redun- dancy while too few can remove critical context. We choose large set of candidate keyframes to bal- these considerations, which then dynami- cally prune. We employ off-the-shelf keyframe extractor instead of learning model ourselves. We choose to use K KATNA² as the baseline keyframe extraction tool as open-sourced and easy to onboard. KATNA selects keyframes by leverag- ing the differences in LUV colorspuce, brightness, contrast, blur, and k-means clustering of images.
To construct the dataset, we first outsource the video corpus to stem from the YouTube-8m dataset (Abu-El- El-Haijaa al., 2016), whose diverse, realistic videos with human-labeled topics align well with our desiderata. To effectively enable multi-frame video reasoning, we downsample visually static categories such as weather, which may not contain much change throughout the video. The weights of each category² is described in Figure 2.
Then, reduce the computational complexity of video processing, reduce video into set of keyframes that seek capture the video's mean- ing ($3.1). To accommodate the limitations of existing models, we also generate two forms of discally- textual representations- un- structured dense captions and FAMOuS scene de- scriptions ($3.2). Figure 3 summarizes this auto- mated pipeline, which reduces the cost ordinarily spent to manually construct such dataset.
"Vidoos may be in multiple categorics.

---

Object Detection Model
Object list
!
Keyframe Extraction & Pruning
Dense Caption Model
Dense Caption List
?
Video
Keyframes
Image Captioning Model
Caption
1. Converting video keyframes
2. Extracting Information from each keyframe
Object Detection Model
Confidence Scores for A object-info extracted from each keyframe
Unstructured, Dense Scene Description
Inputs: D I
GPT-4
Dense Caption Model
Wiki Descriptions of Video Topics
GPT44
Grounding Object (
Structured Scene Description
GPT-4
3. Extractinn Grounding Information
4. Generating Scene Descriptions for each keyframe
Figure 3:
Overview the pipeline to generate scene descriptions provided in VIP Dataset. We first process a video and extract the important frames ($3.1), then generate scene descriptions by extracting visual information from each keyframe, along with grounding information from the video to offset model hallucinations. We then feed the extracted information into GPT-4 generate the dense captions structured scene descriptions. ($3.2).
Algorithm frame extract(e, c, f)
(Radford et al., 2021) to create embeddings for the keyframe image and its list of detected objects and positions from the previous step in the pipeline. This combination helps compare frames using pixel similarity and object invariance. We take the average cosine similarity score for the keyframe's ' image and object embeddings compared the sur- rounding keyframes and prune the frames with the highest similarity. As people tend to be the primary subject of these videos, we add an additional check only to prune keyframes containing people if either of the surrounding frames also includes people.
Data:
video ints c, of the candidate and finalized keyframe counts, respectively.
Result: List of f finalized keyfrumes from v.
Extract initial frames and emheddings:
ki...., Knp f- K Katna(e, v.c)
f1..... ln C1.JJ P( (etie(ka,.... CLIP(LPP,........ km.)
while lem(k) do
Remove frame with highest adjacent similarity: for j in range(len(k)) do costezt t- cus(L;, 44-1, (5411 co8tmage +- cos(t;; 11-1, scores mean(cos co8image)
k[s] where argmax, (scoresss) 10 return ky.....ky
3.2 Textual Representations of Keyframes
Figure 4: frame_extrael extract returns a list of selected keyframes from video u. First, we extract candidates using Katna. These keyframes CLIP in the image space; Detic extracts objects from the keyframes into textual representation, which also embedded with CLIP. Then, we iteratively prune the keyframe with the highest cosine similarity with adjacent frames until f keyframes remain.
Next, to complement the keyframe images, we con- struct two textual representations scenes: an unstructured, dense caption that provides visually descriptive insight into the scene; and second, a FAMOuS scene description that offers a structured approach to the reasoning process. We first gener- the dense caption, which then to extract specific information for the structured scene de- scription. These frame descriptions allow for lever- aging existing LLM/VLM capabilities for video reasoning and generation.
embedded using
Pruning Redundant Frames. Once baseline candidate keyframes are selected, we prune them by removing low-quality, semantically similar frames. First, we remove blurry keyframes with low Laplacian scores, which indicate the absence of intensity changes. Then, we use object detection models DETIC (Zhou et al., 2022) and GRIT (Wu et al., 2022) to filter keyframes that contain minimal objects, which indicates blurriness these models are quite sensitive to all background objects.
Unstructured, Dense Captions. To create visu- ally descriptive frame descriptions, we first extract three things from each keyframe: caption, ob- ject list, and dense caption list. Together, these outputs paint a visual description of the keyframe- the object list and the dense captions describe the focus, objects, and setting, while the caption details the focus, action, and mood. We specifically use
After removing luw-quality frames, we use CLIP

---

Dataset
Frame
Structured
Domain
Vid. Len.
Cap. Len.
Test Samples
MSR-VTT (Xu al., 2016)
X
Open
20.7s
9.6
3K
YouCook2 (Zhou al., 2018)
X
Cooking
5.26m
8.8
2K
ActyNet-Cap (Krishma al., 2017)
X
Open
2m
13.5
5K
HowTo100M (Miech al., 2019)
X
Instructional
18s
24K
VATEX (Wang
2019)
X
Open
10s
15.2
6K
VideoStory (I. et al., 2020)
Events
12.6m
12.1
WebVid-2M 2M (Bain et al., 2021)
X
Open 18s
VIP
Open 3.6m
114.2
1.5K
Table Statistics of video datasets: whether they include frame-level (Frame) and structured descriptions (Structured) as well as the average video token length (Vid. Len.), average caption token length (Cap. Len.). ) and number of inference samples (Test Samples). VIP is the only video reasoning dataset that has an open video domain and frame-level descriptions and provides novel structured frame descriptions.
DETIC (Zhou et al., 2022), a tool that accurately de- tects objects without much detail, to simply list the frame's s objects. To extract more descriptive, object- level detail for high-quality scene descriptions, we use GRIT (Wu et al., 2022), which returns dense captions describing each object. Finally, obtain the keyframe's 's overall caption using LLAVA (Liu et al., 2023).
an average caption length of 114 tokens) compared to traditional video caption datasets (Table 1). This enables reasoning on more specific visual and se- mantic changes which occur between frames, more closely mimicking how humans process videos by thinking frame by frame.
To ensure the quality of our collected dataset, we verify correctness crowdsourcing on Amazon Mechanical Turk (Appendix A). Workers are paid to evaluate the quality of structured scene descrip- tions and edit those of low quality. Unstructured dense captions are corrected using the validated structured scene descriptions with GPT-4 and veri- fied with another round of human evaluation.
Because these individual models are prone to ballucination (Dai al., 2023), we ensure the accu- racy of our unstructured descriptions by engineer- ing DETIC and GRIT return confidence scores for each of their outputs. Additionally, we utilize the Wiki descriptions of each video topic in the YouTube-8h datasct to extract a grounding list of baseline objects using GPT-4. Finally, we feed in all of the extracted outputs into GPT-4 to generate the final dense caption.
Video Reasoning Tasks
Taking inspiration from existing natural language tasks, we propose two tasks for videos that ex- plore a model's multi-frame reasoning capabilities. The Video Infilling task requires models to pre- dict set of keyframes given the preceding and following frames, akin to masked language model- ing for keyframes. Video Prediction tasks models to predict the most likely sequence of frames follow a given series of frames parallel to the text completion task. Video infilling and prediction of keyframes are two general tasks with several downstream contexts that can benefit from video understanding and completion.
FAMOUS Structured Scene Descriptions. Structure can improve the reasoning ability of a model by providing concrete targets. To provide structure, we take inspiration from scene plays which clearly label and describe the scene. Specifically, we identify and extract the focus, action, mood, objects, and setting from the dense caption using GPT-4 categories which should capture the most important visual information in concise, structured manner.
To concretely define the tasks below, repre- sent the sequence of chronological keyframes k1, ... .. kns their respective unstructured dense cap- tions as .1,.... Un. and FAMOuS structured scene descriptions as . 8p-
3.3 Dataset Contributions
The VIP dataset is the first to evaluate multi-hop vidco reasoning via video-chain of thought. This novel paradigm promotes efficient and robust video reasoning through automated keyframe extraction (Algorithm 1) over a breadth of domains (Figure 2). Our two textual representations of keyframes (Fig- ure !) add significantly granularity to videos (with
4.1 Video Infilling Task
Suppose a subsequence frames hi,..., ... ky is masked. In the video infilling task, the target is for

---

Frame 1
Frame
Frame 3
Frame
Frame 5
Vip ground truth scane descriptions
Focus:
tone
Horse and rder
harse
Fecus: kider
Fqy sboending
Action: fiakig gallopping
Actions
absade
Action: kiding
Action
Meed:
opovoting
Moodi Dynomi, locsed
Enerpetic locused
Mood. Competiivve, focused
Cooppetiiiee engaged
Objects frder home ourse
Objectt
Objects:
Objects fides
Objects Horss, specztoss
Setting: Ousdoar situauue
hanke polo
aren cats
Sonting: Indocr arina
setting: porppettion
Setting: cring ueewaur stidacle couns
Setting Faramition cumpu Hine
Hose
kider horse
Paler and appraaciiii
Action: Landing
Vicuna genemted
followed
Fider tarring
Action
Objects: Hors, ridet cowboy
aportaching thei jump
Prompt:
deternined
You are given the scene description of video keyframes (frames 1, 2).
Setting: Showjjrrping
Objects: likle, hros, conday lud,
Objects: Rider hoise cawbay hatt
compet foon
sacilli, burier er
sondie, bonter or pole
Setting: Show umminn coerpetition
Setting Show Saanjuumpan jumping compettioon
Focas: Rida: and hosa
Fecus: fider and hane
Fecus: Hoisa sier
Let's think frame by frame: predict the next scene descriptions of the next keyframes (frames 3, and 5).
inril olher jumo
Action: Gulleiing knwwth the
Actin suczosstally dully jumping
GPT-4 generated
Action: Todar hosa Lending
cbsade
nnl obstide
Detemirer,, bocsed
Moed: frtumphent easted
Moad: accempl andd
Objects: Hoise 16er
Objects Home ridrt unwlozy
objects horee, comboy
sced ditt Setting: Show jump ng competttonn
seodia
Setting: jumping competittt
Setting: Show Lumping competition
Figure 5: Given number of context frames, the frame prediction task requires models to predict the following n frames. In this example, we provide two FAMOUS scene descriptions and use VICUNA and GPT-4 44 to predict the next three frames. Results emphasized in red differ from the ground truth.
a model to learn to reconstruct these masked frames using preceding context frames ki-n.... k and following context frames ký1 1...., kjin where is the number of frames provided as context. With- loss of generality, this task follows for both textual representations using and inputs and outputs instead of In the multimodal setting, we can use pairs (ke, u) or (k, s) as inputs and outputs.
Metric
Mean + STD
ROUGE,
17.75 0.43
BERTSCORE
18.97 + 0.47
SENTENCEBERT
53.50 1 0.50
Table 2: Average GPT-4 performance across three
prompts (Figure 6) on Infilling-1 task reported as mean + one standard deviation. The low standard devi- ation indicates prompt stability.
4.2 Video Prediction Task
This task requires models to capture a scene's temporal variations and transitions, including changes in visual elements, object positions, and contextual factors. Furthermore, the task's diffi culty scales two-fold. First, decreasing the context window will reduce the ability leverage hints from surrounding keyframes infill informatively; combined with the necessity to perform multi-hop reasoning between each pair of frames in sequence, insufficient context could result in training diver- gence. Second, increasing the number of frames to predict in sequence between i, also raises similar challenges too large gap could add several de- grees of freedom, resulting in significant infilling variability. Successfully predicting intermediate keyframes may illuminate models' abilities rea- son through the dynamic evolution of scenes and identify critical deltas in videos.
Suppose we are given a sequence of context frames ki-n....,k; k; In the video prediction task, we aim to predict the f following frames hi ky. Without loss of generality, this task follows for the unimodal text and multimodal representations. Much like the infilling task, the difficulty increases by decreasing the context window or increasing the prediction span. Since the prediction task only pro- vides past context, predicting a longer sequence fol- lowing may be harder as the possibilities increase exponentially.
Experiments
Setup. Although it would be ideal to benchmark multi-modal language models on our proposed tasks, the current pre-trained models (c.g., Open Flamingo and Otter (Awadalla et al., 2023; Li et al.,

---

Infilling-1
Infilling-2
Prediction-1
Prediction-2
Metric
Model
FAMOuS
Dense
FAMOuS
Dense
FAMOuS
Dense
FAMOuS
Dense
GPT-4
17.34
24.92
18.44
25.25
15.52
23.31
16.66
25.22
ROUGE₂
GPT-3
17.83
26.26
19.34
25,50
16.39
28.43
17.20
25.96
VICUNA
17.37
25.34
18.85
26.69
15.86
23.75
16.59
25.88
GPT-4
18.61
24.81
19.66
25.67
16.24
20,57
17.24
24.79
BERTSCORE
GPT-3
18.20
26.24
19.56
23.10
16.60
29.96
17.24
23.47
VICUNA
17.67
24.07
18.98
28.14
15.80
19.76
16.68
24.34
GPT-4
53.05
58.53
53.87
58,22
50.57
53.55
51.54
57.06
SENTENCEBERT
GPT-3
52.95
58.54
54.57
55.69
51.04
59.83
51.81
53.99
VICUNA 52.19
54.66
53.33
58.80
50.40
51.86
50.96
54.86
Table 3: Model performance infilling and prediction tasks when outputting three frames, with the best results underlined.. vary the number of context frames as indicated by dash task name (e.g.. Infilling-2 uses previous and two future context frames to predict three intermediate masked keyframes; rrediction-22 uses two preceding keyframes to predict the subsequent three keyframes). We evaluate models using both FAMOUS structured scene descriptions and unstructured dense captions. Models show weak performance overall.
2023)) are not designed to accommodate multiple image inputs off-the-sheef. As result, we chose to benchmark the video infilling and prediction tasks as a language task, generating keyframes as represented by dense captions FAMOUS de- scriptions. We use GPT-3, GPT-4, and VICUNA as leading models, with in-context inference using one demonstration both our infilling and predic- tion tasks. To mitigate hallucination, we leverage greedy decoding. In cach task, the goal is to infill or predict three intermediate subsequent keyframes, respectively. Evaluation metrics are computed as the mean of these three generated keyframes com- pared to the ground truth. Results are reported using one prompt, but a follow-up analysis shows prompt stability through variiaiition among other prompts (Table 2).
5.1 Primary Results
We break down the primary results (Table 3) into four key points.
Number of Context Frames. Although the out- put size is fixed, we investigate how varying the input size affects the complexity of the VIP tasks. Consistent with intuition, we observe higher per- formance given additional context. However, the performance boost with each additional keyframe marginal. With scores for all three metrics sig- nificantly lower than other tasks of similar spirit, it appears that our multi-hop, multi-fram prediction task is quite challenging using only textual rep- resentations for existing state-of-the art language models. This low baseline performance may over- shadow the change in difficulty as result of vary- ing context frames.
Dense Captions vs FAMOUS Descriptions. Dense captions consistently show stronger perfor- mance using our selected metrics than FAMOUS descriptions. As our evaluation metrics empha- size word similarity, they may favor dense captions which contain filler words used to form complete sentences. In the FAMOUS structure, descriptions are broken down by category, which reduces the verbosity, thereby increasing the difficulty for word comparison metrics.
Metrics. We use three standard text compari- son metrics: ROUGE,, BERTSCORE (Zhang* et al., 2020), and SENTENCEBERT (Reimers and Gurevych, 2019). ROUGE, is best suited for tasks aimed to generate text that exactly matches the ground truth. BERTSCORE leverages BERT em- beddings, which utilize the surrounding context. SENTENCEBERT is similar to BERTSCORE but computes the similarity of texts using sentence- level embeddings instead of word embeddings. These metrics combined provide initial scope into keyframe generation performance from both the semantic and contextual perspective.
Infilling Prediction Tasks. We consistently observe that models have stronger performance on the infilling task compared to the prediction task. To most fairly compare the two tasks, compare the Infil offilling-11 task, which aims to predict three in- termediate frames given one predecessor and one
Wee use the pre-trained VICUNA-13B checkpoint.

---

Metric
Model
FAMOuS
Dense
Focus
Action
Mood
Objects
Setting
GPT-4
17.14
26.05
13.12
9.64
19.56
28.82
14.52
ROUGE,
GPT-3
17.44
29.35
14.96
10.47
19.09
27.85
14.82
VICUNA
17.37
26.99
15.31
10.59
19.53
28.14
13.30
GPT-4
17.63
26.48
20.70
13.26
18.49
19.86
15.81
BERTSCORE
GPT-3
17.43
30.95
22.25
14.08
16.54
19.00
15,26
VICUNA
17.21
26.91
21.77
13.57
18.80
18.12
13.79
GPT-4
51.87
57.89
46.83
44.48
51.34
58.84
57.88
SENTENCEBERT
GPT-3
52.15
56.40
47.72
45.71
51.43
58.35
57.51
VICUNA
51.57
55.44
47.65
45.36
51.44
57.33
56.12
Table 4: Model performance the Prediction-3 task; the two leftmost numerical columns report the aggregate results, while the five rightmost columns report the individual FAMOUS component results. The hest results on the component level is underlined for each metrie. Language models perform worse predicting Action and Setting with textual representation inputs, highlighting where video representations could be most beneficial.
successor keyframe, with the Prediction-2 task, which aims to predict the three keyframes follow- ing the two context frames. Aside from a few non- significant outliers, these models perform better across all metrics and both textual representations. This is inline with intuition bidirectional context reduces the complexity of the problem.
highlight that reasoning through textual representa- tions for basic components such as keyframe focus and objects is a strength of language models, while reasoning about more dynamic components such as the action necessitates more intricate under- standing of the keysrames and could benefit from video representation.
Individual Model Performance. Across mod- els, the performance does not follow any obvi- ous trends, which is surprising considering the size difference in the open-source VICUNA com- pared to the human-reinforced GPT-3 and GPT-4 models. By metrics, we observe GPT-3 and VI- CUNA performs slightly better performances on ROGUE, compared to GPT-4, suggesting exact word consistency may be better in these earlier models. GPT-4's and GPT-3's edge in SENTENCE- BERT suggest their generations may align more semantically than an off-the-shelf VICUNA.
5.3 Causal Aspect Analysis
We examine the difference in performance between physical and social causal reasoning (Table 5). A task necessitates physical causal reasoning when the video changes stem from external, tangible forces, like wave crashing on beach. Conversely, task involves social causal reasoning when video changes result from social cues, such character becoming joyful during a surprise party. Observa- tion of the results show that social causal reasoning tasks scored higher on BERTSCORE while physi- cal causal reasoning tasks scored higher on SEN- TENCEBERT. These results may be an outgrowth of the FAMOuS Component Analysis $5.2, where consistent character focus and objects present in many social scenarios yield higher token-level similarity with BERTSCORE. By contrast, the consistent environmental qualities like action or mood- present in many physical scenarios- result in a greater SENTENCLBERT score.
5.2 FAMOUS Component Analysis
We decompose model performance on FAMOUS structured scene descriptions component level (Table 4) for the Prediction-3 task, which aims predict the three keyframes following the three input keyframes. Through ROUGE,, we ob- serve models perform significantly better identi- fying objects, compared to the other four compo- nents. Comparing BERTSCORE, models appear to semantically align with the ground truth on the focus component better and more poorly in derstanding of the keyframe's action. Finally, the SENTENCEBERT results suggest that models bet- ter maintain overall sentence similarity when con- sidering components the image's 's environment, such as mood, objects, and setting. These trends
5.4 Domain Analysis
We also outline the overall results from all exper- iments corresponding to the different visual do- mains of our videos in Table 6. Although found several categories to be noisy due to low sample sizes, certain categories like Games perform well, while others like Jobs Education fall behind. We hypothesize that the availability domain-

---

Metric
Social
Physical
fort could be worthwhile through fine-tuning more traditional train-validate-tes paradigm learn skills beyond the general pre-trained learnset. In this vein, collecting additional data samples could improve the feasibility of these research threads.
ROUGEL 18.68
17.70
BERTSCORE
20.81
16.61
SENTENCEBERT
52.70
55.44
Table GPT-4 performance videos partitioned by the physical (changes due outside, real forces) and social (changes related to social cues) on the Infilling-1 task with structured, FAMOUS descrip- tions. Significant differences between partitions underlined.
Beyond the language modality, bridging the video reasoning task end-to-end with video is a longer-term research direction with immediate ben- efits in animation. Our paper reduces the video rea- soning task into a language task with textual out- put. Image synthesis would be an immediate step reconstruct the keyframe image. Then, video syn- thesis from a set of images would naturally follow. Finally, unifying these disjoint tasks could benefit from error reduction and improved usability.
specific training data well intrinsic dimen- sionality needed model interactions within these topies jointly contribute to such observations.
5.5 Qualitative Observations
Figure 5 provides visual depiction of the outputs generated by GPT-4 and VICUNA for the prediction task. Inspecting the depicted outputs from both models, it's evident that they lack some semantic congruence with the ground truth, underscoring the limitations that language model-based approaches face in in video reasoning. Figure and Figure 5 further demonstrate impressive early performance using video-chain of thought, though the examples' strikingly similar output suggests some overfitting to training data. Despite the observable limitations, it's clear that the language models have clear base- line vidco understanding. Still, both the quantita- tive and qualitative axes highlight that only using unimodal language doesn't generalize their strong language task performance to VIP's video reason- ing tasks, which naturally opens several areas for subsequent research threads.
As video reasoning is a new space, developing robust evaluation metrics would be a valuable con- tribution. Some desirable but difficult properties to consider in this area include the ability to capture both the spatial and temporal invariance that could occur through videos, multiple interchangeable actions are plausible within different areas of the frame and sequences.
Finally, our general video reasoning tasks pose the prospect of efficient transfer learning where improving on such task could benefit several new applications, similar to the contemporary boom of language technologies.
Conclusion
We present the inference-tin time Vidco Infilling and Prediction dataset to evaluate models' video rea- soning abilities by performing video chain-of- thought on video keyframes. To collect this dataset, we introduce a novel pipeline to systematically ex- tract keyframes and generate corresponding tex- tual representations unstructured dense captions and structured FAMOUS scene descriptions. We benchmark state-of-the-art language models on VIP tasks and evaluate their ability to generate these textual representations of keyframes through video tfain-of-thought. These models display potential to perform video-related reasoning yet have significant potential improve. By test- ing multi-hop, multi-frame reasoning abilities on sparse keyframes, we hope to promote research into developing models that support real-world video understanding and video generation while being resource efficient.
6 Future Work
In this paper, we aim to lay the groundwork for ex- ploring the challenging topic of multi-frame, multi- hop reasoning within the existing capabilities of deep learning models. Naturally, this opens several directions for exciting future work.
In the language model space, we benchmark the performance of several leading models using tex- tual representations of keyframes for video-related reasoning following a standard in-context infer- ence procedure with few-shot demonstrations. This invites the opportunity to discover more targeted inference-time techniques using language models linionelanguage models to improve the perfor- mance of video reasoning tasks beyond a general paradigm. Similarly, additional training-time ef-

---

Limitations
Acknowledgements
Model Selection for Benchmarking, The pri- mary limitation of our work is that current multi- modal models do not support multiple image in- puts per input query as trivial use-case. As a result, despite our paper proposing novel dataset intended for video-related reasoning, we currently only benchmark large language models. We en- courage future research explore the reasoning capabilities for video scene description generation.
We thank our reviewers for their supportive feed- back. This material is based upon work supported in part by the National Science Foundation under Grants #1821415 and #2048 122 REU. The authors are solely responsible for the contents of the paper, and the opinions expressed in this publication do not necessarily reflect the official policy or position of associated funding agencies or past or present employers of the authors. The contents of this pa- per are not intended to provide, and should not be relied upon for, investment advice.
Automation Process. While our work aims to systematically generate samples to evaluate the video-related reasoning capabilities of existing Al systems, we acknowledge the potential for error when using other Al systems to generate such ex- amples. As a result, we add a layer of human supervision where crowd workers are used to first to classify whether generated scene descriptions are sufficiently correct. Then, we must make use of expert annotators to manually correct the generated scene descriptions that were flagged as poor quality for quality assurances purposes of this dataset.
References
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaanarasimman. 2016. Youtube- 8m: A large-scale video classification benchmark.
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-237366 2
Ethics Statement
Potential for Bias. We acknowledge the poten- tial for bias in the data collection process and in- ference task design. We have taken number of steps to mitigate bias, including ensuring diversity in video content selection, and regularly reviewing and refining the annotation process to minimize any unconscious bias. In addition, we are committed to addressing and correcting any biases that may arise during the evaluation and analysis of VIP model performance. Our dataset is restricted to English captions, thereby containing a bias toward cultur- ally Western scenes. In multilingual setting differ- ential behaviors between language classes would probably be observed (Saxon and Wang, 2023).
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hes- sel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Je- nia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Openflamingo: An open- framework for training large autoregressive Vision-languee models. arXiv preprint arXiv:2308.00
6
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zis- serman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. IEEE In- ternational Conference on Computer Vision.
Aanisha Bhattacharya, Yaman K Singla, Balaji Krishna- murthy, Rajiv Ratn Shah, and Changyou Chen. 2023. A video is worth 4096 tokens: Verbalize story videos to understand them in zero shot.
Crowdsoureing. Crowdsourcing via the Amazon Mechanical Turk platform was utilized for conduct- ing human evaluation. To ensure reliable results, we restricted participation to workers based in Aus- tralia, Canada, New Zealand, the United Kingdom, or the United States, with minimum HIT approval rating of 98%. Compensation for scene description checking and correction was set rate of $15 per hour. Our data collection is classified an ap- proved exempt protocol from the Institutional Re- view Board. Details of the interface can be found in Appendix A.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc. 2

---

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. 2
Kyung-Ming Min Kim. Min-Oh Heo, Scong-Ho Choi, and Byoung- Tak Zhang. 2017. Deepstory: Video story qa by deep embedded memory networks. In Pro- ceedings of the 26th International Joint Conference on Artificial Intelligence, 17, page 2016-2022. AAAI Press. 3,3
Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. 2023, Plausible may not be faithful: Probing object ballucination vision- Saiigag pre-training- In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 2136 22662148 Dubrovnik, Croatia. Association for Computational Linguistics.
Wonjae Kim, Bokyung Son, and Iidoo Kim. 2021. Vilt: Vision-and-language transformer without convolu- tion or region supervision. 2
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In International Conference on Computer Vision (ICCV).
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lchter. Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023. Palm-e: An embodied multimodal language model.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. TVQA: Localized, compositional video ques- tion answering. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 1369-1379 1379, Brussels, Belgium. Association for Computational Linguistics. 1,2
Jie Lei. Licheng Yu, Tamara Berg, and Mohit Bansal. 2020a. TVQA+: Spatio-temporaa grounding for video question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8211-8225, Online, Association for Computational Linguistics. 2
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang, and Zicheng Liu. 2021. VIOLET: End-to-Enn End LanguaLa Trans- formers with Masked Visual-token Modeling. In arXiv:2111.266.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal, 2020b. What is more likely to happen next? video- und-language future event prediction. arXiv preprint 3
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu, 2023a. An Empirical Study of End-to-End Video- Language Transformers with Masked Visual Model- ing. In Conference on Computer Vision and Pattern Recognition (CVPR).
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint 2305.03000033726 6
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng- Yang Fu, Long-Chyi Su, William Yang Wang, and Scan Bell. 2023b. Tell Me What Happened: Unifying Text- guided Video Completion via Multimodal Masked Video Generation. In Conference on Computer Vi- sion and Pattern Recognition (CVPR). 3
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified lasionn understanding and generation. In ICML. 2
Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkamennar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2013. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In Proceedings of the 2013 IEEE Inter- national Conference on Computer Vision, ICCV '13, page 27 2712-2719, USA. IEEE Computer Society. 2, 3
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. 2020. Video storytelling: Textual sum- maries for events. IEEE Transactions on Multimedia, 22(2):554-565. 5
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. 2023. Visual instruction tuning. 2, 5
T. Maharaj. N. Ballas, A. Rohrbach, A. Courville, and C. Pal. 2017. A dataset and exploration of models for understanding video data through question- answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7359-7368, Los Alamitos, CA. USA. IEEE Computer Society. 2
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3
Kyung- MMn Kim, Seong-He Ho Choi, Jin Hin- Kim, and Byoung-Tat Tak Zhang. 2018. Multimodal dual attention memory for video story question answering. In Pro- ceedings of the European Conference on Computer Vision (ECCV).
Antoine Miech, Dimitri Zhukov, Jean- Bran-Baptistte Alayrac. Makarand Tapaswi, Ivan Laptev, and Josef Sivie. 2019. HowTol Learning a Text-Video Embed- ding by Watching Hundred Million Narrated Video Clips. In /CCV. 2, 3,5

---

Jonghwan Mun, Paul Hongsuck Sco, Ilchae Jung, and Bohyung Han. 2017. Marioqa: Answering questions by watching gameplay videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2867-2875. 2,3
video-language learners. Advances Neural In- formation Processing Systems. 3
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems. 2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. 4
Jialian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2022. Grit: A generative region-to- to-oettt transformer for ob- ject understanding. arXiv preprint arXiv:2212.002880 4.5
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNL"), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics." 7
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr- vtt: A large video description dataset for bridging video and language. In IEEE International Confer- ence on Computer Vision and Pattern Recognition (CVPR). IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). 2,
Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang. Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chin- may Sonar, Diba Mirza, and William Yang Wang. 2023. Visual chain of thought: Bridging logical gaps with multimodal infillings.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenen- baum. 2020. CLEVRER: collision events for video representation and reasoning. In ICIR. 3
Michael Saxon and William Yang Wang. 2023. Multi- lingual conceptual coverage in text-to-image models. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics ( (Volume
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu. Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. AAAI, pages 9127-9133
1: Long Papers), pages 4831 1-4848, Toronto, Canada. Association for Computational Linguistics. 10
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. In Advances in Neural Information Process- ing Systems, volume 34, pages 23634-23651, Curran Associates, Inc. 1,3
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. Weakly supervised memory networks. ArXiv, has//5033088955
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR). 2
Kuo-Hao Zeng, Tseng- Chen, Ching-Yar Yao Chuang, Kuan-Hong Liao, Juan Carlos Niebles, and Min Sun. 2017. Leveraging video descriptions to learn video question answering. Proceedings of the AAAI Con- ference on Artificial Intelligence, 31(1). 2
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. Ofa: Unifying architectures, tasks, and modalities through a simple sequennce-t--sequeee learning framework. CoRR, abs/2202.03052.
Tianyi Zhang Varsha Kishore* Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations. 7
Luowei Zhou, Chenliang Xu, and Jason J. Corso. 2018. Towards automatic learning of procedures from web instructional videos. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/AAI
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan- Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, nigh-quality multilingual dataset for aaleo-and-langgage research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
18/EAAI 18/AAA''1//AAT"" 18, AAAI Press.
Zhenhailong Wang. Manling Li, Ruochen Xu. Lu- owei 7hou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2022b. Language models with image descriptors are strong few-shot
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. 2022. Detecting thousand classes using image-level supervi- sion. In ECCV. 4,5

---

Wanrong Zhu, An Yan. Yujie Lu, Wenda Xu, Xin Wang. Miguel Eckstein, and William Yang Wang. 2023. Vi- sualize before you write: Imagination-guide ended generation. In Findings the Association for Computational Linguistics: EACL 2023, pages 78-92, Dubrovnik, Croatia. Association for Compu- tational Linguistics. 2
open-
A Appendix
A.1 Human Evaluation Interface
We demonstrate the interface of our human eval- uation in scene description checking in Figure 9, We employ manual procedures guarantee the ex- clusion of personal information and the absence of offensive content during human evaluations.

---

Prompt 1 (Used for experimentation): You are given the scene descriptions of video keyframes (frames and 5). Let's think frame by frame: identify the scene descriptions of the intermediate keyframes that occur between the 2 given scene descriptions (frames 2, 3, and 4). Copy the structure and length exactly.
Your job is to interpolate the scene descriptions for the three intermediate keyframes (frames 2, 3, and 4) using the scene descriptions of the first and fifth keyframes. Ensure that your descriptions match the provided structure and length.
Prompt 2:
Prompt Examine the scene depictions of two video keyframes labeled Scene Description 1 (Frame and Scene Description 5 (Frame 5). Let's break down the progression Ahronologically: deduce the scene depictions for the three intermediate keyframes (Frames 2, 3, and 4) situated between the two provided scenes. Maintain the identical structure and length precisely.
Figure 6: Input prompt templates used for prompt variation (Table To reduce complexity, all subsequent results are reported using Prompt I, which was designed to align with the widely recognized chain-of- thought prompting approach, adapted for video: "Let's think frame by frame." "
You are given the scene descriptions of 2 video keyframes (frames 1 and 5). Let's think frame by frame: identify the scene descriptions of the 3 intermediate keyframes that occur between the 2 given scene descriptions (frames 2, 3, and 4). Copy the structure and length exactly.
Scene Description 1: Focus: Man on fishing boat; Action: Man standing and looking at river; Mood: Focused, calm, enjoyable; Objects: Man, boat, fish food, river, trees, sky; Setting: River in the middle of forest
Scene Description 5: Focus: Man releasing fish into river; Action: Man pushing fish back into water; Mood: Kind, happy, excited; Objects: Man, fish, river, trees, boat, rod; Setting: River surrounded by trees"
Scene Description 2: Focus: Man on boat with fishing rod; Action: Man fishing with rod; Mood: Focused, calm, enjoyable; Objects: Man, fishing rod, boat, river, trees; Setting: River the forest
Scene Description 3: Focus: Fish in the river; Action: Fish swimming; Mood: Peaceful, calm, still;
Objects: Fish, river; Setting: River
Scene Description 4: Focus: Man holding fish on boat; Action: Man taking fish hook out of fish;
Mood: Proud, focused; Objects: Man, fish, boat, rod, river; Setting: River
Figure 7: Input prompt with single in in-context demonstration for the Infilling" task.

---

ROUGEL
BERTScore
SentenceBERT
Visual Domain
Model
FAMOuS
Dense
FAMOuS
Dense
FAMOuS
Dense
GPT-4
0.1570
0.2444
0.1640
0.2337
0.4774
0.5101
Jobs & Education
VICUNA
0.1662
0.2570
0.1694
0.2599
0.4829
0.5391
GPT-4
0.1793
0,2519
0.2476
0.2619
0.5776
0.6794
Games
VICUNA
0.1821
0.2625
0.2251
0,2892
0.5929
0.6344
GPT-4
0.1686
0,2410
0.1940
0.2337
0.5440
0.5808
Sports
VICUNA
0.1768
0.2512
0.2000
0.2415
0.5507
0.5546
GPT-4
0.1644
0.2426
0.1828
0.2366
0.5173
0.5535
Pets Animals
VICUNA
0.1642
0,2492
0.1735
0,2309
0.5127
0.5152
GPT-4
0.1484
0.2434
0.1685
0.2223
0.5091
0.5521
Law & Government
VICUNA
0.1501
0,2522
0.1628
0.2285
0.5016
0.5304
GPT-4
0.1835
0.2590
0.1948
0.2660
0.5511
0.6209
Hobbies & Leisure
VICUNA
0.1835
0.2640
0.1848
0.2617
0.5434
0.5967
GPT-4
0.1656
0.2526
0.1754
0.2430
0.5060
0.5628
Home & Garden
VICUNA
0.1601
0.2544
0.1634
0.2282
0.4955
0.5350
GPT-4
0.1709
0.2484
0.1855
0.2429
0.5248
0.5539
Travel
VICUNA
0.1758
0.2481
0.1816
0.2272
0.5165
0.4978
GPT-4
0.1647
0.2482
0. 0.1476
0.2381
0.5171
0.5587
Food & Drink
VICUNA
0.1651
0,2509
0.1397
0.2258
0.5048
0.5206
GPT-4
0.1656
0.2535
0.1716
0.2448
0.5183
0.5820
Business & Industrial
VICUNA
0.1648
0,2570
0.1654
0.2366
0.5134
0.5501
GPT-4
0.1667
0.2476
0.1795
0.2393
0.5396
0.5845
Autos & Vehicles
VICUNA
0.1668
0.2509
0.1795
0.2351
0.5311
0.5575
GPT-4
0.1975
0,2517
0.2242
0.2668
0.5568
0.6402
People & Society
VICUNA
0.2149
0.2638
0.2263
0.2803
0.5672
0.6304
GPT-4
0.1704
0,2590
0.1944
0.2434
0.5147
0.5704
Reference
VICUNA
0.1691
0,2550
0.1800
0.2449
0.5072
0.5480
GPT-4
0.1687
0.2590
0.1795
0.2440
0.5270
0.6144
Real Estate
VICUNA
0.1698
0.2647
0.1740
0.2459
0.5172
0.6144
GPT-4
0.2407
0.2442
0.2219
0.2262
0.5141
0.5528
Health
VICUNA
0.2175
0.2551
0.1992
0.2260
0.4894
0.5090
GPT-4
0.1934
0.2436
0.2026
0,2362
0.5492
0.5923
Science
VICUNA
0.1906
0.2448
0.1922
0.2210
0.5434
0.5482
Table Average model performance across the Infilling-1, Infilliiiiii 2, Prediction-1, Prediction-2, and Prediction-3 tasks, partitioned by domain. Categories such Games, Sports, People & Society, Health, and Science perform better on average, while others such as Jobs & Education, Law & Govermment, and Food & Drink perform less well. The Best results for each metric and textual representation are underlined.

---

You are given the scene descriptions of video keyframes (frames 1). Let's think frame by frame: predict the scene descriptions of the next keyframes (frames 2, and 4). Copy the structure and length exactly.
Scene Description 1: Focus: Man on fishing boat, Action: Man standing and looking at river; Mood: Focused, calm, enjoyable; Objects: Man, boat, fish food, river, trees, sky; Setting: River in the middle of forest
Scene Description 2: Focus: Man on boat with fishing rod; Action: Man fishing with rod; Mood: Focused, calm, enjoyable; Objects: Man, fishing rod, boat, river, trees; Setting: River in the forest
Scene Description 3: Focus: Fish in the river; Action: Fish swimming; Mood: Peaceful, calm, still;
Objects: Fish, river; Setting: River
Scene Description 4: Focus: Man holding fish on boat; Action: Man taking fish hook out of fish;
Mood: Proud, focused; Objects: Man, fish, boat, rod, river; Setting: River
Figure 8: Input prompt with single in-context demonstration for the Prediction-1 task.
Instructions
image of description given image.
given a
image
objects, setting.
the
activity ottention image Action: primary actin occuring image
Focus:
clmosphere present image
image
Objects: Katlect racters
Setting:
place and circumstanees Image
Task
Pocus: Man playing gull
Action: Hitling golf bail
Objects: Golf ball, goll club, 14 peoplo, tall grees
Setting: Green field, goll course
The scene description provided may have incorrect missing information. Rate scale of 1-5 on how well the scene description matches the image.
011 (very pear)
O (poor)
O (reutral)
O44 (good)
O (very good)
Figure 9; Amazon Mechanical Turk Platform Interface, where crowdworkers are asked to qualitatively judge the accuracy of FAMOuS scene description.


Reviews:
[{"review_id": "8OoslQF0p2", "reviewer": "Reviewer_JM3S", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you reviewer JM3S for your thoughtful and positive response! We are pleased to hear your positive feedback regarding our paper’s organization and the novel pipelined approach for extracting keyframes and generating visually descriptive scene descriptions. Thank you for noting the novelty of our VIP tasks and the uniqueness of our contribution to the video reasoning literature. \n\nYou are right that our experimental results show significant room for improvement in video understanding that future systems can aim for. We share your excitement about the future development of video reasoning abilities in LLMs and hope that VIP will inspire advancements in complex video understanding research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-organized and presents a clear introduction to the problem of video reasoning and the motivation behind VIP.\n\nThe paper introduces a pipelined approach to extract the most important frames and generate unstructured scene description. This pipeline can provide more visually-descriptive information enhancing the reasoning capabilities of models in the vision-language domain. \n\nThe proposed Video Infilling and Prediction dataset offers a unique perspective on video chain-of-thought reasoning, providing two new tasks for evaluating model performance in generating intermediate and future keyframes. These tasks are valuable additions to the video reasoning research landscape and can lead to advances in video understanding and generation.\n\nThe authors perform extensive experiments, benchmarking GPT4 and VICUNA on VIP. The evaluation of existing state-of-the-art models on the proposed tasks highlights the current limitations in complex video reasoning and sets a foundation for future improvements in video understanding models.\n", "reasons_to_reject": "none", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces VIP, a novel dataset designed to explore video reasoning capabilities through a video chain-of-thought using keyframes. The authors propose two forms of keyframe descriptions: unstructured dense captions and FAMOuS (Focus, Action, Mood, Objects, and Setting) structured scene descriptions. The dataset includes two tasks: Video Infilling, where models are evaluated on generating intermediate keyframes, and Video Prediction, which tests their ability to predict future keyframes. The authors benchmark GPT4 and VICUNA on VIP and demonstrate the difficulty of complex video reasoning tasks for existing state-of-the-art models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IzVMd9SUUK", "reviewer": "Reviewer_6krL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks, reviewer 6krL, for your thoughtful response! We are heartened by your appreciation for our pipeline and interest in our novel evaluation direction. Please consider our following responses to your reasons to reject, as we have completed supplemental experiments based on your insightful recommendations that will go into the camera-ready version.\n\n> 1. The evaluation of LLMs might be criticized … [they are] highly sensitive to various prompt formats. … Perhaps, it would be good to use some existing techniques to generate more diversified prompts (e.g., zero-shot, or few-shot) with different wordings, e.g., LPAQA w/ paraphrasing for more comprehensive evaluations.\n\nTo validate the stability of our baseline results to prompt variations, we re-ran a subset of our VIP infilling experiments using an additional two new prompts for GPT-4. While our previous prompt used a single few-shot example and prompt phrasing, our two new prompts use two few-shot examples, and two different wordings of the task description generated automatically with GPT-4. Here are the preliminary results:\n\n**Infilling 1 Performance on various paraphrased prompts with Structured, FAMOuS Descriptions**\n| Metric        | Old Prompt Score | New Prompt 1 Score | New Prompt 2 Score |\n| ------------- | ---------------- | ------------------ | ------------------ |\n| Rogue         | 17.34          | 17.70             | 18.20             |\n| Bertscore     | 18.61          | 18.80             | 19.50             |\n| SentenceBERT  | 54.00           | 53.00             | 53.30     |\n\nThe FAMOuS results appear to hold across prompt phrasings and one- and two-shot learning. *In the camera ready we will add an additional set of prompt phrasings generated similarly to LPAQA, and we will provide these prompt-specific results for each (model, metric, experiment) combination in an appendix, and averages + std dev for each of these in Tables 2 and 3.* It appears that **overall, the performance of the evaluated baselines holds with simple variations in prompt.**\n\nHere are preliminary results comparing the dense unstructured infilling results from zero-shot and one-shot prompting.\n\n**Infilling 1 Performance on one-shot and zero-shot prompts with Unstructured, Dense captions**\n| Metric        | One-shot score | Zero-shot score |\n| ------------- | ---------------- | ------------------ |\n| Rouge         |         24.92        | 24.72             |\n| Bertscore     |     24.81            | 24.96             |\n| SentenceBERT  |   53.05              | 58.36             |\n\n*The infilling 1 task includes predicting the in-between scene description given one context frame description before and after the scene description to be predicted*\n\nAs you can see, the metrics are once again relatively robust to these small variations. Our decision to exclude zero-shot prompting from the original draft is because doing so on the unique FAMOuS schema performs poorly. However, doing so for the dense, unstructured scene description is reasonable. Thus *zero-shot prompting will also be included in our camera ready for the dense infilling and prediction tasks.*\n\nOur selection of prompting techniques is grounded in prior work (Wei et al, 2022). Our use of “let’s think frame by frame”-style prompting is inspired from the well-established “let’s think step-by-step” from Kojima et al (2022).  **We leave deeper exploration of prompting techniques to further optimize performance on this task for future work, as our central goal in this work is producing a benchmark for this novel video understanding task.**\n\n> 2. In addition to … \"Focus/ Action/ Mood/ Objects/ Setting\", [consider analyzing] fine-grained skills (e.g., visual causal reasoning, understanding dynamics/situations, etc), and visual domains [for] a better intuitive understanding of LLM failures.\n\n While we have considered breaking our domains down into further categories, ultimately, we aimed to achieve an automatic process such that future work can easily train and evaluate an LLM’s video reasoning capabilities. However, to provide more fine-grained evaluation insights, we have subsampled from our dataset on various categories and provided their results below, which we will add to the camera ready version.\n\nWe separated our full studies into the labeled visual domains seen in Figure 4 and found some interesting and consistent trends in performance. The results for the FAMOuS experiment with GPT-4 are highlighted in the table below. We see that categories such as Health & Science, Games, Sports, and Hobbies & Leisure have better performance, while Food & Drink, Law & Government, Jobs & Education don’t perform as well. These results can provide us more insight into which domains future work on LLMs’ video reasoning abilities can focus on.\n\n| Visual Domain      | Rogue   | Bertscore |\n|--------------------|---------|-----------|\n| Jobs & Education   | 15.70   | 16.40     |\n| Games              | 17.93   | 24.76     |\n| Sports             | 16.86   | 19.40     |\n| Pets & Animals     | 16.44   | 18.28     |\n| Law & Government   | 14.84   | 16.85     |\n| Hobbies & Leisure  | 18.35   | 19.48     |\n| Home & Garden      | 16.56   | 17.54     |\n| Travel             | 17.09   | 18.55     |\n| Food & Drink       | 16.47   | 14.76     |\n| Business & Industrial | 16.56 | 17.16     |\n| Autos & Vehicles   | 16.67   | 18.59     |\n| People & Society   | 19.75   | 22.42     |\n| Reference          | 17.04   | 19.44     |\n| Real Estate        | 16.87   | 17.95     |\n| Health             | 24.07   | 22.19     |\n| Science            | 19.34   | 20.26     |\n\nWe also labeled a sample of our experiments in our dataset that pertain to two categories of fine-grained skills, namely physical (changes due to outside, real forces) and social (changes related to social cues) causal reasoning. From the results (reported in table below), it's clear that the performance of both types of fine-grained skills is quite similar, showing that there might not be any differences in LLM performance corresponding to these different skills.\n\n| Fine-grained skill | Rouge   | Bertscore | Sentence-BERT |\n|--------------------|---------|-----------|---------------|\n| Social             | 18.68   | 20.81     | 52.70         |\n| Physical           | 17.70   | 16.61     | 55.44         |\n\n\nReferences:\n- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” https://arxiv.org/abs/2201.11903 2022\n- Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa, \"Large language models are zero-shot reasoners.\" https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of multi-modal CoT reasoning (particularly, video modality) is interesting, and the pipeline with unstructured captions looks convincing, by combining the power of several large foundation models. \n2. Nice paper structure and organizations.", "reasons_to_reject": "1. The evaluation of LLMs might be criticized a little bit, since these LLMs are highly sensitive to various prompt formats. Now, one few-shot prompt is used to test their capability in these tasks. Perhaps, it would be good to use some existing techniques to generate more diversified prompts (e.g., zero-shot, or few-shot) with different wordings, e.g., LPAQA w/ paraphrasing for more comprehensive evaluations. \n2. In addition to testing on dimensions like \"Focus/ Action/ Mood/ Objects/ Setting\", it would be good to separate the targeted studies into more fine-grained skills (e.g., visual causal reasoning, understanding dynamics/situations, etc), and visual domains. Therefore, when obtaining the results on your benchmarks, practitioners could get a better intuitive understanding of LLM failures.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Reasoning in text is \"reasoning step-by-step\". Similar, in video domain, it could be \"reasoning frame-by-frame\", but only on key frames. In this work, they show how to enable video CoT reasoning w/ existing visual-language models. Specifically, Vision Modules are deployed to extract dense caption descriptions. Language Models are used to perform such few-shot CoT reasoning, with specific two designed tasks (Video Infilling, and Video Prediction) on their *proposed benchmark (VIP, main contribution)*. The experiments with GPT-4 and VICUNA show the difficulties of such LLMs on these video reasoning tasks, asking for more research on video chain-of-thought reasoning w/ current LLMs or VLMs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NPRKBER0nP", "reviewer": "Reviewer_VoFV", "summary": "", "questions": "1. Refer to Reasons To Reject.\n2. Have you tried to explore the prediction frame description to generate real frame through Stable Diffusion. Some examples would be more interesting.\n3. Have your tried different way to select keyframe? E.g, [1]\n\n[1] An Efficient Keyframes Selection Based Framework for Video Captioning, ACL2021", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you Reviewer VoFV for your response! We are happy to hear your appreciation for VIP as well as your shared sentiment on the need for more work in this field. Please consider the following responses to your reasons to reject and your questions as we have conducted supplemental studies based on your insightful recommendations that will go into the camera-ready version.\n\n> 1. Only an inference-time dataset for VideoCOT is not enough for this task. Although GPT4 and Vicuna have shown great power in language understanding, the video prediction is still missing during training them, which makes the video understanding and prediction far beyond simple COT design. Hope the author can propose a larger dataset for instruction-tuning.\n\nWe primarily crafted VIP to provide a new evaluation framework and to propose novel tasks for multi-frame video reasoning which is why we focused on making VIP an inference-only dataset. Nonetheless, we see the reviewer’s point on the lack of training on video reasoning tasks in LLMs, therefore we will release an instruction-tuning dataset for the camera ready version of our paper.\n\n> 2. The evaluation of BERT-Score and Rouge-L may still pay much attention to token/word-level similarity. Maybe Sentence-BERT is more applicable for this task.\n\nWe had initially omitted metrics like Sentence-BERT because it’s been known to struggle with evaluating fine-grained semantic distinctions, which we had deemed most important for our short FAMOuS descriptions. Still, we are happy to add Sentence-BERT for the camera ready version to benchmark the overall video generation abilities for LLMs. \n\nWe ran Sentence-BERT over our full experiments, and the results for infilling 1 (predicting three in-between scene description given one context frame description before and after the scene description to be predicted), infilling 2 (same as infilling 1 but given 2 context frames before and after scene descriptions to be predicted), prediction 1(predicting the next frame description given one context frame description), and prediction 2 (same as prediction 1 but given two context frame descriptions) are in the table below. We found that trends observed with the prior metrics remain consistent with Sentence-BERT, namely on average increasing scores with more provided frames (infilling 1 to infilling 2), and higher scores for the dense caption evaluations. This confirms our initial hypothesis that the performance on VIP tasks depends on the short context phrases in our FAMOuS descriptions, and demonstrates that our observed trends in performance between models and conditions is signal, not noise. Though, Sentence-BERT scores are visibly higher, likely because of a higher level comparison of sentence similarity. \n\n\n**Sentence-BERT scores for Prediction 1 and Prediction 2 tasks**\n| Model   | FAMOuS Prediction 1 | FAMOuS Prediction 2 | Dense Prediction 1 | Dense Prediction 2 |\n|---------|---------------------|---------------------|---------------------|---------------------|\n| GPT-4   | 50.57               | 51.54               | 53.56               | 57.06               |\n| Vicuna  | 50.40               | 50.96               | 51.86               | 54.86               |\n\n\n\n**Sentence-BERT scores for Infilling 1 and Infilling 2 tasks**\n| Model   | FAMOuS Infilling 1 | FAMOuS Infilling 2 | Dense Infilling 1 | Dense Infilling 2 |\n|---------|-------------------|-------------------|-------------------|-------------------|\n| GPT-4   | 53.05             | 53.87             | 58.53             | 58.22             |\n| Vicuna  | 52.19             | 53.33             | 54.67             | 54.86             |\n\n\n> Q2. Have you tried to explore the prediction frame description to generate real frame through Stable Diffusion. Some examples would be more interesting.\n\nThis is a point we’ve considered. However, the token limit of Stable Diffusion along with the challenges of semantic generated image evaluation pushed us to focus mainly on textual evaluation. Current image evaluation metrics include using pixel variability and structural similarities which may not accurately capture the semantic meaning of the image. Additionally, image quality is a very subjective topic further complicating image evaluation. For these reasons, we emphasize this for future work. \n\n> Q3. Have you tried different ways to select keyframes?\n\nWe thank the reviewer for bringing this up as we put in a lot of thought on our keyframe selection process. We looked into the paper the reviewer suggested, and while we agree strongly with this paper’s goal of reducing redundant information in keyframe extraction, their goal of video captioning is one we differentiate from.  Instead, we focus on multi-frame understanding, which requires maintaining enough frames that still capture the video story. \n\nFurthermore, of this and other available open-source keyframe extraction models, we chose Katna because it is well-documented, high-performing, efficient, and doesn’t require lengthy setup or GPU requirements.  Still, Katna uses state-of-the-art extraction strategies: differences in LUV color space, scores of brightness, contrast, and blur, as well as k-means clustering of image histograms (like in the paper the reviewer suggested). Similarly to the paper, we compare similarity of frames to succinctly capture concepts, though we prune frames to a unique number of keyframes that preserves the videos’ content and length. This is a key difference between our method and what the paper proposes – extracting 3-4 keyframes per video. After much experimentation, we believe that our method effectively selects keyframes that reduce computational power and maintain the integrity of the video for the VIP tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The construction of the VIP dataset is valid.\n2. The proposed Video Infilling and Prediction inference-time dataset is meaningful to convert video generation into a language-central task, which would be helpful to both video understanding and video generation in the future.\n3. Experimental results shows there are still a long way for us in this field.", "reasons_to_reject": "1. Only an inference-time dataset for VideoCOT is not enough for this task. Although GPT4 and Vicuna have shown great power in language understanding, but the video prediction is still missing during training them, which makes the video understanding and prediction are fur beyond simple COT design. Hope the author can proposed a larger dataset for instruction-tuning.\n2. The evaluation of BERT-Score and Rough-L may still pay much attention on token/word-level similarity. May be Sentence-BERT is more applicable for this task.\n\n-------\n\nAfter rebuttal:\nThe author have solved my concern.", "questions_to_authors": "1. Refer to Reasons To Reject.\n2. Have you tried to explore the prediction frame description to generate real frame through Stable Diffusion. Some examples would be more interesting.\n3. Have your tried different way to select keyframe? E.g, [1]\n\n[1] An Efficient Keyframes Selection Based Framework for Video Captioning, ACL2021", "paper_topic_and_main_contributions": "This paper develop a pipelined approach to extract keyframes and generate a unstructured dense captions scene description and FAMOuS (Focus, Action, Mood, Objects, and Setting) structured scene descriptions.\nMoreover, this paper propose the Video Infilling and Prediction inference-time dataset to evaluate video chain-of-thought reasoning with two new tasks in frame generation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers presents VIP, an inference-time dataset designed to explore models' reasoning capabilities through video chain-of-thought. To ease the video processing, two forms of textual descriptions are used to describe extracted keyframes, unstructured dense captions and structured scene descriptions that identify the focus, action, mood, objects, and setting. Two tasks are created for the datasets, Video Infilling and Video Prediction. The evaluation is conducted using GPT-4 and Vicuna shows current LLMs still struggle with complex video reasoning. Overall, as pointed out by the reviewers, the paper is well organized with clear motivation, the idea is interesting, the dataset and the pipeline that constructed the dataset could be useful in assisting research in this direction.
Decision: Accept-Main