CALVIN: Improved Contextual Video Captioning via Instruction Tuning
Authors: Gowthami Somepalli
Arkabandhu Chowdhury
Jonas Geiping
Ronen Basri
Tom Goldstein
David W. Jacobs
Venue: NeurIPS 2024 poster (2024)
Keywords: contextual captioning
video understanding
ai for inclusivity
Primary area: machine_vision

Abstract: The recent emergence of powerful Vision-Language models (VLMs) has significantly improved image captioning. Some of these models are extended to caption videos as well. However, their capabilities to understand complex scenes are limited, and the descriptions they provide for scenes tend to be overly verbose and focused on the superficial appearance of objects. Scene descriptions, especially in movies, require a deeper contextual understanding, unlike general-purpose video captioning. To address this challenge, we propose a model, CALVIN, a specialized video LLM that leverages previous movie context to generate fully "contextual" scene descriptions. To achieve this, we train our model on a suite of tasks that integrate both image-based question-answering and video captioning within a unified framework, before applying instruction tuning to refine the model's ability to provide scene captions. Lastly, we observe that our model responds well to prompt engineering and few-shot in-context learning techniques, enabling the user to adapt it to any new movie with very little additional annotation.

CALVIN: Improved Contextual Video Captioning via Instruction Tuning
Gowthami Somepalli
Arkabandhu Chowdhury $
Ronen Basri
Jonas Geiping
Tom Goldstein
David Jacobs
University of Maryland, College Park
Meta AI
ELLIS Institute, MPI for Intelligent Systems
Weizmann Institute of Science
Abstract
The recent emergence of powerful Vision-Language models (VLMs) has signifi- cantly improved image captioning. Some of these models are extended to caption videos well. However, their capabilities to understand complex scenes are lim- ited, and the descriptions they provide for scenes tend to be overly verbose and focused the superficial appearance of objects. Scene descriptions, especially in movies, require a deeper contextual understanding unlike general-purpose video captioning. To address this challenge, propose model, CALVIN, specialized video LLM that leverages previous movie context to generate fully "contextual" descriptions. To achieve this, train model integrate both image-based question-answering and video captioning within a uni- fied framework, before applying instruction tuning refine the model's ability provide scene captions. Lastly, we observe that our model responds well to prompt engineering and few-shot in-context learning techniques, enabling the user to adapt it to any new movie with very little additional annotation.
suite of tasks that
Ground Truth: Lenihan knocks backwards into pool
VideoLlama:
First, the video shows soldier holding a rifle standing on stairs. Next, soldier holding a gun standing on rooftop in the rain.....the soldier holding bow and arrow and rifle.
VideoLlaVas The video shows
in military uniform holding rifle and aiming it target. He then fires the Here several guns visible in the image, including rifle held by soldier, handgun, and few other guns scattered around the area.
MovieChat: The video shows various scenes of an army man in a military setting, including a helicopter flying, a soldier standing in front of a building, a soldier standing in front of tree, and soldiers walking
CALVIN (Ours): Lenihan shoots the alien in the head
Figure 1: A scene from MAD [80]-eval split. We present the captions generated by model, represented CALVIN against various off-the-shelf LLMs, with hallucinations highlighted in red. First, our model utilizes context well, by understanding the name of the character is 'Lenihan' and that there is an alien in the scene, and second, our model has less hallucination and verbosity compared to other models.
Work done during internship at Meta. Correspondence: gowthami gowthami@und. @umd. edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The volume of video data on the internet is increasing every year, now representing the largest portion of internet traffic [2]. To make this wealth of visual content accessible vision impaired individuals requires audio descriptions. Audio descriptions (AD) describe and narrate videos or segments of videos in natural language, but are often still manually curated for only few select videos.
The emergence of large, multi-modal vision-language models (vLLMs) has led to exciting progress on supplementing AD with automatically generated descriptions (AAD), but current models still suffer from notable weaknesses in videos. Consider the concrete example of a video of a woman smiling and waving the scene, like a human, might caption it as A happy woman waving at the bus, perhaps receiving loved one, while vLLM trained image-caption pairs would provide a more literal description, such as A woman wearing a red dress. A woman is smiling. A bus stopped. The scene has people in the background. This illustrates key shortfall: vision LLMs focus on superficial details like the properties of physical objects, which are often irrelevant to the broader context, because they are trained on static images. Several of these models process videos only as individual images, simply stringing together descriptions from each image while failing to capture the overarching narrative.
bus carrying friend. An observer with understanding of the context of
Secondly, current systems struggle to incorporate prior context into their description. Using the same scenario, if we inform human that the woman's 's name is Mary and her husband, soldier, is returning from war, a human would caption it as, Mary is overflowing husband, war hero, the bus stop. Here, the human might choose to omit less signifi- cant details like 'waving' in favor of emphasizing her happiness, showing nuanced understanding of the context. In contrast, many vision L.MMs lack the ability integrate such context with visual data, often ignoring it entirely. This is a significant gap, as useful interpretations naturally prioritize the in- tentions of characters and the results of their interactions through time, rather than the mere presence of objects and their movements which would make the video hard to follow for listener reliant on AD,
with joy to receive her
Drawing inspiration from the way humans utilize context in captioning, introduce novel contextual captioning model, CALVIN, that is designed and instruction-tunes to generate audio descriptions. Our primary objective is to develop model that, given appropriate context, can generate captions closely resembling those crafted by humans. To accomplish this, we train our model using the Movie Audio Descriptions(MAD) dataset [80], which includes human-generated annotations for movie scenes, complete with timestamps. This enables us to construct a "text context" for each scene, based on preceding scenes. However, the MAD dataset alone limited in scope and insufficient for fully training a video-LLM. To address this, we incorporate image-VQA datasets, which significantly enhance the model's visual understanding. We provide detailed discussion on training methodologies, including the optimal combinations of data to use at different stages.
In addition to producing scene deseriptions that are more human-like and useful than the existing vision models, CALVIN, achieves state-of-thh-an (SOTA) performance in captioning on the MAD- eval dataset, with major improvements (~266% improvement CIDEr and ~68% improvement on BertScore) over the previously established SOTA model. CALVIN also performs significantly better than all the recent off-the-shelf video I.LMs studied zero-shot evaluations the TVC dataset [43]. We illustrate the model's capabilities in Figure 1. Here, while most video- LLMs generically describe the scene as a soldier holding a rifle or an army man in military setting missing the narrative nuance of the ground truth caption Lenihan knocks it backward into the pool. CALVIN stands out by not only recognizing the character as 'Lenihan' but also incorporating prior information about the presence of an alien. Unlike the overly verbose captions of other models, our captions are crisp and narrative, focusing on actions and outcomes that are important to the plot, underlining their usefulness for the task of automated audio description.
We also introduce two test-time adaptation strategies, prompting and few-shot tuning, that are particularly useful in scenarios where additional contextual information is lacking. Finally, we describe some of the limits of our current model and outline potential avenues for future research to further advance this field.

---

2 Related Work
Video Understanding. The key objective of parsing spatiotemporal information in videos can be achieved through hand-crafted features [15, 18, 22, 69], recurrent networks [19, 36, 107], convolu- tional networks [24, 27, 51, 70, 90, 95], and more recently Vision Transformers (ViTs) [7, 9. 21, 23]. ViT's [21] treat an image as set of patches and use transformer architecture to model their interac- tions. Some works also effectively add multi-scale hierarchies [23, 28, 60, 73, 1 106] or local structures [14, 20, 60] to the transformers. Naturally these models can be extended to videos where video is treated as sequence of independent image frames, and subsequent temporal pooling layer or temporal transformer is added [7. 16]. There have been more generalized video modeling approaches [9, 23, 66, 68, 73, 84] that directly work on a video clip by dividing it into 3D spatio-temporal patches. While most of these models have proven to work well on short videos (<5 seconds), longer- video (>30 seconds) understanding is still an active research area. Some existing methods include pre-compuiigg features and separately training backbones [3, 19, 26, 94], increasing model efficiency to include more frames [35, 38, 95, 110]. and building memory model that can reference prior context [13, 41, 42, 96, 97]. Recent studies show that video-text pretraining [4, 5, 25, 29,37, 45, 46, 49, 63, 71, 75, 82, 88, 89, 101, 103] can greatly help in long video understanding [83]. temporal localization tasks [12, 44, 55, 100], text-video retrieval [63], video question answering [104], and video clip captioning [5, 75].
Generalist Video LLMs. Recent breakthroughs in language modeling have also spurred flurry of work incorporating first image data, and then video data, as additional input modalities [5. 61]. While some works try to train large-scale video-text models directly from scratch [47, 65, 67, 87], most work focuses on continued pretraining and finetuning of base language models [102, 108]. Models may be generic multi-modal models [98, 105], or branch off from existing image-tex. models, such as LLaVA [52], targeting video understanding [54], and initial work on long video understanding [54]. Instruction tuning for videos was considered in [58, 59] and [78], and, with a particular emphasis on interactions with long(er) videos in [48, 81]. Understanding long videos is not only a learning problem, but also technical challenge, necessitating engineering improvements, such as Ring to even process long videos [57].
Contextual Video Captioning. What are use cases for these video LLMs? A particularly interesting one is automated video captioning, which effectively converts video content back into text. This is useful, not only for tasks such as search, retrieval, but also essential to generate audio descriptions (AD) of video content. Automated audio descriptions describe the content of videos verbally, and are central to making videos accessible to anyone with visual impairment [64, 76, 93]. Seminal work, such as CineAD [11], argued that automated audio deseriptions of even highly contextual content such cinema, should be feasible, leading to a broader interest in automated audio descriptions (AAD). The most recent work in this direction are AutoAD I and II [30, 31], generating audio descriptons based on CLIP features processed with smaller language models. Auto-AD III [32] is a concurrent work that introduces a large-scale dataset and a 7b model to perform this task. AD is particularly focused on contextual descriptions and allows a listener make sense of long-form video, such as a movie. This separates this task from the more generic task of dense video captioning [33, 39, 77, 82, 91].
3 CALVIN: A contextual video captioner
We now discuss the proposed architecture, datasets, and training process.
3.1 Method & Architecture
Consider a collection, V. of movie scenes [V1, V vn). Each scene contains k image frames such that Vj- = [0] (u)...., w/), and comes with annotations v in the form of a sequence of text tokens (a), az --,"- These annotations are usually obtained by converting audio text, where the audio comes from the movie dialog person describing the scene. We seek model (-) that effectively predicts the audio description given the past visual and text tokens.
We parameterize (-))) with an LLM. While the pre-trained LLM can easily handle all the text tokens a1:1-1. it must be adapted to handle vision tokens corresponding to frames vi. We construct a video projection layer to process the video component and project it into the LLM's representation space. The overall causal problem formulation of contextual video captioning becomes
a] = So (go(I(u)) ...., I(04)), 14-11
(1)
3

---

Stage-2
Stage-1
LLM f$(.)
LLM f
trained
B
frozen
Vision tokens
Context tokens
Question tokens
Caption tokens
The architecture has main components. (1) A frozen image embedding extractor (2) Non-lincar projection module Q. and (3) LLM. We train model 2 stages. Stage we train only the projection module Q on image caption data. Stage 2, we use instruction formatted higher quality image-vidco data and finetune the parameters of Q and LLM. Refer to Sec. 3 for more details. Image and video examples shown here are synthetically generated using Meta Imagine [1].
Figure 2: CALVIN:
Data format for training:
Stage 1: <visual><<>> <riaua><<capton>>
Stage 2: <visual> <context> <question> <caption>
Examples:
Stage I: (IMAGE_TOKENS A cat playing tennis in a saree.
TOKENS] Leonard the panda has day off. He went for swim. How would you describe key visual elements and actions this video? Leonard is taking a stroll down the beach
Stage 2: [VKENS]
Figure Stage-wise data format example. We present the templates and example of data format for different stages as shown in Figure 2. The colors of the text in this example match the token type colors in Fig. 2. The underlined text represents the segments that the language model must predict at all stages, and it is upon these predictions that the loss is computed.
where I is frozen image embedding model that accepts image frame uf and outputs fixed dimeennsional embedding. The function go(-) is a learned non-linear projection module that takes in multiple image embeddings and projects them into the LLM latent space. We use CLIP ViT-h/14 [99] vision encoder as the frozen feature extractor I.
We train the system to predict the next text token based on all available context in the scene:
8"
arg min
>> (a)
0,0
where å; as defined Equation (1) and is cross-entropy loss.
Our projection module (+(-) is comprised of two sub-components: Q-Former layer [47], and a linear layer. The Q-Former begins with fixed number of learnable embeddings, which are cross-attended by the video-frame embeddings during training. This formulation provides flexibility in handling variable number of CLIP embeddings, up a maximum defined by the position embedding, hence allowing training with mixed image and video datasets. The Q-Former layer is initialized with the pre-trained weights of BERT base [17]. We add position embeddings the video-frame embeddings before the cross-attention to capture the temporal dynamics in videos. The second part of the projection module linear layer that projects the output of the Q-Former into the latent space of the LLM. We use a Llama-2 7b variant [85] model as our base LLM module.
3.2 Data and Stage-wise Training
The MAD dataset [80] stands out as the only extensive audio-description dataset currently available that includes aligned visual data, making it uniquely suitable for training contextual captioning

---

valjean strokes cosette's hair CALVIN: carriage, sictor sits with his wife's head resting on his lap as be gently strokes her bair
GT: the moon shines through
GT: marius exchanges a glance with onjoiras and follows her
GT: cosetle sees acar on his wrist
CALVIN: a moon shines the night sky as malks throuugh a dense forest
CALVIN: in the tavern, william glances woman he walks past her
CALVIN: dloe--pp, the muc's is hand grips the woman's wrist
CALVIN-3S: valjean strokes her hair
CALVIN-3S: valjean and cosette ride through the nighht in cirriage
CALVIN-3S: murius follows
CALVIN-35; valjean takes cosette's hand
Figure 4: Qualitative results: Captions based on previous context are denoted CALVIN-3S. Captions without any context are represented as CALVIN. Contextual captions are close ground truth and can get the names right based on the context. Without context, we see hallucinations (underlined), especially with the names.
models. However, the dataset's scale is relatively modest, especially when considering the extensive data requirements for effectively training a video-based LLM. To address this limitation, we train jointly on both image and video data. We use single unified CLIP and Q-former vision pipeline for both modalities, enabling us to seamlessly transfer knowledge from the image to the video domain. We two-stage training process for the video I.L,, as illustrated in Fig. 2.
In the first stage, we align the projection component (as denoted by Q in Figure 2), which consists of the Q-Former and a linear layer, while keeping both the CLIP and LLM frozen. In this stage, we use an internal dataset of data curated by LLaVA [58, 59] which is built on top of MS-COCO [56].
pairs (= 100M pairs), CC- 12M [76] and the pretraining stage
For the second stage of training, we fine-tune the projection component as well as the LLM as shown in Fig. 2. This stage also marks our transition to employing higher-quality vision datasets. The data mixture for this phase is significantly enriched, comprising CC-3M [76], instruction tuning data from LLaVA 1.5 [58]- itself curation based on a range of public image-caption or QA datasets including OKVQA [62], A-OKVQA [74], TextCaps [79], Visual Genome [40]. Additionally, we incorporate the WedVid-3M [8] video caption dataset and MAD [80] dataset train split into this stage. As such, this data mix is of higher quality as the majority of it is -anan-annotatte.
Contrary to the approach of AutoAD [30] where the MAD dataset was trained separately, our ablation studies, which will be discussed later, reveal that integrating the MAD dataset into the second stage speeds up and simplifies training without significantly impacting the performance of the final model.
Turning Stage-2 data into instruction data: The dataset from LLaVA-1.5 comes pre-formatted for instruction tuning, simplifying its integration into our training process. For the other datasets, we adapt them to align with this instruction format. Typically, these datasets follow <image/vides >caption>> structure. We restructure them into a more comprehensive format: cimage/viche In this format, for all datasets except MAD, the <context> component is simply a placeholder space. For MAD, we use the annotation from the previous few scenes as <context>. We sample from curated set of template questions regarding the <question> component. For detailed view of these template questions, please refer to the Appendix.
Experiments & Results
Training details: All models are trained on a single A100 node with 8 GPUs. In Stage 1, we train only the projection module (Q-Former and linear layer) for 400, 000 iterations, with gradient accumulation over steps and Per-GPU batch size of 32. The learning rate has a cosine schedule and warmup phase of 2,500 steps with min LR 1e 6 and max LR le 4. In Stage-2 we train Q-Former, linear projection, and the LLM. We train each model for 120, 000 iterations with a cosine learning rate with min LR of le 6 and max LR of le 4. The per- GPU batch size is 12 for image datasets and 6 for video datasets. Across all stages, weight decay of 0.05 was applied. We adopt the Low-Rank Adaptation (LoRA) A) [34] approach for training the LLM. We set the LoRA rank to 32 and use LoRA on the QKVO (Query, Key, Value, and Output) in the attention layers. Unless stated, the Q-Former contains 4 layers with 32 learned embeddings and is trained to accept a maximum of 32 frames from the video. During the training, we sample 32 frames from the video. The datasets used in each stage are discussed in detail in Sec. 3.
Metrics: We evaluate our models and compare against baselines on four metrics. The first is BertScore [109], which measures the similarity in BERT representations between the ground truth and the generated captions. The remaining three are traditional captioning evaluation metrics, CIDEr [86]. ROUGE-L [53] and, SPICE [6] each offering unique evaluation perspective.
Evaluation details: We benchmark against the current state-of-the-a model, AutoAD-II [31], using the MAD named-eedtt split. This test split has captions that include character names. Owing to
5

---

the unavailability of trained models for both Auto-AD [30] and AutoAD-II [31], we present the best-reported results from these papers. Other baselines include ClipCap [65] and CapDec [67] which were discussed in Auto-AD
limitation of the MAD dataset [80] is the absence of raw videos; only embeddings of the CLS token of the VIT model are shared. This constraint prevents the evaluation of models that require data in formats other than these specific embeddings. To provide a broader assessment of our model's performance relative to general-purpose open-source video language models, such as MovieChat [8 ]. VideoLLaVA [52] and VideoLlama [108], we have conduct zero-shot comparisons on the TV. captioning (TVC) dataset [43].
4.1 Results
MAD-eval. Our main objective is to do contextual captioning. Hence we compare CALVIN against the SOTA models on the MAD-eval dataset Tab. 1. Across all the metries, we see CALVIN has significantly higher performance compared to the SOTA model. Even though model is trained with context, we present a baseline case where we evaluate without context. In this case, we see ~90% improvement over the best baseline BertScore. Among models with context, we further ~66 improvement on CIDEr and ~68% improvement on BertScore. We present a few qualitative results in Figure CALVIN-3S represents the with 3-scene context and CALVIN is the evaluation with no context. When evaluated with no context, the model hallucinates names some scenarios (scenes- a, c). However, these hallucinations disappear once provide context the model and the captions are quite close to the ground truth. For scene b, our model's generation differs from the ground-truth caption but is technically correct. We encounter few such false negative cases where a human would find this generation acceptable but it is hard to identify such cases without human intervention. We present some such scenarios in the Appendix.
TV-Captioning. We also benchmark CALVIN against off-the shelf SOTA vidco-LLMs on the TVC captioning [43] task in Table 2. All models evaluated in shot fashion, where none of the models (including ours) are fine-tuned on this dataset. Context is not available in this dataset, but this dataset's captions refer to the characters in the scene with names, and we add the character names in the prompt when we are querying all the models. CALVIN outperforms all the baselines across all metrics despite not being trained on this dataset. The distinction is quite visible on the CIDER metric where CALVIN is ~>3x better than the next best model. This shows the generality of our model for movie/TV captioning tasks. We discuss few ways of contextualizing captioning when the context data is not available in Section
Table Evaluation on MAD-named-eval split. The top half represents models evaluated without context. The bottom half shows the models trained/evaluated with context. Context column the numbers in brackets show the number of scenes used in context. f- The numbers are from the original papers as the models are not public.
Model
Context
BertScore t CIDER
ROUGE-L f
SPICE
ClipCap [65]
No
11.8
4.4
8.5
1.1
CapDec [67]
No
14.3
6.7
8.2
1.4
CALVIN (Ours)
No
27.25
14.74
11.9
3.89
AutoAD-I [30]
Yes (6S)
23.8
21.9
13.9
4.8
AutoAD-II [31]*
Yes (Char.)
19.5
13.4
CALVIN (Ours)
Yes (3S)
39.08
25.47
16.30
7.33
CALVIN (Ours)
Yes (5S)
40.18
27.71
16.83
7.76
Table 2: Zero-Shot evaluation on TV-Caption dataset. All the models are provided with names of characters in the scene. All the models use 7B LLMs.
Model
Zero-Shot
BertScore t
CIDER
ROUGE-L 1
SPICE
Videol Jama [108]
Yes
28.29
3.34
5.61
3.52
MovieChat [81]
Yes
38.11
8.43
12.09
9.21
VideoLLaVA [52]
Yes
48.44
12.4
15.8
10.9
CALVIN (Ours)
Yes
52.16
38.9
20.1
14.27

---

Table 3: Model Ablations: Unless otherwise stated, all models trained for the number of iterations
on same dataset. Unless otherwise stated, all the models are evaluated with a 3-scene context on MAD-named-eval split. -refers to the CALVIN 7B variant which we discuss throughout the paper
Ablation type Config-
BertSeore
CIDER
ROUGE-L
SPICE f
All *
39.08
25.47
16.30
7.33
(-) MAD
17.94
3.52
7.26
1.44
Stage-2 Data
()) WebVideo
37.35
22.07
13.5
6.7
(-) Image VQA data
34.99
20.21
14.53
6.3
(-) All stage-2 data
16.20
2.24
7.14
1.24
Q-Forner
33.59
15.91
14.22
5,28
Component
LLM
11.18
0.3
finetuning
Q-Former LLM
39.08
25.47
16.30
7.33
32 tokens, width=1 *
39.08
25.47
16.30
7.33
Q-Former tokens
64 tokens, width=4
38.83
24.27
16.22
6.94
128 tokens, width=4
38.97
23.10
16.35
6.97
LLM prefix tuning
28.99
11.46
11.89
3.87
LLM Lora (rank=16)
38.63
24.67
16.08
7,09
LLM tuning
LLM Lora (rank=32)
*
39.08
25.47
16.30
7.33
LLM Lora (rank=64)
39.27
24.73
16.57
7.18
MAD in Stage-2 *
39.08
25.47
16.30
7.33
Stagewise
MAD Stage-3
39.34
23,46
16.34
7,05
Stage-2 Learning
LR=le
39.08
25.47
16.30
7.33
Rute
LR le
34.59
15.91
14.22
5.28
4.2 Ablations
We share few ablations and some insights gained. Since there are many moving parts in the system, the search for our final configuration is mostly greedy and looks one component at time. We present the captioning metrics of the resulting models in Table 3. All models in the table are trained for the same number of iterations with context three previous scenes along with the video. These models are also evaluated with 3-scene context, and other hyperparameters are kept constant.
Data mixture in Stage-2. As discussed Section 3.2, we combine multiple datasets in this stage. We remove one data type at a time. Removing Stage-2 training impacts the performance the most, followed by removing the MAD dataset. This makes sense since AD captions tend be crisp and more contextual, while others are more descriptive. There is a clear domain shift and we need the MAD dataset in training to bridge this gap. One interesting observation is that removing the image VQA data impacted the performance a bit more than removing the only video dataset other than MAD. This confirms that the image VQA data can contribute to video understanding.
Component tuning. As discussed in Section 3.2, we tune both Q-Former and LLM in Stage-2. We check performance when just one of these components is present. It turns that training only the LLM leads to an extreme drop in performance while training only Q-Former dropped performance slightly, especially the CIDER score. We believe in LLM training case, the model completely ignores the vision embedding, attending only to the context, causing generated captions that are hallucinations with no grounding in the video. the case of only Q-Former training, we believe that the model does not learn to utilize the context well, hence slight degradation in performance.
Q-Former tokens. We examine the effect of the number of Q-Former tokens, which in turn controls the parameter count. More tokens causes a slight degradation in performance. We believe this is because the smaller size of the stage-2 dataset causes overfitting.
LLM tuning parameters. We examined two types of efficient LLM fine-tuning techniques LoRA and Prefix Tuning [50]. Prefix-Tuning adds few additional trainable parameters to each transformer block. In LoRA, a chosen set of parameters is updated by a low-rank approximation. First, we see LoRA training is significantly better than Prefix-Tuning Second, we increase the LoRA rank hyperparameter, we see slight performance improvement.
Other ablations. We depart from previous works by mixing the MAD dataset into stage II rather than separately fine-tuning on it. we instead fine-tune on MAD with 40,000 iteration stage III, we do not see much improvement. Additionally, we observe that reducing the max learning rate from le to in Stage 2 improves the performance by non-trivial amount. We refer the reader to the Appendix for additional train-time and inference time ablations.
7

---

Prompt formats:
No context: [VISION_ TOKENS]] Describe this visual.
Entity context: [VISION TOKENS] The characters and entities in scene are <ENT1> <ENT2>. Describe this visual.
Example model outputs:
No context: She talking blonde Entity context: <ENT1> is talking <ENT2>.
Figure 5: Entity Prompting. Instead the previous scene context, we prompt the models with only the entity information. Refer to Section 5.1 for more details.
Video-language models are also few-shot learners
It was shown in Brown et.al [10] that LLMs, trained diverse set of data, can benefit from prompt engineering and in-context learning. Despite our video representations being trained in the specific domain of video captioning, they still inherit the emergent properties of the parent LLM. This includes their ability to adapt to prompt engineering, in-context learning, and fine-tuning for specific tasks, In this section, we propose and evaluate two realistic strategies for customizing the model at test time. These strategies have shown improved performance in captioning compared to scenarios where no context is provided.
Regularization data: don't recognize the characters entities in the scene. Hence the final caption is: COODOFFEDDDRIGIAALLCCO
Test-time movie data: The characters and entities in the scene are <ENT1> and <ENT2>, Hence the final caption is: SEEIGINAL_CAPPTON
Figure 6: Few-shot fine-tuning. We rewrite the captions in CoT [92] style and fine-tune the model them.
5.1 Prompt engineering
It is hard for the models (and humans) to associate actors with their characters' names without priori information. We observe that our model can be enhanced further by providing characters' or entities' information in the context. Assuming can access entity information for given scene, we prompt the model as illustrated in Figure We noticed that just adding this information about entities in the scene to the context improves performance. See Section 5.3 for results.
5.2 Fewshot finetuning
Works like Dreambooth [72] "personalize" a large pre-trained model to object category with test-time training. We test this idea by fine-tuning our model for every movie the test set, showing it 20 scenes (~55 minutes) of each movie. This paradigm enables -uman-in-the-loop annotation where the humans can correct the model's mistakes while the model performs the bulk of the work.
To avoid over-fitting on the test time data, propose to use additional regularization data from the train set (i.e. aee.pre-annotteed other movies) where we strip away the entity details using LLaMA-2 70B to get rephrased caption without proper nouns identifying details. See Figure for an example of test-time movie data and regularization data used for test-time training. This is analogous to soain-of-thougee prompting [92], where model is trained to output its reasoning. In we show that with fine-tuning, the model recognizes some of the main characters as well as adapting to the tone of annotation for the movie. Note that annotators use different styles languages based on the type of the movie. For instance, animated films have simplistic audio descriptions as they are directed at children. We share results in Section 5.3.
case,
Experiments. first sample 20 scenes from the test-time movie such that the scenes which main characters first appear are included (see Table 4 for 50 instead of 20). Then we add the same number of samples for the regularization data, which are sampled randomly from the MAD-train split. We rephrase the captions of the whole mix in chain-of-Hee oftthhughhh (CoT) style present in Figure 6. We start with CALVIN Stage-2 checkpoint and finetune it for 100 iterations with a constant learning rate of 1e 5, 2-step gradient accumulation, and batch size of 10 on a single A6000 GPU. Each model. training run takes less than minutes.
8

---

Tisa takes steps toward him and smiles.
shaking head, george shrugs his hands and smiles.
CALVIN: She smiles at him and walks away.
CALVIN:
smiling man wearing blue suit and pink tie
walks down hallway office building
She wears tight-fiting black dress with plunging neckline, and her blonde hair styled in loose curls
CALVIN-E:
CALVIN-EE E: George smiles and shakes his head as he walks away.
CALVIN-FS: smiles him.
CALVIN- FS; George smiles and nods.
Figure 7: Qualitative results: We present two adjacent scenes from one of the test movies. CALVIN corresponds the result with context. CALVIN-E refers to context containing only entity information. CALVIN-FS corresponds to the model few-shot trained on 20 random scenes from this movie (not the ones shown here.)
5.3 Results
We present the average metrics across all 10 movies Tab. 4. We present results on CALVIN without context and CALVIN with entities in the prompt discussed in Sec. 5.1. We present fine-tuning scenarios trained with 20 to 50 samples from each test movie. Note that this accounts for less than 8% of total scenes with AD in all of the test movies.
We see performance improvement in both the personalization scenarios over the model without context. This shows that test time personalization can be cheap and efficient way to improve contextual captioning. We present qualitaaive example for continuous scenes in Fig.7. The model without context often describes scenes in a generic albeit correct way. CALVIN with entities in the context is able to caption the scene a bit better, however, the model does not always use this information, e.g., the left scene caption does not refer to the character 'Lisa'. CALVIN-FS (fewshot) captions well with correct character names. See Appendix for ablations on the amount of training and number of few-shot examples. As downside, few-shot finetuning seems to reduce caption diversity and length and occasionally associates wrong names with characters.
Table 4: Test-time adaptation results: (first row) CALVIN evaluation without context. (second row) 'Entities'
means the context has just the list of entities in the scene discussed in See. 5. (third and fourth rows) For few-shot training from Sec. 5.2, the number brackets counts examples used in finetuning. Both strategies improve performance over the no co-context.(' Numbers differ slightly from Tab. since the numbers presented here are an average of metrics computed one movie at a time, and some metrics like CIDEr depend on the word distribution of evaluation set.)
Model variant
Context
BertScore
CIDER f
ROUGE-L f
SPICE 1
CALVIN 1
None
26.91
10.85
11.76
3.73
Prompting $5.1
CALVIN
Entities
39.14 (+12,23)
16.27 (+5.42)
15.56 (+3.8)
7.13 (+3.4)
Few-shot training $5.2
CALVIN ES (20)
None
34.14 (47.23) 36.19 (+9.28)
17.74 (+6.89)
14.06 (+2.3)
7.17 (43.44)
CALVIN FS (50)
None
20.06 (+9.21)
15.22 (43.46)
7.21 (+3.48)
6 Discussion and Conclusion
We present a state-of ofate-of-thh-ar the-art model for contextual scene captioning. Our model introduces several innovations, including instruction tuning and stage-wise training for various components, along with tailored data mixes at each stage. We demonstrate that our model, CALVIN, exhibits superior generalization capabilities and outperforms existing off-the-shelf video-LLMs at zero-shot evaluation on a TVVcaptioning dataset. We additionally propose novel test-time adaptation strategies involving prompting and few-shot tuning.
There are still ample opportunities for enhancement. These include processing longer videos, intelligent frame sampling for better and faster video representations, and audio processing. The biggest challenge to these advancements is data scarcity; There is pressing need for public video datasets of high quality.
7 Acknowledgements
This work was made possible by National Science Foundation (NSF) grant #2213335. Further support was provided by the National Science Foundation (IIS-2212182). and by the NSF TRAILS Institute (2229885).

---

References
[1] Meta imagine. https://imagins wet.. com/ Accessed: 2024-01-10.
[2] Sandvine's 2023 global internet phenomena report. https://ww.. wttp:/iiwww...iii.e...w.w....... com/
press- releases/ sandvines- 2023-022222222
-mmpmmp---p-p-- in- video- traf.
Accessed: 2024-02-01.
[3] Sami Abu-El-Haija. Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakr- ishnan Varadarajan, and Sudheendra Vijayanarasinhan. Youtube-Sm: A large-scale video classification benchmark. arXiv preprint (rXXvv:60900867555 2016.
[4] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal suler-sue learning from raw video, audio and text. Advances Neural Information Processing Systems, 34:24206-24221, 2021.
[5] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi. Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot aj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems, volume 35, pages 23716-23736. Curran As- sociates, Inc., 2022. URL 2022/file
wwwpeeeeddiiinnnnnnneeeeeeeeeeee neurips. cwtppppppppplllccpppee sateerrfffleppppeeeeeeeeeeeeeeee /apper/ pdf.
[6] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part V 14, pages 382-398. Springer, 2016.
[7] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836-6846, 2021.
[8] Max Bain, Arsha Nagrani, Güil Varol, and Andrew Zisserman. Frozen time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[9] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[11] Virginia P Campos, Tiago MU de Araújo, Guido I. de Souza Filho, and Luiz MG Gonçalves. Cinead: a system for automated audio description script generation for the visually impaired. Universal Access in the Information Society, 19:99-111, 2020.
[12] Meng Cao, Tianyu Yang, Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. Loevtp: Video-text pre-training for temporal localization. In European Conference Computer Vision, pages 38-56. Springer, 2022.
[13] Yihong Chen, Yue Cao, Han Hu, and Liwei Wang. Memory enhanced global-local aggregation for video object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10337-10346, 2020.
[14] Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In Proceedings of the IEFF/CVF international conference on computer vision, pages 589-598, 2021.
10

---

[15] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR '05). volume 1, pages 886-893. leee, 2005.
[16] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference Machine Learning, pages 7480-7512. PMLR, 2023.
[17] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1Xiv:1810.04805. 2018.
[18] Piotr Dollár, Vincent Rabaud, Garrison Cottrell, and Serge Belongie. Behavior recognition via sparse teapool features. In 2005 IEEE international workshop on visual surveillance and performance evaluation of tracking and surveillance, pages 65-72. IEEE, 2005.
[19] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2625-2634, 2015.
[20] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12144-1213,, 2022.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[22] Efros, Berg, Mori, and Malik. Recognizing action at a distance. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 726-733. IEEE, 2003.
[23] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6824-6835, 2021.
[24] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211. 6211, 2019.
[25] Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridg- ing video-text retrieval with multiple choice questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16167-16176, 2022.
[26] Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic. and Bryan Russell. Actionvlad: Learning spatio-temporal aggregation for action classification. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 971-980, 2017.
[27] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action trans- former network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 244-253, 2019.
[28] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision. pages 12259-12269, 2021.
[29] Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long- term video. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 2906-2916, 2022.

---

[30] Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisserman. Autoad: Movie description in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18930-18940, 2023.
[31] Tengda Han, Max Bain, Arsha Nagrani, Gul Varol, Weidi Xie, and Andrew Zisserman. Autoad ii: The sequel-who, when, and what in movie audio description. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13645-13655, 2023.
[32] Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisserman. Autoad
ii:: The prequel-back to the pixels. arXiv preprint arXiv:2404.14412, 2024.
[33] Mehrdad Hosseinzadeh and Yang Wang. Video captioning of future frames. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 980-989, January 2021.
[34] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, arXiv preprint (tt::2006.09660 2021.
[35] Noureldien Hussein, Efstratios Gavves, and Amoll WM Smeulders. Timeception for complex action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 263, 2019.
[36] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2000-2009, 2019.
[37] Dohwan Ko, Joonmyung Choi, Juyeon Ko, Shinyeong Noh, Kyoung- Woon On, Eun-Sol Kim, and Hyunwoo J Kim. Video-text representation learning via differentiable weak temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5016-5025, 2022.
Mooung-Woonnen
[38] Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsampler: Sampling salient clips from video for efficient action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6232-6242, 2019.
[39] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[40] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[41] Sangho Lee, Jinyoung Sung, Youngjae Yu, and Gunhee Kim. A memory network approach for story-based temporal summarization of 360 videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1410-1419, 2018
[42] Sangmin Lee, Hak Gu Kim, Dae Hwi Choi, Hyung-Il Kim, and Yong Man Ro. Video prediction recalling long-term motion context via memory alignment learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3054-3063, 2021.
[43] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XXI 16. pages 447-463. Springer, 2020.
[44] Jie Lei, Tamara I. Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 1846- 11858, 2021.
[45] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Cliphert for video-and-languus learning via sparse sampling. In Proceedings of the IEEF/CVE conference on computer vision and puttern recognition, pages 7331-7341, 1021.
12

---

[46] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Vido-and-langues pre-traiming with entity prompts. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 4953 4963, 2022.
[47] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301112597, 2023.
[48] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint WrXiv:2305.06355 2023.
[49] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hi- erarchical encoder for videot language onni-representation pre-training. arXiv preprint arXiv:2005.000 2020.
[50] Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00199, 2021.
[51] Zhenyang Li, Kirill Gavrilyuk, Efstratios Gavves, Mihir Jain, and Cees GM Snoek. Videolstm convolves, attends and flows for action recognition. Computer Vision and Image Understanding, 166:41-50, 2018.
[52] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 2023.
[53] Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, pages 74-81, 2004.
[54] Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17949-17958, 2022.
[55] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric lideo-laggaage pretraining. Advances in Neural Information Processing Systems, 35:7575-7586, 2022.
[56] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[57] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268. 2024.
[58] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[59] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08844 2023.
[60] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[61] Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint 23X6v:2306.09000 2023.
[62] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
13

---

[63] Antoine Miech, Jean-Baptiss Alayrac. Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9879-9889, 2020.
[64] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oaa: Visual question answering by reading text in images. In 2019 international conference document analysis and recognition (ICDAR), pages 947-952. IEEF, 2019.
[65] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09755 2021.
[66] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In Proceedings of the IEEE/CVF international conference computer vision, pages 3163-3172, 2021.
[67] David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected clip. arXiv preprint arXiv:2211.00575. 2022.
[68] Mandela Patrick, Dylan Camphell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Fe- ichtenhofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. Advances in neural information processing systems, 34: 12493-12506, 2021.
[69] Xiaojiang Peng, Changqing Zou, Yu Qiao, and Qiang Peng. Action recognition with stacked fisher vectors. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V 13, pages 581-595. Springer, 2014.
[70] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533-5533555 5541,,51,111 2017.
[71] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[72] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-drives generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[73] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffmmn, et al. Hiera: A hierar- chical vision transformer without the bells-and-whistae arXiv preprint arXiv:2306.0099 2023.
[74] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146-162. Springer, 2022.
[75] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. End-to-end genera- tive pretraining for multimodal video captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17959-17968, 2022.
[76] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.
[77] Zhiqiang Shen, Jianguo Li, Zhou Su, Minjun Li, Yurong Chen, Yu-Gang Jiang, and Xiangyang Xue. Weakly supervised dense video captioning. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 1916-1924, 2017.
14

---

[78] Nina Shvetsova, Anna Kukleva, Xudong Hong, Christian Rupprecht, Bernt Schiele, and Hilde Kuehne. Howtocaption: Prompting llms transform video annotations scale. arXiv preprint arXiv:2310.04900, 2023.
[79] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer AisionEECVV 2020: 16th European Conference. Glasgow, UK. August 23-28, 2020. Proceedings, Part II 16, pages 742-758. Springer, 2020.
[80] Mattia Soldan, Alejandro Pardo, Juan León Alcázar. Fabian Caba, Chen Zhao, Silvio Giancola, and Bemnard Ghanem. Mad: scalable dataset for language grounding in videos from movie audio descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5026-5035, 2022.
[81] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Nenq-Neng Hwang, al. Moviechat: From dense token sparse memory for long video understanding. arXiv preprint arXiv:2307.16449 2023.
[82] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings the IEEE/CVF international conference on computer vision, pages 7464-7473, 2019.
[83] Yuchong Sun, Hongwei Xue, Ruihua Song. Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. Advances in neural information processing systems, 35:38032-38045. 20222.
[84] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078-10093, 2022.
[85] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09286 2023.
[86] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider:
Consensus- based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4165,4475,22 2015.
[87] Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 2022.
[88] Jinpeng Wang. Yixiao Gc, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Object-aware Iado-lan pre- pre-training for retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3313-3322,
2022.
[89] Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jiamping Wu, Ying Shan, al. All in one: Exploring unified video-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6598 6608, 2023.
[90] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36. Springer, 2016.
[91] Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6847-6857, 2021.
[92] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning large language models. Advances in Neural Information Processing Systems, 35:24824-2448 2022.
15

---

[93] Spencer Whitehead, Heng Ji, Mohit Bansal, Shih-Fu Chang, and Clare Voss. Incorporat- ing background knowledge into video description generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3992-4001. 2018.
[94] Chao- Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1884-1894, 2021.
[95] Chao- Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl. Compressed video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018.
[96] Chao- Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 284-293, 2019.
[97] Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmente multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587-13597, 2022.
[98] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, al. mplug-2: modularized multi-modal foundation model across text, image and video. arXiv preprint ar arXivv:330..00402 2023.
[99] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309. 16671, 2023.
[100] Mengmeng Xu, Erhan Gundogdu, Maksim Lapin, Bernard Ghanem, Michael Donoser, and Loris Bazzani. Contrastive language-action pre-training for temporal localization. arXiv preprint arXiv:2204. 12293, 2022.
[101] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang. Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5036-5044 2022.
[102] Hongwei Xue, Yuchong Sun. Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting gre-trained image-text model to video-language representation alignment. arXiv preprint arXiv: 200422000006000 2022.
[103] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979, 2022.
[104] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learn- ing to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1686-1697, 2021.
[105) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.144: 141788 2023.
[106] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEF/CVF international conference on computer vision, pages 558-567, 2021.
[107] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4694-4702, 2015.
16

---

[108] Hang Zhang, Xin Li, and Lidong Bing. Video- llama: An instruction tuned audio-visum language model for video understanding. arXiv preprint arXiv:2306.02858, 2023.
[109] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: urXiv:190.006777 2019.
[110] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European conference on computer vision (ECCV), pages 113-818,7 2018
17

---

CALVIN: Improved Contextual Video Captioning via Instruction Tuning
Supplementary Material
A
Data curation additional details
All the human annotations in this paper are done exclusively by the authors. Data Cleanup: During the initial training of our models, we encountered unexpected challenge. Despite using datasets generally regarded as high-quality, we observed that the models were outputting repetitive patterns of numerical data. A more detailed examination revealed that the captions in the WebVideo dataset often included specific dates or video resolutions, like 1930x 1080 or 4HD. These elements were inadvertently leading the model to generate dates and resolutions in its output. We scrubbed numerical data from Web Video using hand-crafted (regex) functions, resulting in a marked improvement in the quality of the model's 's output.
We also identified that the bounding box coordinate questions in the instruction fine-tuning dataset of LLa [58], which are uncharacteristic of our problem domain, proved detrimental to the downstream captioning task. Hence, we excluded these.
Lastly, we noticed that MAD includes movie credits at the beginning or end of some films, We used LLaMA-70B [85] with a few human-annotatee in-context examples to tag training examples as either credit-related or not. Removing scenes with credits caused subtle improvement in the quality and relevance of the generated captions.
B Stage-2 training additional details
In Tab. 5, we present the question templates used in training of CALVIN Stage-2 model, to convert video-caption data into instruction data.
Table 5: Stage-2 question templates
Question template
What this video about?
Describe the video, including the actions and scenes.
Provide a description of the given video, capturing its key moments. Give concisc explanation of the video you sec, including the events and characters. Summarize the contents of the vidco, focusing on the main events and participants.
Detail the scenes and characters present in the provided video clip.
Narrate the sequence of events in the video, including significant actions.
Report on the events and individuals portrayed in the video clip.
Highlight the pivotal scenes and actions observed in the video.
What's 's going on in video?
Descrihe the scene the video
Summarize the key visuals and events of this video. What's 's happening in the video?
Detail the primary actions and visuals of this footage. Provide brief account of the scenes and characters within this video.
Elucidate the main events Give an overview of the storyline and characters present in this vidco.
visuals this clip.
Narrate the visual clements and the main events showeased in this vidco.
Chronicle the character dynamics and seene transitions in this vidco. Describe the aesthetic elements and main occurrences in this footage.
Highlight the significant actions and interactions within this video. Offer a concise breakdown of the scenes and events in this clip.
Detail the setting, characters, and major events of this video. Explain the visual motifs and character roles within this video. Decode the main story and visuals in this footage.
Provide an interpretation
character relationships and visuals present.
18

---

C Qualitative analysis of CALVIN vs ground truth
In Tab. we present few examples of predicted caption vs ground truth caption, one from each movie in MAD-eval split. Clearly CALVIN-3S captions are good as GT caps in many scenarios.
In the main paper Fig. 4(b), we have seen that the GT caption and CALVIN-3S captions differ quite bit, however, the caption is an acceptable alternate when we looked at the video. We present few cases from of the MAD-eval movies, HOW DO YOU KNOW? in Tab. 7. This shows that the maximum achievable bert-score on this task is lower than 100 due to the subjective nature of this task.
Table 6: Ground truth vs Predicted captions on CALVIN-3S model: We present an example from each movie
the MAD-eval -vaa dataset.
Movie
Ground Truth Coption
Predicted Caption
IDES OF MARCH
stepbacn closes the door behind him
stephen shuts the door bohind him
HOW DO YOU KNOW?
presses the elevator betton
lisa button the elevator's control
ROOMMATE BATTLE LOS ANGELES
sarah
stephen joe nods at mantz
kisses
kisses stephen
joc nods, then turms nantz dumbledore hands to harry charlie's
Harry Poterr and the of fire
dumbledore thrasts it into harry hand charlie eyes pool with
CHARLIE ST CLOUD
up with punches zabriel the
LEGION
gabriel punches
LES MISERABLES
marius gonily kisses sponine brow
munios and eponine kiss passionalely
HANSEL GRETEL WITCH HUNTERS
bansel falls the ground
bansel climbs down from the tree and collapses the ground grahamn's gaze drops the Blour
SIGNS
graham stares blearily down the Blor
Table Human acceptable predictions: Some ground truth captions vs CALVIN-3S generated captions which the authors felt are acceptable despite being different from the GT. All the examples are from the MAD-eval movie, How do you know?
Ground Truth Caption
Predicted Caption
the elevator door shut george accepts the letter lisa nods
matty watches her go she hands him the letter lisa's 's mouth hangs open as she stares at him the bespectacled man shakes his head george sprints down the sidewalk she sets the bag on the kitchen counter george hangs up the phone
in george office
in his suit and dress shoes
annie loads the fridge now on the phone
19

---

Bert Score
CIDE:
Metric
00-
00
21-
Lenggh conlext
Nom.
Video
Framce
LLM Sampling Temperature
Figure 8: Inference time ablations. Effect of inference-tine hyperparameters. (Left) Number of previous scenes in the context (Middle) Number of frames sampled per scone (Right) LLM sampling temperature.
D Additional ablations.
Inference-time ablations. In Figure show how BertScore and Cider metrics change with context length, number of video frames sampled, and LLM sampling temperature. We see that increasing the number of previous scenes the context improves performance across both the metries; The model was trained with just scenes in the context and yet generalizes to more.
Regarding the number of frames sampled from each clip. we see performance improvement we go from frame to more, indicating that the model uses motion information in caption generation. However, we see no improvement in performance beyond frames. L.LM sampling uses beam search. We do not see much difference in BertScore at lower temperature, but we see slight bump in CIDEr.
E Fewshot-training ablations
We conducted an ablation on one movic from the MAD-trai train split 3041 JUST GO WITH IT to understand the right number of samples and training iterations needed for few-shot experiments. We ablate the number of iterations in Fig. 9 and the number annotated samples in Fig. 10. We observe that the performance does not improve beyond 50 annotated samples in training. And we that training for more than 100 iterations does not improve the performance.
variable = Bert.Score
variable = CIDEr
wariab'e = ROUGE.L
variable = SPICE
30
40
80
120
100
80
120
160
80
120
100
40
80
120
100
Figure 9: Number of training iterations Metries.
Broader Impacts.
Broader Impacts By leveraging previous scene contexts, CALVIN is well-positioned to enhance the accessibility visual media for individuals with visual impairments, offering them a more immersive and contextually rich experience. This improvement could make entertainment and educational content more inclusive, promoting equal access to information and enjoyment regardless of visual capability. Furthermore, the technology could be applied in various other domains such as automated content moderation, where understanding the context of scenes can improve the detection of inappropriate content, and in digital humanities, where researchers analyze films at scale to study cultural representations and evolution.
20

---

Bert. Score
ROUGEL
CTDEr
SPICE
35
30
Metric
25
20
15
10
25
50
75
Number of test samples
Figure 10: Number of ground-truth samples Metrics.
21

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " generally preferable to "[No] it is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too used"). In
computationally expensive" "we unable to find the license for the dataset general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims in the abstract and introduction are supported by empirical evidence presented the Experiments section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
We discuss the limitations of the method in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on a few datasets with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and refereeefer....
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We referenced all the publicly available datasets used in the training and explained detail the model architecture and training details. We will release the code upon acceptance.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We referenced all the publicly available datasets used in the training and explained in detail the model architecture and training details.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
pottps://nipas cc ppublic/guiee Soniedee//oeSuubiii ss sioonolicyy) for more details. While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines pestps://nips. http:://c/ppcccccpppiipccccc caooolcccguue /gguddei////ee/ Sguudde//odeeubmii Coo///oooeeeeeeeeeeeee ssisnooliicyys for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
24

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We share the hyperparameters and ablations in the main and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We could train models only once due to limited compute availability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar the standard deviation or the standard emror of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the verified.
hypothesis of Normality of errors
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss this in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: https::/ http://eeerrspp..ee www.eeerppsp.cccppi/eeeeeeeeeeee wctpublic//thiceeeeeeec delpppiiieeeiiiouudeeleeleeeeeee cetuiiciiiieeeeeeeee ines?
Answer: [Yes]
Justification: To the best of our understanding, the paper conforms to the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We discuss this in the appendix.
26

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
Licenses for existing assets
Are the creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: To the best of our knowledge, all the assets used are properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question:
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "Z14WUbiaO1", "reviewer": "Reviewer_jqA2", "summary": "Note: Raised score by 1 point after reading reviews, responses and the concerns addressed in the rebuttal phase.\n\n----\nThis work introduces a video LLM model that can describe movie scenes in context incorporating names of characters and generate short contextual descriptions. They train a model on data from image question answering datasets and video description using context from previous frames, these enable their model to generate better contextual descriptions of events.\nThey evaluate their model on Movie Audio Description (MAD) and TV-Caption datasets and show improved performance.", "questions": "1. Have you evaluated on the CMD-eval dataset[1,2]? do you find the performance to be different? If not, is there a specific reason for not utilizing the CMD-AD eval dataset?\n2. Do you evaluate on any closed multmimodal LLMs? If you tried, did you face any issues? If not, is there a reason not to try?\n\n[1]Bain, Max, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. \"Condensed movies: Story based retrieval with contextual embeddings.\" In Proceedings of the Asian Conference on Computer Vision. 2020.\n[2] https://www.robots.ox.ac.uk/~vgg/research/autoad/", "limitations": "Limitations are discussed at the end of the appendix. The reviewer’s understanding is that this woul dhave to move to the main paper. Please check the guidelines.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Their model tuning strategy of using context of the video is well motivated.\n* Their tuned model shows good performance on both the MAD-eval dataset and TV-caption datasets.\n* Ablations are explained clearly and evaluated well.", "weaknesses": "* With regard to evaluations, for movie audio description task, the CMD-train and and CMD-eval [1,2] datasets are also available. It’s not specified why the CMD dataset has not been used in this work, at least for eval. \n* It seems valuable to have a baseline with the closed MultiModal LLMs e.g. GPT-4o and Gemini 1.5 Pro all of which claim to have superior video captioning capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jqA2, thank you for your positive review of our work, and your interest in our modeling strategy. There were a few questions raised regarding dataset choices and evaluation of API models that we're happy to answer below:\n\n**[Evaluation of the CMD dataset]**\nCMD is a classic video dataset, but we did not evaluate this dataset for several reasons. First, and most pragmatic, the download links to the raw data are no longer publicly available. Second, the CMD dataset’s descriptions are metadata from the YouTube clip, Wikipedia, and IMDb. While these descriptions give a general sense of the scene, they are not strictly scene audio descriptions. Finally, in the CMD paper, the authors evaluate models for retrieval tasks but not for text generation tasks. While we hold the dataset in high regard, it is not so clear to us how well performance on the CMD eval split measures the fine-grained AADs that we're looking for with modern models.\n\nHowever, similar to another related dataset CinePile[1], the CMD dataset might prove useful in the pre-training stage of the model.  While pretraining is a bit out of scope for us within the rebuttal phase, we will provide the results of the model variants of these two datasets in the mix for the camera-ready version.\n\n**[Evaluation of closed MultiModal LLM for the task]** Due to copyright reasons, the MAD dataset provides only CLIP embeddings but not the raw videos thus preventing us from evaluating closed MM-LLM models directly on MAD. This makes it impossible to test this dataset with general-purpose multi-modal APIs, which only accept video input (not CLIP embedding input). Nevertheless, we acquired scenes from 2 movies of the test set from YouTube - How Do You Know and Battle Los Angeles (approx 300 AD scenes) for this rebuttal and ran Gemini Pro. We present Google Gemini Pro and CALVIN’s results on this subset below. We chose Gemini Pro as a commercial model since it performs better than GPT-4o according to recent video understanding benchmarks[1,2].  While Gemini’s results are better, CALVIN’s numbers are close behind for a much smaller model, trained with research computing and public data. Since it is not known what is in its training mix, Gemini may have some level of memorization of the evaluated data while CALVIN never saw this data in training. We will add these results to the appendix in camera-ready.\n\n\nModel | BertScore |  CIDER |  ROUGE-L | SPICE\n| -------- | ------- |  -------- | ------- | ------- |\nGemini | 43.23 | 28.11 | 18.54 | 8.99\nCalvin | 41.42 |  26.90 | 17.88 | 8.23\n\n[1] - Rawal, Ruchit, et al. \"Cinepile: A long video question answering dataset and benchmark.\" arXiv preprint arXiv:2405.08813 (2024).\n\n---------\nOverall, we appreciate the reviewer's insights, and these additional experiments have added depth to our analysis. We hope we addressed your questions sufficiently and we would be grateful if you would consider raising your score. Do you have any additional questions we can address?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HopXb31LeO", "reviewer": "Reviewer_suoE", "summary": "This paper addresses the task of contextual video captioning, with a particular application emphasis on settings for film and tv, where audio descriptions can be useful for making these mediums more broadly accessible. Standard vision-language models are often verbose (problematic since audio descriptions must fit between dialogue) and are prone to hallucinations (e.g., of actor names, or of entities not present in the scene but may be thematically related). The authors propose CALVIN, a model that incorporates pretraining on a range of vision-language tasks (image QA, video captioning, etc.), instruction tuning to improve captioning behavior, and some final adaptations (e.g., prompt engineering, few-shot adaptation, etc.). They observe improvements on the MAD-eval (for film) and TVC (for television) datasets, and qualitative results show more terse descriptions with fewer hallucinations.", "questions": "Overall, this work is borderline, with a preliminary rating of borderline+. The discussion phase will be important for this work, so if the authors could address some of the areas for clarification that are raised above in weaknesses, particularly with respect to prior work in this space, this would be helpful towards finalizing the rating. Note that for many of the points the review above notes a specific example for illustration, but the comments do extend more generally.", "limitations": "The authors have mentioned some limitations of their method, but not in great detail (there is a short, general statement in the final paragraph of the conclusion). This can be expanded in the supplement, for example, addressing the limitation that the model does not seem to output a clear localization of the caption.\n\n---\n\n**Post-rebuttal update:** The authors largely addressed the concerns raised in the rebuttal phase. After consideration of this and other reviewers' comments, I've raised my rating further.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, there are a number of strengths to this work:\n\n`+` The domain of contextual description for films and television is important and can have broad positive impacts. It is also a relatively new space in the broader space of dense captioning.\n\n`+` The proposed model examines some sensible explorations of the VLM design space for this task setting. Among other things, the adjustments to the training recipe (i.e., which data goes in which training stage) seem to have a good impact, in ablations.\n\n`+` Benchmark results on two different datasets, important since MAD-eval only provides pre-computed features (due to copyright), so having an additional dataset without this restriction helps to better clarify differences with other SOTA VLM models.\n\n`+` Shows good performance results, quantitatively and qualitatively.", "weaknesses": "However, there are some key areas of weakness, as follows:\n\n`-` The paper focuses on reducing hallucinations and verbosity, but the metrics provided (across the model and model-free/n-gram based ones) do not necessarily correlate or show that this is the reason that the metrics have gone up. Similar prior work (e.g., AutoAD series) have done extensive explorations of metrics around character relevance, and stronger model-based metrics. Given that many of these metrics/analyses have been released by prior work, and the research focus of this paper, the empirical results would be significantly stronger if such analyses were performed for here (and ideally, in a consistent way to make an explicit comparison with prior work).\n\n`-` The paper presents a large set of ideas, but the individual novelty of each element is not clear, and it would be good to have clearer comparisons with the closest prior work that introduced the relevant idea (in as much of an apples-to-apples fashion as is possible). As one example, the incorporation of IMDB metadata (for actors) as a small set of exemplars has been explored by prior work (specifically, AutoADII); an ablation based on this approach seems like it would be an important reference point for characterizing the relative improvement with the slightly different style proposed here.\n\n`-` The overall task of audio description also cares about the localization of the captions along with the captions themselves. While prior work (e.g., in the AutoAD series) also considered this aspect, this is notably missing from this architecture. It is also a bit unclear why, given that time tokens have been explored for similar VLM models already. Relatedly, the paper would benefit from improving the clarity of the exact inputs, in terms of the temporal regions that are provided to the model for captioning and how these are selected. (There is some language around the choice of the few-shot examples, but this comment is more broadly speaking across all settings).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer suoE, thank you for your detailed review. We're glad that you found our contextual captioning model insightful. We do think that our thorough analysis of all parts of the video LLM pipeline, from improved data cleaning and handling, to clearly ablated training recipes and architectural simplifications are a key contribution of our paper taken together as a whole, and we believe that this paper would be a good resource for next-gen contextual video LLMs.\n\n**[Character analysis]** The character analysis performed in Auto-AD II is to check the performance of  their character recognition module not the character recognition in the final caption. We noticed that Bert score is highly correlated to getting the character name right in the audio description (as Bert embeddings seem to be biased towards proper nouns). Prompted by your question, we also performed an analysis on the percentage of times the model predicts the name correctly, and we saw a score of 72% for our best model. Please note that in the rest of the 28% of the time, in high proportion, the names are replaced with “he” or “she” which are not technically incorrect.  We are unable to compute this Auto-AD series of models since the model checkpoints nor the model outputs are available. \n\n**[Verbosity]**  Prompted by your question, we computed the token count using TikToken for Table 2 (since outputs of baselines from Table 1 are not available), we present the average token counts on the test set for the models in the table below\nModel | Avg num of tokens (lower better)\n| -------- | ------- | \nVideoLlama | 82\nMovieChat | 126\nVideoLlava | 64\nOurs  | 24\n\n**[Hallucinations]** What we noticed is, that most of the models suffer from “factual errors” as discussed in Liu et al [1]. To the best of our knowledge, there are no benchmarks or tools to evaluate hallucinations in a free-form generated caption on a random video. However, to address your question, we used Gemini Vision Pro as a judge to give a hallucination score between 1 to 10 for a given generation. We prompted Gemini with some examples from [1]. We took the video clip from Fig 1 of the main paper (a scene from the Battle Los Angeles movie), which in turn has 6 ADs associated with it. We present the average scores given by Gemini in the table below. We see Calvin is significantly better than other OS models in terms of the level of hallucination according to the Gemini model as judge rating. \n\nModel | Avg hallucination score by Gemini (lower better)\n| -------- | ------- | \nVideoLlama | 8.7\nMovieChat | 9.1\nVideoLlava | 6.8\nOurs  | 4.3\n\nWe will extend all the above-mentioned analyses with full details to the appendix of the final version of the paper\n\n**[CALVIN comparisons to prior work]** Calvin and Auto-AD II share most of the training data and whatever differs, the scale of the data is the same. While AutoAD II has additional aspects of incorporating character and identity information, ours is a simplified VideoLLM that takes a scene representation and pure text context. We do think that it is a key strength of our work to look for general-purpose improvements only through contextual information, without specializing the system further with e.g. actor information. While using IMDB actor data in Calvin is an interesting thought, we would like to point out that, despite using that information, Auto-AD II’s results are worse than Calvin’s, highlighting the advantages of the generic approach. We believe our model’s high performance is due to data cleanup, conversion of existing data into instruction tuning data, and a well-tuned model due to our exhaustive ablations. \n\n**[Temporal localization of AD]** Thanks for raising this point. Both AutoAD II and Calvin take a truncated video clip from the movie and previous context to produce Audio Descriptions. We observed that whether an AD should be written for a given clip or not depends on three factors 1) the subjectivity of the annotator 2) the Length of the clip 3) How different the clip is from the previous scene. \n\nThe AutoAD II paper also observed that pauses longer than 10 sec have an AD while pauses less than 4 sec most likely do not have an AD. We also observed similar trends. AutoAD II looks at 30-sec proposals along with audio and video data in the time interval and makes a prediction for each 5-seconds whether AD exists or not. \n\nWe approached this problem slightly differently as we believed a simpler solution was possible. First, we only look at the “pauses” in the audio of the movie.  We trained a simple classifier head on top of our Q-former layer which classifies whether a clip needs AD or not. Along with the vision embeddings, we also input the length of the clip and the L2 distance of the current clip with the previous and next clip. We finetuned the classifier head for 2 epochs on MAD training data. We saw slightly better results compared to AutoAD-2. We present the results in the table below. Since AutoAD II code of temporal classification is not available, we present the best numbers from the corresponding paper for this model. \n\nModel | AUROC (higher better)\n| -------- | ------- | \nAutoAD II | 0.78 \nOurs | 0.8 \n\nWe believe it is not possible to 100% accurately predict whether AD is needed in a given pause or not for shorter durations since it is quite subjective and we noticed differences even within movies in the eval-split (which are most likely annotated by different humans). We will add this additional analysis with exhaustive details to the camera-ready version.\n\n[1] - Liu, Hui, and Xiaojun Wan. \"Models see hallucinations: Evaluating the factuality in video captioning.\" arXiv preprint arXiv:2303.02961 (2023).\n\n-------\nThank you again for your thoughtful review. We hope we addressed your concerns/questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vagqnk3aG8", "reviewer": "Reviewer_Y2oh", "summary": "This paper introduces a specialized video LLM named CALVIN, which leverages previous movie context to generate contextual scene descriptions. The authors achieve this by training their model on both image QA tasks and video captioning tasks within a unified framework. Experiments demonstrate that with mixed training data and context during inference, CALVIN outperforms the previous state-of-the-art in the audio description task.", "questions": "See weaknesses.", "limitations": "The authors have discussed the limitation of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-written and easy to follow.\n2.\tThis paper explores the use of video LLM in audio description, demonstrating significant performance gains over the previous SOTA.\n3.\tThe paper presents detailed ablation studies, which are helpful in understanding the effectiveness of CALVIN.", "weaknesses": "1.\tThe method appears similar to VideoLLaMA except for differences in data usage and the base LLM. The authors should provide a more thorough discussion on how their approach differs from previous methods.\n2.\tThe experimental results in Table 1 and Table 2 are not entirely fair due to the use of different training data and pre-trained models. The data seems critical for CALVIN's performance, as shown in Table 3. Therefore, it would be better to align data usage with the compared methods or at least highlight the types of data used by other models.\n3.\tIs it practical to use context from captions, given that ground-truth captions do not exist? What is the performance of using generated captions as context for CALVIN?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback, Y2oh, we're happy to answer your questions regarding a better description of the architectural differences between CALVIN and VideoLLaMA, and regarding further dataset ablations.\n\n**[Clarification of Differences to VideoLLaMA and other recent video architectures]** The first main difference between CALVIN and other models is how the video representations are projected onto the Language model space. While previous methods such as videoLLama used 2 projection Q-formers (with one component frozen and another trained from scratch) in sequence, we simplified the pipeline with a single Q-former that is trained from scratch. The second main difference is the LLM finetuning and the usage of LoRA. Our experiments showed that unfreezing the LLM in stage 2 can lead to significant performance gains, while previous methods such as VideoLlama had that component frozen throughout the training.\n\nMany recent works such as MM-1 [1], Cambrian-1[2], and PaliGemma [3] have shown that while some critical architectural choices such as positional embeddings or certain activations could be important, most of the gains came from the data mixtures. Compared to previous works, we are also the first ones to propose the use of image VQA data in training and the conversion of caption data into synthetic VQA via LLMs and to show that adding these into training can improve performance. \n\nOverall, thank you for bringing up this question. We will add this discussion to our related work section.\n\n**[Dataset comparisons]** \nWe provide a short table on the types of data used in the last stage of training of important baselines in the following table. \n\nModel | Datasets used\n| -------- | ------- |\nVideoLlama | Webvid-2M, CC595k\nAuto-AD I |  CC3M, Webvid-2M, AudioVault-AD (3.3M), MAD\nCALVIN | CC3M, Webvid-2M, Llava (600k), MAD\n\nWhile the scale of data used in the training of all the models is almost the same, how the data is used varies across the models. We shared our data learnings in the paper (Appendix A), showing that cleaning the data or removing certain types of data can improve performance significantly. We also modified the data into instruction-tuning format and this led to gains. \n\nWe also thoroughly ablated different types of training setups and shared our learnings in our paper. We believe while our architecture and data slightly differ from the baselines, they are of similar scale, and our smart data curation and our thorough studies led to the best freeze-train configuration to achieve the final model.\n\nWith even more compute, we would have liked to run additional ablations of the other models with our data, but this is not something we could budget in. In any case, we've included the exact data mix for all models in Table 1 and Table 2 with an additional column, to clarify to readers that these models do differ in their pre-training data.\n\n**[How to do contextual captioning when ground truth context doesn’t exist?]** Excellent question. This is quite a plausible scenario and this is the inspiration for Section 5.  We saw a drop in performance when we used generated captions in context, however, the performance is still better than the baseline model as well as Calvin with no context. We present the contextual captioning with self-generated ground truth in the following table. (We only presented Cider since that is the only metric available in Auto-AD paper for this scenario.)\n\nModel | Context length| Cider\n| -------- | ------- | ------- |\nAuto-AD I | 3 | 14.5 \nAuto-AD 2 | 3 | 19.5\nCalvin | 3 | 19.9\n\nWe observed the main reason for this performance drop is that many ADs contain either the first names of the characters or locations and without this knowledge in context, it is hard for models to predict correct names, and they sometimes hallucinate. Please refer to Section 5 to see our proposals to get around this issue with few-shot finetuning.\n\n[1] - McKinzie, Brandon, et al. \"Mm1: Methods, analysis & insights from multimodal llm pre-training.\" arXiv preprint arXiv:2403.09611 (2024).\n\n[2] - Tong, Shengbang, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" arXiv preprint arXiv:2406.16860 (2024).\n\n[3] - Beyer, Lucas, et al. \"PaliGemma: A versatile 3B VLM for transfer.\" arXiv preprint arXiv:2407.07726 (2024).\n\n-------\nWe thank the reviewer for a thoughtful review. We hope we addressed your concerns/questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DhCc2TgUB2", "reviewer": "Reviewer_GsDL", "summary": "The paper focuses on captioning movie scenes, which unlike typical captioning tasks, videos from movies can be captioned in a way that tells the story of the movie. For example, captions from movie scenes should convey how the characters felt, and not just the detail in the image.  To be able to caption in this way, the authors propose and train CALVIN, a 7B LVLM, on MAD (a movie-audio dataset) as well as other datasets. They then experiment with a variety of techniques to experiment with CALVIN in various settings.", "questions": "What do you do with frames that introduce new information specific to the movie? For example, how could a model correctly predict the ground truth for frames that introduce names of characters?", "limitations": "I do not see where in the paper core limitations of CALVIN or the methodology are discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The core contribution of this paper is a large model to caption movie scenes, which is meaningfully different from captioning typical videos. The weights of the model can contribute to further study this difference, or further explore movie captioning.\n* Big improvements over existing work \n* There are lots of experiments and ablations, testing the model in prompting, test-time setups, and even few-shot finetuning (i.e., personalization toward a specific movie)", "weaknesses": "* I am not sure I can say there’s technical novelty in this work (this is not a deal breaker, since existing knowledge was applied to create a research artifact that did not exist prior). However, I do not see a statement about releasing weights (only code), and if it will not be released, I am not sure what the contribution of the paper is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback, GsDL. We're happy to answer your questions regarding our paper's key contributions, model weight release, and how to adapt the model to new movies.\n\n**[Main Contribution of the paper]** While the model weights are important (and we are working to release them), our work also includes a number of key contributions concerning architectural choices for video LLM, model training, and dataset choice and setup. On the architectural side, we show that we can simplify/unify complicated input pipelines from prior work with a single Q-former, and we highlight the importance of continued training in the LLM part of the model.  \n\nWe believe this can help video-understanding researchers make calls while training the next-gen models. For example, we are one of the few video-understanding papers that calls out the importance of data cleaning and data mixtures which is also echoed by recent works such as MM-1 [1], and PaliGemma [2]. We further introduce novel few-shot adaptations of the model for new movies which can make the model useful in practice and take it beyond the academic realm. \n\n**[Weights release]** This issue is currently under legal review. As a backup option, we are also currently working with a partner to release a non-official version of the weights for the camera-ready. \n\n**[How to predict new information specific to the movie?]** Great question. This is exactly what we tried to answer in Section 5. In our experiments, we found CALVIN was predicting character emotions and physical states well but it is hallucinating the names and locations when we do not provide the context on a new movie. Hence in section 5.2, we few-shot finetune on some scenes with main characters and ground-truth ADs, this led to non-trivial improvement in the performance even without context. We believe this could be a way to adapt CALVIN for new movie audio descriptions. \n\n[1] - McKinzie, Brandon, et al. \"Mm1: Methods, analysis & insights from multimodal llm pre-training.\" arXiv preprint arXiv:2403.09611 (2024).\n\n[2] - Beyer, Lucas, et al. \"PaliGemma: A versatile 3B VLM for transfer.\" arXiv preprint arXiv:2407.07726 (2024).\n\n-------\nThank you again for your thoughtful review. We hope we addressed your concerns/questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all four reviewers for their highly constructive feedback! We very much appreciate their assessment that our paper has “**thorough ablations**”[jqA2,suoE,Y2oh,GsDL], “**good performance**”[jqA2,suoE,Y2oh,GsDL], “well motivated”[jqA2], “important…novel problem”[suoE, GsDL], “well-written”[Y2oh].

The reviewers raised many excellent and thought-provoking questions and we conducted many experiments to address these questions. This interaction added more depth to the paper. 

### [A brief summary of questions and additional results] 
Here we summarize the additional analyses we conducted as part of the discussion. All the new analyses will be added to the appendix of the camera-ready version of the paper.
1. **Evaluation of closed MultiModal LLM for the task:** Since MAD dataset does not provide “raw” video data, we collected videos from YouTube for 2 movies from the test set and evaluated Gemini-Pro on the same. Results show Calvin has comparable performance (albeit slightly worse than Gemini) despite being a much smaller model. (For more details, see response to )
2. **Character analysis, verbosity, and hallucination aspects of captions:** For verbosity, we computed the average number of tokens of predictions (from Table 2). For character analysis, we computed the percentage of times the model can predict the proper nouns correctly. For hallucinations, we used the Gemini model as a judge to predict the level of hallucinations in the prediction against the ground truth. In all the metrics, our model Calvin did better than other open-source models. (For more details, see response to Reviewer suoE)
3. **Temporal localization of AD:** We trained a simple classifier head on top of Q-former and are able to get comparable results to AutoAD-II (For more details, see response to Reviewer suoE)
4. **How to do contextual captioning when ground truth context doesn’t exist?** Section 5 in the main paper is dedicated to this. Additional results show Calvin outperforms baselines in this scenario as well. (For more details, see response to Reviewer Y2oh)

Here are two points raised by couple of reviewers that we want to re-emphasize. We will add this discussion to the related work section in the camera-ready version.

### [Are baseline comparisons reasonable?]
We believe the comparisons with baselines are reasonable due to the following two factors:

1. The scale of the training data is the same, although the data mixtures differ, which is a core contribution of our paper. We also discussed in detail how to clean the data to improve performance, which was not addressed in previous works.
2. The size of the previous models and Calvin is the same. Videollama and Calvin have a similar number of parameters. Our improvements result from thorough experimentation and study of different components of the model pipeline.

### [How is Calvin different from previous methods?] 
The first main difference between CALVIN and other models is how the video representations are projected onto the Language model space. While previous methods such as videoLLama used 2 projection Q-formers (with one component frozen and another trained from scratch) in sequence without any justification for this choice. We simplified the pipeline with a single Q-former that is trained from scratch. Merging of parameters into single module simplifies the architecture for thorough ablations, and we also found that the results are slightly better if there is 1 Qformer vs 2 Qformer blocks, perhaps due to slight reduction in number of parameters. The second main difference is the LLM finetuning and the usage of LoRA. Our experiments showed that unfreezing the LLM in stage 2 can lead to significant performance gains, while previous methods such as VideoLlama had that component frozen throughout the training.

Many recent works such as MM-1 [1], PaliGemma [2] have shown that while some critical architectural choices such as positional embeddings or certain activations could be important, most of the gains came from the data mixtures. Compared to previous works, we are also the first ones to propose the use of image VQA data in training and the conversion of caption data into synthetic VQA via LLMs and to show that adding these into training can improve performance.


-----
In conclusion, our work includes a number of key contributions concerning architectural choices for video LLM, model training, and dataset choice and setup. On the architectural side, we show that we can simplify/unify complicated input pipelines from prior work with a single Q-former, and we highlight the importance of continued training in the LLM part of the model.
We believe this can help video-understanding researchers make calls while training the next-gen models. For example, we are one of the few video-understanding papers that calls out the importance of data cleaning and data mixtures which is also echoed by recent works such as MM-1 [1], and PaliGemma [2]. We further introduce novel few-shot adaptations of the model for new movies which can make the model useful in practice and take it beyond the academic realm.

[1] - McKinzie, Brandon, et al. "Mm1: Methods, analysis & insights from multimodal llm pre-training." arXiv preprint arXiv:2403.09611 (2024).
[2] - Beyer, Lucas, et al. "PaliGemma: A versatile 3B VLM for transfer." arXiv preprint arXiv:2407.07726 (2024).

Meta-review: This paper proposes a video LLM that uses previous movie context to generate contextual scene descriptions. Experiments on two datasets demonstrated the effectiveness of the approach.

Initially, the paper received four borderline accept scores. Reviewer jqA2 pointed out the lack of evaluation datasets especially the CMD dataset and no closed MLLMs as baselines. Reviewer suoE, Y2oh and GsDL all raised major concern on the novelty of the approach. Additionally, reviewer suoE questioned the evaluation metrics and reviewer Y2oh requested to align data for comparison and ablations w/o relying on groundtruth contexts.
The rebuttal provides more result comparisons, evaluations across different aspects and further analysis, which effectively addressed most of these concerns. 

Based on this, three reviewers increased their scores to weak accept and one remained borderline accept. The AC thus recommends accepting the paper.
The authors should ensure that all promised revisions are included in the final version.
Decision: Accept (poster)