CALVIN: Improved Contextual Video Captioning via Instruction Tuning

Authors: somepalli ⋆ gowthami
♠  chowdhury
ronen  basri
jonas  geiping
tom  goldstein
david  jacobs
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:16:37

Abstract: The recent emergence of powerful Vision-Language models (VLMs) has significantly improved image captioning. Some of these models are extended to caption videos as well. However, their capabilities to understand complex scenes are limited, and the descriptions they provide for scenes tend to be overly verbose and focused on the superficial appearance of objects. Scene descriptions, especially in movies, require a deeper contextual understanding, unlike general-purpose video captioning. To address this challenge, we propose a model, CALVIN, a specialized video LLM that leverages previous movie context to generate fully "contextual" scene descriptions. To achieve this, we train our model on a suite of tasks that integrate both image-based question-answering and video captioning within a unified framework, before applying instruction tuning to refine the model's ability to provide scene captions. Lastly, we observe that our model responds well to prompt engineering and few-shot in-context learning techniques, enabling the user to adapt it to any new movie with very little additional annotation.

Main paper: 
 introduction: 
 The volume of video data on the internet is increasing every year, now representing the largest portion of internet traffic [2] . To make this wealth of visual content accessible to vision impaired individuals requires audio descriptions. Audio descriptions (AD) describe and narrate videos or segments of videos in natural language, but are often still manually curated for only a few select videos.The emergence of large, multi-modal vision-language models (vLLMs) has led to exciting progress on supplementing AD with automatically generated descriptions (AAD), but current models still suffer from notable weaknesses in videos. Consider the concrete example of a video of a woman smiling and waving at a bus carrying a friend. An observer with understanding of the context of the scene, like a human, might caption it as A happy woman waving at the bus, perhaps receiving a loved one, while a vLLM trained on image-caption pairs would provide a more literal description, such as A woman wearing a red dress. A woman is smiling. A bus stopped. The scene has people in the background. This illustrates a key shortfall: vision LLMs focus on superficial details like the properties of physical objects, which are often irrelevant to the broader context, because they are trained on static images. Several of these models process videos only as individual images, simply stringing together descriptions from each image while failing to capture the overarching narrative.Secondly, current systems struggle to incorporate prior context into their description. Using the same scenario, if we inform a human that the woman's name is Mary and her husband, a soldier, is returning from war, a human would caption it as, Mary is overflowing with joy to receive her husband, a war hero, at the bus stop. Here, the human might choose to omit less significant details like 'waving' in favor of emphasizing her happiness, showing a nuanced understanding of the context. In contrast, many vision LLMs lack the ability to integrate such context with visual data, often ignoring it entirely. This is a significant gap, as useful interpretations naturally prioritize the intentions of characters and the results of their interactions through time, rather than the mere presence of objects and their movements -which would make the video hard to follow for a listener reliant on AD.Drawing inspiration from the way humans utilize context in captioning, we introduce a novel contextual captioning model, CALVIN, that is designed and instruction-tuned to generate audio descriptions. Our primary objective is to develop a model that, given appropriate context, can generate captions closely resembling those crafted by humans. To accomplish this, we train our model using the Movie Audio Descriptions(MAD) dataset [80] , which includes human-generated annotations for movie scenes, complete with timestamps. This enables us to construct a "text context" for each scene, based on preceding scenes. However, the MAD dataset alone is limited in scope and insufficient for fully training a video-LLM. To address this, we incorporate image-VQA datasets, which significantly enhance the model's visual understanding. We provide a detailed discussion on training methodologies, including the optimal combinations of data to use at different stages.In addition to producing scene descriptions that are more human-like and useful than the existing vision models, CALVIN, achieves state-of-the-art (SOTA) performance in captioning on the MADeval dataset, with major improvements (∼26% improvement on CIDEr and ∼68% improvement on BertScore) over the previously established SOTA model. CALVIN also performs significantly better than all the recent off-the-shelf video LLMs we studied on zero-shot evaluations on the TVC dataset [43] . We illustrate the model's capabilities in Figure 1 . Here, while most video-LLMs generically describe the scene as a soldier holding a rifle or an army man in a military setting missing the narrative nuance of the ground truth caption Lenihan knocks it backward into the pool. CALVIN stands out by not only recognizing the character as 'Lenihan' but also incorporating prior information about the presence of an alien. Unlike the overly verbose captions of other models, our captions are crisp and narrative, focusing on actions and outcomes that are important to the plot, underlining their usefulness for the task of automated audio description.We also introduce two test-time adaptation strategies, prompting and few-shot tuning, that are particularly useful in scenarios where additional contextual information is lacking. Finally, we describe some of the limits of our current model and outline potential avenues for future research to further advance this field. 
related work: 
 Video Understanding. The key objective of parsing spatiotemporal information in videos can be achieved through hand-crafted features [15, 18, 22, 69] , recurrent networks [19, 36, 107] , convolutional networks [24, 27, 51, 70, 90, 95] , and more recently Vision Transformers (ViTs) [7, 9, 21, 23] . ViTs [21] treat an image as a set of patches and use a transformer architecture to model their interactions. Some works also effectively add multi-scale hierarchies [23, 28, 60, 73, 106] or local structures [14, 20, 60] to the transformers. Naturally these models can be extended to videos where a video is treated as a sequence of independent image frames, and a subsequent temporal pooling layer or a temporal transformer is added [7, 16] . There have been more generalized video modeling approaches [9, 23, 66, 68, 73, 84] that directly work on a video clip by dividing it into 3D spatio-temporal patches. While most of these models have proven to work well on short videos (<5 seconds), longer-video (>30 seconds) understanding is still an active research area. Some existing methods include pre-computing features and separately training backbones [3, 19, 26, 94] , increasing model efficiency to include more frames [35, 38, 95, 110] , and building a memory-model that can reference prior context [13, 41, 42, 96, 97] . Recent studies show that video-text pretraining [4, 5, 25, 29, 37, 45, 46, 49, 63, 71, 75, 82, 88, 89, 101, 103] can greatly help in long video understanding [83] , temporal localization tasks [12, 44, 55, 100 ], text-video retrieval [63] , video question answering [104] , and video clip captioning [5, 75] .Generalist Video LLMs. Recent breakthroughs in language modeling have also spurred a flurry of work incorporating first image data, and then video data, as additional input modalities [5, 61] . While some works try to train large-scale video-text models directly from scratch [47, 65, 67, 87] , most work focuses on continued pretraining and finetuning of base language models [102, 108] . Models may be generic multi-modal models [98, 105] , or branch off from existing image-text models, such as LLaVA [52] , targeting video understanding [54] , and initial work on long video understanding [54] . Instruction tuning for videos was considered in [58, 59] and [78] , and, with a particular emphasis on interactions with long(er) videos in [48, 81] . Understanding long videos is not only a learning problem, but also a technical challenge, necessitating engineering improvements, such as RingAttention to even process long videos [57] .Contextual Video Captioning. What are use cases for these video LLMs? A particularly interesting one is automated video captioning, which effectively converts video content back into text. This is useful, not only for tasks such as search, retrieval, but also essential to generate audio descriptions (AD) of video content. Automated audio descriptions describe the content of videos verbally, and are central to making videos accessible to anyone with visual impairment [64, 76, 93] . Seminal work, such as CineAD [11] , argued that automated audio descriptions of even highly contextual content such as cinema, should be feasible, leading to a broader interest in automated audio descriptions (AAD). The most recent work in this direction are AutoAD I and II [30, 31] , generating audio descriptons based on CLIP features processed with smaller language models. Auto-AD III [32] is a concurrent work that introduces a large-scale dataset and a 7b model to perform this task. AD is particularly focused on contextual descriptions and allows a listener to make sense of a long-form video, such as a movie. This separates this task from the more generic task of dense video captioning [33, 39, 77, 82, 91] . 
calvin: a contextual video captioner: 
 We now discuss the proposed architecture, datasets, and training process.3.1 Method & Architecture Consider a collection, V, of movie scenes [v 1 , v 2 , . . . , v n ]. Each scene contains k image frames such that v j = [v 1 j , . . . , v k j ],and comes with annotations v j in the form of a sequence of text tokens [a 1 j , a 2 j , .., a m j ]. These annotations are usually obtained by converting audio to text, where the audio comes from the movie dialog or a person describing the scene. We seek a model f ϕ (•) that effectively predicts the audio description given the past visual and text tokens.We parameterize f ϕ (•) with an LLM. While the pre-trained LLM can easily handle all the text tokens a 1:i-1 , it must be adapted to handle vision tokens corresponding to frames v i . We construct a video projection layer to process the video component and project it into the LLM's representation space. The overall causal problem formulation of contextual video captioning becomes We present the templates and an example of data format for different stages as shown in Figure 2 . The colors of the text in this example match the token type colors in Fig. 2 .EQUATIONThe underlined text represents the segments that the language model must predict at all stages, and it is upon these predictions that the loss is computed.where I is a frozen image embedding model that accepts an image frame v k i and outputs a fixed d-dimensional embedding. The function g θ (•) is a learned non-linear projection module that takes in multiple image embeddings and projects them into the LLM latent space. We use a CLIP ViT-h/14 [99] vision encoder as the frozen feature extractor I. We train the system to predict the next text token based on all available context in the scene:θ * , ϕ * = arg min θ,ϕ n j=1 m i=1 L âi j , a i jwhere âi is as defined in Equation ( 1) and L is cross-entropy loss.Our projection module g θ (•) is comprised of two sub-components: a Q-Former layer [47] , and a linear layer. The Q-Former begins with a fixed number of learnable embeddings, which are cross-attended by the video-frame embeddings during training. This formulation provides flexibility in handling a variable number of CLIP embeddings, up to a maximum defined by the position embedding, hence allowing training with mixed image and video datasets. The Q-Former layer is initialized with the pre-trained weights of BERT base [17] . We add position embeddings to the video-frame embeddings before the cross-attention to capture the temporal dynamics in videos. The second part of the projection module is a linear layer that projects the output of the Q-Former into the latent space of the LLM. We use a Llama-2 7b variant [85] model as our base LLM module.The MAD dataset [80] stands out as the only extensive audio-description dataset currently available that includes aligned visual data, making it uniquely suitable for training contextual captioning models. However, the dataset's scale is relatively modest, especially when considering the extensive data requirements for effectively training a video-based LLM. To address this limitation, we train jointly on both image and video data. We use a single unified CLIP and Q-former vision pipeline for both modalities, enabling us to seamlessly transfer knowledge from the image to the video domain.We use a two-stage training process for the video LLM, as illustrated in Fig. 2 .In the first stage, we align the projection component (as denoted by Q in Figure 2 ), which consists of the Q-Former and a linear layer, while keeping both the CLIP and LLM frozen. In this stage, we use an internal dataset of image-caption pairs (≈ 100M pairs), CC-12M [76] and the pretraining stage data curated by LLaVA [58, 59] which is built on top of MS-COCO [56] .For the second stage of training, we fine-tune the projection component as well as the LLM as shown in Fig. 2 . This stage also marks our transition to employing higher-quality vision datasets. The data mixture for this phase is significantly enriched, comprising CC-3M [76] , instruction tuning data from LLaVA 1.5 [58] -itself a curation based on a range of public image-caption or QA datasets including OKVQA [62] , A-OKVQA [74] , TextCaps [79] , Visual Genome [40] . Additionally, we incorporate the WebVid-3M [8] video caption dataset and MAD [80] dataset train split into this stage. As such, this data mix is of higher quality as the majority of it is human-annotated.Contrary to the approach of AutoAD [30] where the MAD dataset was trained separately, our ablation studies, which will be discussed later, reveal that integrating the MAD dataset into the second stage speeds up and simplifies training without significantly impacting the performance of the final model.The dataset from LLaVA-1.5 comes pre-formatted for instruction tuning, simplifying its integration into our training process. For the other datasets, we adapt them to align with this instruction format. Typically, these datasets follow a <image/video ><caption> structure. We restructure them into a more comprehensive format: <image/video ><context><question><caption>. In this format, for all datasets except MAD, the <context> component is simply a placeholder space. For MAD, we use the annotation from the previous few scenes as <context>. We sample from a curated set of template questions regarding the <question> component. For a detailed view of these template questions, please refer to the Appendix. 
experiments & results: 
 Training details: All models are trained on a single A100 node with 8 GPUs. In Stage 1, we train only the projection module (Q-Former and linear layer) for 400, 000 iterations, with gradient accumulation over 4 steps and per-GPU batch size of 32. The learning rate has a cosine schedule and a warmup phase of 2,500 steps with min LR 1e -6 and max LR 1e -4. In Stage-2 we train Q-Former, linear projection, and the LLM. We train each model for 120, 000 iterations with a cosine learning rate with min LR of 1e -6 and max LR of 1e -4. The per-GPU batch size is 12 for image datasets and 6 for video datasets. Across all stages, a weight decay of 0.05 was applied. We adopt the Low-Rank Adaptation (LoRA) [34] approach for training the LLM. We set the LoRA rank to 32 and use LoRA on the QKVO (Query, Key, Value, and Output) in the attention layers. Unless stated, the Q-Former contains 4 layers with 32 learned embeddings and is trained to accept a maximum of 32 frames from the video. During the training, we sample 32 frames from the video. The datasets used in each stage are discussed in detail in Sec. 3.We evaluate our models and compare against baselines on four metrics. The first is BertScore [109] , which measures the similarity in BERT representations between the ground truth and the generated captions. The remaining three are traditional captioning evaluation metrics, CIDEr [86] , ROUGE-L [53] and, SPICE [6] each offering a unique evaluation perspective.Evaluation details: We benchmark against the current state-of-the-art model, AutoAD-II [31] , using the MAD-named-test split. This test split has captions that include character names. Owing to the unavailability of trained models for both Auto-AD [30] and AutoAD-II [31] , we present the best-reported results from these papers. Other baselines include ClipCap [65] and CapDec [67] which were discussed in Auto-AD I.A limitation of the MAD dataset [80] is the absence of raw videos; only embeddings of the CLS token of the ViT model are shared. This constraint prevents the evaluation of models that require data in formats other than these specific embeddings. To provide a broader assessment of our model's performance relative to general-purpose open-source video language models, such as MovieChat [81] ,VideoLLaVA [52] and VideoLlama [108] , we have conduct zero-shot comparisons on the TVcaptioning (TVC) dataset [43] .MAD-eval. Our main objective is to do contextual captioning. Hence we compare CALVIN against the SOTA models on the MAD-eval dataset in Tab. 1. Across all the metrics, we see CALVIN has significantly higher performance compared to the SOTA model. Even though our model is trained with context, we present a baseline case where we evaluate without context. In this case, we see ∼90% improvement over the best baseline BertScore. Among models with context, we see a further ∼26% improvement on CIDEr and ∼68% improvement on BertScore. We present a few qualitative results in Figure 4 . CALVIN-3S represents the case with 3-scene context and CALVIN is the evaluation with no context. When evaluated with no context, the model hallucinates names in some scenarios (scenes-a, c). However, these hallucinations disappear once we provide context to the model and the captions are quite close to the ground truth. For scene b, our model's generation differs from the ground-truth caption but is technically correct. We encounter a few such false negative cases where a human would find this generation acceptable but it is hard to identify such cases without human intervention. We present some such scenarios in the Appendix.TV-Captioning. We also benchmark CALVIN against off-the shelf SOTA video-LLMs on the TVC captioning [43] task in Table 2 . All models are evaluated in zero-shot fashion, where none of the models (including ours) are fine-tuned on this dataset. Context is not available in this dataset, but this dataset's captions refer to the characters in the scene with names, and so we add the character names in the prompt when we are querying all the models. CALVIN outperforms all the baselines across all metrics despite not being trained on this dataset. The distinction is quite visible on the CIDER metric where CALVIN is ∼3x better than the next best model. This shows the generality of our model for movie/TV captioning tasks. We discuss a few ways of contextualizing captioning when the context data is not available in Section 5. We share a few ablations and some insights gained. Since there are many moving parts in the system, the search for our final configuration is mostly greedy and looks at one component at a time. We present the captioning metrics of the resulting models in Table 3 . All models in the table are trained for the same number of iterations with a context of three previous scenes along with the video. These models are also evaluated with a 3-scene context, and other hyperparameters are kept constant.Data mixture in Stage-2. As discussed in Section 3.2, we combine multiple datasets in this stage. We remove one data type at a time. Removing Stage-2 training impacts the performance the most, followed by removing the MAD dataset. This makes sense since AD captions tend to be crisp and more contextual, while others are more descriptive. There is a clear domain shift and we need the MAD dataset in training to bridge this gap. One interesting observation is that removing the image VQA data impacted the performance a bit more than removing the only video dataset other than MAD. This confirms that the image VQA data can contribute to video understanding.Component tuning. As discussed in Section 3.2, we tune both Q-Former and LLM in Stage-2. We check performance when just one of these components is present. It turns out that training only the LLM leads to an extreme drop in performance while training only Q-Former dropped performance slightly, especially the CIDER score. We believe in LLM training case, the model completely ignores the vision embedding, attending only to the context, causing generated captions that are hallucinations with no grounding in the video. In the case of only Q-Former training, we believe that the model does not learn to utilize the context well, hence a slight degradation in performance.Q-Former tokens. We examine the effect of the number of Q-Former tokens, which in turn controls the parameter count. More tokens causes a slight degradation in performance. We believe this is because the smaller size of the stage-2 dataset causes overfitting.LLM tuning parameters. We examined two types of efficient LLM fine-tuning techniques -LoRA and Prefix Tuning [50] . Prefix-Tuning adds a few additional trainable parameters to each transformer block. In LoRA, a chosen set of parameters is updated by a low-rank approximation. First, we see LoRA training is significantly better than Prefix-Tuning. Second, as we increase the LoRA rank hyperparameter, we see slight performance improvement.Other ablations. We depart from previous works by mixing the MAD dataset into stage II rather than separately fine-tuning on it. If we instead fine-tune on MAD with a 40,000 iteration stage III, we do not see much improvement. Additionally, we observe that reducing the max learning rate from 1e -4 to 1 -5 in Stage 2 improves the performance by a non-trivial amount. We refer the reader to the Appendix for additional train-time and inference time ablations. No context: She is talking to a blonde man. Entity context: <ENT1> is talking to <ENT2>. 5 Video-language models are also few-shot learners!It was shown in Brown et.al [10] that LLMs, trained on a diverse set of data, can benefit from prompt engineering and in-context learning. Despite our video representations being trained in the specific domain of video captioning, they still inherit the emergent properties of the parent LLM. This includes their ability to adapt to prompt engineering, in-context learning, and fine-tuning for specific tasks. In this section, we propose and evaluate two realistic strategies for customizing the model at test time. These strategies have shown improved performance in captioning compared to scenarios where no context is provided.Regularization data: I don't recognize the characters or entities in the scene. Hence the final caption is: <MODIFIED_ORIGINAL_CAPTION>.The characters and entities in the scene are <ENT1> and <ENT2>. Hence the final caption is: <ORIGINAL_CAPTION>.Figure 6 : Few-shot fine-tuning. We rewrite the captions in CoT [92] style and fine-tune the model on them. 
prompt engineering: 
 It is hard for the models (and humans) to associate actors with their characters' names without a priori information. We observe that our model can be enhanced further by providing characters' or entities' information in the context. Assuming we can access entity information for a given scene, we prompt the model as illustrated in Figure 5 . We noticed that just adding this information about entities in the scene to the context improves performance. See Section 5.3 for results.Works like Dreambooth [72] "personalize" a large pre-trained model to an object or a category with test-time training. We test this idea by fine-tuning our model for every movie in the test set, showing it 20 scenes (∼5 minutes) of each movie. This paradigm enables human-in-the-loop annotation where the humans can correct the model's mistakes while the model performs the bulk of the work.To avoid over-fitting on the test time data, we propose to use additional regularization data from the train set (i.e.pre-annotated other movies) where we strip away the entity details using LLaMA-2 70B to get a rephrased caption without proper nouns or identifying details. See Figure 6 for an example of test-time movie data and regularization data used for test-time training. This is analogous to chain-of-thought prompting [92] , where a model is trained to output its reasoning. In our use case, we show that with fine-tuning, the model recognizes some of the main characters as well as adapting to the tone of annotation for the movie. Note that annotators use different styles or languages based on the type of the movie. For instance, animated films have simplistic audio descriptions as they are directed at children. We share results in Section 5.3.Experiments. We first sample 20 scenes from the test-time movie such that the scenes in which main characters first appear are included (see Table 4 for 50 instead of 20). Then we add the same number of samples for the regularization data, which are sampled randomly from the MAD-train split. We rephrase the captions of the whole mix in chain-of-thought (CoT) style as present in Figure 6 . We start with CALVIN Stage-2 checkpoint and finetune it for 100 iterations with a constant learning rate of 1e -5, 2-step gradient accumulation, and batch size of 10 on a single A6000 GPU. Each model training run takes less than 6 minutes. We present the average metrics across all 10 movies in Tab. 4. We present results on CALVIN without context and CALVIN with entities in the prompt as discussed in Sec. 5.1. We present 2 fine-tuning scenarios trained with 20 to 50 samples from each test movie. Note that this accounts for less than 8% of total scenes with AD in all of the test movies.We see performance improvement in both the personalization scenarios over the model without context. This shows that test time personalization can be a cheap and efficient way to improve contextual captioning. We present a qualitative example for 2 continuous scenes in Fig.  
discussion and conclusion: 
 We present a state-of-the-art model for contextual scene captioning. Our model introduces several innovations, including instruction tuning and stage-wise training for various components, along with tailored data mixes at each stage. We demonstrate that our model, CALVIN, exhibits superior generalization capabilities and outperforms existing off-the-shelf video-LLMs at zero-shot evaluation on a TV-captioning dataset. We additionally propose novel test-time adaptation strategies involving prompting and few-shot tuning.There are still ample opportunities for enhancement. These include processing longer videos, intelligent frame sampling for better and faster video representations, and audio processing. The biggest challenge to these advancements is data scarcity; There is a pressing need for public video datasets of high quality.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We share the hyperparameters and ablations in the main and in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We could train models only once due to limited compute availability.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We discuss this in the experiments section.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We discuss this in the appendix.Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Tab. 6, we present a few examples of predicted caption vs ground truth caption, one from each movie in MAD-eval split. Clearly CALVIN-3S captions are as good as GT caps in many scenarios.In the main paper Fig. 4 (b), we have seen that the GT caption and CALVIN-3S captions differ quite a bit, however, the caption is an acceptable alternate when we looked at the video. We present a few cases from one of the MAD-eval movies, HOW DO YOU KNOW? in Tab. 7. This shows that the maximum achievable bert-score on this task is lower than 100 due to the subjective nature of this task. Length of context Inference-time ablations. In Figure 8 , we show how BertScore and Cider metrics change with context length, number of video frames sampled, and LLM sampling temperature. We see that increasing the number of previous scenes in the context improves performance across both the metrics; The model was trained with just 3 scenes in the context and yet generalizes to more.Regarding the number of frames sampled from each clip, we see performance improvement as we go from 1 frame to more, indicating that the model uses motion information in caption generation. However, we see no improvement in performance beyond 8 frames. LLM sampling uses beam search. We do not see much difference in BertScore at lower temperature, but we see a slight bump in CIDEr.We conducted an ablation on one movie from the MAD-train split -3041 -JUST GO WITH IT to understand the right number of samples and training iterations needed for few-shot experiments. We ablate the number of iterations in Fig. 9 and the number of annotated samples in Fig. 10 . We observe that the performance does not improve beyond 50 annotated samples in training. And we see that training for more than 100 iterations does not improve the performance. F Broader Impacts.Broader Impacts By leveraging previous scene contexts, CALVIN is well-positioned to enhance the accessibility of visual media for individuals with visual impairments, offering them a more immersive and contextually rich experience. This improvement could make entertainment and educational content more inclusive, promoting equal access to information and enjoyment regardless of visual capability. Furthermore, the technology could be applied in various other domains such as automated content moderation, where understanding the context of scenes can improve the detection of inappropriate content, and in digital humanities, where researchers can analyze films at scale to study cultural representations and evolution. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: This paper proposes a video LLM that uses previous movie context to generate contextual scene descriptions. Experiments on two datasets demonstrated the effectiveness of the approach.

Initially, the paper received four borderline accept scores. Reviewer jqA2 pointed out the lack of evaluation datasets especially the CMD dataset and no closed MLLMs as baselines. Reviewer suoE, Y2oh and GsDL all raised major concern on the novelty of the approach. Additionally, reviewer suoE questioned the evaluation metrics and reviewer Y2oh requested to align data for comparison and ablations w/o relying on groundtruth contexts.
The rebuttal provides more result comparisons, evaluations across different aspects and further analysis, which effectively addressed most of these concerns. 

Based on this, three reviewers increased their scores to weak accept and one remained borderline accept. The AC thus recommends accepting the paper.
The authors should ensure that all promised revisions are included in the final version.

Reviews: review: paper_summary: paper_summary: Note: Raised score by 1 point after reading reviews, responses and the concerns addressed in the rebuttal phase.

----
This work introduces a video LLM model that can describe movie scenes in context incorporating names of characters and generate short contextual descriptions. They train a model on data from image question answering datasets and video description using context from previous frames, these enable their model to generate better contextual descriptions of events.
They evaluate their model on Movie Audio Description (MAD) and TV-Caption datasets and show improved performance.
strength_weakness: strength_weakness: * With regard to evaluations, for movie audio description task, the CMD-train and and CMD-eval [1,2] datasets are also available. It’s not specified why the CMD dataset has not been used in this work, at least for eval. 
* It seems valuable to have a baseline with the closed MultiModal LLMs e.g. GPT-4o and Gemini 1.5 Pro all of which claim to have superior video captioning capabilities.
questions: questions: 1. Have you evaluated on the CMD-eval dataset[1,2]? do you find the performance to be different? If not, is there a specific reason for not utilizing the CMD-AD eval dataset?
2. Do you evaluate on any closed multmimodal LLMs? If you tried, did you face any issues? If not, is there a reason not to try?

[1]Bain, Max, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. "Condensed movies: Story based retrieval with contextual embeddings." In Proceedings of the Asian Conference on Computer Vision. 2020.
[2] https://www.robots.ox.ac.uk/~vgg/research/autoad/
limitations: limitations: Limitations are discussed at the end of the appendix. The reviewer’s understanding is that this woul dhave to move to the main paper. Please check the guidelines.
paper_summary: paper_summary: This paper addresses the task of contextual video captioning, with a particular application emphasis on settings for film and tv, where audio descriptions can be useful for making these mediums more broadly accessible. Standard vision-language models are often verbose (problematic since audio descriptions must fit between dialogue) and are prone to hallucinations (e.g., of actor names, or of entities not present in the scene but may be thematically related). The authors propose CALVIN, a model that incorporates pretraining on a range of vision-language tasks (image QA, video captioning, etc.), instruction tuning to improve captioning behavior, and some final adaptations (e.g., prompt engineering, few-shot adaptation, etc.). They observe improvements on the MAD-eval (for film) and TVC (for television) datasets, and qualitative results show more terse descriptions with fewer hallucinations.
strength_weakness: strength_weakness: However, there are some key areas of weakness, as follows:

`-` The paper focuses on reducing hallucinations and verbosity, but the metrics provided (across the model and model-free/n-gram based ones) do not necessarily correlate or show that this is the reason that the metrics have gone up. Similar prior work (e.g., AutoAD series) have done extensive explorations of metrics around character relevance, and stronger model-based metrics. Given that many of these metrics/analyses have been released by prior work, and the research focus of this paper, the empirical results would be significantly stronger if such analyses were performed for here (and ideally, in a consistent way to make an explicit comparison with prior work).

`-` The paper presents a large set of ideas, but the individual novelty of each element is not clear, and it would be good to have clearer comparisons with the closest prior work that introduced the relevant idea (in as much of an apples-to-apples fashion as is possible). As one example, the incorporation of IMDB metadata (for actors) as a small set of exemplars has been explored by prior work (specifically, AutoADII); an ablation based on this approach seems like it would be an important reference point for characterizing the relative improvement with the slightly different style proposed here.

`-` The overall task of audio description also cares about the localization of the captions along with the captions themselves. While prior work (e.g., in the AutoAD series) also considered this aspect, this is notably missing from this architecture. It is also a bit unclear why, given that time tokens have been explored for similar VLM models already. Relatedly, the paper would benefit from improving the clarity of the exact inputs, in terms of the temporal regions that are provided to the model for captioning and how these are selected. (There is some language around the choice of the few-shot examples, but this comment is more broadly speaking across all settings).
questions: questions: Overall, this work is borderline, with a preliminary rating of borderline+. The discussion phase will be important for this work, so if the authors could address some of the areas for clarification that are raised above in weaknesses, particularly with respect to prior work in this space, this would be helpful towards finalizing the rating. Note that for many of the points the review above notes a specific example for illustration, but the comments do extend more generally.
limitations: limitations: The authors have mentioned some limitations of their method, but not in great detail (there is a short, general statement in the final paragraph of the conclusion). This can be expanded in the supplement, for example, addressing the limitation that the model does not seem to output a clear localization of the caption.

---

**Post-rebuttal update:** The authors largely addressed the concerns raised in the rebuttal phase. After consideration of this and other reviewers' comments, I've raised my rating further.
paper_summary: paper_summary: This paper introduces a specialized video LLM named CALVIN, which leverages previous movie context to generate contextual scene descriptions. The authors achieve this by training their model on both image QA tasks and video captioning tasks within a unified framework. Experiments demonstrate that with mixed training data and context during inference, CALVIN outperforms the previous state-of-the-art in the audio description task.
strength_weakness: strength_weakness: 1.	The method appears similar to VideoLLaMA except for differences in data usage and the base LLM. The authors should provide a more thorough discussion on how their approach differs from previous methods.
2.	The experimental results in Table 1 and Table 2 are not entirely fair due to the use of different training data and pre-trained models. The data seems critical for CALVIN's performance, as shown in Table 3. Therefore, it would be better to align data usage with the compared methods or at least highlight the types of data used by other models.
3.	Is it practical to use context from captions, given that ground-truth captions do not exist? What is the performance of using generated captions as context for CALVIN?
questions: questions: See weaknesses.
limitations: limitations: The authors have discussed the limitation of their work.
paper_summary: paper_summary: The paper focuses on captioning movie scenes, which unlike typical captioning tasks, videos from movies can be captioned in a way that tells the story of the movie. For example, captions from movie scenes should convey how the characters felt, and not just the detail in the image.  To be able to caption in this way, the authors propose and train CALVIN, a 7B LVLM, on MAD (a movie-audio dataset) as well as other datasets. They then experiment with a variety of techniques to experiment with CALVIN in various settings.
strength_weakness: strength_weakness: * I am not sure I can say there’s technical novelty in this work (this is not a deal breaker, since existing knowledge was applied to create a research artifact that did not exist prior). However, I do not see a statement about releasing weights (only code), and if it will not be released, I am not sure what the contribution of the paper is.
questions: questions: What do you do with frames that introduce new information specific to the movie? For example, how could a model correctly predict the ground truth for frames that introduce names of characters?
limitations: limitations: I do not see where in the paper core limitations of CALVIN or the methodology are discussed.

Comments: title: Thank you Reviewer GsDL
Thank you Reviewer suoE!
Thank you Reviewer Y2oh
Thank you!
Satisfactorily addressed raised concerns
Thank you
comment: Thank you to the authors for the additional details and reply! Want to acknowledge that I have read this and will be incorporating into the final rating update after the next discussion phase.
We thank the reviewer for your thoughtful feedback. We note your concerns and we will update the abstract and introduction to highlight a bit more on architectural and training innovations (LLM continued training and Fewshot fine-tuning) we introduce in the paper. 

Please let us know if you have any additional questions we can address.
We are glad to hear the reviewer is satisfied with our rebuttal. We address your queries below-

> when the model does not mention the correct character, what are examples of the incorrect outputs that the model does provide instead? (also to confirm, are the pronoun substitutions aligned to the character identity?). 

We used Spacy for extracting Proper Nouns from each ground-truth AD to conduct character analysis. From the qualitative analysis of mistakes, we noticed that many a time, the character name is replaced with pronouns. We also observed a small percentage of instances where the model incorrectly refers to characters, for example, calling `Lisa` as `Susan` even though there is no character named `Susan` exists in the movie. This is perhaps a bias inherited from the training data or LLM itself. 

To quantify the correctness of predicted pronouns, we did not find any off-the-shelf tools to associate character names with pronouns. From qualitative analysis, the outputs seem reasonable most of the time. We will add this analysis, all the experiments done for rebuttal and brief discussion regarding the open problems in evaluations to the appendix of the paper. 

>  If the authors have additional examples that qualitative show the different scores that the model-based method outputs and can include that here/in their final supplement, that would be much appreciated.

To give you a sense of scores, we provide some scenes from the analysis, CALVIN's predictions and Gemini's score in the table below. We see in current form VLM judge penalizes when a name is missing or the description diverges too far from the original description. We believe an in-depth analysis is needed to perfect the prompting of VLM-judge by looking at many diverse scenarios (which is a study on its own and too out of scope for this paper). We will add these examples and a discussion to the appendix of the camera-ready version of the paper.

Ground Truth | Calvin prediction | VLM Judge score (lower better)
--- | ----| ----| 
Swinging around, Lenihan aims his gun at the sky but sees nothing. | The soldier points the gun and looks at an apartment | 5
Behind him, an alien emerges from the pool. | A robot is in the pool | 3
Lenihan wheels around. | Lenihan turns | 1

 Please let us know if you have any additional questions we can address.
We thank the reviewer for raising the score. We agree that the new experiment has further enhanced the analysis. We will include all the new results in the appendix of the camera-ready version.
We are glad to hear that you are satisfied with our response. We will include all the additional studies conducted for the rebuttal in the appendix of the paper.

Additionally, we will add a discussion in the conclusion section addressing the limitations of the current MAD dataset and the need for a new 'raw video' dataset.
Thank you to the authors for their detailed response.

Regarding the first concern, although the authors listed some differences, I still find the implementation of CALVIN not novel enough. However, I appreciate the efforts made in optimizing the method through data and structure tuning, which indeed contribute to better performance in practice.

For the second question, I understand the challenges in making a fair comparison. However, including at least one such ablation study would strengthen the paper.

Lastly, I am pleased to see the new results in this setting, as they will serve as a valuable baseline for future studies.

Overall, I am inclined to raise my score to 6.
Thanks for the response and additional numbers! I also looked at the concerns raised by other reviewers, thanks for thoughtful responses to those reviews as well!

It's unfortunate that there aren't other datasets to properly evaluate CALVIN against Multimodal LLMs. I definitely appreciate the  quick experiment with 2 movies that you were able to get hold of, and it is good to be able to put CALVIN's performance in perspective with these much larger and expensive off-the-shelf API models.

Please include these discussions in the main paper or the supplement along with some notes on future direction of how one can evaluate models on this task. I will increase my score based on the response
Thank you for the response.

**Main Contribution of the paper** Thank you for clarifying. I do maintain my belief however, that the implementation of CALVIN is not novel. Careful data cleaning (and data mixtures) is not a specific issue for video and there are countless papers that study its importance in adjacent fields, such as LLM pretraining. To the best of my understanding, the core contribution of this paper is not to study the importance of quality data in training, nor does it contribute tools to clean data specific to video, so I do not see how this furthers the field of video understanding on the axis of data cleaning. Similarly, if the main point of the paper is to show a single Q-former is enough to replace a complex pipeline, then I do not believe the text reflects that. For example, the abstract does not indicate any type of novelty on the training method, but it does emphasize the utility of CALVIN.

Regarding the few-shot adaptations, I agree they are useful, but this is not data cleaning or related to the pretraining of the model (including the dreambooth-like setup).

**Weights Release** I am glad to hear that, I do believe it has a non-trivial impact on the value of this work.

**How to predict new information specific to the movie?** Thanks, this does answer my question.

For the reasons above, I choose to maintain my score.
Thank you to the authors for the detailed response to my review! I will be updating and finalizing my rating after all discussion periods are complete, but this response is good overall and I believe strengthens the quality of the work. The temporal localization and additional discussions are much appreciated. Below, a quick response to one subset of the rebuttal above:

*Re: analysis, verbosity, and hallucinations*: The analyses provided by the author are helpful to better contextualize the method. 
- I recognize that there are not good hallucination metrics for open-ended video captions, but the character analysis is one that (at least in terms of the subject of the caption) does help to address this, especially given the movie domain. For general context, when the model does not mention the correct character, what are examples of the incorrect outputs that the model does provide instead? (also to confirm, are the pronoun substitutions aligned to the character identity?). 
- It's unclear how well-correlated the hallucination scores (with the VLM "judge") are with human judgements (especially since VLMs can hallucinate themselves), but this analysis can still be useful. If the authors have additional examples that qualitative show the different scores that the model-based method outputs and can include that here/in their final supplement, that would be much appreciated.