NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative
Authors: Asmar Nadeem
Faegheh Sardari
Robert Dawes
Syed Sameed Husain
Adrian Hilton
Armin Mustafa
Venue: ICLR 2025 Poster (2025)
Keywords: Video Captioning
Video Narrative
Video Storytelling
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: The new video captioning benchmark and method, CTN captions and CEN network, improve understanding of video narratives by embedding causal-temporal narrative elements in new captions and then, training the network on these captions respectively.

Abstract: Existing video captioning benchmarks and models lack causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models’ ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, we propose NarrativeBridge, an approach comprising of: (1) a novel Causal-Temporal Narrative (CTN) captions benchmark generated using a large language model and few-shot prompting, explicitly encoding cause-effect temporal relationships in video descriptions; and (2) a Cause-Effect Network (CEN) with separate encoders for capturing cause and effect dynamics, enabling effective learning and generation of captions with causal-temporal narrative. Extensive experiments demonstrate that CEN significantly outperforms state-of-the-art models in articulating the causal and temporal aspects of video content: 17.88 and 17.44 CIDEr on the MSVD-CTN and MSRVTT-CTN datasets, respectively. Cross-dataset evaluations further showcase CEN’s strong generalization capabilities. The proposed framework understands and generates nuanced text descriptions with intricate causal-temporal narrative structures present in videos, addressing a critical limitation in video captioning. For project details, visit https://narrativebridge.github.io/.

Published as a conference paper at ICLR 2025
NARRATIVEBRIDGE: ENHANCING VIDEO CAPTIONING WITH CAUSAL-TEMPONE NARRATIVE
Asmar Nadeem-, Faegheh Sardari', Robert Dawes², Syed Sameed Husain', Adrian Hilton', Armin Mustafa'
CVSSP, University of Surrey, Guildford, UK
BBC Research and Development, UK
( (asmar. nadeem, a armin inn mustafa @eurrey. ac..u..ccu
ABSTRACT
Existing video captioning benchmarks and models lack causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models' ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, we propose NarrativeBridge, an Narrative (CTN) captions
approach comprising of: (1) novel Kausal-Tempone benchmark dataset generated using large language model and few-shot prompting, explicitly encoding cause-effect temporal relationships in video descriptions; and (2) a Cause-F.ffect Network (CEN) with separate encoders for capturing cause and effect dynamics, enabling effective learning and generation of captions with causal- temporal narrative. Extensive experiments demonstrate that CEN significantly outperforms state-of-ther offthoo--he-rrt models in articulating the causal and temporal aspects of video content: 17.88 and 17.44 CIDEr on the MSVD-CTN and MSRVTT-CTV datasets, respectively. Cross-dataset evaluations further showcase CEN's strong generalization capabilities. The proposed framework understands and generates nuanced text descriptions with intricate causal-temporal narrative structures present in videos, addressing a critical limitation in video captioning. For project details, visit basittttt////arrrtivvoddd githbb....
INTRODUCTION
Video captioning aims generate textual descriptions that capture the visual information and temporal dynamics in videos Venugopalan et al. (015). Research has primarily focused on developing new methods to improve the accuracy of video captioning models on well-established benchmark datasets,
MSR-VTT Xu et al. (2016) and MSVD Chen & Dolan (2011). State-of-the-art (SOTA) approaches
lashin & Rahtu (2020) Tian et al. (2019); Xu al. (017): Nadeem et al.] (202); ); Wang al.
(202)): Chen et al. (2023) proposed new architectures to better align the generated captions with
the provided ground truth. However, these efforts enhance models' accuracy on existing evaluation
criteria, without modifying the benchmark datasets or their ground truth annotations to address the lack of coherent representations of causal-temporal narrative. -ausal-temporal narrative the construction and interpretation of sequence of events linked through cause and effect, unfolding over time and space, and often driven by entities (characters or agents) acting with intention Wilkens al. (2003).
As discussed by Wilkens et al. (2003), causal relationships Granger 1969) are scenarios where one event (cause) directly influences the occurrence of another event (effect), while temporal under- standing Berrevoets et al. (2023) involves recognizing the chronological order of events, a crucial component in establishing causality. Existing ground-trutt captions in popular benchmarks lack causal, temporal and narrative information. Figure Iillustrates the importance of causal- causal-temporal narrative on the MSR- VTT dataset Xu al. (0016). In the input video sequence, a car first shown driving recklessly through an open field and flipping over, which represents the cause event. This then followed by the effect event showing the severely damaged car, and subsequently, a group of guys starting to play beer pong, which represents the temporal sequence. The causal-temporal narrative would connect these events in a temporal order, linking the reckless driving and resulting crash to the group's 's subsequent behavior of playing beer pong. This example aligns with the definition of narrative provided by Wilkens et al. Wilkenss et al. (2003). which emphasizes the role of cause-effect relationships, the perception of narrativity in videos, and highlights the temporal

---

Published as a conference paper at ICLR 2025
Input Video:
Original Captions
'a car crashes and play boor
being fliooed car driving through open sold Kilking diff. Nipping over" 1 dirt vehicle riding and rolling, dune bugay flipiigg over',
car gel wracked. wheeler wrecking monster truck Rips its side then several young men shout while playing beer pongf. lä person drivas an affraad car around a fiaks parson flpping a go kait white a crowd choars' la race truck crashing!, in truck roils over itseff and boys choer on a friend la truck tumbles over Itself, aa tumbler crashos on dirl road and then
of guys play beer pong" vehicle Tipe over, vehicle crashingl, 'crashinn
monster truck crashes and driving,, Nootage from
men are shown celebrating. 'an monster truck style event followed by a party!
LLM
Cause: 's car drove recklessly through an open field flipping over' Effect: 'the car was severely damaged and a group of guys started playing beer pong'
Caption
Figure 1:
Comparison of Original captions vs. Causal Temporlll Narrative (CTN) caption to illustrate the inclusion of causal-temporal narrative.
sequence of events that is crucial for understanding the causal-temporal narrative. Figure I highhlights the limitations of the original captions in existing benchmark datasets, such as MSR- V VTT Xu et al. (2016). The original captions focus on isolated events or actions, such as "a car flipping over" or monster truck flips on its side," lacking contextual narrative and causal-temporal relationships between events. Consequently, models trained on these captions suffer from the same limitations. To bridge this gap, we introduce NarrativeBridge, novel framework encompassing a new benchmark dataset and architecture tailored for causal -tumpolll narrative learning in video captioning. Our Temporall Narrative (CTN), novel captions benchmark dataset, leverages a large language model (LLM) and few-shot prompting to generate enhanced video descriptions that explicitly encode causal and temporal sequences, as shown in Figure [ This establishes clear connection between the cause (reckless driving) and the effect (damaged car and subsequent behavior of the group). Our CTN captions benchmark dataset enables models better understand and articulate the causality, sequence, and significance of events within the broader video context Wilkens et al.) (200)). To ensure the quality and relevance of the generated captions, we employ an automatic evaluation framework that compares the CTN captions with the video content, keeping or discarding them based on score threshold. Additionally, we conduct a human evaluation study that further validates the high quality of our CTN captions, demonstrating their accuracy, temporal coherence, and relevance. This CTN captions benchmark dataset addresses the limitations of existing benchmark datasets and emphasizes the importance of incorporating causal-temporal narrative understanding into video captioning models to generate accurate, informative, and contextually relevant descriptions.
Existing SOTA video captioning methods that different architectures such LSTM Nadeem eet al. 2023), GNN Hendria et al. (2023), and Transformer Wang et al.] (2022), struggle to effectively learn ceusal-temporal narrative from the CTN captions. These architectures are designed to capture the overall semantics in videos but lack dedicated mechanisms to explicitly model the cause-effect relationships and temporal sequences. As result, the captions generated by these methods fail to articulate the complex causal-temporal dynamics in the videos. as demonstrated in results (Section
To address this challenge, we propose the Cause-Fffees Network (CEN) that captures cause and effect dynamics using dedicated encoders. By separately encoding the entire video through the cause and effect encoders, without requiring explicit cause-effect segmentation, and then combining them to generate the final caption, CEN is able to better understand the causal-temporal narrative. The primary contributions our work are:
Introducing for the first time CTN captions benchmark dataset with -enaal-emppoal narrative captions, automatically evaluated and humanly validated. A novel CEN network for learning causal- causal-temporal narrative information from the videos explicitly.
Extensive experiments demonstrating CEN's superior performance on CTN, setting a new SOTA in causal-temporal narrative learning.
In addition, our work demonstrates strong generalization capabilities through cross-dataset evaluations and outperforms even fine-tuned SOTA vision- models such as VideoL Lin et al. (2023) and ShareGPT4Video Chen et al. (2024) in generating causal-temporal narratives. Our work marks a significant step forward in video captioning research by explicitly addressing the challenges of causal- temporal narrative understanding. The CTN captions benchmark dataset and the CEN architecture provide comprehensive framework for learning and generating narrative-based video descriptions.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 BENCIMARKS
MSVD Chen & Dolan 2011 is a benchmark focused on human activities, providing a platform for evaluating video captioning models. The captions in MSVD often describe the observable actions without delving into the underlying motivations or the cause- relationships between the events. MSR-VTT Xu et al. (016) large-scale benchmark with diverse video content, encompassing a wide range of topics and genres. The captions often focus on describing the observable content without capturing the causal links between the events or the temporal progression of the narrative. As a result, models trained on MSVD and MSR-VTT may struggle to generate descriptions that accurately reflect the causal and temporal dynamics in the videos.
While recent benchmarks like NEXT-QA Xiao et al. (2021) and EgoSchema Mangallm et al. (2024) have made strides in incorporating causal and temporal reasoning in video understanding, they focus primarily question-answering tasks rather than generating comprehensive causal-temporal narratives. NEXT-QA introduces multi-choice and open-ended question-answert specific ireestionanswwerr pairs that often target single events actions. In contrast, our CTN captions provide a more comprehensive narrative that captures the causal and temporal relationships across the entire video sequence (see Appendix A.7 a detailed comparison). EgoSchema Mangallm et al.) (2024), on the other hand, emphasizes long-form video understanding and temporal reasoning but does not explicitly focus on causal-temporal narrative.
tasks focusing on
Similarly, efforts like VCR Zellers et al. (2019). V2C et al. 2020), and Motivation Vondrick al. (2016) integrate causality into their analysis of visual description or question- aattiionnnn relying heavily on commonsense reasoning for generating predictions. VCR Zellers et al. (2199) focuses on visual commonsense reasoning. V2C[Fang et al. (2020) aims to generate commonsense descriptions for video captioning, and Motivation Vondrick al. (2016) explores the prediction of motivations behind actions in videos. However, these works primarily rely on commonsense reasoning and do not delve into the causal and temporal structures underpinning video narratives. Our CTN goes beyond existing benchmarks by explicitly modeling causal-temporal narrative in a single, coherent caption, enabling a more comprehensive understanding of video content.
2.2
VIDEO CAPTIONING
Video captioning techniques have evolved from LSTM-based Gao et al. (0117) Song et al. (017));
Nadeem et al. 2023) frameworks to the latest designs using SOTA GNNs Hendria et al. (2023) ; Zhang
(2020) Pan al. (020) and Transformers WWan et al. )022); : Lin et al. (002 Yang al. et al.
(2023) with a focus on enhancing the complexity of captions through the injection of multimodal data. Despite these advancements, current architectures often struggle capture the intricate temporal sequences and causal relationships in video storytelling. To bridge this gap, video captioning can benefit from crots-ffetilizzaton with ideas and strategies developed in related fields, such as action
recognition Sun et al. (2022); Waag et al. 2 Kazakos et al. (2019): Xiao et al. (2020); Chen &
Ho (002)): Gao et al.
(2020g Panda et al. (202T); Sardari et al. (2023): Alfasly et al. (2022), ); Awan
et al.1
[2224]
Planamente et al. (202 ); Zhang et al. (2022), event localization Tian et al. (2018) Lin
jet al.
)
Duan et al.2 Ty: Lin et al. (2021). ) and question-answeereeee
Alamri et al. (20T9): Hori
ctt al.
b; Yun et al. (202 1); Li et al. (2022a); Shah 2022); Nadeem et al. (2024). The integration of causal reasoning Liu et al. (2022); Xue et al.
[2019;
Schwartz al. (2019); Geng et al. (2021);
cct al.
(2023) has shown promise in enhancing the ability of neural networks to discern causal relationships,
leading to improved performance in image captioning Liu et al.) (2022) and visual question answering
Xue et al. (2023). However, current SOTA models still struggle to effectively handle the narrative
complexity in videos.
Recent advancements in Dision-language models (VLMs) such as VideoLLaVA Lin et al. (2023) and
ShareGPT4Video Chen et al. (2024) have shown promising results in various video understanding tasks. However, as our experiments show (see Section 4.3). even these advanced models struggle with generating accurate causal- temporal narratives. In light of these challenges, our work explicitly addresses the limitations of the current approaches and provides a platform for causal-temporal narrative learning by introducing NarrativeBridge, comprehensive framework that encompasses the CTN benchmark dataset and the CEN architecture.
3 METHOD
NarrativeBridge addresses the limitations of existing benchmarks and models in capturing causal- temporal narrative. It consists of two key components: (i) the CTN captions benchmark, which
3

---

Published as a conference paper at ICLR 2025
Input Video:
Original Captions
LLM (CTN Caption
Caption Evaluation
Score
Final CTN
Prompt
Caption
Generation)
Score
Figure 2: CTN caption generation pipeline. indicates a threshold.
provides a rich representation of cause-and-effect relationships and event sequences in video content, and (ii) the CEN architecture, designed learn and articulate these causal-temporal narrative elements.
Problem Statement: Existing video captioning benchmarks lack the intricate causal-temporal narrative inherent in video content. SOTA video captioning methods also fall short in articulating the cause-and-effect relationships and temporal sequences that drive the events in the video (see Section
3.1 CTN CAPTIONS BENCHMARK DATASET
To address the challenges of existing benchmarks in representing causal and temporal relationships within video content, we propose an approach that harnesses the potential of LLMs through the few-shot prompting technique Brown et al. (2020). Our method generates CTN captions without the need for model fine-tuning, leveraging the inherent generative capabilities LLMs to produce captions that encapsulate causal-temporal narrative structures. Figure 2 shows the CTN caption generation pipeline, which consists two key steps: prompt design, LLM-based caption generation and evaluation.
with generating causal-tanporal narrative captions video. However, you cannot directly access the video itself. Instead, you will be provided with series of captions that outline the events and scenes in the video. Your task is to generate concise Cause and Effect scenario, based the information provided should fulfill following reguizenanta:
language
deseriptive captions.
and Effect Your narrative should be groundad the information provided by descriptive captions.
careful, your generated
2. Cause and Eéfect acenaric is relevant.
3. It should not introduce any new events or details not mentioned.
4. Avoid implying conclusions.
provided captions. 6. Use plain English and direct sentences.
temporal consistency
7. Cause and Effect statements each limited to maximum of 15 words.
0. not include additional before after the JSON object.
exapples
[Sxanples]:
[[' ('Cuuse':: : 'the atudent overslapt due to malfunctioning alarn clack', 'Bffect': 'nissed catching the bas to school'), :'Cause': 'see absentaindedly skipped applying noisturizer after taking long hot shower', 'Etfect': 'ber skin became and flaky'), ('Cause': 'be carelessly neglected prescribed allergy medication', soccer player recklessly fouled an opponent in the penalty area', 'Bffect': 'the opposing tean was avarded crucial panalty kick'), ), ('Cause': 'due to unforeseen road closures they found thenselves stuck in heavy traffic'', 'Réfact': 'missed out expariencing the opening act: of the concart' please generate only descriptive captions.
his severe sneering fit'), ('Cause': :
presented
following
[Descriptive Captions]:
cnberipttie_ccatiion [Causal Temporal Narrative]:
Prompt 1: LLM Prompt used in CT'N captions generation.
The prompt design step is crucial in guiding the LLM to generate causal-temporal narrative captions. The input to the CTN caption generation are the original video captions from existing benchmarks. We design prompt that include small set of carefully selected example captions, illustrating the desired output structure and highlighting the key aspects of causal-temporal narrative. We design Prompt which guides L.M to generate causal-temporal narrative. The prompt
Prompt Design:
4

---

Published as a conference paper at ICLR 2025
Stage 1
Fleeheo the nut mis sinaril ramppe
Stage
Diilarty
Stage
Skellarity
de
Pooiing
Tresdw
freede:
Paeking
Heir
lediiny
Cauase Encader
Eifech Ereeder Teleo
(Tanatarmer)
(Thealermet
Crass Video Eneadar Scnae
TTainabla
Siour Villo Secadar Eatos (Prezsined cuPNm))
Pretrneed culvm
Ingel Kerred
Imper car
Sald Elppiin avea'
group gags sticted plcying beir pain
Figure The two-stage Cause-Effect Network (CEN) architecture. Stage 1: Separate Cause (Eeause) and Effect feet)) video encoders, pretrained using CLIP-ViT, learn specialized video representations. Corresponding text encoders (Teause and Tef fect) encode the cause and effect portions of the CTN caption. Contrastive losses are applied to align the video and text embeddings. Stage 2: The learned cause and effect video features are encoded separately (Enceaase and fnce.foott and concatenated before being input to the decoder, which generates the final CTN caption.
sets clear requirements for grounding the captions in the provided descriptive context maintains temporal consistency, and avoids unsupported details. The illustrative examples demonstrate the desired format, facilitating the generation of plainly written, length-come constrained Cause and Effect statements that capture the video's 's causal- temporal narrative. By explicitly outlining instructions and constraints, the carefully designed prompt steers the LLM's generative capabilities towards producing CTN captions. Further details are provided in Appendix and
CTN Caption Generation and Automatic Evaluation: We send Prompt into the Mixtral of Experts LLM Diang et al. (2024). which generates CTN captions. The LLM's advanced natural language understanding and generation capabilities, combined with the few-shot prompting approach, enable the production of captions that encapsulate complex causal and temporal relationships. To ensure the quality and relevance of the generated CTN captions, we employ the EMScore [hi et al. (2022) metric for evaluation. EMScore directly measures the consistency between caption and the video content, and has been shown to be more effective than other metrics de Souza Inácio & Lopes) (2023) in evaluating video relevance without referenced captions. We set a threshold (0 = 0.2) for the EMScore value, indicating adequate relevance to the video. Captions with an EMScore above the 0 are kept, while those below the 0 are discarded, and the LLM generates a new caption. This iterative refinement process continues until caption meets the EMScore threshold, ensuring that the final captions efficiently describe the relevant events in the vidco. Further details are provided in Appendix A.4
Human Evaluation: To further validate the quality of our CTN captions, we conduct a human evaluation study using standard practice Chen & Dolan (201 Xu et al. (2016). We randomly sample 100 videos out of 11,970 videos (10,000 MSRVTT-CTN and 1,970 MSVD-CTN) using proportional stratified random sampling, which yields a margin of error of 8.2% at 90% confidence level. Five independent domain experts evaluate the videos and the CTN captions on three criteria: causal accuracy the degree to which the caption correctly identifies and describes cause-effect relationships; temporal coherence the extent to which the caption accurately represents the sequence of events; and relevance how well the caption reflects the overall content. Each rater does 300 evaluations and each criterion is rated on a 5-point Likert scale. Further details are provided in Appendix A.5
3.2 PROPOSED CAUSE-EFFECE NETWORK (CEN)
Existing SOTA video captioning methods based on LSTM&/Nddem al. (2023). GNNcHendria
et al. 2023), and Transformers Wang et al. (2022) lack dedicated mechanisms to explicitly model the intricate causal-temporal narrative sequences. As shown in Figure 3 our proposed CEN employs a two-stage approach to address this limitation (see Section 3 In Stage (Section B.2.Ip. CEN learns specialized cause and effect video representations using separate encoders trained on the corresponding portions of the CTN captions. The input is raw video frames from the entire video without requiring explicit segmentation of cause-effect regions, and the output is the learned cause and effect features extracted from the encoders. The learned video features from Stage I are then passed to Stage 2 (Section 3.2.2). which utilizes a sequence-to-sequenee to-sequence transformer based network with two new encoders to generate the final captions. The parameters of stage I encoders are frozen at this stage. The cause and effect video features are separately encoded and then concatenated before being input to the decoder. This allows the model to synthesize the specialized representations
5

---

Published as a conference paper at ICLR 2025
learned in Stage and generate captions that accurately capture the causal and temporal relationships in the videos.
3.2.1 STAGE 1: CAUSAL VISUAL ENCODING
For the CEN network to learn causal-temporal narrative, we design two separate visual encoding models trained on two parts of the captions (see Fi.ure
(1) Cause Video Encoder Feause: Trained on the cause captions part describing the initiating events;
(2) Effect Video Encoder feffect: Trained on the effect part of the captions capturing the conse- quential outcomes.
We instantiate Eeause and feff as two separate instances of the CLIP-ViT ViT (ViT-B/32) model Radford et al. (202I), adapted from CLIP4Clip Luo al. (2022) (see Appendix (r.rendix(A66joe for more details). For both Cause and Effect text encoders (T (Foause and T feet respectively), ), we employ a 12-layer transformer with hidden size of 512 and 8 attention heads, using the weights from the pre-trained CLIP Radford et al. (002)) text encoder. As in CLIP Radford et al. (202I), the [EOS] token's s representation from the last layer of the text encoders is used as the feature representation for the input text. Similar to CLIP4Clip, we use mean pooling for the video embedding and then, adopt cosine similarity to measure the similarity between the video embedding (mean pooled) and the text embedding. Given a video embedding f; and a text embedding fj. the similarity function is defined
e A.
as s(fr, (2) E.Il Radford et al. (222I) over its respective cause/effect portion from CTN captions. For a batch of N video-cause text pairs (ui, c.) and video-effect text pairs (v), e.). the contrastive losses for the cause (Leause) and effect (Leffect) video encoders are defined [ 2nd aespereeecti
During training, each model is optimized using contrastive loss, as in CLIP
respectively:
( Leause = Lozz( (Eenune(y:) (emuee)) + Lizo (Beass (e), (e)n)
(I)
Luffact = (oo((EE)foa (vi), Tef Lez- (E.ffea((a)) (ef
(2)
where Lu and Li2v are the video-to-text and text-to-video contrastive losses, respectively, defined as:
Cuzi N
log
exp(s(fi,A=)
(3)
Liz.
Σ log
N
exp(s(ft,
(4)
Ee= exp(s(fr. 4))
Σ exp(s(f;,4))
F=1
After training, we freeze the weights of Ecouse and Eeffect for the second stage, where we extract the cause and effect video features, respectively.
3.2.2 STAGE 2: CTN CAPTION GENERATION
In the second stage (see Figure B) we employ a sequence-to-sequeenee transformer-base network, consisting of two separate encoders (EnCeause and fnceffeet) for processing cause and effect video features, respectively, and a decoder Dec for generating the final CTN captions. The encoded cause and effect video representations are concatenated before being sent to the decoder. Both encoders and the decoder are initialized with the weights from the pre- pre-trained Uni-VI model et al. (2020). Given video v, we first extract the cause and effect video features using the pre-trained Eeause and feffect encoders:
(6) The extracted features Feause and Feffect separately encoded using EnCeause and Ence fact- each consisting of two transformer layers, to obtain the respective encoded representations: heause Enceause (Feause) and heffect Enceffeet (eeffect.
Feause (v),se(v),
(5)
Fefteat (effec(()).
The encoded cause and effect representations are concatenated to form a single representation hementt = [heasse; f feat], which is then input to the decoder Dec, consisting of two transformer layers, to generate the CTN caption ý by attending to heanent at each time step &::
(7) where Uंج denotes the previously generated words. For training and evaluation, we combine the cause and effect parts of the ground tround-truth CTN caption with a space in between form single, coherent caption. We opt for space separation to maintain consistency with the raw input format of the training datasets (MSRVTT and MSVD), which contain no separators and also, for a fair comparison with the SOTA. This combined caption serves as the target for the decoder during training and the reference for evaluating the generated captions using diffferent evaluation metrics. The entire network is trained end-to-end using the cross-entropy loss between the generated caption ý and the ground-truth CTN caption y::
f. Dec(heeneas) (c+);
T
M
Leaption
He,e log(()..))
(8)
T

---

Published as a conference paper at ICLR 2025
where: T is the length of the caption. [V] is the size of the vocabulary, yt.e is the ground-truth label for the o-th word at time step (1 if the word is present, otherwise). Dt,cc is the predicted probability of the e-th word at time step Finally, our total loss Liotal is:
(9) This two-stage approach first encodes the causal aspects into the video representations, facilitating the generation of coherent causal-eempoaa narrative descriptions in the second stage (see Experiments section to generate CTN captions as final output.
Ltotal Leause + feet + Leuption
EXPERIMENTS
We conduct extensive experiments to validate our proposed CEN architecture on the new CTN video captioning benchmark dataset. We present comprehensive evaluation against the state-of-the-an video captioning methods and vision-language models (VLMs) demonstrating the superior performance of the proposed network on the CTN benchmark dataset.
4.1 EVALUATION METRICS
For quantitative comparison, we use three metrics as per Celikyilmaz et al. (2020): CIDEr (C) Vedantam et al. (2013); [Shi al. (2022): Measures aligmment of generated captions with references by highlighting frequently occurring terms, capturing the ability to reproduce salient causal-temporal narrative elements. ROUGE-L (R-L) Lin (2004): Evaluates the longest common subsequence between generated and reference captions, considering both precision and recall for assessing semantic similarity, including temporal dynamics and causal relationships. SPICE (S) (ndersso et al. (2016); Evaluates semantic quality by considering the overlap of scene graphs between generated and reference captions, effectively assessing the ability to capture causal relationships and event sequences.
4.2 IMPLEMENTATION DETAILS
We generate the CTN captions using the Mixtral of Experts LLM Jiang et al. 2024), running A100-80GB GPUs For the SOTA models, we follow the hyperparameter settings specified their respective methods for training the MSR-VT VTT and MSVD datasets. Our CEN model is trained using the Adam optimizer with learning rates of X 10 (stage 1) and x 10-6 (stage 2) and batch size of 64 for 10 epochs (stage 1) and 50 epochs (stage 2). For comparison with recent Vision- Language Models (VLMs), we fine-tune VideoLLa Lin et al. (2023) and ShareGPT4Vider Chen al.) (2024) using both LoRA and simple tinettuning approaches on our CTN benchmark dataset. We use the recommended hyperparameters for each model during fine-tuning. Further details are provided in the Appendix A.
4.3 RESULTS AND DISCUSSION
We comprehensively evaluate the performance of our proposed CTN generation pipeline and also, the CEN architecture against several SOTA methods, including SEM-POS Nadeem et al. (2023). AKGNN Hendria et al. (2023), and GIT Wang al. (2022) and the VLMs Videol Lin et al. and ShareGPT44 Video Chen et al. 2024) on our generated CTN captions benchmark datasets.
(2023)
4.3.1
QUANTITATIVE RESULTS
CTN Captions Benchmark Dataset: We generate our CTN captions (1 caption per video) using two widely-used video captioning datasets: I) MSRVTT Xu et al. (2016), consists of 10,000 videos with 20 human-annotated captions per video, and MSVD Chen & Dolan (011). with 1,970 videos focused on human activities with approx. 50 captions per video. The length of the caption is on avg. 19 words (cause=10 words, effect=9 words) for MSRVTT-CTN and 17 words (cause=8 words, effect=7 words) for MSVD-CTN. Then, we train as well as test our CEN, SOTA methods and VLMs on the MSVRTT-CTN and MSVD-CTN.
Our CTN generation approach generates good quality captions with 52. 1% CTN captions exceeding 0.27 EMScore (threshold=0.2), Then, in the human evaluation study, the overall mean quality score of 4.8 (a 0.40) on a 5-inont Likert scale, indicates high performance across all criteria; 93% of captions receive scores of 4 or higher across all three dimensions, with 84% achieving perfect scores. To ensure reliability across raters, we calculate the intraclass correlation coefficient (ICC). The ICC for absolute agreement among raters is 0.87 (95% CI: 0.83-0.91), indicating high inter-rater reliability. These results validate our automatic generation and evaluation process, demonstrating that our CTN captions benchmark dataset provides high-quality, coherent representations of causal-tone -esaaarltmprall narratives in videos. Further details are provided in Appendix A.5
7

---

Published as a conference paper at ICLR 2025
Table Comparison of our CEN architecture against SOTA methods and VLMs on the MSVD-CTN and MSRVTT-CTN datasets. The best results in each category are in bold. R-L,, C, and S denote ROUGE-L, CIDEr, and SPICE scores, respectively.
MSVD-CTN
MSRVTT-CTN
Method
R-L (1)
C(T)
(T)
R-I(TT
c(t)
ST)
SEM-POS
23.39
37.16
14.46
20.11
26.01
12.09
AKGNN
25.11
35.08
14.55
21.42
25.90
11.99
GIT
27.51
45.63
15.58
24.51
32.43
13.70
VideoI.LaVA (Zero-shot)
21.80
30.55
14.67
19.33
16.24
12.49
VideoLLaVA (LoRA FT)
24.56
34.98
15.41
21.21
18.97
13.28
VideoLLaVA (Simple FT)
25.61
36.12
16.09
22.18
19.98
13.07
ShareGPT4Video (Zero-shol)
21.66
27.06
14.06
20.27
17.08
12.21
ShareGPT4Video (LoRA FT)
24.39
30.72
14.83
22.09
19.83
13.02
ShareGPT4Video (Simple FT)
25.32
31,67
14.92
23.01
20.76
13.28
CEN (Ours)
31.46
63.51
19.25
27.90
49.87
15.76
CTN vs Original captions MSRVTT
CTN captions Cause-Effect Mapping MSRVTT
CTN
Cause
Original
Effect
Figure 4: (a) UMAP visualization video features learned from CTN (red) and original (blue) captions MSR-VTT. showing non-overlapping feature spaces, (b) UMAP visualization of video features learned from cause (black) and effect (orange) parts of CTN captions on MSRVTT-CTN, showing complee overlap.
Cause-Effect Network (CEN): CEN outperformed all SOTA methods and the VLMS across all metrics and datasets shown in Table [ On MSVD-CTN, CEN surpassed the next best (GIT) by 3.95 ROUGE-L, 17.88 CIDEr, and 3.67 SPICE points. On MSRVTT-CTN, CEN led GIT by 3.39 ROUGE-L, 17.44 CIDEr, and 2.06 SPICE points. These significant gains highlight CEN's effectiveness in capturing causal narratives and temporal dynamics.
We compare CEN's performance against zero-shot and fine-tuned (FT) versions of VideoLLa and ShareGPT4Video, CEN consistently outperforms these models, even after fine-tuning, demonstrating the effectiveness of our specialized architecture for causal-tempon narrative generation. For example, on MSRVTT-CTN, CEN achieves CIDEr score of 49.87, compared to 19.98 for VideoL (Simple FT) and 20.76 for ShareGPT4Video (Simple FT).
To gain further insights into the effectiveness of the CTN captions in capturing causal-temporal narratives, we visualize the video feature representations learned by CLIP-VIT encoders trained on CTN and Original captions using UMAP McInnes et al. (2018) dimensionality reduction from high dimension onto 2D plane. In Figure 4 a), we compare the representations learned from CTN captions (red) and Original captions (blue) on the MSR-VTT dataset. The non-overlapping feature spaces indicate that CTN captions capture the causal and temporal relationships not present in the original captions. In Figure 4b), we visualize the representations learned from the cause (black) and effect (orange) parts of the CTN captions. The near-complete overlap suggests a strong correlation between the cause and effect components, aligning with the inherent structure of causal-tena -emmpoaa narratives and supporting the design of the CEN architecture.
8

---

Published as a conference paper at ICLR 2025
Table 2: Ablation study results on the MSVD-CTI and MSRVTT-CTE datasets. Econtssed, w/o FT CLIPs. Only Eauee and Only Egflcc are baselines of our CEN architecture, while Zero Shot X and Fine-tune X repre- sent cross-dataset evaluation settings. The best results in each category are bold.
EMScore Distributiion w/o CTN Automatic Evaluation
Disinsution B-02
MSVD-CTN
MSRVTT-CTN
Method
R-L
C
(1] T
Epombined
30.93
55.72
17.04
27.34
45.97
15.07
w/o FT Fiiinngle CLIP
27.81
46.23
15.82
25.34
32.31
14.27
0.0
0.1
0.2
0.3
0.4
FT Two CLIIP
28.40
53.84
16.58
26.10
40.92
14.55
EMScore
Only Eeuose
30.72
56.42
18.43
27.19
47.10
15.21
Only Eeffect
30.70
57.14
17.89
27.24
45.58
15.19
Figure 5: EMScore distribution of initial CTN
Zero Shot X
27.16
39.65
14.45
24.73
29.76
12.26
captions showing 66.4% exceed quality thresh-
Fine-tune X
31.78
63.60
19.39
27.47
47.74
15.65
old 0=0.2
CEN (Ours)
31.46
63.51
19.25
27.90
49.87
15.76
4.3.2
ABLATION RESULTS
CTN Captions Benchmark Dataset: Figure demonstrates the distribution of caption quality without automatic evaluation filtering. The EMScore distribution across 11,970 videos shows 66.4% of initially generated captions achieve scores above 8=0.2, with peak around EMSccor==0.24 The remaining 33.6% of captions fall below our quality threshold, highlighting the importance of our automatic evaluation and regeneration process in maintaining CTN quality.
Cause-Effect Network (CEN): To examine the effectiveness of CEN, we compare with 6 baselines in Table 2
a) Ecombined: Instead of using two separate video encoders for cause and effect, we train one CLIP-VIT using the combined cause and effect captions separated by space for the text encoder. The performance drops across all metrics and datasets compared to the CEN architecture, which underscores the benefits of dedicated encoders for capturing cause and effect dynamics.
b) w/o FT Single CLIP: This ablation is performed using one CLIP encoder and no Fine-tuning (FT) on cause, effect, or cause+effect (combined). The results show the effectiveness of CTN fine-tuning. e) w/o FT Two CLIP: This ablation is performed using two CLIP Eeause and Eeftset encoders (as in our CEN) without performing the Stage I fine-tuning (FT) on cause and effect. This demonstrates the effectiveness of CTN fine-tuning and also, the performance increase in comparison to "w/o FT Single CLIP" demonstrates the effectiveness of CEN.
d) Only Ecause and Only Eeffect: These ablations employ only the cause encoder only the effect encoder, respectively. While retaining reasonable performance, both variants fall short of the performance of the full model across all metrics and datasets.
e) Zero Shot X: In this zero-shot setting, we evaluate the models trained on one dataset (MSVD-CTN or MSRVTT-CT CTN against the other dataset, without any fine-tuning on the other dataset. The results are either better than comparable with SOTA and VLMs in Table [
f) Fine-tune X: Similar to the Zero Shot X, this variant involves fine-tuning the models trained on cross-datasets Notably, this fine-tuning process leads to improved performance the MSVD dataset. This observation demonstrates the potential for transfer learning of CEN model from larger datasets i.e. MSRVTT-CTN and the ability to leverage their knowledge effectively on smaller datasets i.e. MSVD-CTN through fine-tuning Lin et al. (2022): ; Ventura et al.] (2024)
Overall, the ablation study validates the efficacy of our proposed CEN architecture and approach, emphasizing the significance of dedicated encoders for capturing cause and effect relationships independently as the performance drops are observed in Fcombined- w/o FT Single CLIP, w/o FT Two CLIP, Only Ecause. and Only Eeffect baselines. We also provide LLM-based evaluations in Appendix A.9
4.3.3 QUALITATIVE RESULTS
Qualitative examples, in Figure 6 demonstrate CEN's strength in accurately articulating causal relationships and event sequences, while SOTA methods and recent VLMs struggle. For example: (a) Video game scene: CEN accurately captures the causality ("performs fatality move" causing "killing another"), while other methods miss this crucial relationship. SEM-POS and GIT provide overly simplistic descriptions, while AKGNN generates irrelevant details. VLMs Viideo-LLaV and ShareGPT4Video misinterpret the scene entirely.; (b) Paper folding: CEN correctly identifies both the cause (folding paper) and effect (creating a paper airplane). Other methods either miss

---

Published as a conference paper at ICLR 2025
CTN caplion Gh: player Wlled in the gome'
lombad wrathie dienactr
CTN caption GT: Koking piece paper arp is buing crealeed
GEN (OURS): vide game character porforns a fasa more in monal kombet siced game liling snother SEM POS: a player is plaing pame AKGNN: Treddy knueger plyinng a game teddy leueger
paper axplann
game featuring meddy krusger and
lnusger Fadby and
praying womant
person making paper oirgionl GTT man making a paper plane" LLaVA: papeeeee faidiigg is paece pecae into sheascatapp atappe ShareGP GPVVidee penson lokts a ploce of paper in half and then haT ogoin make small square"
Video-LLaVA uplauu minin SharsGPT4We 4Video: man and black strped shot hokiing Inite and shabinn woman in red drees!
CTN caplion decided
the sudience watched
vot gual
Io tex
GEN (OURS) la secoer olaye lasked the ball with procesion the sall wore mi the 203) esoring in point PPS:: player le caing ball AKGNNN susee player eccend
CEN (ouRs] l'e decited in sing seeg on stage the ausienna waished Isloned Nimi POB: a is singing song AKSNN: man singing senging song' Video-LLaVVA yourn siagje ShareGPP SharaGPieVidee. piano'
gool and scored aoall soccor players are playing scccer
jeraay nurning
mamphorne whi amuther maer the
played with
while uniforme'
Figure 6: Qualitative examples across scenarios like video games, paper folding, soccer, and singing. CEN (Ours) captions accurately capture causal narratives and temporal sequences from ground truth, outperforming SOTA video captioning methods.
the causal relationship or misinterpret the action (e.g., ShareGGT4Video's "small square" instead of an airplane).: (c) Soccer scene: CEN accurately links the precise kick to the goal, capturing both cause and effect. Other methods either focus on a single action or provide generic descriptions of soccer game.; (d) Singing performance: CEN captures the boy's decision to perform (cause) and the audience's 's reaction (effect). Other methods mostly describe the act of singing without the causal- temporal context.
Overall, quantitative and qualitative results showcase CEN's superior performance in understanding and generating temmpoll narrative video captions compared to existing SOTA methods and recent VLMs. More results are provided in Appendix A.10
5 LIMITATIONS
While our work represents significant step forward in causal-temporal narrative video captioning, complex onvoluted causal relationships are still a challenge. This necessitates further architectural enhancements for improved generalization. Explicit integration of spatial reasoning, multi-agent interactions, and long-term dependencies could further enhance the robustness and applicability of our approach. Despite these limitations, our work opens up new avenues for innovative applications and research directions, further solidifying the importance of nuusll-temporal narrative understanding in video analysis tasks.
6 CONCLUSION
In this paper, we introduced for the first time a Causal-Temporal Narrative (CTN) captions benchmark dataset and proposed a novel Cause-Fffect Network (CEN) tailored for causal-temporal narrative video captioning. This work finds vast applications in automated vidco summarization, question- answering, assistive technologies, surveillance, and educational content creation. The CTN captions benchmark dataset provides comprehensive testbed for evaluating video understanding models' ability to grasp complex temporal and causal dynamics, which will be released for research purpose on https//narrativebriggg wtt.rraaiiiebbdddeeeiuuu.u.bb/// And CEN explicitly models cause-effect relationships and temporal dynamics to generate rich, contextually relevant descriptions capturing nuanced causal-temporal narrative in videos, demonstrating significant performance improvement over SOTA methods (Section 4. NarrativeBridge lays the foundation for a paradigm shift, where models comprehend underlying causal-temporal narrative driving events, unlocking new frontiers in contextually aware human- machine interactions with video.
For future work, we aim to integrate CTN caption generation with existing image captioning tech- niques, to annotate unlabeled videos with causal-temporal narrative labels. A few frames of the video will be labelled using fff-the-heelf image captioning methods, and CTN caption generation will exploit the labelled frames to generate coherent caption for the unlabelled video (see Ap- pendix A..). This synergistic approach opens new avenues for comprehensive video understanding and annotation, enabling more robust and accurate video analysis pipelines.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research was partly supported by the British Broadcasting Corporation Research and De- velopment (BBC R&D), Engineering and Physical Sciences Research Council (EPSRC) Grant EP/V038087/1 "BBC Prosperity Partnership: Future Personalised Object- Based Media Experiences Delivered at Scale Anywhere".
ETHICS STATEMENT
Our work on NarrativeBridge focuses on improving video captioning through causal-temporal narrative understanding. We use publicly available datasets (MSR-VTT and MSVD) and have cited their creators appropriately. We will release our new CTN captions benchmark dataset on our project webpage https://narraiiverrddg.. gittdggeggttubioo// where we also mention the licenses of all assets used. Our research does not involve human subjects or crowdsourcing, and we do not use or curate data containing personally identifiable information or offensive content. We have read and adhered the ethics review guidelines for ICLR submissions. We acknowledge potential ethical concerns (e.g.. privacy in surveillance, risk of misleading content) but are committed to responsible development.
REPRODUCIBILITY STATEMENT
We will release the CTN captions benchmark dataset and CEN model weights upon paper publication on project website https:/narrativeeridge. www.iggg..gtuub.oo/oo We provide comprehensive information datasets (Section 443. I. evaluation metrics (Section I. and implementation details (Section 4.2 and Appendix A.Ip. We specify data splits, hyperparameters, and other key details necessary for reproducing our results. We include information about the type of GPUs and compute resources used in Section 4.2 and Appendix A.I] We properly cite all existing datasets (MSR-VTT, MSVD) and models (IT,,-VTT, Uni-VL.) used in our work.
REFERENCES
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, al. Audio visual scene-aware dialog. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Saghir Alfasly, Jian Lu, Chen Xu, and Yuru Zou. Learnable irrelevant modality dropout for multimodal action recognition Conference on Computer Vision and Putlern Recognition, pp. 20208-20211 2022.
modality-speciffi annotated videos. In Proceedings of the IEEE/CVF
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Ams- terdam, The Netherlands, October 11-4, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, and Syed Sameed Husain. Attend-fusion: Efficient audio-visual fusion for video classification. arXiv preprint a40iv:2408. 14441, 2024.
Jeroen Berrevoets, Krzysztof Kacprzyk, Zhaozhi Qian, and Mihaela van der Schaar. Causal deep learning. arXiv preprint arXiv:2303.02 186, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey. arXiv preprint arXhv:2006.14799 2020.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190-200, 201

---

Published as a conference paper at ICLR 2025
Jiawei Chen and Chiu Man Ho. Mm-vit: Multi-modal video transformer for compressed video action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1910-1921, 2022.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprini arXiv:2406.04322 2024.
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: arXiv:2304.088 08344 2023.
omni-perception pretraining model and dataset. arXiv preprint
Andrei de Souza Inácio and Heitor Silvério Lopes. Evaluation metries for video captioning: A survey. Machine Learning with Applications, 13:100488, 2023.
Bin Duan, Hao Tang, Wei Wang, Ziliang Zong, Guowei Yang, and Yan Yan. Audio-visual event localization via recursive fusion by joint co-attention. In Proceedings the IEEF/CVE Winter Conference Applications of Computer Vision, pp. 4013-4022, 2021.
Zhiyuan Fang. Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Video2commonsense: Generating commonsense descriptions to enrich video captioning. arXiv preprint arXiv:2003.0516: 2020.
Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen. Video captioning with attention-based Istm and semantic consistency. IEEE Transactions on Multimedia, 19(9):2045- 2055, 2017.
Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10457-10467, 2020.
Shijie Geng, Peng Gao, Moitreya Chatterjee, Chiori Hori, Jonathan Le Roux, Yongfeng Zhang, Hongsheng Li, and Anoop Cherian. Dynamic graph representation learning for video dialog via mudti-modal shuffled transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1415-1423, 2021.
Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: journal of the Econometric Society, pp. 424-438, 1969.
Willy Fitra Hendria. Vania Velda, Bahy Helmi Hartoyo Putra, Fikriansyah Adzaka, and Cheol Jeong. Action knowledge for video captioning with graph neural networks. Journal of King Saud University- Compuutrr and Information Sciences, 35(4)500-62, 2023.
Chiori Hori, Anoop Cherian, Tim K Marks, and Takaaki Hori. Joint student-teacher learning for audio-visual scene-aware dialog. In INTERSPEECH, pp. 1886-1890, 2019.
Vladimir lashin and Esa Rahtu. Multi-modal dense video captioning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 958-959, 2020.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 2401.0088, 2024.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5492-5501, 2019.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu, Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108-19118 2022a.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-ima -mage pre- training for unified languae understanding and generation. In International conference on machine learning, pp. 12888-1200 PMLR, 2022b.
12

---

Published as a conference paper at ICLR 2025
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXi::2311. 2023.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 17949-17958, 2022.
Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2002 2006. IEEE, 2019.
Yan-Bo Lin, Hung- Yu Tseng, Hsin- Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. Exploring cross- audio- visual video parsing, Advances in
video and mross-mod signals for Neural Information Processing Systems, 34: 191114499911 - 11461, 2021.
Bing Liu, Dong Wang, Xu Yang, Yong Zhou, Rui Yao, Zhiwen Shao, and Jiaqi Zhao. Show, deconfound and tell: Image captioning with causal inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18041-18050, 2022.
Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.0633 2020.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and 2018.
projection for dimension reduction. arXiv preprint
Asmar Nadeem, Adrian Hilton, Robert Dawes, Graham Thomas, and Armin Mustafa. Sem-pos: Grammatically and semantically correct video captioning. arXiv preprint arXiv:2303.14829 2023.
Asmar Nadeem, Adrian Hilton, Robert Dawes, Graham Thomas, and Armin Mustafa. Cad-contextual multi-modal alignment for dynamic avqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7251-7263, 2024.
Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-tempont graph for video captioning with knowledge distillation. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pp. 10870-10879, 2020.
Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris. Adamml: Adaptive multi-modal learning for efficient video recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7576-7585, 2021.
Mirco Planamente, Chiara Plizzari, Emanuele Alberti, and Barbara Caputo. Cross-domain first person audio-visual action recognition through relative norm alignment. arXiv preprint arXiv:2 106.01689,, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Faegheh Sardari, Armin Mustafa, Philip JB Jackson, and Adrian Hilton. Pat: Position-aves aware transformer for dense multi-label action detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20882998899 29972999.200 2023.
Idan Schwartz, Alexander G Schwing, and Tamir Hazan. A simple baseline for audio-visual scene- aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12548-12554 2019.
Ankit Shah, Shijie Geng, Peng Gao, Anoop Cherian, Takaaki Hori, Tim K Marks, Jonathan Le Roux, and Chiori Hori. Audio-visual scene-aware dialog and reasoning using audio-visual transformers with joint student-teacher learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7732-7736. IEEE, 2022.
Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. Emscore: Evaluating video captioning via coarse-grained and fine-grained embedding matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17929- 17938, 2022.
Jingkuan Song, Zhao Guo. Lianli Gao, Wu Liu, Dongxiang Zhang, and Heng Tao Shen. Hicrarchical 1stm with adjusted temporal attention for video captioning. arXiv preprint arXiv: 01231, 2017.
Zehua Sun, Qiuhong Ke, Hossein Rahmani, Mohammed Bennamoun, Gang Wang, and Jun Liu. Human action recognition from various data modalities: A review. IEEE transactions on pattern analysis and machine intelligence, 2022.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 247-263, 2018.
Yapeng Tian, Chenxiao Guan, Justin Goodman, Mare Moore, and Chenliang Xu. Audio-visual inter- pretable and controllable video captioning. In IEEE Computer Society Conference on Computer Vision and Puttern Recognition workshops, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099: 092888 2023.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image
description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Lucas Ventura, Cordelia Schmid, and Gül Varol. Learning text-to-video retrieval from image captioning. International Journal of Computer Vision, pp. 1-21, 2024.
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. Proceedings of the IEEE international conference on computer vision, pp. 4542-4542, 2015.
Carl Vondrick, Deniz Oktay, Hamed Pirsiavash, and Antonio Torralba. Predicting motivations of actions by leveraging text. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2997-3005, 2016.
Cheng Wang, Haojin Yang. and Christoph Mcinel. Exploring multimodal video representation for action recognition. In 2016 International Joint Conference on Neural Networks (IJCNN), pp. 1924 1931. IEEE, 2016.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint 2205v:2205.1 220.:22000..00. 2022.
Todd Wilkens, Anthony Hughes, Barbara M Wildemuth, and Gary Marchionini. The role of narrative in understanding digital video: An exploratory analysis. Proceedings of the American Society for Information Science and Technology, 40(1):323-33) 2003.
14

---

Published as a conference paper at ICLR 2025
Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovi- sual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering explaining temporal actions. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 9777-9786, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Jun Xu, Ting Yao, Yongdong Zhang, and Tao Mei. Learning multimodal attention Istm networks for video captioning. In Proceedings of the 25th ACM international conference on Multimedia, pp-
537-545 2017.
Dizhan Xue, Shengsheng Qian, and Changsheng Xu. Variational causal inference network for explanatory visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2515-2525, 2023.
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10714- 10726, 2023.
Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2031-2041, 2021.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi, From recognition cognition: Visual commonsense reasoning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6720-6731, 2019.
Yunhua Zhang. Hazel Doughty, Ling Shao, and Cees GM Snoek. Audio-adaptive activity recognition across video domains. In Proceedingy of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 13791-13800, 2022.
Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng Jun Zha Object relational graph with gaacherreccmmeende learning for video captioning. In Proceedings of the IEFF/CVF conference computer vision and paltern recognition, pp. 13278-13288 2020.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLRMENTATION DETAILS
We first extract frames from the video clip and then encode them using the Cause and Effect Video Encoders. We adopt sampling strategy where we sample one frame per second, with maximum of 20 frames. This approach ensures that the sampled frames cover the entire duration of the video, providing a comprehensive representation of the video content while maintaining computational efficiency. For the fine-tuning experiment in Table 2 all the hyperparameters remain the same except the learning rate i.e. 0.0000005. For our experimentation of the Cause and Effect Network (CEN), we use standard splits of the MSVD(Chen & Dolan (2011) and MSRVTT Xu et al. (2016) datasets. For MSVD, use 1200 videos for training. 100 for validation, and 670 for testing. For MSRVTT, use 6513 videos for training, 497 for validation, and 2990 for testing. These splits are commonly used in video captioning research ensure fair comparison with other methods. All the experiments in Stage I and Stage 2 of our CEN are run using A 1000-80GB and RTX 3090-24GB GPUs respectively. We implement SEM-POS Nadeem et al. (2023), AKGNN Hendria et al. (202)) and GIT Wang et al.) (2022) using RTX 3090-24GB, A 100-80GB and A 1000800B GPUs respectively.
For comparison with Vision-Language Models (VLMs), we implement two fine-tuning approaches:
LoRA Fine-Tuning and Simple Fine-Tuning using A A100-80GB GPUs LoRA Fine-Tuning is applied specifically to the LLM component, with a learning rate of 2e-4 for LoRA parameters. Simple Fine-Tuning is applied to the entire model, using an AdamW Loshchilov (2017) optimizer with cosine learning rate schedule (initial learning rate: 1e-3, warmup ratio: 0.03).
A.2 PROMPT DESIGN PROCESS
we introduce our approach for generating Causal-Temporal Narrative (CTN) captions using a large language model (LLM) and few-shot prompting. The prompt design played a crucial role in guiding the LLM to generate captions that accurately capture the cause-effect relationships and temporal dynamics in the video content. Figure I showcases the effectiveness of our few-shot based prompt, which resulted in a coherent and contextually relevant CTN caption for the given video. To further illustrate the importance of prompt design and the benefits of few-shot learning, additional experiments with zero-shot prompting. In these experiments, we evaluate the performance of the LLM generating CTN captions without providing any example captions in the prompt. Zero-Shot Prompting: Zero-shot prompting refers to the process of providing large language models (LLMs) with a task prompt without any accompanying examples or demonstrations, requiring the model to generate relevant responses based solely on its pre- trained knowledge and understanding of the prompt. First, we aim to evaluate the performance of LLMs in generating causal-tent -emmoall narrative (CTN) captions for videos without providing any few-shot examples.
In Section 3.1
conduct
To achieve this, we construct 4 distinct prompts, each with varying degrees of quality, and experi- mented with all of them to assess their effectivenees in zero-shot setting, starting with the most challenging prompt and progressively improving the quality. 2oompt2present an extremely terse and poorly structured set of instructions. It contains minimal context and information, merely instructing the model to "Make Cause Effect thing from captions" and listing a set of numbered requirements without providing clear explanations. As expected, the resulting captions (Figure 7 are of low quality and often lack coherence and relevance to the video content. Moreover, the LLM generated multiple captions instead of single coherent caption, and the output format does not adhere to the specified structure (a single JSON object containing the Cause and Effect statements).
Make Cause Effect thing from captions. Follow: 1. Use only caption info 2, Be relevant 3. No new stutr 4. No conclusions 5. Same time 6. Plain English short sentences 7. 15 words nax 8, Just JSON. [Descriptive Captions]:
edescriptive captions>
[Causal Temporal Narrative]:
Prompt 2: Zero-shot LLM Prompt with minimal context and unexplained requirements for generating cause-effect from captions.
16

---

Published as a conference paper at ICLR 2025
"1"; ("Cause"; "A car crashes", "Effact"; "Cuys play beer pong"),
"2": (*Cause": "A drivea through open Mell"" "Rffect": *Kicka up
dixt"),
"3": ("Cause": "Unclear", "Bffect":
flips over"), "Unclear", "Effect": "A car geta wracked"), "Unclear", "Effect";
"4"; (*Cause":
"5": (*Cause";
car is being flipped over"), "A dirt vehicle rides", "sffect"; "it rolls*),
"6": (*Cause":
"t"; ("Cause":
"Unclear", "Effect": "A dune buggy flips over"), "Unclear", "Etfect";
"B": ("Cause":
tour "A monster truck flips on its side", "Effect*; "Several young pong"),
wrecks"),
"g": (*Cause":
playing
"10": :"Cause";
"À person drives offread around field", "ECTect";
"Unclesr"),
"11": '"Cause::
"A person flips kart", "affect"; "A growd cheers"),
"12": ("Cause": "thalear", *#ffect*; "A race truck is crashing"),
"13": ("Cause";
Lruck
itself", "Etfect"; "Boys
Eriend";
"14": :"Cause": "unslear", "affect";
tumbles over itself"),
"15": ("Cause": "A tumbler crashes dirt road", "Effect*; "Then group
guys play beer pong"),
"16": ["Cause:: "A type of monster truck crashes", "Effent": "Men shown
celebrating"),
"17": :"Cause"; "Uacleac", "Bffect"; vehicle flips over"),
"18": ("Cause": "Uaclear", "Effect"; "Aa off read vehicle crashes"),
"19": "Cause": "Unslear", "Effect";
while driving"), "20": ("Cause": "Fostage from monster truck event", "EEfect":
"Fellowed by frat party")
Figure 7: Caption generated using LLM for the example in Figure Jsiing Prompt
We then progressively improve the prompt quality, moving to a more concise set of instructions in Prompt 3 and relatively well-structured prompt in Prompt While the captions generated by these improved prompts (Figures 8 ) show some enhancements, they still fall short in terms of capturing the temporall narrative accurately and consistently. Additionally, the LLM continues to generate multiple captions and deviate from the desired output format.
descriptive captions about a video. The scenario must follow these rules: stick to the given information, be relevant, don't add new aventa/detaila or draw conclusions, naintain time consistency, use simple language in short sentences, and limit each atatement to worda. Don't extra text outside [Descriptive Captions]:
ddeseriptive_captione [Causal Temporal Narrative]:
Prompt 3: Zero-shot LLM Prompt with reasonable structure conveying core task of creating effect scenario from captions.
( "gvent"
"Car crashes", "gffect": "Guys play beer pong"), "car drives in open field", *Sffect"; "Dirt is kicked up*),
"Evant":
"Event"; :
"Car flipa over", "Stfect"; "None specified"),
gets wracked", "Bffect"; "Noae specified"),
"Event": :
"car is flipped ovar", "Iffect"; "Nore speci
"Bvent"
"Dirt vehicle rides and rolls", "BCiect"; *None specifled"),
"Event":
"Dune buggy flips
"Effect": "None specified"),
"gvent"
"Four wheeler wrecks", "sffect": "Nere specified"),
"Evaent
"Nonater truck flips", *#ffent": "Young ahout play
pong") 1,
"Person
offreed car", "Iffect":
specified"),
""vent": *
"Peraon flips go kart*, "Etfect": "Crowd cheers"),
"Bvvnt :
"Race truck crashes", "BSfect"; *None specifled"),
"gvent":
"Izuck rolls over", "gffect": "soys cheer friend";,
"Event":
"Tumbler crashes on dirt read", "gifeet:: *Guys play baer pang"),
"Bvent*:
"Nonater
"Erfect": "Men
celebrating"),
"Bvent":
"Vehicle flips over", "BCfect"; *None specifled"),
"Event":
vehicle creshes", "Iffect": "Nene specified";,
"Bvent":
"Car crashes while driving", "E[Ject"; "Rone apeci
"Event":
"Konster sruck event footsge", "Bffect": "Peat party is show?")
Figure 8: Caption generated using LLM for the example in Figure using Prompt
17

---

Published as a conference paper at ICLR 2025
You are tasked with generating causal narrative caption for video based on series of provided descriptive captions. Your generated Cause and Effect scenario should be grounded in the given information, relevant, avoid introducing details implying conclusions, maintain temporal consistency, plain Inglish 15 words maximun. Do not include any additional text outside the JSon object. [Descriptive Captions]:
ddsceriptive_captiiie captions>
[Causal Temporal Narrative]:
Prompt 4: Zero-shot LLM Prompt clearly describing caption-based cause-effect generation task with specified requirements.
[""vent*
"Car crashes", "#ffect": *Guys play pong"),
"gvent": :
"car drives in open field", "=ffect"; "pirt is kicked up"),
"Bvent"
"car flips ovar", *Effect": "Nore apecified"),
"Event":
"Car gets wracked", "Effect": "None specified"),
"Bvent" :
"Caz is flipped over", "Iffect"; "None specified"),
"Event":
"virt vehicle rides and rolls", "gffect": *None specified"),
"Event" :
"Dune huggy flips aver", "#ffect": "Nona specified"),
"Event
"Four
wrecks", "Strect": "Nore speciried"), "Nonster truck flips", "Effact"; "Young men shout and play beer
"gwent": :
pong") ),
"Bvent":
"Peraon drives offread cas", "E[fect"; "Kone speci[ied"),
"Event" :
"Person flips go kart", "Effect": "Crowd cheers"),
"gvent": "Race truck crashes", "géfect": *None specified"),
"Event" :
"Truck ralls aver", "#ffect": "Roys chear on friend"),
"Bvent":
"Tumbler
read", "Effect"; *Gays play pong"), "Konster truck crashes", "Effect": "ee are shown celebreting"),
on
"Bventt:
"Bvent":
"Vehicle flips over", "Brfect"; *None specifled"),
"Bvent":
"cff road vehicle creshes", "Iffect"; "None specified"),
"Event": :
"Car crashes while driving", "Effect": "Kone specified"),
"Evant":
"Xonster truck
footage", "Effact"; "Frat party is shown")]
Figure 9: Caption generated using LLM for the example in Figure using Prompt
Finally, we experiment with a zero-shot prompt (Prompt 5) that closely resembles our few-shot based Prompt from Section 3.T but without the example captions. The captions generated by this prompt (Figure TOp demonstrate improved coherence and relevance compared to the previous zero-shot prompts. However, they still lack the level of detail, accuracy, and contextual understanding exhibited by the captions generated using our few-shot based prompt (Prompt T. Furthermore, the LLM persists in generating multiple captions.
language Lasked with generating video. Hoveves, you cannot directly access the video sacies of itself, Instead, you will be provided with captions that outline the kay eventa and scenes is the video. Your task is to generate based
temporal narrative captions
concise and Effect
information provided
descriptive following zequirements:
careful, your generated
1. Your narrative should be grounded in the information provided by the descriptive captions.
2. Cause and Effect scenario is relevant.
3. should introduce
events details not mentioned.
implying
temporal consistency the provided captions.
6. Use plain English and direct sentences.
7. Cause and Effect statements each limited to maximun of 15 worda. include additional before after the JSON object.
[Descriptive Captions]:
dessoriptive_cattions
[Causal Temporal Macrative]:
Prompt 5: Zhoo-Shot Prompting for Causal Narrative Caption Generation
18

---

Published as a conference paper at ICLR 2025
"Cause and Effect Statements"; "Car crashes in field, causing it to #lip kart flipped person,
"bune buggy fl,,,, resulting in wreck. leading to crowd cheers,", "Race truck crashes, boys cheer in reaponse, ", "Truck tumbles, friends celebrate alterwards.", "off-road vehiole crashes, causing tumble.", "Car craah occurs, followed by game of beez pong."
playing beez pong.",
crashes,
Figure 10: Caption generated using LLM for the example in Figure
| Jusing Prompt
Prompt | Ablations: Here, we present systematic ablation experiments analyzing the impact of individual requirements in the design of Prompt
[Sae Prompt
using only information captioos]
Prompt 6: Removing the grounding requirement from Prompt
"Cause": "a seckless driver speeda through field in an unstable off-road vehicle', "affect": vehicle crashes dramatically group with beer nearby"
Figure 11: Caption generated without grounding requirement
Removing the grounding requirement results in the introduction of speculative details that not present in the source captions. This demonstrates how the grounding requirement maintains factual accuracy.
Prompt
temporal consistency)
Prompt 7: Removing temporal consistency requirement from Prompt
"Cauae*: "people oe celebrate anthusisations leally at a gathering", "#ffect": flips while driving through open field"
Figure 12: Caption generated without temporal consistency requirement
The absence of the temporal consistency requirement leads incorrect event ordering, with the celebration incorrectly positioned before the car crash. This highlights the importance of maintaining proper temporal sequence in causal narratives.
[Same as Prompt but
without rule about word limit]
Prompt 8: Removing word limit requirement from Prompt
"Cause": "an adventurous individual decided drive their vehicle recklessly through an open field without proper consideration for safaty or stability, leacing dangerous situation", "BSfect"; *the vehiole inevitably lest control and dramatically flipped over, resulting in serious accident while nearby, a group people were engaging in casual game beer pong, seemingly unaware of the incident"
Figure 13: Caption generated without word limit requirement
Removing the word limit results in verbose descriptions with unnecessary elaboration and speculation about circumstances and motivations.
[Same Prompt
without
rule about plain English]
Prompt 9: Removing plain English requirement from Prompt
19

---

Published as a conference paper at ICLR 2025
"Cause": "the automobile's operator executed an ill-advised travarsal of the paatora) terratn", "orfect": "oulminatingg in vehi cular inverstin whi 1st ensued in proximily"
Figure 14: Caption generated without plain English requirement
Without the plain English requirement, the model employs unnecessarily complex vocabulary and formal language that reduces clarity and accessibility.
[Same as Prompt
without
rule about avoiding conclusions]
Prompt 10: Removing the requirement to avoid conclusions from Prompt
"Cause": "driver's obvious Inexperience and pcor judgment led to dengerous driving field", "Iffect": "predictable crash cccurred while others carelessly continued their party, showing clear negligence"
Figure 15: Caption generated without avoiding conclusions requirement
Removing the requirement to avoid conclusions leads to subjective interpretations about driver inexperience and partygoer negligence that cannot be definitively determined from the video content.
[Same as Prompt
but without rule
about relevance]
Prompt 11: Removing relevance requirement from Prompt
"Cause": "a vehicle driives through field on sunny day and crashes", "Bffect"; "people enjoy various outdoor activities including beer pong and socializing"
Figure 16: Caption generated without relevance requirement
The absence of the relevance requirement results in the inclusion of peripheral details. This ablation analysis validates the prompt design choices and underscores the importance of main- taining all requirements for generating effective causol-temporal narrative captions. conclusion, these findings underscore the importance of our few-shot based prompt design in the CTN caption generation process and demonstrate superiority over zero-shot prompting approaches. By lever- aging the power of few-shot learning, enable the LLM to generate high-quality CTN captions that effectively capture the cause-effect relationships and temporal dynamics in video content, as evidenced by the results presented in Figure TT Additionally, our few-shot based prompt along with requirements ensures that the generated captions adhere to the specified JSON format, facilitating seamless integration and usability in downstream applications.
A.3 COMPARISON LARGE LANGUAGE MODELS CTN CAPTION GENERATION
To evaluate the effectiveness of large language models (LLMs) in generating high-quality Causal- Temporal Narrative (CTN) captions, we compare the performance of two open source state-of-the-art LLMs at the time of experimentation: Mixtral of Experts Tiang et al. (2024), which we utilized in our CTN caption generation pipeline, and Llama2-700 70b Touvron et al. (2023).
We provide the original video captions input to both models and assess the quality of their generated CTN captions. In the video game example (Figure 17). Mixtral of Experts accurately captures the causal relationship between the fatality move performed by one character and the consequent death of another character in the game. In contrast, Llama2-70b focuses on a specific effect (blood spurting from the character's neck) without explicitly linking the cause.
For the paper folding tutorial (Figure 18). both models correctly identify the causal relationship between folding the paper and creating paper craft. However, Mixtral of Experts provides a more precise description, specifying that the paper craft being created is paper airplane.
In the soccer highlight scenario (Figure [9). Mixtral of Experts successfully captures the causal link between the soccer player's precise kick and the ball successfully going into the goal. Ilama2-70b.
20

---

Published as a conference paper at ICLR 2025
Input Video:
animated cartoon cuts another's throaf'
a character a video game gets his throst cut open'
aa fatality from Mortal Kombat is shown'
Original Captions
Kombat fatality sequence'
5. 'a person playing video game'
playing Mortal Kombat" 7. la scene from the video game of fatality'
'a
'a video game character kills another character'
a video game scene is going on fighting'
10. 'a violent scene from video game'
through
12. "gameplay with lots of blood
LLM Evaluation
video game,
attacks with knife'
14. video game, woman is killing man'
15. "one character wins perishing the other'
fatality Mortal Kombat'
17. 'someone is playing game'
CTN Caption
'someone is playing video game'
19. 'there is a woman killing someone In game'
20. 'Skarlet fatality in Mortal Kombat'
Mixtral of Experts (Our CTN) ("Cause': player performs a fatality move in mortal kombat', 'Effect': 'another character is killed in the game')
Llama2-70b : ('Cause': 'a character in video game gets his throat cut open', 'Effect': 'blood shown spurting out of the characters neck))
Figure 17: Comparison of Mixtral of Experts and Llama2-70b on a video game sequence involving a fatality move.
Input Video:
1. man folding paper"
folding paper"
3. man is showing how to fold paper'
Original Captions
4. showing some crafts'
person folds paper relaxing music'
6. person is folding white paper'
7. person folding white sheet of paper"
person folding paper
9. person showing people how to fold origami'
10. young man folding piece of paper"
'craft work piace paper'
12. 'paper plane folding tutorial video clip'
Evalustion
LLM
'somebody folds piece paper
fold paper plane' 15. 'someone is folding origami'
'someone explain
'someone is folding paper'
17. 'someone is making paper craft'
CTN
18. 'someone is folding paper'
Caption
'paper plane folding tutorial video
20. person folding white paper'
Mixtral of Experts CTN) ('Cause": 'a folding piece of paper', 'Effect': 'a paper airplane being created")
Llama2-70b ('Cause:: 'a person folds paper', 'Effect': the person is making paper craft")
Figure 18: Comparison Mixtral Experts and Llama2-70b on paper folding tutorial create paper airplane.
21

---

Published as a conference paper at ICLR 2025
Input Video:
professional soccer game that set music'
2.'s soccer highlight video'
match is happening and of the players just kicked ball the goal! à soccer player hops over a wall"
Original Captions
'a soccer player is scoring the goal for his team"
player scoring goal'
l soccer player makes goal
'a soccer player scores goal'
'compilation of popular soccer
10. 'highlights soccer game'
11. 'people are playing sports the ground" players running
Prompt
field'
13. 'soccer players playing soccer'
14. 'sports highlights displayed'
are people playing soccer scores goal"
16. 'there ls some footage a soccer game'
17. 'this video from
18. video of athietes performing amazing plays'
Caption
19. la soccer player hops over wall'
20.
player makes goal'
Mixtral of Experts (Our CTN) ('Cause": 'a soccer player kicked the ball with precision', 'Effect': 'the ball successfully went the goal')
Llama2-70b ('Cause:: soccer player scores goal', 'Effect': people are cheering')
Figure 19: Comparison of Mixtral of Experts and Llama2-700b on highlights from a soccer match.
the other hand, mentions the effect (people cheering) without explicitly connecting it the cause (the player scoring a goal).
Input Video:
'a boy giving stage performance'
boy is singing"
3. la boy is singing song'
Original Captions
boy singing
stage'
'a boy is singing on scene front of the crowd'
6. la boy is singing on stage in front of an audience'
boy singing stage in frant crowd"
'a boy sings front of judges and audience'
la guy Is singing on stage'
singing'
11. la man singing a song on stage'
Prompt Evallaation
12. ta man plays plano while young boy sings stage'
is singing stage'
14. 'a young boy signing on tv front of an audience'
15. 'a young
is singing crowd stage'
'guy singing stage'
17. Iaa is singing in live show'
CTN
18. young boy singing song'
Caption
singing a song stage'
20. 'guy singing on stage'
Experts (Our CTN) : ('Cause": 'a boy decided perform stage', 'Effect': audience listened to his singing')
Llama2-70b ('Cause': 'the boy is singing', 'Effect': the crowd enjoys the performance')
Figure 20: Comparison of Mixtral of Experts and Llama2-70 70b on a boy singing on stage front of audience.
Lastly, for the singing performance (Figure 20). both models accurately describe the cause-effect relationship between the boy's decision to perform on stage and the audience's reaction watching and listening. However, Mixtral of Experts provides more detailed description of the audience's 's response.
22

---

Published as a conference paper at ICLR 2025
Overall, the CTN captions generated by Mixtral of Experts consistently demonstrate better derstanding of the causal-temporal narrative in the videos compared to Llama2-70b. Our approach using Mixtral of Experts effectively captures the cause-effect relationships and temporal dynamics, resulting in more accurate and contextually relevant captions.
These results highlight the importance of selecting an appropriate LLM and designing effective prompts for generating high-quality CTN captions. The superior performance of Mixtral of Experts can be attributed to its architecture and training, which enable better understand and articulate the complex temporall narratives present in video content.
A.4 IMPACT OF AUTOMATIC EVALUATION ON CTN CAPTION GENERATION
In Section 3.1 we discuss the importance using automatic evaluation to ensure the quality and relevance of the generated Causal Temporal Narrative (CTN) captions. We employ the EMScore metric to measure the consistency between the generated captions and the video content, discarding captions that fell below predefined threshold. This appendix visually demonstrates the impact of the automatic evaluation step on the quality of the generated CTN captions.
P
CTN caption GT -urOurs: ('Cause': 'a female sportscaster interviewed male athlete',
Ours:
'Effect": 'she asked him questions")
CTN caption (w/o automatic evaluation): ('Cause': 'a man carelessly neglected taking
his prescribed allergy medication', 'Effect': 'he suffered a severe sneezing fit')
Figure 21: CTN caption comparison for a video female sportscaster interviewing a male athlete.
CTN caption GT Ours: ('Cause': 'bus ticket prices increased in Brazil', 'Effect': woman was interviewed about her participation in the protests")
CTN caption (w/o automatic evaluation): ('Cause': 'protest caused by a rise in bus tickets', 'Effect': 'mass gathering of people in rio de janeiro and sao paulo")
Figure 22: CTN caption comparison for video of protests in Brazil due to increased bus ticket prices.
CTN caption GT Ours: ("Cause": "a terror attack occurred in Texas", "Effect": "news anchors discussed freedom of speech issues")
CTN caption (w/o automatic evaluation): ('Cause': 'a band performs a song on stage', 'Effect': 'the audience cheers and enjoys the music")
Figure 23: CTN caption comparison for a video discussing a terror attack Texas.
Figure 21 shows an example where the CTN caption generated with automatic evaluation accurately captures the causal relationship between the female sportscaster interviewing the male athlete and her asking him questions. contrast, the caption generated without automatic evaluation is irrelevant to the video content, discussing neglecting his allergy medication and suffering sneezing fit.
23

---

Published as a conference paper at ICLR 2025
CTN caption GT Ours: ("Cause": "the Air Force used a surveillance system in Iraq", "Effect": woman talked about the man who created it")
CTN caption (w/o automatic evaluation): ('Cause": 'a group starts dancing at an event', 'Effect': 'the atmosphere becomes more lively")
Figure 24: CTN caption comparison for video about the Air Force using surveillance system in Iraq.
Similarly, in Figure 22, the CTN caption generated with automatic evaluation correctly identifies the cause of the protests Brazil as the increase in bus ticket prices and links it to the effect of woman being interviewed about her participation in the protests. The caption generated without automatic evaluation, while mentioning the protest and its cause, fails to capture the specific effect of the woman's 's interview. Figure Figure[23demmn demonsttaates how the CTN caption generated with automatic evaluation accurately describes the cause of a terror attack in Texas and its effect on news anchors discussing freedom of speech issues. The caption generated without automatic evaluation is completely unrelated, mentioning a band's performance and the audience's 's reaction.
Lastly, in Figure 24 the CTN caption generated with automatic evaluation correctly links the cause of the Air Force using a surveillance system in Iraq to the effect a woman talking about the man who created it. The caption generated without automatic evaluation is again irrelevant, discussing group starting to dance at an event and the atmosphere becoming more lively. These examples clearly illustrate the importance of incorporating automatic evaluation in the CTN caption generation process. By ensuring that the generated captions are consistent with the video content, we can significantly improve the quality and relevance of the CTN captions, enabling the CEN model learn more meaningful causal and temporal relationships from the videos.
A.5
HUMAN EVALUATION STUDY DETAILS
To validate the quality of CTN captions, conduct a rigorous human evaluation study. This section details the methodology, results, and analysis of this evaluation.
A.5.1
METHODOLLGGY
We employ the following methodology for our human evaluation:
Sample size: 100 validation instances (50 each from MSR-VTT and MSVD datasets)
Evaluators: Five independent domain experts
Evaluation criteria:
1. Causal Accuracy: Degree of correctly identifying and describing cause-effect relation- ships
2. Temporal Coherence: Accuracy in representing the sequence of events
3. Relevance: How well the caption reflects the overall content and context
Rating scale: 5-point Likert scale (0-5, with 5 being the highest quality)
A.5.2 RESULTS
Table presents the results of our human evaluation study.
Table 3: Human Evaluation Results for CTN Captions
Criterion
Mean Score
Std Dev
% Perfect Scores
Causal Accuracy
4.8
0..42
82%
Temporal Coherence
4.7
0.46
78%
Relevance
4.9
0.31
91% %
Overall Quality
4.8
0.40
84%
Key findings from the evaluation:
24

---

Published as a conference paper at ICLR 2025
Overall mean quality score: 4.8/5.0 (o 0.40)
93% of captions received scores of or higher across all dimensions
84% of captions achieved perfect scores
A.5.3 INTER-RATER RELIABILITY
To ensure the consistency of ratings across evaluators, we calculate the Intraclass Correlation Coeffi- cient (ICC) for absolute agreement among raters:
ICC: 0.87 (95% CI: 0.83-0.91)
This high ICC value indicates strong inter-rater reliability, supporting the robustness our evaluation process.
A.5.4 ANALYSIS
The results of our human evaluation study strongly validate the quality and consistency of our CTN captions. With mean quality score of 4.8/5.0 and 93% of captions receiving high scores across all dimensions, we can confidently assert that our CTN captions effectively capture causal-temporal narratives in videos. The high inter-rater reliability further strengthens the credibility of these results. These findings demonstrate that our automatic generation and evaluation process, as described in Section [.I] produces high-quality CTN captions that accurately represent the causal and temporal dynamics in video content. This human evaluation complements our automatic evaluation metrics, providing comprehensive validation of our CTN caption benchmark dataset.
A.6 ENCODER COMPARISON FOR CEN
To explore the versatility of our CEN architecture, we conduct experiments using BLIP Li et al. (2022b) encoders place of CLIP encoders. This comparison allows us assess the adaptability of our method to different visual encoding architectures. Table presents the results of this comparison on both MSVD-CTN and MSRVTT-CTN datasets.
Table 4: Comparison of CLIP and BLIP Encoders for our CEN
Dataset
Encoder
ROUGE-L
CIDEr
SPICE
CLIP (Ours)
31.46
63.51
19.25
MSVD-CTN
BLIP
28.79
57.23
18.06
CLIP (Ours)
27.90
49.87
15.76
MSRVTT-CTN
BLIP
25.34
43.01
14.89
While the performance with CLIP encoders is higher, the results with BLIP encoders are still competitive. This demonstrates the flexibility of our CEN architecture and its potential to work effectively with various visual encoding methods.
A.7 COMPARISON WITH NEXT-QA
To highlight the differences between our CTN benchmark dataset and the NExT-QA dataset, we provide detailed comparison table and illustrative examples:
Table 5 presents comprehensive comparison between NExT-QA and our NarrativeBridge approach.
Table 5: Comparison of NExT-QA and NarrativeBridge
Aspect
NEXT-QA
NarrativeBridge (Ours)
Task
muestion-Answeree
Video Captioning
Output
Short answers
Coherent narratives
Temporal
Specific points
Multiple events
Causal
Implicit
Explicit
Narrative
Not present
Explicit
Focus
Info retrieval
Video description
This comparison highlights the fundamental differences approach and output between NEXT-QA and our NarrativeBridge method.
Figure proviide examples from the NExT-QA dataset, illustrating the nature of its question-answer pairs. As in these examples, NEXT-QA focuses specific question-answer pairs that often
25

---

Published as a conference paper at ICLR 2025
around
adult after
What the boy pulling? Rope
the start?
cravling awayy? A: Follow the baby
A: Waifing for the lady
Figure 25: Examples of question-anie answer pairs from the NEXT-QA dataset.
target single events actions. In contrast, CTN captions provide
comprehensive narrative
that captures the causal and temporal relationships across the entire video sequence.
A.8 APPLICATION OF CTN CAPTION GENERATION FOR LABELING UNLABELED VIDEOS
Our CTN caption generation approach, as described in Section 3.1 can be effectively applied to the task of labeling unlabeled videos. This application leverages the power of our few-shot based prompt and the LLM's ability to generate coherent and contextually relevant captions that capture the narrative in video content.
Video:
Sample
Fraine
Frame
Input Images from unlabelled video
Zaro-shot Image Captions
tractor driving'
are four
tractor upside down!
game in being playad
"woo people are celebrating'
CTN Caption
Goneration
Pronpe
Evaluction
CTN Caption
Effect: 'causing accident, while others celebrated unaware"
Cause: "a tractor driving recklessly flips over and ends upside down
Figure 26: Application of our CTN caption generation approach for labeling unlabeled videos. Given an unlabeled video, we extract frames and generate image captions using a state-of-the the-art image captioning model (GIT) Wang et al, (2022)). These captions are then used input CTN caption generation pipeline, which produces a CTN caption for the entire video. The generated CTN caption captures the cause-effect relationships and temporal dynamics in the video, enabling effective labeling of unlabeled video content.
LLM-based
Figure 26 illustrass the pipeline for labeling an unlabeled video using our CTN caption generation approach. First, extract frames from the unlabeled video. In this example, we extract five frames which are equally spaced in the video. Next, we generate image captions for each of the extracted frames using state-of-the-art image captioning model. For this demonstration,
employ the GIT Wang et al. (2022), which has shown impressive performance in generating accurate captions for individual images. The GIT model generates captions such as "a tractor driving". "there are four trucks", "a tractor is upside down". "a game is being played", and "two people are celebrating" for the five frames. These image captions serve as the input to our L.Md- CTN caption generation pipeline (see Figure 2)
We use Prompt T in Section 3.1 replacing the original descriptive captions with the image captions generated by the GIT model. The LLM then generates CTN caption based these image captions, following the specified requirements and format. In this example, the generated CTN caption is: "Cause: aa tractor driving recklessly flips over and ends up upside down' Effect: 'causing accident,
26

---

Published as a conference paper at ICLR 2025
while others celebrated unaware"". This caption effectively captures the key events and their causal- temporal relationships in the video, providing a concise and informative summary the video content.
To ensure the quality and relevance the generated CTN caption, we employ the same evaluation framework described in Section The caption is compared against the video content using the EMScore Shi et al. (2022) metric, and only captions that meet a specified threshold are retained. This application demonstrates the versatility and effectiveness of our CTN caption generation approach in labeling unlabeled videos. By leveraging the power of state-of-the-an art image captioning models and our few-shot based prompt, we can generate high-quality CTN captions that accurately capture the causal-temporal narrative in video content, even in the absence of human-annotated captions. This approach has the potential to significantly streamline the process of labeling large-scale video datasets and enable more effective video understanding and retrieval tasks.
A.9 LLM-BASED EVALUATION GENERATED CTN CAPTIONS
We conduct additional evaluations on MSVD-CTN and MSRVTT-CTN (combined) to evaluate further the quality of generated CTN captions, using Llama-3.2-3B Instruct in the default configuration. Our evaluation focused on two key aspects:
Temporal Order Analysis
Input: Ground truth and generated captions
Task: LLM compares the temporal sequence events
Score: Binary (I: correct sequence, O: incorrect)
Causal Chain Analysis
Input: Ground truth and generated captions
Task: LLM evaluates preservation of cause-effect relationships
Score: Binary (I: preserved, O: not preserved)
Table the performance comparison between CEN and the best baseline (GIT).
Table 6: LLM-based evaluation results comparing CEN with GIT
Model
Temporal Order (%)
Causal Chain (%)
CEN (Ours)
81.2
84.5
GIT
52.1
48.3
We further analyze all ablations of Table usiig the same LLM-based metrics, as shown in Table
Table 7: LLM-based evaluation results for ablation study
Method
Temporal Order (%)
Causal Chain (%)
CEN (Full)
81.2
84.5
Ecombined
72.4
75,6
w/o FT single CLIP
58.3
54.2
w/o - Two CL.IP
63.7
61.8
Only Eeause
73.9
76.5
Only Beffect
74.8
77.2
Zero Shot
55.2
51.7
Fine-tune X
79.4
82.1
Following are the key findings:
CEN outperforms GIT by 29.1% % in temporal and 36.2% in causal accuracy
Dual encoder architecture provides 15% improvement over single encoder
Fine-tuning improves temporal understanding by 20% compared to frozen models
27

---

Published as a conference paper at ICLR 2025
A.10 ADDITIONAL QUALITATIVE RESULTS FOR CEN ARCHITECTURE
In this section, present additional qualitative results comparing the ground truth Causal-Temporal Narrative (CTN) captions with the captions generated by our CEN architecture. These examples further demonstrate the effectiveness of our approach in capturing the causal-temporal relationships and generating accurate and contextually relevant captions.
CTN caption GT: 'a person working on a rubiks cube the rubiks cube gets solved piece by piece'
CEN (OURS): 'a person is playing with a rubiks cube and the person successfully solved the cube'
Figure 27: Comparison of CTN and CEN captions for a video a person solving a Rubik's cube.
CTN caption GT: 'a man is playing the piano music fills the room'
CEN (OURS): 'the man is playing the piano and he is demonstrating how to play the piano'
Figure 28: Comparison of CTN and CEN captions for video of man playing the piano.
CTN caption GT: 'man shared sad story on stage audience became emotionally engaged'
CEN (OURS): 'the man gave lecture on stage and the audience listened intently'
Figure 29: Comparison of CTN and CEN captions for video of man giving lecture on stage.
CTN caption GT: 'a band performs song in a video people watch and listen to the music'
CEN (OURS): 'a band is performing a song and the crowd enjoying the music'
Figure 30: Comparison of CTN and CEN captions for a video a band performing a song. In Figure [7 both the CTN and CEN captions accurately capture the causal relationship between the person working on the Rubik's cube and the cube being solved piece by piece. Similarly, in Figure
28

---

Published as a conference paper at ICLR 2025
28
both captions correctly describe the cause-effect relationship between the playing the piano and the music filling the room.
Figure 29 demonstrates the ability of our CEN architecture generate captions that capture the audience's S engagement in response to the man's lecture on stage. While the CTN caption specifically mentions the emotional engagement of the audience due the sad story, the CEN caption more generally describes the audience listening intently to the lecture.
Lastly, in Figure [30] both the CTN and CEN captions accurately depict the causal relationship between the band performing song and the crowd enjoying the music. The CEN caption, in particular, directly states the crowd's enjoyment as a result of the band's performance. These additional qualitative examples further validate the effectiveness of our CEN architecture in understanding and articulating the causal-temporal narratives present in videos, generating captions that are coherent, accurate, and contextually relevant.
29


Reviews:
[{"review_id": "vYq8QAMpSr", "reviewer": "Reviewer_qdoh", "summary": "In this paper, a video captioning dataset named Causal-Temporal Narrative (CTN) is proposed. Compared to previous video captioning datasets which lack in causal-temporal  information which connects a cause and an effect, CTN, built with LLMs, includes caption that directly connects cause and effect events. Also, a new architecture named Cause-Effect Network (CEN) is proposed, which consists of two independent encoders which capture cause and effect events, respectively.", "questions": "Please refer to weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the work is clear, while the problem of improving the causal-and-effect understanding capability of a vision-language model is an important research direction.\n2. Experimental results show that both the proposed methods, CTN and CEN contribute to better performance on MSVD and MSRVTT benchmarks.", "weaknesses": "1. Including detailed statistics about the constructed CTN dataset (e.g., # of samples, average caption length, # of captions per sample, or more..) may enable readers to easily understand the dataset.\n2. Questions about Table 2.\n    1. Results show that Fine-tune v.v. shows better performance than CEN on the MSVD benchmark under the cross-dataset evaluation setting, compared to CEN trained with both datasets. The result of cross-dataset evaluation performing better than whole dataset training is somewhat counter-intuitive. Can you provide an explanation for these results?\n    2. What does the abbreviation of v.v. stands for? It would be better to mention the full name.\n3. Is there additional experimental results or analyses that could show CTN and CEN contributes to better cause-and-effect modeling besides results on MSVD and MSRVTT? Since the quantitative results of Tables 1 and 2 are not directly aimed at measuring cause-and-effect capability, some additional experiments may be required.\n4. These are some **minor** issues about the presentation of the work. These are **not** critical weaknesses, and somewhat personal opinions.\n    1. Personally, I think the example in Figure 1 is not very intuitive to explain the importance of causal-temporal narrative. Cause: “a car drove recklessly” and Effect: “the car was severely damaged” seems to be a valid cause-and-effect event, but the following event of “a group of guys started playing beer pong” seems like it’s not a direct effect of ‘cause'. This is a personal opinion, but you could consider finding a better example to demonstrate the idea.\n    2. I think ‘dataset’ instead of ‘benchmark’ might be a better word for CTN, as it is only used for training but not for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dONYuzCDfe", "reviewer": "Reviewer_8P6x", "summary": "The paper highlights that current video captioning benchmarks lack causal-temporal narratives and proposes a new benchmark to address this gap. This benchmark is generated by prompting a large language model (LLM) with a series of captions extracted from a video. To ensure relevance, EMScore is adapted to estimate the similarity between the generated captions and the video, discarding samples with low scores. The paper further introduces a two-stage framework to generate causal-temporal narrative (CTN) captions. In the first stage, two contrastive losses are applied to train Causal and Effect Video Encoders with their respective captions. In the second stage, the encoded causal and effect features are concatenated to decode a cohesive caption.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a benchmark focused on causal-temporal narratives, adapting an evaluation score for automated assessment.\n\nIt introduces a novel captioning model that encodes causal and effect information separately, then fuses them to decode the final caption.\n\nExperiments demonstrate that the proposed CEN framework outperforms comparative methods in performance.", "weaknesses": "The paper includes a human evaluation of the generated CTN captions; however, the assessment is less convincing due to the limited sample size. Despite the large dataset size—approximately 300,000 samples in MSRVTT and MSVD—only 100 samples were selected for human evaluation, reducing the reliability of the results.\n\nFor supervision during caption decoding, the paper simply concatenates the causal and effect text, which disrupts the original sentence grammar and lowers the quality of the synthetic supervision. The rationale behind this approach is unclear. An alternative method could involve inputting the causal and effect text into an LLM to fuse them into a more natural and coherent sentence. With ground truth captions generated by simple concatenation, evaluation metrics such as SPICE and CIDEr may not accurately reflect the quality of the generated captions. This approach could lead to discrepancies between metric scores and the true readability and coherence of the captions.\n\nThe structure of the feature encoder in stage 2 is not clearly depicted. Typically, encoded features from transformer encoders consist of a series of representations. It is unclear whether h_cause and h_effect are lists of representations or single representations. If h_cause and h_effect are single representations, it is not specified how they are generated—whether by selecting a specific token, or by applying mean pooling or max pooling. Furthermore, the specific dimension along which these features are concatenated is not detailed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50nRH0gT2t", "reviewer": "Reviewer_8bE2", "summary": "This paper tackles the problem of captioning videos taking into account causal-temporal narratives. It does so by presenting two main contributions: a dataset and benchmark (CTN) with causes and effects per video, and a network architecture (CEN) for extracting such narratives directly from videos.\n\nWhen applying the CTN benchmark to CEN, the network gets better metrics when compared to similar methods, potentially establishing it as SOTA in the domain.", "questions": "* The fatality video may be too graphic for some readers. I suggest switching it to a video on another topic.\n\n* In figure 5, CTN captions appear too similar to CEN. Is this a symptom of overfitting?\n\n* What does the system do when a video with no apparent cause and effect is used as input? For example, a person is singing but there's no audience reaction. Or a person is just sitting in a chair. The paper mentions \"our method assumes a certain level of causal and temporal coherence within the video content\"; maybe it would benefit from providing a probability value on the confidence for the existence of cause and effect in the input video?\n\n* `As expected, the resulting captions (Figure 6) are of low quality`, how is this measured?\n\n* Has the team considered the encoder of effect to also receive the encoded cause as input? How would that compare with the current approach?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance:\n\n* The problem discussed is important and relevant to much of the field of video captioning and video generation. It's true that typical captioning methods don't typically expose causal information, which makes generated captions much less useful than they could be.\n\nOriginality:\n\n* The work appears original though the novel aspects of it are a bit of straightforward applications of existing methods.\n\nQuality:\n\n* The work produces good results. While the paper discusses both results on existing benchmarks, and in CTN, the former being more interesting because they've been established independently.\n* I appreciate that both automatic benchmarks and human evaluations were used to evaluate this work.\n* The proposed CEN network architecture is interesting and original.\n\nClarity:\n\n* The work is mostly clear, except for some aspects that are discussed in the weaknesses section.", "weaknesses": "* The automatic evaluation process is found to filter out most bad cause and effect generations. More complete analysis on the distribution of the qualities of the generated captions is missing. I can imagine a histogram that displays the distribution of qualities with a vertical line showing the threshold that's been picked. Also the paper would benefit from analysis on whether the automatic evaluation filtering improves on the method, or is a patch for more fundamental issues in the prior step.\n\n* The contribution of CTN depends strongly on the prompt used to zero shot cause and effect generation. Given that this prompt is so central to the paper, the paper would benefit from more complete ablations regarding different aspects of the prompt. Especially important, the list of requirements that are presented in the prompt are justified in the paper but no corresponding ablation per requirement is done to validate the explanation. (the ablation donefor 4 prompts of varying degrees of quality appears insufficient for this purpose)\n\n* When building CTN, the captions are 1) generated from frames, and then 2) a model combines multiple captions into cause and effect captions without looking at frame information again until 3) the automatic evaluation step. Has the team considered using a model that can read text + image in step 2 as well? \n\n  Concluding cause and effect just from the individual frame captions is a lot more error prone than it could be if image information was also used. I think this possibility could be mentioned / analyzed as a potential alternative.\n\n* Why are causes and effects only separated by a space in CTN? Have other choices been ablated? In particular, I can imagine using a \"comma\" or an \"and\" between both to work slightly better.\n\n    * Has the team considered using an LLM to combine both captions together, instead of using concatenation?\n\n* For A5.1: more discussion on the human evaluation process would be very beneficial. How were the experts chosen? How is information presented to them? How were the 100 videos chosen?\n\n  For example it would be ideal to have an independent expert set; and it would be ideal if they were presented with the captions without labels that identify the models, and in such a way that each the models are independently ordered for each video (so annotators cannot be biased to always vote for the first model, for example)\n\n* Clarity: The paper, especially in the first few pages, is a bit too repetitive or verbose. It may benefit from some proofreading for succinctness. For example, the following two paragraphs are very close to each other, and both express a similar idea:\n\n```\nIt means that our CTN benchmark differs significantly by focusing on generating comprehensive causal-temporal narratives, capturing broader temporal relationships within a single caption, and providing a more holistic view of video content\n```\n\n  and\n\n```\nWe introduce a new benchmark CTN specifically designed to capture and evaluate causal-temporal narrative in video captioning. Our approach goes beyond existing benchmarks by explicitly modeling causal-temporal narrative in a single, coherent caption, enabling a more comprehensive understanding of video content.\n```\n\n  In other cases ideas are repeated in the same paragraph:\n\n```\nHowever, as our experiments show (see Section 4.4), even these advanced models struggle with generating accurate causal-temporal narratives. To address this limitation, there is a need to develop new video captioning models that can overcome the shortcomings of current frameworks. This also underscores the need for specialized architectures like our proposed CEN that are explicitly designed to capture and generate causal-temporal narratives in videos\n```\n\n  one could summarize this text as:\n\n```\nHowever, as our experiments show (see Section 4.4), even these advanced models struggle with generating accurate causal-temporal narratives. This also underscores the need for specialized architectures like our proposed CEN that are explicitly designed to capture and generate such narratives in videos\n```\n\n  Such verbosity makes the paper hard to read, despite the main ideas being explained very clearly.\n\n* The paper would benefit from a more complete discussion on possible downsides of this approach. For example for CTN how it compares to using frame data in step 2 as discussed above, or what downsides could arise from the concrete proposed architecture.\n\n* The paper mentions surveillance; maybe analyze this in the context of ethic concerns?\n\n* `We acknowledge potential ethical concerns (e.g., privacy in surveillance, risk of misleading content) but are committed to responsible development`, could this be made more concrete?\n\n* The paper mentions `lays the foundation for a paradigm shift`, which appears potentially grandiose in my opinion. Better to let the community arrive to this conclusion, than write it in the text.\n\nI'll be glad to revise my score depending on the response to this review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DAI8uvffAH", "reviewer": "Reviewer_YiKX", "summary": "This paper focuses on incorporating causal-temporal narrative into video captioning.  The authors create a new dataset called Causal-Temporal Narrative (CTN) captions, using a LLM to generate captions that explicitly describe cause-effect relationships and maintain temporal consistency within videos based on multiple annotated ground-truths for the same video. The quality of these CTN captions is assessed both automatically (using EMScore) and through human evaluation.\n\nFurthermore, the authors propose a novel model architecture called the Cause-Effect Network (CEN), specifically designed to learn from the CTN captions. CEN decouples the cause and effect captions by finetuning one visual encoder for each of them. The representations of both encoders are then used during the LLM finetuning stage.\n\nExperiments on MSVD and MSR-VTT datasets demonstrate that CEN significantly outperforms state-of-the-art video captioning methods for generating CTN captions. Ablation studies further validate the design choices of CEN.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- While most of the causal video understanding focuses on video question answering, this paper proposes to bring causal understanding into video captioning. \n- The data generation pipeline looks sound.\n- The proposed approach achieves strong results on the proposed benchmark.", "weaknesses": "- The weakest ablated baseline already performs well above state-of-the-art approaches. It would be great to understand what the differences are. For instance, how does a baseline with the CLIP ViT features used directly perform?\n- The presentation can be improved, for instance it is not clear that the evaluation is on the CTN captions in Table 1, and the ablation study includes method names which are hard to interpret when having a first look at the table.\n- The data generation approach largely relies on the multiple GT captions present for each video in MSR-VTT and MSVD. \n- Captioning metrics which do not move much like Rouge-L are hard to interpret.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)