NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative

## Abstract

Abstract Existing video captioning benchmarks and models lack coherent representations of causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models’ ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, we propose NarrativeBridge, an approach comprising of: (1) a novel Causal-Temporal Narrative (CTN) captions benchmark generated using a large language model and few-shot prompting, explicitly encoding cause-effect temporal relationships in video descriptions, evaluated automatically to ensure caption quality and relevance and validated through human evaluation; and (2) a dedicated Cause-Effect Network (CEN) architecture with separate encoders for capturing cause and effect dynamics independently, enabling effective learning and generation of captions with causal-temporal narrative. Extensive experiments demonstrate that CEN significantly outperforms state-of-the-art models, including fine-tuned vision-language models, and is more accurate in articulating the causal and temporal aspects of video content than the second best model (GIT): 17.88 and 17.44 CIDEr on the MSVD and MSR-VTT datasets, respectively. Cross-dataset evaluations further showcase CEN’s strong generalization capabilities. The proposed framework understands and generates nuanced text descriptions with intricate causal-temporal narrative structures present in videos, addressing a critical limitation in video captioning. For project details, visit https://narrativebridge.github.io/ .

## 1 Introduction

Video captioning aims to generate textual descriptions that capture the visual information and temporal dynamics in videos Venugopalan et al. (). Research has primarily focused on developing new methods to improve the accuracy of video captioning models on well-established benchmark datasets, MSR-VTT Xu et al. () and MSVD Chen & Dolan (). State-of-the-art (SOTA) approaches Iashin & Rahtu (); Tian et al. (); Xu et al. (); Nadeem et al. (); Wang et al. (); Chen et al. () proposed new architectures to better align the generated captions with the provided ground-truth. However, these efforts enhance models’ accuracy on existing evaluation criteria, without modifying the benchmark datasets or their ground truth annotations to address the lack of coherent representations of causal-temporal narrative - construction and interpretation of a sequence of events linked through cause and effect, unfolding over time and space, and often driven by entities (characters or agents) acting with intention Wilkens et al. ().

As discussed by Wilkens et al. (), causal relationships Granger () are scenarios where one event (cause) directly influences the occurrence of another event (effect), while temporal understanding Berrevoets et al. () involves recognizing the chronological order of events, a crucial component in establishing causality. Existing ground-truth captions in popular benchmarks lack causal, temporal and narrative information. Figure 1 illustrates the importance of causal-temporal narrative on the MSR-VTT dataset Xu et al. (). In the input video sequence, a car is first shown driving recklessly through an open field and flipping over, which represents the cause event. This is then followed by the effect event showing the severely damaged car, and subsequently, a group of guys starting to play beer pong, which represents the temporal sequence. The causal-temporal narrative would connect these events in a temporal order, linking the reckless driving and resulting car crash to the group’s subsequent behavior of playing beer pong. This example aligns with the definition of narrative provided by Wilkens et al. Wilkens et al. (), which emphasizes the role of cause-effect relationships, the perception of narrativity in videos, and highlights the temporal sequence of events that is crucial for understanding the causal-temporal narrative. Figure 1 highlights the limitations of the original captions in existing benchmark datasets, such as MSR-VTT Xu et al. (). The original captions focus on isolated events or actions, such as "a car flipping over" or "a monster truck flips on its side," lacking contextual narrative and causal-temporal relationships between events. Consequently, models trained on these captions suffer from the same limitations.

Figure: Figure 1: Comparison of Original captions vs. Causal-Temporal Narrative (CTN) caption to illustrate the inclusion of causal-temporal narrative.
Refer to caption: x1.png

To bridge this gap, we introduce NarrativeBridge, a novel framework encompassing a new benchmark and architecture tailored for causal-temporal narrative learning in video captioning. Our Causal-Temporal Narrative (CTN) captions benchmark leverages a large language model (LLM) and few-shot prompting to generate enhanced video descriptions that explicitly encode cause-effect relationships and temporal sequences, as exemplified in Figure 1. Our CTN captions benchmark, generated using the approach detailed in Section 3, establishes a clear connection between the cause (reckless driving) and the effect (damaged car and subsequent behavior of the group). By generating a single, coherent caption that encapsulates the causal-temporal narrative of the video, our CTN captions benchmark enables models to better understand and articulate the causality, sequence, and significance of events within the broader video context Wilkens et al. ().
This approach addresses the limitations of existing benchmark datasets and emphasizes the importance of incorporating causal-temporal narrative understanding into video captioning models to generate accurate, informative, and contextually relevant descriptions. To ensure the quality and relevance of the generated captions, we employ an automatic evaluation framework that compares the CTN captions with the video content, keeping or discarding them based on a score threshold. Additionally, we conduct a human evaluation study that further validates the high quality of our CTN captions, demonstrating their accuracy, temporal coherence, and relevance. This CTN captions benchmark addresses the limitations of existing benchmark datasets and emphasizes the importance of incorporating causal-temporal narrative understanding into video captioning models to generate accurate, informative, and contextually relevant descriptions.

Existing SOTA video captioning methods that use different architectures such as LSTM Nadeem et al. (), GNN Hendria et al. (), and Transformer Wang et al. (), struggle to effectively learn causal-temporal narrative from the CTN captions. These architectures are designed to capture the overall semantics in videos but lack dedicated mechanisms to explicitly model the cause-effect relationships and temporal sequences. As a result, the captions generated by these methods fail to articulate the complex causal-temporal dynamics in the videos, as demonstrated in results (Section 4.4).
To address this challenge, we propose the Cause-Effect Network (CEN), an architecture specifically designed to capture cause and effect dynamics using dedicated encoders. By separately encoding the entire video through the cause and effect encoders, without requiring explicit cause-effect segmentation, and then combining them to generate the final caption, CEN is able to better understand and articulate the causal-temporal narrative. The primary contributions of our work are:

- •
Introducing for the first time CTN captions benchmark with high-quality, coherent causal-temporal narrative captions, automatically evaluated and humanly validated, to address the lack of coherent representations in video captioning benchmarks.
- •
A novel CEN network for learning causal-temporal narrative from videos to address limitation of SOTA video captioning methods to learn narrative information on CTN.
- •
Extensive experiments demonstrating CEN’s superior performance on CTN, setting a new SOTA in causal-temporal narrative learning.

In addition, our work demonstrates strong generalization capabilities through cross-dataset evaluations and outperforms even fine-tuned SOTA vision-language models such as VideoLLaVA Lin et al. () and ShareGPT4Video Chen et al. () in generating causal-temporal narratives. Our work marks a significant step forward in video captioning research by explicitly addressing the challenges of causal-temporal narrative understanding. The CTN captions benchmark and the CEN architecture provide a comprehensive framework for learning and generating contextually relevant and temporally coherent video descriptions.

## 2 Related Work

### 2.1 Benchmarks

MSVD Chen & Dolan () is a benchmark focused on human activities, providing a platform for evaluating video captioning models. The captions in MSVD often describe the observable actions without delving into the underlying motivations or the cause-effect relationships between the events.
MSR-VTT Xu et al. () is a large-scale benchmark with diverse video content, encompassing a wide range of topics and genres. The captions often focus on describing the observable content without capturing the causal links between the events or the temporal progression of the narrative. As a result, models trained on MSVD and MSR-VTT may struggle to generate descriptions that accurately reflect the causal and temporal dynamics in the videos.

While recent benchmarks like NExT-QA Xiao et al. () and EgoSchema Mangalam et al. () have made significant strides in incorporating causal and temporal reasoning in video understanding, they focus primarily on question-answering tasks rather than generating comprehensive causal-temporal narratives. NExT-QA introduces multi-choice and open-ended question-answering tasks that target causal action reasoning but do not address the generation of holistic video descriptions. It focuses on specific question-answer pairs that often target single events or actions. In contrast, our CTN captions provide a more comprehensive narrative that captures the causal and temporal relationships across the entire video sequence. It means that our CTN benchmark differs significantly by focusing on generating comprehensive causal-temporal narratives, capturing broader temporal relationships within a single caption, and providing a more holistic view of video content (see Appendix A.7 for a detailed comparison). EgoSchema Mangalam et al. (), on the other hand, emphasizes long-form video understanding and temporal reasoning but does not explicitly focus on causal-temporal narrative for video captioning.

Similarly, efforts like VCR Zellers et al. (), V2C Fang et al. (), and Motivation Vondrick et al. () integrate causality into their analysis of visual description or question-answering, relying heavily on commonsense reasoning for generating predictions. VCR Zellers et al. () focuses on visual commonsense reasoning, V2C Fang et al. () aims to generate commonsense descriptions for video captioning, and Motivation Vondrick et al. () explores the prediction of motivations behind actions in videos. However, these works primarily rely on commonsense reasoning and do not delve into the causal and temporal structures underpinning video narratives.
We introduce a new benchmark CTN specifically designed to capture and evaluate causal-temporal narrative in video captioning. Our approach goes beyond existing benchmarks by explicitly modeling causal-temporal narrative in a single, coherent caption, enabling a more comprehensive understanding of video content.

### 2.2 Video Captioning

Video captioning techniques have evolved from LSTM-based Gao et al. (); Song et al. (); Nadeem et al. () frameworks to the latest designs using SOTA GNNs Hendria et al. (); Zhang et al. (); Pan et al. () and Transformers Wang et al. (); Lin et al. (); Yang et al. (), with a focus on enhancing the complexity of captions through the injection of multimodal data. Despite these advancements, current architectures often struggle to capture the intricate temporal sequences and causal relationships in video storytelling. To bridge this gap, video captioning can benefit from cross-fertilization with ideas and strategies developed in related fields, such as action recognition Sun et al. (); Wang et al. (); Kazakos et al. (); Xiao et al. (); Chen & Ho (); Gao et al. (); Panda et al. (); Sardari et al. (); Alfasly et al. (); Planamente et al. (); Zhang et al. (), event localization Tian et al. (); Lin et al. (); Duan et al. (); Lin et al. (), and question-answering Alamri et al. (); Hori et al. (); Schwartz et al. (); Geng et al. (); Yun et al. (); Li et al. (); Shah et al. (); Nadeem et al. ().
The integration of causal reasoning Liu et al. (); Xue et al. () has shown promise in enhancing the ability of neural networks to discern causal relationships, leading to improved performance in image captioning Liu et al. () and visual question answering Xue et al. (). However, current SOTA models still struggle to effectively handle the narrative complexity in videos, particularly in terms of causal and temporal progression.

Recent advancements in vision-language models (VLMs) such as VideoLLaVA Lin et al. () and ShareGPT4Video Chen et al. () have shown promising results in various video understanding tasks. However, as our experiments show (see Section 4.4), even these advanced models struggle with generating accurate causal-temporal narratives. To address this limitation, there is a need to develop new video captioning models that can overcome the shortcomings of current frameworks. This also underscores the need for specialized architectures like our proposed CEN that are explicitly designed to capture and generate causal-temporal narratives in videos.

In light of these challenges, our work aims to bridge the gap in existing video captioning benchmarks and models by introducing NarrativeBridge, a comprehensive framework that encompasses the CTN captions benchmark and the CEN architecture. By explicitly addressing the limitations of current approaches and providing a platform for causal-temporal narrative learning, our work paves the way for the development of more sophisticated video captioning models capable of generating contextually relevant and temporally coherent descriptions.

## 3 Method

NarrativeBridge is a comprehensive approach to video captioning that addresses the limitations of existing benchmarks and models in capturing causal-temporal narrative. It consists of two key interconnected components: (i) the CTN captions benchmark, which provides a rich representation of cause-and-effect relationships and event sequences in video content, and (ii) the CEN architecture, which is specifically designed to learn and articulate these causal-temporal narrative elements.

Problem Statement:
Existing video captioning benchmarks and models struggle to coherently capture the intricate causal-temporal narrative inherent in video content. As shown in Figure 1, original captions from benchmark datasets often focus on isolated events, failing to capture the causal and temporal relationships that drive the video’s narrative. SOTA video captioning methods also fall short in articulating the cause-and-effect relationships and temporal sequences that drive the events in the video (see Section 4.4).

Figure: Figure 2: CTN caption generation pipeline. $\theta$ indicates a threshold.
Refer to caption: x2.png

### 3.1 CTN Captions Benchmark

To address the challenges of existing benchmarks in representing causal and temporal relationships within video content, we propose an approach that harnesses the potential of LLMs through the few-shot prompting technique Brown et al. (). Our method generates CTN captions without the need for model fine-tuning, leveraging the inherent generative capabilities of LLMs to produce captions that encapsulate causal-temporal narrative structures. Figure 2 shows the CTN caption generation pipeline, which consists of two key steps: prompt design, LLM-based caption generation and evaluation.

Prompt Design: The prompt design step is crucial in guiding the LLM to generate causal-temporal narrative captions. The input to the CTN caption generation are the original video captions from existing benchmarks. We design a prompt that include a small set of carefully selected example captions, illustrating the desired output structure and highlighting the key aspects of causal-temporal narrative. We design Prompt 1 which guides LLM to generate accurate and concise causal-temporal narrative. The prompt sets clear requirements for grounding the captions in the provided descriptive context, maintains temporal consistency, and avoids unsupported details or conclusions. The illustrative examples demonstrate the desired format, facilitating the generation of plainly written, length-constrained Cause and Effect statements that effectively capture the video’s causal-temporal narrative. By explicitly outlining instructions and constraints, the carefully designed prompt steers the LLM’s generative capabilities towards producing contextually relevant and narratively coherent CTN captions. Further details are provided in Appendix A.2 and A.3.

Figure: Prompt 1: LLM Prompt used in our CTN captions generation.

CTN Caption Generation and Automatic Evaluation: We send Prompt 1 into the Mixtral of Experts LLM Jiang et al. (), which generates CTN captions that accurately reflect the causal-temporal narrative in the video. The LLM’s advanced natural language understanding and generation capabilities, combined with the few-shot prompting approach, enable the production of captions that encapsulate complex causal and temporal relationships.
To ensure the quality and relevance of the generated CTN captions, we employ the EMScore Shi et al. () metric for evaluation. EMScore directly measures the consistency between a caption and the video content, and has been shown to be more effective than other metrics de Souza Inácio & Lopes () in evaluating video relevance without referenced captions.
We set a threshold ($\theta$ = 0.2) for the EMScore value, indicating adequate relevance to the video. Captions with an EMScore above the $\theta$ are kept, while those below the $\theta$ are discarded, and the LLM generates a new caption. This iterative refinement process continues until a caption meets the EMScore threshold, ensuring that the final captions efficiently describe the relevant events in the video. Further details are provided in Appendix A.4.

Human Evaluation: To further validate the quality of our CTN captions, we conduct a human evaluation study. Five independent domain experts evaluate 100 randomly sampled captions (50 each from MSRVTT-CTN and MSVD-CTN) on three criteria: causal accuracy - the degree to which the caption correctly identifies and describes cause-effect relationships; temporal coherence - the extent to which the caption accurately represents the sequence of events; and relevance - how well the caption reflects the overall content. Each criterion is rated on a 5-point Likert scale. Following are the results: the overall mean quality score of 4.8 ($\sigma$ = 0.40) indicates high performance across all criteria; 93% of captions receive scores of 4 or higher across all three dimensions, with 84% achieving perfect scores. To ensure reliability across raters, we calculate the intraclass correlation coefficient (ICC). The ICC for absolute agreement among raters is 0.87 (95% CI: 0.83-0.91), indicating high inter-rater reliability. These results validate our automatic generation and evaluation process, demonstrating that our CTN captions benchmark provides high-quality, coherent representations of causal-temporal narratives in videos. Further details are provided in Appendix A.5.

Figure: Figure 3: The two-stage Cause-Effect Network (CEN) architecture. Stage 1: Separate Cause ($E_{cause}$) and Effect ($E_{effect}$) video encoders, pretrained using CLIP-ViT, learn specialized video representations. Corresponding text encoders ($T_{cause}$ and $T_{effect}$) encode the cause and effect portions of the CTN caption. Contrastive losses are applied to align the video and text embeddings. Stage 2: The learned cause and effect video features are encoded separately ($Enc_{cause}$ and $Enc_{effect}$) and concatenated before being input to the decoder, which generates the final CTN caption.
Refer to caption: x3.png

### 3.2 Proposed Cause-Effect Network (CEN)

Existing SOTA video captioning methods based on LSTMsNadeem et al. (), GNNsHendria et al. (), and TransformersWang et al. () lack dedicated mechanisms to explicitly model the intricate causal-temporal narrative sequences. Our proposed CEN employs a two-stage approach to address this limitation.
In Stage 1 (Section 3.2.1), CEN learns specialized cause and effect video representations using separate encoders trained on the corresponding portions of the CTN captions. The input is raw video frames from the entire video without requiring explicit segmentation of cause-effect regions, and the output is the learned cause and effect features extracted from the encoders.
The learned video features from Stage 1 are then passed to Stage 2 (Section 3.2.2), which utilizes a sequence-to-sequence transformer-based network with two new encoders to generate the final captions. The parameters of stage 1 encoders are frozen at this stage. The cause and effect video features are separately encoded and then concatenated before being input to the decoder. This allows the model to synthesize the specialized representations learned in Stage 1 and generate captions that accurately capture the causal and temporal relationships in the videos.

#### 3.2.1 Stage 1: Causal Visual Encoding

For the CEN network to learn causal-temporal narrative, we design two separate visual encoding models trained on two parts of the captions (see Figure 3).

(1) Cause Video Encoder $E_{cause}$: Trained on the cause captions part describing the initiating events;

(2) Effect Video Encoder $E_{effect}$: Trained on the effect part of the captions capturing the consequential outcomes.

We instantiate $E_{cause}$ and $E_{effect}$ as two separate instances of the CLIP-ViT (ViT-B/32) model Radford et al. (), adapted from CLIP4Clip Luo et al. () (see Appendix A.6 for more details). For both Cause and Effect text encoders ($T_{cause}$ and $T_{effect}$ respectively), we employ a 12-layer transformer with a hidden size of $512$ and $8$ attention heads, using the weights from the pre-trained CLIP Radford et al. () text encoder. As in CLIP Radford et al. (), the [EOS] token’s representation from the last layer of the text encoders is used as the feature representation for the input text.
Similar to CLIP4Clip, we adopt cosine similarity to measure the similarity between the video embedding (mean pooled) and the text embedding. Given a video embedding $\hat{r}_{i}$ and a text embedding $t_{j}$, the similarity function is defined as $s(\hat{r}_{i},t_{j})=\frac{t_{j}^{\top}\hat{r}_{i}}{|t_{j}||\hat{r}_{i}|}$.
During training, each model is optimized using a contrastive loss, as in CLIP Radford et al. (), over its respective cause/effect portion from CTN captions.
For a batch of $N$ video-cause text pairs $(v_{i},c_{i})$ and video-effect text pairs $(v_{i},e_{i})$, the contrastive losses for the cause and effect video encoders are defined as:

$$ $\mathcal{L}_{cause}=\mathcal{L}_{v2t}(E_{cause}(v_{i}),T_{cause}(c_{i}))+ \mathcal{L}_{t2v}(E_{cause}(v_{i}),T_{cause}(c_{i})),$ (1) $$

$$ $\mathcal{L}_{effect}=\mathcal{L}_{v2t}(E_{effect}(v_{i}),T_{effect}(e_{i}))+ \mathcal{L}_{t2v}(E_{effect}(v_{i}),T_{effect}(e_{i})),$ (2) $$

where $\mathcal{L}_{v2t}$ and $\mathcal{L}_{t2v}$ are the video-to-text and text-to-video contrastive losses, respectively, defined as:
$\mathcal{L}_{v2t}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(s(\hat{r}_{i},t_{i}
))}{\sum_{j=1}^{N}\exp(s(\hat{r}_{i},t_{j}))},$
(3)
$\mathcal{L}_{t2v}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(s(\hat{r}_{i},t_{i}
))}{\sum_{j=1}^{N}\exp(s(\hat{r}_{j},t_{i}))}$
(4)

After training, we freeze the weights of $E_{cause}$ and $E_{effect}$ for the second stage, where we extract the cause and effect video features, respectively.

#### 3.2.2 Stage 2: CTN Caption Generation

In the second stage (see Figure 3), we employ a sequence-to-sequence transformer-based network, consisting of two separate encoders ($Enc_{cause}$ and $Enc_{effect}$) for processing cause and effect video features, respectively, and a decoder $Dec$ for generating the final CTN captions. The encoded cause and effect video representations are concatenated before being sent to the decoder. Both encoders and the decoder are initialized with the weights from the pre-trained Uni-VL model Luo et al. ().
Given a video $v$, we first extract the cause and effect video features using the pre-trained $E_{cause}$ and $E_{effect}$ encoders:

$$ $F_{cause}=E_{cause}(v),$ (5) $$

$$ $F_{effect}=E_{effect}(v).$ (6) $$

The extracted features $F_{cause}$ and $F_{effect}$ are separately encoded using $Enc_{cause}$ and $Enc_{effect}$, each consisting of two transformer layers, to obtain the respective encoded representations: $h_{cause}=Enc_{cause}({F}_{cause})$ and $h_{effect}=Enc_{effect}({F}_{effect})$.

The encoded cause and effect representations are concatenated to form a single representation $h_{concat}=[h_{cause};h_{effect}]$, which is then input to the decoder $Dec$, consisting of two transformer layers, to generate the CTN caption $\hat{y}$ by attending to $h_{concat}$ at each time step $t$:

$$ $\hat{y}_{t}=Dec(h_{concat}],\hat{y}_{<t}),$ (7) $$

where $\hat{y}_{<t}$ denotes the previously generated words. For training and evaluation, we combine the cause and effect parts of the ground-truth CTN caption with a space in between to form a single, coherent caption. This combined caption serves as the target for the decoder during training and the reference for evaluating the generated captions using diffferent evaluation metrics.
The entire network is trained end-to-end using the cross-entropy loss between the generated caption $\hat{y}$ and the ground-truth CTN caption $y$:

$$ $\mathcal{L}_{caption}=-\frac{1}{T}\sum_{t=1}^{T}\sum_{c=1}^{|V|}y_{t,c}\log( \hat{y}_{t,c}),$ (8) $$

where:
$T$ is the length of the caption. $|V|$ is the size of the vocabulary. $y_{t,c}$ is the ground-truth label for the $c$-th word at time step $t$ (1 if the word is present, 0 otherwise). $\hat{y}_{t,c}$ is the predicted probability of the $c$-th word at time step $t$. Finally, our total loss $\mathcal{L}_{total}$ is:

$$ $\displaystyle\mathcal{L}_{total}=\mathcal{L}_{cause}+\mathcal{L}_{effect}+ \mathcal{L}_{caption}$ (9) $$

This two-stage approach first encodes the causal aspects into the video representations, facilitating the generation of coherent causal-temporal narrative descriptions in the second stage (see Experiments section 4) to generate CTN captions as final output.

## 4 Experiments

We conduct extensive experiments to validate our proposed CEN architecture and the new CTN video captioning benchmark. This section details the datasets used, the experimental setup, and the evaluation metrics. We also present comprehensive results and evaluation against the state-of-the-art video captioning methods and vision-language models (VLMs) demonstrating the performance of our network on the CTN benchmark.

### 4.1 Datasets

We evaluate our CTN captions and CEN on two widely-used video captioning datasets: 1) MSR-VTT Xu et al. (), consists of 10,000 videos with 200,000 human-annotated captions, and MSVD Chen & Dolan (), with 1,970 videos focused on human activities with 78,800 captions. The captions in these datasets lack explicit representations of the causal relationships, and temporal dynamics which we address by generating CTN captions as detailed in Section 3.1.

### 4.2 Evaluation Metrics

For quantifying the performance of our method CEN against the SOTA methods, we use three metrics as per Celikyilmaz et al. (); CIDEr (C) Vedantam et al. (): Measures alignment of generated captions with references by highlighting frequently occurring terms, capturing the ability to reproduce salient causal-temporal narrative elements. ROUGE-L (R-L) Lin (): Evaluates the longest common subsequence between generated and reference captions, considering both precision and recall for assessing semantic similarity, including temporal dynamics and causal relationships. SPICE (S) Anderson et al. (): Evaluates semantic quality by considering the overlap of scene graphs between generated and reference captions, effectively assessing the ability to capture and express underlying causal relationships and event sequences.

### 4.3 Implementation Details

We generate the CTN captions using the Mixtral of Experts LLM Jiang et al. (), running on A100-80GB GPUs. All models, including our proposed CEN architecture, the SOTA models (SEM-POS Nadeem et al. (), AKGNN Hendria et al. (), and GIT Wang et al. ()) and the VLMs VideoLLaVA Lin et al. () and ShareGPT4Video Chen et al. () are implemented using PyTorch. For the SOTA models, we follow the hyperparameter settings specified in their respective methods for training on the MSR-VTT and MSVD datasets.
Our CEN model is trained using the Adam optimizer with learning rates of $1\times 10^{-4}$ (stage 1) and $1\times 10^{-6}$ (stage 2) and a batch size of 64 for 10 epochs (stage 1) and 50 epochs (stage 2). For comparison with recent Vision-Language Models (VLMs), we fine-tune VideoLLaVA Lin et al. () and ShareGPT4Video Chen et al. () using both LoRA and simple fine-tuning approaches on our CTN benchmark. We use the recommended hyperparameters for each model during fine-tuning. Further details are provided in the Appendix A.1.

### 4.4 Results and Discussion

We comprehensively evaluate the performance of our proposed CEN architecture against several SOTA methods, including SEM-POS Nadeem et al. (), AKGNN Hendria et al. (), and GIT Wang et al. () and the VLMs VideoLLaVA Lin et al. () and ShareGPT4Video Chen et al. () on our generated CTN captions benchmark datasets.

| Table 1: Comparison of our CEN architecture against SOTA methods and VLMs on the MSVD and MSR-VTT datasets. The best results in each category are in bold. R-L, C, and S denote ROUGE-L, CIDEr, and SPICE scores, respectively.<br>Method<br>MSVD<br>MSRVTT<br>R-L $(\uparrow)$<br>C $(\uparrow)$<br>S $(\uparrow)$<br>R-L $(\uparrow)$<br>C $(\uparrow)$<br>S $(\uparrow)$<br>SEM-POS<br>25.39<br>37.16<br>14.46<br>20.11<br>26.01<br>12.09<br>AKGNN<br>25.11<br>35.08<br>14.55<br>21.42<br>25.90<br>11.99<br>GIT<br>27.51<br>45.63<br>15.58<br>24.51<br>32.43<br>13.70<br>VideoLLaVA (Zero-shot)<br>21.80<br>30.55<br>14.67<br>19.33<br>16.24<br>12.49<br>VideoLLaVA (LoRA FT)<br>24.56<br>34.98<br>15.41<br>21.21<br>18.97<br>13.28<br>VideoLLaVA (Simple FT)<br>25.61<br>36.12<br>16.09<br>22.18<br>19.98<br>13.07<br>ShareGPT4Video (Zero-shot)<br>21.66<br>27.06<br>14.06<br>20.27<br>17.08<br>12.21<br>ShareGPT4Video (LoRA FT)<br>24.39<br>30.72<br>14.83<br>22.09<br>19.83<br>13.02<br>ShareGPT4Video (Simple FT)<br>25.32<br>31.67<br>14.92<br>23.01<br>20.76<br>13.28<br>CEN (Ours)<br>31.46<br>63.51<br>19.25<br>27.90<br>49.87<br>15.76 | Table 2: Ablation study results on the MSVD and MSR-VTT datasets. $\text{E}_{\text{JointCE}}$, Only $\text{E}_{\text{cause}}$ and Only $\text{E}_{\text{effect}}$ are variants of our CEN architecture, while Zero Shot v.v. and Fine-tune v.v. represent cross-dataset evaluation settings. The best results in each category are in bold.<br>Method<br>MSVD<br>MSRVTT<br>R-L $(\uparrow)$<br>C $(\uparrow)$<br>S $(\uparrow)$<br>R-L $(\uparrow)$<br>C $(\uparrow)$<br>S $(\uparrow)$<br>$\text{E}_{\text{JointCE}}$<br>30.93<br>55.72<br>17.04<br>27.34<br>45.97<br>15.07<br>Only $\text{E}_{\text{cause}}$<br>30.72<br>56.42<br>18.43<br>27.19<br>47.10<br>15.21<br>Only $\text{E}_{\text{effect}}$<br>30.70<br>57.14<br>17.89<br>27.24<br>45.58<br>15.19<br>Zero Shot v.v.<br>27.16<br>39.65<br>14.45<br>24.73<br>29.76<br>12.26<br>Fine-tune v.v.<br>31.78<br>65.60<br>19.39<br>27.47<br>47.74<br>15.65<br>CEN (Ours)<br>31.46<br>63.51<br>19.25<br>27.90<br>49.87<br>15.76 |
| --- | --- |

Figure: Figure 4: (a) UMAP visualization of video features learned from CTN (red) and original (blue) captions on MSR-VTT, showing non-overlapping feature spaces. (b) UMAP visualization of video features learned from cause (black) and effect (orange) parts of CTN captions on MSR-VTT, showing near-complete overlap.
Refer to caption: x4.png

#### 4.4.1 Quantitative Results

CEN outperformed all SOTA methods and the VLMs across all metrics and datasets as shown in Table 2. On MSVD, CEN surpassed the next best (GIT) by 3.95 ROUGE-L, 17.88 CIDEr, and 3.67 SPICE points. On MSR-VTT, CEN led GIT by 3.39 ROUGE-L, 17.44 CIDEr, and 2.06 SPICE points. These significant gains highlight CEN’s effectiveness in capturing causal narratives and temporal dynamics.

We compare CEN’s performance against zero-shot and fine-tuned (FT) versions of VideoLLaVA and ShareGPT4Video. CEN consistently outperforms these models, even after fine-tuning, demonstrating the effectiveness of our specialized architecture for causal-temporal narrative generation. For example, on MSR-VTT, CEN achieves a CIDEr score of 49.87, compared to 19.98 for VideoLLaVA (Simple FT) and 20.76 for ShareGPT4Video (Simple FT).

To demonstrate generalization capability, we perform cross-dataset evaluations (Table 2). CEN shows strong zero-shot performance (Zero Shot v.v.) and significant improvements after fine-tuning (Fine-tune v.v.), outperforming models trained directly on the target datasets. For instance, when trained on MSR-VTT and tested on MSVD, CEN achieves a CIDEr score of 65.60 after fine-tuning, compared to 45.63 for GIT trained directly on MSVD.

To gain further insights into the effectiveness of the CTN captions in capturing causal-temporal narratives, we visualize the video feature representations learned by CLIP-ViT encoders trained on CTN and Original captions using UMAP McInnes et al. () dimensionality reduction from high dimension onto a 2D plane.
In Figure 4(a), we compare the representations learned from CTN captions (red) and Original captions (blue) on the MSR-VTT dataset. The non-overlapping feature spaces indicate that CTN captions capture the causal and temporal relationships not present in the original captions. In Figure 4(b), we visualize the representations learned from the cause (black) and effect (orange) parts of the CTN captions. The near-complete overlap suggests a strong correlation between the cause and effect components, aligning with the inherent structure of causal-temporal narratives and supporting the design of the CEN architecture.

Figure: Figure 5: Qualitative examples across scenarios like video games, paper folding, soccer, and singing. CEN (Ours) captions accurately capture causal narratives and temporal sequences from ground truth, outperforming SOTA video captioning methods.
Refer to caption: x6.png

#### 4.4.2 Qualitative Results

Qualitative examples, in Figure 5, demonstrate CEN’s strength in accurately articulating causal relationships and event sequences, while SOTA methods and recent VLMs struggle. For example:
(a) Video game scene: CEN accurately captures the causality ("performs a fatality move" causing "killing another"), while other methods miss this crucial relationship. SEM-POS and GIT provide overly simplistic descriptions, while AKGNN generates irrelevant details. VLMs (Video-LLaVA and ShareGPT4Video) misinterpret the scene entirely.
(b) Paper folding: CEN correctly identifies both the cause (folding paper) and effect (creating a paper airplane). Other methods either miss the causal relationship or misinterpret the action (e.g., ShareGPT4Video’s "small square" instead of an airplane).
(c) Soccer scene: CEN accurately links the precise kick to the goal, capturing both cause and effect. Other methods either focus on a single action or provide generic descriptions of a soccer game.
(d) Singing performance: CEN captures the boy’s decision to perform (cause) and the audience’s reaction (effect). Other methods mostly describe the act of singing without the causal-temporal context.

Overall, quantitative and qualitative results showcase CEN’s superior performance in understanding and generating causal-temporal narrative video captions compared to existing SOTA methods and recent VLMs. More results are provided in Appendix A.9.

#### 4.4.3 Ablation Results

To examine the effectiveness of CEN, we compare it with 4 baselines in Table 2:

a) $\text{E}_{\text{JointCE}}$: Instead of using two separate video encoders for cause and effect, we train one CLIP-ViT using the combined cause and effect captions separated by space for the text encoder. The performance drops across all metrics and datasets compared to the CEN architecture, which underscores the benefits of dedicated encoders for capturing cause and effect dynamics.

b) Only Ecause and Only Eeffect: These ablations employ only the cause encoder or only the effect encoder, respectively. While retaining reasonable performance, both variants fall short of the performance of the full model across all metrics and datasets.

c) Zero Shot v.v.: In this zero-shot setting, we evaluate the models trained on one dataset (MSVD or MSR-VTT) against the other dataset, without any fine-tuning on the other dataset. The substantial performance drop highlights the importance of dataset-specific fine-tuning for effective causal-temporal narrative learning.

d) Fine-tune v.v.: Building upon the zero-shot case, this variant involves fine-tuning the models trained on one dataset (MSVD or MSR-VTT) against the other dataset. Notably, this fine-tuning process leads to improved performance on the MSVD dataset, even surpassing the CEN model. This observation demonstrates the potential for transfer learning from larger datasets and the ability to leverage their knowledge effectively on smaller datasets through fine-tuning.

Overall, the ablation study validates the efficacy of our proposed CEN architecture, emphasizing the significance of dedicated encoders for capturing cause and effect relationships independently. The performance drops observed in $\text{E}_{\text{JointCE}}$, Only $\text{E}_{\text{cause}}$, and Only $\text{E}_{\text{effect}}$ results, compared to the CEN model, clearly indicate the necessity of explicitly embedding causal-temporal narrative understanding within the architecture through dedicated cause and effect encoders.

## 5 Limitations

While our work represents a significant step forward in causal-temporal narrative video captioning, our method assumes a certain level of causal and temporal coherence within the video content. Highly complex or convoluted causal relationships may challenge our approach like existing approaches, necessitating further architectural enhancements for improved generalization. Also, the current work primarily focuses on understanding and generating captions that capture causal narratives and temporal dynamics. Explicit integration of spatial reasoning, multi-agent interactions, and long-term dependencies could further enhance the robustness and applicability of our approach. Despite these limitations, our work opens up new avenues for innovative applications and research directions, further solidifying the importance of causal-temporal narrative understanding in video analysis tasks.

## 6 Conclusion

In this paper, we introduced for the first time a Causal-Temporal Narrative (CTN) captions benchmark and proposed a novel Cause-Effect Network (CEN) tailored for causal-temporal narrative video captioning.
This work finds vast applications in automated video summarization, question-answering, assistive technologies, surveillance, and educational content creation.
The CTN captions benchmark provides a comprehensive testbed for evaluating video understanding models’ ability to grasp complex temporal and causal dynamics, which will be released for research purpose on [https://narrativebridge.github.io/](https://narrativebridge.github.io/). And CEN explicitly models cause-effect relationships and temporal dynamics to generate rich, contextually relevant descriptions capturing nuanced causal-temporal narrative in videos, demonstrating significant performance improvement over SOTA methods (Section 4).
NarrativeBridge lays the foundation for a paradigm shift, where models comprehend underlying causal-temporal narrative driving events, unlocking new frontiers in contextually aware human-machine interactions with video.

For future work, we aim to integrate CTN caption generation with existing image captioning techniques, to annotate unlabeled videos with
causal-temporal narrative labels. A few frames of the video will be labelled using off-the-shelf image captioning methods, and CTN caption generation will exploit the labelled frames to generate one coherent caption for the unlabelled video (see Appendix A.8). This synergistic approach opens new avenues for comprehensive video understanding and annotation, enabling more robust and accurate video analysis pipelines.

## Acknowledgement

This research was partly supported by the British Broadcasting Corporation Research and Development (BBC R&D), Engineering and Physical Sciences Research Council (EPSRC) Grant EP/V038087/1 “BBC Prosperity Partnership: Future Personalised Object-Based Media Experiences Delivered at Scale Anywhere”.

## Appendix A Appendix

### A.1 Implementation Details

We first extract frames from the video clip and then encode them using the Cause and Effect Video Encoders. We adopt a sampling strategy where we sample one frame per second, with a maximum of 20 frames. This approach ensures that the sampled frames cover the entire duration of the video, providing a comprehensive representation of the video content while maintaining computational efficiency. For the fine-tuning experiment in Table 2, all the hyperparameters remain the same except the learning rate i.e. 0.0000005.
For our experimentation of the Cause and Effect Network (CEN), we use standard splits of the MSVD Chen & Dolan () and MSRVTT Xu et al. () datasets. For MSVD, we use 1200 videos for training, 100 for validation, and 670 for testing. For MSRVTT, we use 6513 videos for training, 497 for validation, and 2990 for testing. These splits are commonly used in video captioning research to ensure fair comparison with other methods.
All the experiments in Stage 1 and Stage 2 of our CEN are run using A100-80GB and RTX 3090-24GB GPUs respectively. We implement SEM-POS Nadeem et al. (), AKGNN Hendria et al. () and GIT Wang et al. () using RTX 3090-24GB, A100-80GB and A100-80GB GPUs respectively.

For comparison with Vision-Language Models (VLMs), we implement two fine-tuning approaches: LoRA Fine-Tuning and Simple Fine-Tuning using A100-80GB GPUs. LoRA Fine-Tuning is applied specifically to the LLM component, with a learning rate of 2e-4 for LoRA parameters. Simple Fine-Tuning is applied to the entire model, using an AdamW Loshchilov () optimizer with a cosine learning rate schedule (initial learning rate: 1e-3, warmup ratio: 0.03).

### A.2 Prompt Design Process

In Section 3.1, we introduce our approach for generating Causal-Temporal Narrative (CTN) captions using a large language model (LLM) and few-shot prompting. The prompt design played a crucial role in guiding the LLM to generate captions that accurately capture the cause-effect relationships and temporal dynamics in the video content. Figure 1 showcases the effectiveness of our few-shot based prompt, which resulted in a coherent and contextually relevant CTN caption for the given video.
To further illustrate the importance of prompt design and the benefits of few-shot learning, we conduct additional experiments with zero-shot prompting. In these experiments, we evaluate the performance of the LLM in generating CTN captions without providing any example captions in the prompt.
Zero-Shot Prompting: Zero-shot prompting refers to the process of providing large language models (LLMs) with a task prompt without any accompanying examples or demonstrations, requiring the model to generate relevant responses based solely on its pre-trained knowledge and understanding of the prompt. First, we aim to evaluate the performance of LLMs in generating causal-temporal narrative (CTN) captions for videos without providing any few-shot examples.
To achieve this, we construct 4 distinct prompts, each with varying degrees of quality, and experimented with all of them to assess their effectiveness in a zero-shot setting, starting with the most challenging prompt and progressively improving the quality.
Prompt 2 presents an extremely terse and poorly structured set of instructions. It contains minimal context and information, merely instructing the model to "Make Cause Effect thing from captions" and listing a set of numbered requirements without providing clear explanations. As expected, the resulting captions (Figure 6) are of low quality and often lack coherence and relevance to the video content. Moreover, the LLM generated multiple captions instead of a single coherent caption, and the output format does not adhere to the specified structure (a single JSON object containing the Cause and Effect statements).

Figure: Prompt 2: Zero-shot LLM Prompt with minimal context and unexplained requirements for generating cause-effect from captions.

Figure: Figure 6: Caption generated using LLM for the example in Figure 1 using Prompt 2

We then progressively improve the prompt quality, moving to a more concise set of instructions in Prompt 3 and a relatively well-structured prompt in Prompt  4. While the captions generated by these improved prompts (Figures 7 and 8) show some enhancements, they still fall short in terms of capturing the causal-temporal narrative accurately and consistently. Additionally, the LLM continues to generate multiple captions and deviate from the desired output format.

Figure: Prompt 3: Zero-shot LLM Prompt with reasonable structure conveying core task of creating cause-effect scenario from captions.

Figure: Figure 7: Caption generated using LLM for the example in Figure 1 using Prompt 3

Figure: Prompt 4: Zero-shot LLM Prompt clearly describing caption-based cause-effect generation task with specified requirements.

Figure: Figure 8: Caption generated using LLM for the example in Figure 1 using Prompt 4

Finally, we experiment with a zero-shot prompt (Prompt 5) that closely resembles our few-shot based Prompt 1 from Section 3.1, but without the example captions. The captions generated by this prompt (Figure 9) demonstrate improved coherence and relevance compared to the previous zero-shot prompts. However, they still lack the level of detail, accuracy, and contextual understanding exhibited by the captions generated using our few-shot based prompt (Prompt 1). Furthermore, the LLM persists in generating multiple captions.

Figure: Prompt 5: Zero-Shot Prompting for Causal Narrative Caption Generation

Figure: Figure 9: Caption generated using LLM for the example in Figure 1 using Prompt 5

These experiments highlight the significance of effective prompt design and the advantages of few-shot learning in generating high-quality CTN captions. By providing carefully crafted example captions in the prompt, we enable the LLM to better understand the desired format, style, and level of detail for the generated captions. The few-shot based prompt allows the LLM to learn from and mimic the provided examples, resulting in captions that accurately capture the causal-temporal narrative of the video content.

Crucially, the few-shot based prompt also guides the LLM to generate a single coherent caption in the specified JSON format, ensuring consistency and usability of the generated captions.
In contrast, the zero-shot prompts, even when well-structured, rely solely on the LLM’s pre-existing knowledge and understanding of the task description. While the LLM can generate relevant captions to some extent, the lack of explicit examples limits its ability to produce captions with the same level of coherence, accuracy, and contextual understanding as those generated using the few-shot based prompt. Moreover, the zero-shot prompts fail to guide the LLM toward generating a single caption in the desired JSON format, resulting in multiple captions and inconsistent output structures.

These findings underscore the importance of our few-shot based prompt design in the CTN caption generation process and demonstrate its superiority over zero-shot prompting approaches. By leveraging the power of few-shot learning, we enable the LLM to generate high-quality CTN captions that effectively capture the cause-effect relationships and temporal dynamics in video content, as evidenced by the results presented in Figure 1. Additionally, our few-shot based prompt ensures that the generated captions adhere to the specified JSON format, facilitating seamless integration and usability in downstream applications.

### A.3 Comparison of Large Language Models for CTN Caption Generation

To evaluate the effectiveness of large language models (LLMs) in generating high-quality Causal-Temporal Narrative (CTN) captions, we compare the performance of two open source state-of-the-art LLMs at the time of experimentation: Mixtral of Experts Jiang et al. (), which we utilized in our CTN caption generation pipeline, and Llama2-70b Touvron et al. ().

Figure: Figure 10: Comparison of Mixtral of Experts and Llama2-70b on a video game sequence involving a fatality move.
Refer to caption: x7.png

We provide the original video captions as input to both models and assess the quality of their generated CTN captions. In the video game example (Figure 10), Mixtral of Experts accurately captures the causal relationship between the fatality move performed by one character and the consequent death of another character in the game. In contrast, Llama2-70b focuses on a specific effect (blood spurting from the character’s neck) without explicitly linking it to the cause.

Figure: Figure 11: Comparison of Mixtral of Experts and Llama2-70b on a paper folding tutorial to create a paper airplane.
Refer to caption: x8.png

For the paper folding tutorial (Figure 11), both models correctly identify the causal relationship between folding the paper and creating a paper craft. However, Mixtral of Experts provides a more precise description, specifying that the paper craft being created is a paper airplane.

Figure: Figure 12: Comparison of Mixtral of Experts and Llama2-70b on highlights from a soccer match.
Refer to caption: x9.png

In the soccer highlight scenario (Figure 12), Mixtral of Experts successfully captures the causal link between the soccer player’s precise kick and the ball successfully going into the goal. Llama2-70b, on the other hand, mentions the effect (people cheering) without explicitly connecting it to the cause (the player scoring a goal).

Figure: Figure 13: Comparison of Mixtral of Experts and Llama2-70b on a boy singing on stage in front of an audience.
Refer to caption: x10.png

Lastly, for the singing performance (Figure 13), both models accurately describe the cause-effect relationship between the boy’s decision to perform on stage and the audience’s reaction of watching and listening. However, Mixtral of Experts provides a more detailed description of the audience’s response.

Overall, the CTN captions generated by Mixtral of Experts consistently demonstrate a better understanding of the causal-temporal narrative in the videos compared to Llama2-70b. Our approach using Mixtral of Experts effectively captures the cause-effect relationships and temporal dynamics, resulting in more accurate and contextually relevant captions.

These results highlight the importance of selecting an appropriate LLM and designing effective prompts for generating high-quality CTN captions. The superior performance of Mixtral of Experts can be attributed to its architecture and training, which enable it to better understand and articulate the complex causal-temporal narratives present in video content.

### A.4 Impact of Automatic Evaluation on CTN Caption Generation

In Section 3.1, we discuss the importance of using automatic evaluation to ensure the quality and relevance of the generated Causal Temporal Narrative (CTN) captions. We employ the EMScore metric to measure the consistency between the generated captions and the video content, discarding captions that fell below a predefined threshold. This appendix visually demonstrates the impact of the automatic evaluation step on the quality of the generated CTN captions.

Figure: Figure 14: CTN caption comparison for a video of a female sportscaster interviewing a male athlete.
Refer to caption: x11.png

Figure: Figure 15: CTN caption comparison for a video of protests in Brazil due to increased bus ticket prices.
Refer to caption: x12.png

Figure: Figure 16: CTN caption comparison for a video discussing a terror attack in Texas.
Refer to caption: x13.png

Figure: Figure 17: CTN caption comparison for a video about the Air Force using a surveillance system in Iraq.
Refer to caption: x14.png

Figure 14 shows an example where the CTN caption generated with automatic evaluation accurately captures the causal relationship between the female sportscaster interviewing the male athlete and her asking him questions. In contrast, the caption generated without automatic evaluation is irrelevant to the video content, discussing a man neglecting his allergy medication and suffering a sneezing fit.

Similarly, in Figure 15, the CTN caption generated with automatic evaluation correctly identifies the cause of the protests in Brazil as the increase in bus ticket prices and links it to the effect of a woman being interviewed about her participation in the protests. The caption generated without automatic evaluation, while mentioning the protest and its cause, fails to capture the specific effect of the woman’s interview.
Figure 16 demonstrates how the CTN caption generated with automatic evaluation accurately describes the cause of a terror attack in Texas and its effect on news anchors discussing freedom of speech issues. The caption generated without automatic evaluation is completely unrelated, mentioning a band’s performance and the audience’s reaction.

Lastly, in Figure 17, the CTN caption generated with automatic evaluation correctly links the cause of the Air Force using a surveillance system in Iraq to the effect of a woman talking about the man who created it. The caption generated without automatic evaluation is again irrelevant, discussing a group starting to dance at an event and the atmosphere becoming more lively.
These examples clearly illustrate the importance of incorporating automatic evaluation in the CTN caption generation process. By ensuring that the generated captions are consistent with the video content, we can significantly improve the quality and relevance of the CTN captions, enabling the CEN model to learn more meaningful causal and temporal relationships from the videos.

### A.5 Human Evaluation Study Details

To validate the quality of our CTN captions, we conduct a rigorous human evaluation study. This section details the methodology, results, and analysis of this evaluation.

#### A.5.1 Methodology

We employ the following methodology for our human evaluation:

- •
Sample size: 100 validation instances (50 each from MSR-VTT and MSVD datasets)
- •
Evaluators: Five independent domain experts
- •
Evaluation criteria:
1.
Causal Accuracy: Degree of correctly identifying and describing cause-effect relationships
2.
Temporal Coherence: Accuracy in representing the sequence of events
3.
Relevance: How well the caption reflects the overall content and context
- •
Rating scale: 5-point Likert scale (0-5, with 5 being the highest quality)

#### A.5.2 Results

Table 3 presents the results of our human evaluation study.

**Table 3: Human Evaluation Results for CTN Captions**
| Criterion | Mean Score | Std Dev | % Perfect Scores |
| --- | --- | --- | --- |
| Causal Accuracy | 4.8 | 0.42 | 82% |
| Temporal Coherence | 4.7 | 0.46 | 78% |
| Relevance | 4.9 | 0.31 | 91% |
| Overall Quality | 4.8 | 0.40 | 84% |

Key findings from the evaluation:

- •
Overall mean quality score: 4.8/5.0 ($\sigma=0.40$)
- •
93% of captions received scores of 4 or higher across all dimensions
- •
84% of captions achieved perfect scores

#### A.5.3 Inter-rater Reliability

To ensure the consistency of ratings across evaluators, we calculate the Intraclass Correlation Coefficient (ICC) for absolute agreement among raters:

- •
ICC: 0.87 (95% CI: 0.83-0.91)

This high ICC value indicates strong inter-rater reliability, supporting the robustness of our evaluation process.

#### A.5.4 Analysis

The results of our human evaluation study strongly validate the quality and consistency of our CTN captions. With a mean quality score of 4.8/5.0 and 93% of captions receiving high scores across all dimensions, we can confidently assert that our CTN captions effectively capture causal-temporal narratives in videos. The high inter-rater reliability further strengthens the credibility of these results.

These findings demonstrate that our automatic generation and evaluation process, as described in Section 3.1, produces high-quality CTN captions that accurately represent the causal and temporal dynamics in video content. This human evaluation complements our automatic evaluation metrics, providing a comprehensive validation of our CTN caption benchmark.

### A.6 Encoder Comparison for CEN

To explore the versatility of our CEN architecture, we conduct experiments using BLIP Li et al. () encoders in place of CLIP encoders. This comparison allows us to assess the adaptability of our method to different visual encoding architectures. Table 4 presents the results of this comparison on both MSR-VTT and MSVD datasets.

**Table 4: Comparison of CLIP and BLIP Encoders for our CEN**
| Dataset | Encoder | ROUGE-L | CIDEr | SPICE |
| --- | --- | --- | --- | --- |
| MSR-VTT | CLIP (Ours) | 27.90 | 49.87 | 15.76 |
| BLIP | 25.34 | 43.01 | 14.89 |  |
| MSVD | CLIP (Ours) | 31.46 | 63.51 | 19.25 |
| BLIP | 28.79 | 57.23 | 18.06 |  |

While the performance with CLIP encoders is higher, the results with BLIP encoders are still competitive. This demonstrates the flexibility of our CEN architecture and its potential to work effectively with various visual encoding methods.

### A.7 Comparison with NExT-QA

To highlight the differences between our CTN benchmark and the NExT-QA dataset, we provide a detailed comparison table and illustrative examples:

Table 5 presents a comprehensive comparison between NExT-QA and our NarrativeBridge approach.

**Table 5: Comparison of NExT-QA and NarrativeBridge**
| Aspect | NExT-QA | NarrativeBridge (Ours) |
| --- | --- | --- |
| Task | Question-Answering | Video Captioning |
| Output | Short answers | Coherent narratives |
| Temporal | Specific points | Multiple events |
| Causal | Implicit | Explicit |
| Narrative | Not present | Explicit |
| Focus | Info retrieval | Video description |

Figure: Figure 18: Examples of question-answer pairs from the NExT-QA dataset.
Refer to caption: x15.png

This comparison highlights the fundamental differences in approach and output between NExT-QA and our NarrativeBridge method.

Figure 18 provides examples from the NExT-QA dataset, illustrating the nature of its question-answer pairs. As seen in these examples, NExT-QA focuses on specific question-answer pairs that often target single events or actions. In contrast, our CTN captions provide a more comprehensive narrative that captures the causal and temporal relationships across the entire video sequence.

### A.8 Application of CTN Caption Generation for Labeling Unlabeled Videos

Our CTN caption generation approach, as described in Section 3.1, can be effectively applied to the task of labeling unlabeled videos. This application leverages the power of our few-shot based prompt and the LLM’s ability to generate coherent and contextually relevant captions that capture the causal-temporal narrative in video content.

Figure: Figure 19: Application of our CTN caption generation approach for labeling unlabeled videos. Given an unlabeled video, we extract frames and generate image captions using a state-of-the-art image captioning model (GIT Wang et al. ()). These captions are then used as input to our LLM-based CTN caption generation pipeline, which produces a CTN caption for the entire video. The generated CTN caption captures the cause-effect relationships and temporal dynamics in the video, enabling effective labeling of unlabeled video content.
Refer to caption: x16.png

Figure 19 illustrates the pipeline for labeling an unlabeled video using our CTN caption generation approach. First, we extract frames from the unlabeled video. In this example, we extract five frames which are equally spaced in the video.
Next, we generate image captions for each of the extracted frames using a state-of-the-art image captioning model. For this demonstration, we employ the GIT Wang et al. (), which has shown impressive performance in generating accurate captions for individual images. The GIT model generates captions such as "a tractor is driving", "there are four trucks", "a tractor is upside down", "a game is being played", and "two people are celebrating" for the five frames.
These image captions serve as the input to our LLM-based CTN caption generation pipeline (see Figure 2).

We use Prompt 1 in Section 3.1, replacing the original descriptive captions with the image captions generated by the GIT model. The LLM then generates a CTN caption based on these image captions, following the specified requirements and format.
In this example, the generated CTN caption is: "Cause: ’a tractor driving recklessly flips over and ends up upside down’ Effect: ’causing an accident, while others celebrated unaware’". This caption effectively captures the key events and their causal-temporal relationships in the video, providing a concise and informative summary of the video content.

To ensure the quality and relevance of the generated CTN caption, we employ the same evaluation framework described in Section 3.1. The caption is compared against the video content using the EMScore Shi et al. () metric, and only captions that meet a specified threshold are retained.
This application demonstrates the versatility and effectiveness of our CTN caption generation approach in labeling unlabeled videos. By leveraging the power of state-of-the-art image captioning models and our few-shot based prompt, we can generate high-quality CTN captions that accurately capture the causal-temporal narrative in video content, even in the absence of human-annotated captions. This approach has the potential to significantly streamline the process of labeling large-scale video datasets and enable more effective video understanding and retrieval tasks.

### A.9 Additional Qualitative Results for CEN Architecture

In this section, we present additional qualitative results comparing the ground truth Causal-Temporal Narrative (CTN) captions with the captions generated by our CEN architecture. These examples further demonstrate the effectiveness of our approach in capturing the causal-temporal relationships and generating accurate and contextually relevant captions.

Figure: Figure 20: Comparison of CTN and CEN captions for a video of a person solving a Rubik’s cube.
Refer to caption: x17.png

Figure: Figure 21: Comparison of CTN and CEN captions for a video of a man playing the piano.
Refer to caption: x18.png

Figure: Figure 22: Comparison of CTN and CEN captions for a video of a man giving a lecture on stage.
Refer to caption: x19.png

Figure: Figure 23: Comparison of CTN and CEN captions for a video of a band performing a song.
Refer to caption: x20.png

In Figure 20, both the CTN and CEN captions accurately capture the causal relationship between the person working on the Rubik’s cube and the cube being solved piece by piece. Similarly, in Figure 21, both captions correctly describe the cause-effect relationship between the man playing the piano and the music filling the room.

Figure 22 demonstrates the ability of our CEN architecture to generate captions that capture the audience’s engagement in response to the man’s lecture on stage. While the CTN caption specifically mentions the emotional engagement of the audience due to the sad story, the CEN caption more generally describes the audience listening intently to the lecture.

Lastly, in Figure 23, both the CTN and CEN captions accurately depict the causal relationship between the band performing a song and the crowd enjoying the music. The CEN caption, in particular, directly states the crowd’s enjoyment as a result of the band’s performance.

These additional qualitative examples further validate the effectiveness of our CEN architecture in understanding and articulating the causal-temporal narratives present in videos, generating captions that are coherent, accurate, and contextually relevant.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to weaknesses part.
soundness: 3
strengths: 1. The motivation of the work is clear, while the problem of improving the causal-and-effect understanding capability of a vision-language model is an important research direction.
2. Experimental results show that both the proposed methods, CTN and CEN contribute to better performance on MSVD and MSRVTT benchmarks.
summary: In this paper, a video captioning dataset named Causal-Temporal Narrative (CTN) is proposed. Compared to previous video captioning datasets which lack in causal-temporal  information which connects a cause and an effect, CTN, built with LLMs, includes caption that directly connects cause and effect events. Also, a new architecture named Cause-Effect Network (CEN) is proposed, which consists of two independent encoders which capture cause and effect events, respectively.
weaknesses: 1. Including detailed statistics about the constructed CTN dataset (e.g., # of samples, average caption length, # of captions per sample, or more..) may enable readers to easily understand the dataset.
2. Questions about Table 2.
    1. Results show that Fine-tune v.v. shows better performance than CEN on the MSVD benchmark under the cross-dataset evaluation setting, compared to CEN trained with both datasets. The result of cross-dataset evaluation performing better than whole dataset training is somewhat counter-intuitive. Can you provide an explanation for these results?
    2. What does the abbreviation of v.v. stands for? It would be better to mention the full name.
3. Is there additional experimental results or analyses that could show CTN and CEN contributes to better cause-and-effect modeling besides results on MSVD and MSRVTT? Since the quantitative results of Tables 1 and 2 are not directly aimed at measuring cause-and-effect capability, some additional experiments may be required.
4. These are some **minor** issues about the presentation of the work. These are **not** critical weaknesses, and somewhat personal opinions.
    1. Personally, I think the example in Figure 1 is not very intuitive to explain the importance of causal-temporal narrative. Cause: “a car drove recklessly” and Effect: “the car was severely damaged” seems to be a valid cause-and-effect event, but the following event of “a group of guys started playing beer pong” seems like it’s not a direct effect of ‘cause'. This is a personal opinion, but you could consider finding a better example to demonstrate the idea.
    2. I think ‘dataset’ instead of ‘benchmark’ might be a better word for CTN, as it is only used for training but not for evaluation.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 3
strengths: The paper proposes a benchmark focused on causal-temporal narratives, adapting an evaluation score for automated assessment.

It introduces a novel captioning model that encodes causal and effect information separately, then fuses them to decode the final caption.

Experiments demonstrate that the proposed CEN framework outperforms comparative methods in performance.
summary: The paper highlights that current video captioning benchmarks lack causal-temporal narratives and proposes a new benchmark to address this gap. This benchmark is generated by prompting a large language model (LLM) with a series of captions extracted from a video. To ensure relevance, EMScore is adapted to estimate the similarity between the generated captions and the video, discarding samples with low scores. The paper further introduces a two-stage framework to generate causal-temporal narrative (CTN) captions. In the first stage, two contrastive losses are applied to train Causal and Effect Video Encoders with their respective captions. In the second stage, the encoded causal and effect features are concatenated to decode a cohesive caption.
weaknesses: The paper includes a human evaluation of the generated CTN captions; however, the assessment is less convincing due to the limited sample size. Despite the large dataset size—approximately 300,000 samples in MSRVTT and MSVD—only 100 samples were selected for human evaluation, reducing the reliability of the results.

For supervision during caption decoding, the paper simply concatenates the causal and effect text, which disrupts the original sentence grammar and lowers the quality of the synthetic supervision. The rationale behind this approach is unclear. An alternative method could involve inputting the causal and effect text into an LLM to fuse them into a more natural and coherent sentence. With ground truth captions generated by simple concatenation, evaluation metrics such as SPICE and CIDEr may not accurately reflect the quality of the generated captions. This approach could lead to discrepancies between metric scores and the true readability and coherence of the captions.

The structure of the feature encoder in stage 2 is not clearly depicted. Typically, encoded features from transformer encoders consist of a series of representations. It is unclear whether h_cause and h_effect are lists of representations or single representations. If h_cause and h_effect are single representations, it is not specified how they are generated—whether by selecting a specific token, or by applying mean pooling or max pooling. Furthermore, the specific dimension along which these features are concatenated is not detailed.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: * The fatality video may be too graphic for some readers. I suggest switching it to a video on another topic.

* In figure 5, CTN captions appear too similar to CEN. Is this a symptom of overfitting?

* What does the system do when a video with no apparent cause and effect is used as input? For example, a person is singing but there's no audience reaction. Or a person is just sitting in a chair. The paper mentions "our method assumes a certain level of causal and temporal coherence within the video content"; maybe it would benefit from providing a probability value on the confidence for the existence of cause and effect in the input video?

* `As expected, the resulting captions (Figure 6) are of low quality`, how is this measured?

* Has the team considered the encoder of effect to also receive the encoded cause as input? How would that compare with the current approach?
soundness: 3
strengths: Significance:

* The problem discussed is important and relevant to much of the field of video captioning and video generation. It's true that typical captioning methods don't typically expose causal information, which makes generated captions much less useful than they could be.

Originality:

* The work appears original though the novel aspects of it are a bit of straightforward applications of existing methods.

Quality:

* The work produces good results. While the paper discusses both results on existing benchmarks, and in CTN, the former being more interesting because they've been established independently.
* I appreciate that both automatic benchmarks and human evaluations were used to evaluate this work.
* The proposed CEN network architecture is interesting and original.

Clarity:

* The work is mostly clear, except for some aspects that are discussed in the weaknesses section.
summary: This paper tackles the problem of captioning videos taking into account causal-temporal narratives. It does so by presenting two main contributions: a dataset and benchmark (CTN) with causes and effects per video, and a network architecture (CEN) for extracting such narratives directly from videos.

When applying the CTN benchmark to CEN, the network gets better metrics when compared to similar methods, potentially establishing it as SOTA in the domain.
weaknesses: * The automatic evaluation process is found to filter out most bad cause and effect generations. More complete analysis on the distribution of the qualities of the generated captions is missing. I can imagine a histogram that displays the distribution of qualities with a vertical line showing the threshold that's been picked. Also the paper would benefit from analysis on whether the automatic evaluation filtering improves on the method, or is a patch for more fundamental issues in the prior step.

* The contribution of CTN depends strongly on the prompt used to zero shot cause and effect generation. Given that this prompt is so central to the paper, the paper would benefit from more complete ablations regarding different aspects of the prompt. Especially important, the list of requirements that are presented in the prompt are justified in the paper but no corresponding ablation per requirement is done to validate the explanation. (the ablation donefor 4 prompts of varying degrees of quality appears insufficient for this purpose)

* When building CTN, the captions are 1) generated from frames, and then 2) a model combines multiple captions into cause and effect captions without looking at frame information again until 3) the automatic evaluation step. Has the team considered using a model that can read text + image in step 2 as well? 

  Concluding cause and effect just from the individual frame captions is a lot more error prone than it could be if image information was also used. I think this possibility could be mentioned / analyzed as a potential alternative.

* Why are causes and effects only separated by a space in CTN? Have other choices been ablated? In particular, I can imagine using a "comma" or an "and" between both to work slightly better.

    * Has the team considered using an LLM to combine both captions together, instead of using concatenation?

* For A5.1: more discussion on the human evaluation process would be very beneficial. How were the experts chosen? How is information presented to them? How were the 100 videos chosen?

  For example it would be ideal to have an independent expert set; and it would be ideal if they were presented with the captions without labels that identify the models, and in such a way that each the models are independently ordered for each video (so annotators cannot be biased to always vote for the first model, for example)

* Clarity: The paper, especially in the first few pages, is a bit too repetitive or verbose. It may benefit from some proofreading for succinctness. For example, the following two paragraphs are very close to each other, and both express a similar idea:

```
It means that our CTN benchmark differs significantly by focusing on generating comprehensive causal-temporal narratives, capturing broader temporal relationships within a single caption, and providing a more holistic view of video content
```

  and

```
We introduce a new benchmark CTN specifically designed to capture and evaluate causal-temporal narrative in video captioning. Our approach goes beyond existing benchmarks by explicitly modeling causal-temporal narrative in a single, coherent caption, enabling a more comprehensive understanding of video content.
```

  In other cases ideas are repeated in the same paragraph:

```
However, as our experiments show (see Section 4.4), even these advanced models struggle with generating accurate causal-temporal narratives. To address this limitation, there is a need to develop new video captioning models that can overcome the shortcomings of current frameworks. This also underscores the need for specialized architectures like our proposed CEN that are explicitly designed to capture and generate causal-temporal narratives in videos
```

  one could summarize this text as:

```
However, as our experiments show (see Section 4.4), even these advanced models struggle with generating accurate causal-temporal narratives. This also underscores the need for specialized architectures like our proposed CEN that are explicitly designed to capture and generate such narratives in videos
```

  Such verbosity makes the paper hard to read, despite the main ideas being explained very clearly.

* The paper would benefit from a more complete discussion on possible downsides of this approach. For example for CTN how it compares to using frame data in step 2 as discussed above, or what downsides could arise from the concrete proposed architecture.

* The paper mentions surveillance; maybe analyze this in the context of ethic concerns?

* `We acknowledge potential ethical concerns (e.g., privacy in surveillance, risk of misleading content) but are committed to responsible development`, could this be made more concrete?

* The paper mentions `lays the foundation for a paradigm shift`, which appears potentially grandiose in my opinion. Better to let the community arrive to this conclusion, than write it in the text.

I'll be glad to revise my score depending on the response to this review.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: See above.
soundness: 3
strengths: - While most of the causal video understanding focuses on video question answering, this paper proposes to bring causal understanding into video captioning. 
- The data generation pipeline looks sound.
- The proposed approach achieves strong results on the proposed benchmark.
summary: This paper focuses on incorporating causal-temporal narrative into video captioning.  The authors create a new dataset called Causal-Temporal Narrative (CTN) captions, using a LLM to generate captions that explicitly describe cause-effect relationships and maintain temporal consistency within videos based on multiple annotated ground-truths for the same video. The quality of these CTN captions is assessed both automatically (using EMScore) and through human evaluation.

Furthermore, the authors propose a novel model architecture called the Cause-Effect Network (CEN), specifically designed to learn from the CTN captions. CEN decouples the cause and effect captions by finetuning one visual encoder for each of them. The representations of both encoders are then used during the LLM finetuning stage.

Experiments on MSVD and MSR-VTT datasets demonstrate that CEN significantly outperforms state-of-the-art video captioning methods for generating CTN captions. Ablation studies further validate the design choices of CEN.
weaknesses: - The weakest ablated baseline already performs well above state-of-the-art approaches. It would be great to understand what the differences are. For instance, how does a baseline with the CLIP ViT features used directly perform?
- The presentation can be improved, for instance it is not clear that the evaluation is on the CTN captions in Table 1, and the ablation study includes method names which are hard to interpret when having a first look at the table.
- The data generation approach largely relies on the multiple GT captions present for each video in MSR-VTT and MSVD. 
- Captioning metrics which do not move much like Rouge-L are hard to interpret.

Meta-review: The paper introduces a new video captioning benchmark focusing on causal-temporal narratives, along with a method tailored for this task. It concluded the review process with three positive evaluations (two 6s and an 8) and a negative review, which might have been improved had the reviewer been more active in the discussion. Two reviewers explicitly supported the acceptance of the paper, with one playing a significant role in driving the discussion forward. The author rebuttal effectively addressed the majority of concerns, leading all initially negative reviewers to increase their ratings. With final sentiments leaning positively, the AC also recommends the acceptance of the paper.

Decision: Accept (Poster)