MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning
Authors: Tieyuan Chen
Huabin Liu
Tianyao He
Yihang Chen
Chaofan Gan
Xiao Ma
Cheng Zhong
Yang Zhang
Yingxue Wang
Hui Lin
Weiyao Lin
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Video understanding; Video reasoning; Causal discovery; Causal inference
Primary area: machine_vision
TL;DR: A multi-event causal discovery task for video reasoning.

Abstract: Video causal reasoning aims to achieve a high-level understanding of video content from a causal perspective. However, current video reasoning tasks are limited in scope, primarily executed in a question-answering paradigm and focusing on short videos containing only a single event and simple causal relationships, lacking comprehensive and structured causality analysis for videos with multiple events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relationships between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD requires identifying the causal associations between these events to derive a comprehensive, structured event-level video causal diagram explaining why and how the final result event occurred. To address MECD, we devise a novel framework inspired by the Granger Causality method, using an efficient mask-based event prediction model to perform an Event Granger Test, which estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to address challenges in MECD like causality confounding and illusory causality. Experiments validate the effectiveness of our framework in providing causal relationships in multi-event videos, outperforming GPT-4o and VideoLLaVA by 5.7% and 4.1%, respectively.

MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning
Tieyuan Chen'r, Huabin Liu'*, Tianyao He', Yihang Chen', Chaofan Gan', Xiao Ma², Cheng Zhong', Yang Zhang?, Yingxue Wang', Hui Lin Weiyao Lin
Shanghai Jíao Tong University, Lenovo Research, AI Lab, China Academic of Electronics and Information Technology (tieyuanchen, huabinliu, wylin]@sjtu edu. edu. cn
https:/ httpss///gthhh...cccyyceeeeeeee/ SJTUMMEEDD Ben
Abstract
Video causal reasoning aims to achieve a high-level understanding of video content from a causal perspective. However, current video reasoning tasks are limited in scope, primarily executed a question-ansee answering paradigm and focusing on short vidcos containing only single event and simple causal relationships, lacking comprchensive and structured causality analysis for videos with multiple events. To fill this gap. introduce (MECD). aims to uncover the causal relationships between events distributed chronologically across long videos. Given visual segments and textual deseriptions of events, MECD requires identifying the causal associations between these events derive comprehensive, structured event-level video causal diagram explaining why and how the final result event occurred. To address MECD, we devise a novel framework inspired by the Granger Causality method, using an efficient mask- based event prediction model to perform an Event Granger Test, which estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to address challenges in MECD like causality confounding and illusory causality. Experiments validate the effectiveness of our framework providing causal relationships in multi-event videos, outperforming GPT-4o and LideoLLa by 5.7% and 4. %, respectively.
task and dataset, Multi-Event Causal Discovery
1 Introduction
Video causal reasoning aims to achieve high-level understanding and analysis of video content from a causal perspective. Video Question Answering (VQA) [1-5] represents one of the most prominent tasks in causal reasoning. where models are tested on their causal ability to understand video content through causal questions such explanations, predictions, and counterfactual assumptions. Recently, some studies have sought to move beyond the single QA task, attempting to construct more complex and challenging video reasoning tasks and methodologies. For example, CLEVRER [5]. V-CDN [6] and CATER [7] explored more difficult causal reasoning tasks in virtual scenes by constructing object-aware features using graph neural networks. Neural-symbolic paradigm AAR [8] and LMLN [9] extended to derive inference rules by symbolizing data. VAR [10] and BiGED [11] extended to daily video causal reasoning by introducing causality during prediction.
However, current video causal reasoning tasks are still limited in scope (primarily QA-based) and mainly focus on short videos containing only single event or a few events. Most importantly, they
Equual Contribution.
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) People:
Premise Events
(b)
A slides chute.
Others watch.
8 pushes intto the chute. throws bowling ball. O feels funny,
Causality
Causal Diagram
Causality confounding
[d]
Illusory temporal causality
Illusory existence causality
Submitting.
bridge
Taking the test.
Obtaining grade:
Adding
Adding vegetables;
Dog enters.
Human attracts. Dog comes pouncing.
Figure 1: (a): Illustration of Multi- Causal Discovery Task, where the 3rd and 5th premise events account for the occurrence of the final event. The objective of our task is to determine whether causal relation exists between events and outputs structured causal diagram. (c): Example of causality confounding. (d)& Illustration of illusory causality.
cannot provide comprehensive and structured causal representation for multi-event video reasoning, which is typically required in real-world scenarios. For instance, traffic surveillance videos, is necessary to cross-analyze events happening different times to determine which events, combinations of events, led to the final traffic accident event.
To address this gap, set up new task:
Multi-Event Causal Discovery (MECD), which aims to uncover causal relationships among events that distribute chronologically long videos. As illustrated in Fig. 1, given multiple chronologically arranged event segments in a video along with their corresponding textual descriptions (Fig. ((a),, MECD requires identifying causal associations between these events to derive comprehensive and structured event-level causal diagram (Fig. 1(b)), indicating why and how the final result event happens. Mcanwhile, we contribute a new dataset for the training and evaluation of MECD by collecting long-form videos involving multiple events and manually annotating real causal relations between events for them. However, to our knowledge, no available solutions directly comprehend causal relationships at the event level, necessitating the development of a new framework to tackle this complex task.
To this end, we draw inspiration from the Granger Causality Method [12-14] for solution, which is widely used in traditional causal discovery weather patterns). The main idea is that temporal causality can be effectively estimated by predictive ability. Specifically, applied to videos, if Event A occurs prior to Event B, we consider to cause of B only if A could facilitate the prediction of B. We term this criterion the Event Causality Test. However, compared to simple low-dimensional data, the inputs of MECD involve much more complex modalities, including both visual and textual content, which may introduce bias in the estimation of causality using such a predictive paradigm. Specifically, we observe that directly applying Event Causality Test to video causal discovery presents two main problems:
low-dimensional time-series data (e-g., stock prices,
(1) Causality confounding indicates that the original causal relationships between events are dis- rupted or interfered with by other relay or adjacent events. Such confounding stems from the fact that many causal relationships flow through an intermediary event that acts as a bridge. As shown in Fig. I(c), event "submitt tting the paper" serves necessary bridge between "taking the test" and "obtaining a grade." In this case, this bridge event might be mistakenly regarded as the only cause of the result event, while another cause, "taking the test," is overlooked. However, the bridge event can only occur if "taking the test" happens first. Therefore, we cannot identify the real causality between events that linked by such bridges following simple predictive criterion, and eliminating such confounding is thus crucial for an accurate causal discovery.
(2) Illusory Causality, which includes illusory temporal and existence causality. Illusory temporal causality exists when events exhibit close correlation in temporal distribution. Such correlation may mislead the test of real causality. As shown in Fig. 1(d), the event "adding oil

---

when cooking" often occurs before "adding vegetables to stir-fry," but there is no real causality between them. As for illusory existence causality, it occurs when some objects in early events may serve as necessary existence conditions of later event. For instance (Fig. (()),, consider determining the causal relation between "a large brown dog enters the room" (at the start of the video) and "the dog runs towards the camera." (at the end of the video). Although the presence of the dog in the former event is a prerequisite for the subsequent event, it does not directly cause the dog to rush towards the camera.
Building upon the preceding discussion, we introduce a novel framework to tackle MECD. This framework executes the Event Granger Test via an efficient mask-based event prediction model. It deduces the causality of premise event by comparing the predicted features of the result event when the premise is either masked or unmasked. Furthermore, to mitigate the challenges of causality confounding and illusory causality discussed earlier, we integrate two additional causal inference techniques- cociquuuss--ro--oooo front-door adjustment [15-17] and counterfactual inference [15, 18, 19]-into our framework. Specifically, these techniques compensate for or remove the causal effects of previous or subsequent adjacent bridge events to eliminate confounding. Simultancously, they address the issue of illusory causality through the incorporation of an extra chain of thought [20-22] and existence-only descriptions during inference. Extensive experiments validate the effectiveness of our proposed framework in predicting structured causal relationships for given long-form videos.
2 Related Work
Video causal reasoning Many tasks in the past have tried to carry out causal reasoning in videos. Among these, the most common task Video Question Answering (VQA) [1-4], aiming to give a reasonable answer to the question, methods such as Se and LocAns [3, 4] made abductions based on the result, they grounded a single reason in previous time. However, VQA does not extend to abduct multiple reasons, merely creating single causal link from reason to result.
Many tasks were based on VQA task for further causal reasoning attempts. CLEVRER [5]. CATER [7] and V-CDN [6] explored causal reasoning based on physics and other basic laws in virtual scenes. However, these tasks haven't 't been committed to extending to the general video causal reasoning. AAR [8] and LMLN [9] symbolized data and derived inference rules using external knowledge, However, they can only reason within a defined symbol domain. The most similar VAR [10] predicted explanation events with premise events, and the causality was introduced during its prediction process. However, firstly it hasn't 't been committed to discovering the complete causal diagram. Besides, there is no explicit utilization of causal methods which constrains its ability.
All tasks above are for causal reasoning in short videos, while ours aims handle long-duration videos. Besides, most of these are coarse video-level tasks, ours is more fine-grained event-level reasoning. Additionally, we want to establish whole causal diagram rather than a single causal link. In conclusion, all these tasks haven't 't been committed to discovering causality among complex multi-event videos. Consequently, there exists necessity need for more comprehensive task.
Causal discovery in low-dimensional temporal data Traditional causal discovery methods of simple temporal data mainly divided into three categories. Constraint-based methods use condi- tional independence tests to identify causal relations [23-25]. Score-based methods search through the space of all possible causal structures optimize a specified metric [26-28]. The Granger Causality method discovers causal relations by calculating the degree to which the earlier occurred event contributes to the prediction of the latter occurred event [29-31]. The constraint-basee and score-based methods require stringent assumptions about data distribution, making them less suitable for video data. The Granger Causality methods are more suitable yet face challenges when applied directly to video data, our method reaches better performance by utilizing causal inference methods.
3 Benchmark
3.1 MECD task settings
Our Multi-Event Causal Discovery (MECD) task is designed to test the ability of causal discovery in multi-event videos. Given a video E that contains chronologically organized N events, fei,..., eN. J. the task aims at determining whether any previous event e, (n < N) has causal
3

---

Burn DO Roriii hhugging LUOIE
running 1ug
1.
Rodo of Causal Relstion with
000
Desributionn Brents
Sports and Competition
Word Clouds
Making and Creating
200
0.2
Daily Activities
Performing Displaying Interaction and Socializing
Posttio of3
(a1)
(a2)
Figure 2:
Constitute of MECD dataset. (al), we present 5 main video categories of the dataset.
The word cloud is also summarized for video types. In (a2), the left chart indicates the impact of positions of events on their causality where we find the second last event tends to be more significant; while the right chart plots the number of events in videos.
relation with the last one (i.e., eN). Specifically, an event e, (Ua, Cr consists of a video clip Un and the corresponding caption Cn. Without loss of generality, relations of previous events to the last one can expressed as r [ri...., -N- where r (k N) set to "1" to indicate the existence of ex 's causal relation with ex, and "0" otherwise. Notably, this setting is generalizable to causal relations of any two events as long as we cut off the video and treat the latter one as the last event.
3.2 MECD task dataset
Data Source The Multi Events Causal Discovery (MECD) task contains videos with multiple events and intricate causal relationships. The ActivityNet Captions dataset [32] built on ActivityNet v1.3 which includes 20k 120-second YouTube untrimmed videos. We carefully reorganize the ActivityNet Captions dataset and select 1. 1,105 lifestyle videos that span diverse scenarios. We call this new dataset MECD dataset, where 806 and 299 videos randomly split for training and testing, respectively. Specifically, each video in the MECD dataset contains to 11 events, with minimum of 2 premise events exhibiting causal relations with the last one. Fig. (al) presents the main categories and word clouds of video types. Please refer to Appendix Sec. B.4 for more dataset examples.
Data Cleaning We further clean our dataset by excluding non-causal videos. For example, videos that describe multiple non-causal action steps such as washing hands and shaving were excluded. Dataset Annotation The annotations of MECD dataset include 4 attributes. The "duration", "sen- tence", and "timestamps" attributes in annotations remain the same as the ActivityNet Captions dataset. Specifically, in the context of our task, a new attribute "relation" is introduced. To obtain this attribute, relations among events are firstly annotated by GPT-4 API [33]. and subsequently refined by five human annotators. Through a cross-annotation process, gt labels are determined by the majority of the annotators' causal relation choices, thus mitigating potential inaccuracies and subjective biases certain extent. We also present the impact of positions of events on their causality and number of events in videos in Fig. 2 (a2), annotation pipeline is in Appendix Sec. B.3.
4 Methodology
In this section, we present our Video Granger Causality Model (VGCM), as shown in Fig. 3. This model establishes the global connections across all events, and deduces the causality of premise event by comparing the output features when it is masked or not, under the concept of the Event Causality Test. However, masking out an event may lead the problem of confounding and illusion. In this context, we further utilize causal inference methods to address these by compensating or removing the effect of previous or subsequent causal events to mitigate the confounding meanwhile during inference the extra chain of thoughts and existence-only descriptions relieve the illusion.
4.1 VGCM: Video Granger Causality Model
Building upon the Granger Causality method introduced in [34-36], our core motivation for con- structing VGCM is Event Causality Test: To compare the prediction result of the last event using all the premise events with or without a certain event in If the results exhibit obvious divergence, indicates that the current premise event is causally related to the result event.

---

6
multi-moda. decoder Dec that shares weights Multi- Modal
Dual-Branch Processing
Information Interaction
Masked Prediction
masked Unmasked
Input
Pretral ned Encoder 4gr
Video
Video Encoder
R
Multi modal
Decoder
Relation Head
query
A
Contexts
U
shane
Mult Mult-modall
nh
Relation?
F
Decader
Semantic
Similarity
CN
Caption
share
Caption Encoder
mask
query
Mult--mdda
Decoder
Caption
Head
Ck Contexts
query
Figure 3: Video Granger Causality Model. Two data streams V and V serve input, video and text embeddings are concatenated after being separately embedded. The VGCM incorporates two classifiers, the caption head takes the unmasked stream accomplish the event-predicting task, while the relation head discovers the causal relations with two embedding streams.
We design VGCM to take in both the video clips and the captions to maximize information utilization. As illustrated in Fig. 3, our proposed VGCM is multi-modal transformer maadedd structure with video encoder and caption encoder, and multi-modaa decoder with causal relation head to discover causal relations through the predicting process and the comparison predicting results.
Based on this, we denote E² as the set of all the premise events E² == IE \ the event where the premise event ej (k: N) is masked. Notably, eN. and Eg == EP \ ex mask the event ex by setting all zeros to its video clip eg and assign constant characters to the caption Cj-
Following [10, 37-39], we firstly pretrain vidco encoder & pre under action recognition task to extract the features of the vidco clips. We essentially create two paths, one for the unmasked event set E" (orange path in Fig. 3) while the other for the set with event (i.e., ex. masked EA"" (green path in Fig. 3). The video clips and captions are first separately encoded using Ency and Encg to obtain compact features, then their features sent for both paths to fuse the information. Afterward, several model heads are employed for feature comparison and loss measurement. VP and C are the video clip and caption matrix concatenated from all premise events set EP, similarly, V and C are from Ey
F Ency @ pre (V")), O [Dec(Cat(F") Enco (ORC((C))lee [ F Eney (
pre ( (VT)), OF [Dec(Cat (FF. Ency (CTT N)
where Ency and Ency represent the encoder module of video clips and captions, respectively. Dec is multi-modal decoder that shares weights for both paths. Cat indicates the concatenate operation, and [-] indicates the (N - 1)-th slice at dimension O. FD and FT are features after encoding, and O" and T are the output features, which are then used for comparison difference. Incorporating both visual and linguistic representations, the decoder conducts cross-modal reasoning and leverages the context from the unmasked premise events to posit a meaningful representation of the most likely explanatory result event.
Subsequently, the feature O" deduced from the unmasked events sent the caption head for prediction. Additionally, in order to compare the difference of the prediction result, O", O are directed to the relation head for causal relation discovery. The result event ex encoded the same way eg to get feature FN Ency ( (o (UN)) and the output ON Dec(Cat(() n (nc )), ON aggregated for reasoning (red path in Fig. 3). The relation head consists module and self-enhancement module, where outputs are concatenated and then passed through the cross-reasoning layer 9r for further interaction. Last but not least, the auxiliary similarity measured between 0" and O supplement to the output information of the relation head. After
semantic query
5

---

the reasoning process, the prediction output of the causal relation f& can be represented by:
Fx- (Cat(CCC.. ate (Cat(O ON), Cat(O", ON)), T (Cat(O) ON))))
(2)
where &h training objective consists of two main directions as previously discussed:
att represents cross-attention, & represents self-attention, 9r represents linear layer. The
To reconstruct the textual and visual representation of the result event eN, we introduce caption loss and reconstruction loss, respectively. Caption loss LC ensures an accurate prediction of the result caption êλ given all the premise events E". Simultancously, visual reconstruction loss Ly forces the encoder to "imagine" representation of the result video clip UN that better aligns with the original representation uy. These losses allow the model to predict visual and textual representations that are close to the original representations, which better supports our method of inferring causal relations by comparing the results of the two-stream predictions.
For the objective of causal discovery, we introduce causal relation loss and an auxiliary semantics similarity loss. Causal relation loss Ln supervised the output relations ft. Meanwhile, the semantics similarity loss L.s is introduced to guarantee the semantics similarity of result event prediction under the presence or absence of a causal-relation-ffee free premise event. The complete loss function is:
-= Lo(eN, ēN) + ARCn(rK, fk) Lv N FN) + {sss Op))
where AR. λv. and As are weights for trade off. La and LR are the cross-entropy losses, Lv and Lg are the mse losses, F is the N-th slice of F?, which represents the encoded feature of eN.
4.2 Causal Inference in VGCM
In Sec. 4.1, we employ the concept of Granger Causality to design our VGCM model under the principle of Event Causality Test which may, however, introduce causality confounding and illusory. Below we introduce these issues in detail, as well as how we manage to solve the problems.
Causality confounding is phenomenon where the original causal relations across events are im- pacted due to modification (i.e., masking) of some intermediate events (i.e., ex). Existing disentangled representation learning works [40, 41] disentangled different attributes of a variable by supervising high-order distribution under strict assumptions but failed in disentangling different variables.
Specifically, when ej is masked for the comparison in VGCM, the causal relations between et's adjacent events and the last event are impacted, leading to confounding of causal effects. Notably, for brevity, we only employ ex 's previous one event c and its subsequent one event Cx+1 for analysis, but the same analysis also applies to all the previous or subsequent events. To be specific, there exist two distinct kinds of confounding when e₁ is absent: 1) Causal effects of ex-1 to eN may be lost, as its connection to eN built upon ex. (green path in Fig. 4 (al)). 2) Causal effects of ex+1 to ex may be redundant, ex must be necessary prior of its causality, (red path in Fig. 4 (al)).
Iluuoor causality is another issue that may lead some spatial temporal misunderstandings, including illusory temporal and existence causality. 1) Illusory temporal causality is the situation that events could have tight temporal ordering, but they in fact have no causal relations. 2) Additionally, illusory existence causality occurs when an object introduced in the premise event is a necessary condition for the result, but the premise event does not semantically serve as a reason. Notably, we find that illusory in multi-event videos is much more significant than two independent events, which also tends to be exacerbated by causality confounding.
Overall, causality confounding and illusory causality both bring difficulties for relation modeling of events in videos. Notably, these two issues coupled in that causality confounding tends to exacerbate illusory causality by misallocating attention to temporal ordering and causal effect. Therefore, illusory causality can be partially relieved by solving the problem of causality confounding.
When considering taking the illusory causality, the chain of thoughts [20-22] has been shown in LLMs to lead the model to logical thinking which is similar to human thought process, the chain of thoughts Tootfes_ p provides a step-by-step process of reasoning the en from ex-1- Specifically, Teodles [=ew] is obtained using GPT-4 API [33] by feeding it with 6g-1, EN along with prompt asking it to provide the probable reasoning chain. We consider utilizing it in causal inference to eliminate the attention bias on temporal correlations introduced by non-causal temporal knowledge.
Besides, as the illusory existence causality is caused by the objects' correlation between the events, we address this influence by keeping objects in the green path in Fig. 3 the same those in the

---

Compersation
Missing
Adjustment
monerg Causal Relaion Removement
Tootjeu-iien)
Causal Inference
Reconstruct
Counterfactual Intervention
(a1)
Existonce only
(a2)
Figure 4: Causal Effect of the Adjacent Events and Causality Diagram. (al) shows the causality of the third event analyzed, the red causal effect needs to be compensated while the green needs to be mitigated. (a2) shows the causal inference methods corresponding to the two causal effects.
orange path. We introduce an alternative event ck = [ug. cf) of e₁ to briefly recaps the objects in ek- Specifically, of obtained using GPT-4 API [33] by feeding it with eg along with a prompt asking to extract the objects from eą and organize them as the sentence such as "There are objects A, B and C. Consequently, we opt to employ oe to approximate eg our VGCM model while omitting vg. as et is sufficient already to convey the information of objects. By providing eg all the necessary objects are still available in this path, thus effectively mitigating the illusory existence causality, facilitating the model to focus more on essential and causality- related semantic information.
To tackle the issues above, we introduce two causal inference methods: the front-door adjustment [42] for the missing causal effect of ék-1 and counterfactual inference [42] for the redundant causal effect of ex+11 Meanwhile, the chain of thoughts Tootler-1.e and the descriptions of existence og are also provided to carefully address illusory causality, which in turn mitigates confounding.
We establish a causality diagram in Fig. 4 (a2) for an improved elaboration. On masking ex. the causality confounding that requires compensation FC or removement F can be expressed as:
FC ((eNlee)) P(ex)do(es)) F P(eN(ex+1) P(eN (do(Ex+1))
where P(eNless lex) and P(eN (x Jrepresents the process by which we predict eN from ex and ex+11 in the orange path in Fig. 3, and do(.) represents do-operation in causal inference [15] that cuts off the causal relation between the event and its causes.
We aggregate the subsequent events 6 the current event eg and the chain of thoughts T pee using a linear layer Ido for aggregation and the cross- rrtssattentioon and self-attention, according to the study [43, 17], Plex (do(c))) can be implemented as:
P(ex (do(ex) = ((Cat are (x+1,11+1)) Ck+1), Ta (ek, Ck2 ex), (Too|jex--ex)))
(5)
Here, we re-use the cross-attention & and the self-attention & modules as in (2) to cut off the causal effect from 6g-1 to G₁ through do-operation, e only interacts with subsequent events in predicting ex. Then the missing causal effect FC can be compensated since the causal-view operation and illusory temporal causality can be suppressed at the same time with the introduction of the chain of thoughts. Similarly, the redundant causal effect FR can be removed by applying counterfactual intervention, then P(ex do(ex+1)) can be represented by:
P(ex (o(x++)(()+)) = P(eN (aaxlle+1)][p) P(eN/ee+:))P(eseelee - P(ex+1le()),
(6)
P(ey (d((e+))) effectively cuts off the redundant causal effect between ex+1 and e for the reason that the causes of ex+1 are replaced with counterfactual description eß. then the illusory existence causality can suppressed at the same time.
To refine the originally decoded feature OF from the path with premise events masking:
O O Dec( FC) ) + (R)
where 0 the refined feature that replaces O for further deduction of the model. With the refinement feature 0t" our VGCM model effectively compensates the connections between e and e that were originally lost due to the removal of eg, and effectively removes the redundant causal effect between ck+1 and ex well.
7

---

Table 1: Main results. Experiments validate the effectiveness of our VGCM framework in reasoning causal relations towards multi-event videos, outperforming GPT-4o and VideoL by 5.7% and 4.1%, respectively. indicates without causal inference. Random results and human performances are also provided.
Table 2: Ablation Study. Adj indicates the front-door adjustment, and inter indicates the counterfactual intervention.
designs
Causal methods
Le
Lv
Es
Adj
Inter
Acc
64,8
Paradigm
Method
Ave SHD [|Acurryy 1
65.1
Rundom Guess
Guess all causal.
6.95
42.4
67.0
65.3
Guess non-causal.
5.36
57.6
68.7
Gemini-1 [44]
4.91
59.3
69.3
LLM Base
GPT-4-0613 [33]
4.92
59.6
71.2
MiniGPT4-video [45]
5.16
56.8
Table 3: Illusory existence causality exper- iment. w/o C. indicates without counterfac-
Few-shot
MiniGPT-4 [46]
5.14
57.5
VLLM Base
Video-Ilama [47]
5.10
tual intervention.
VidcoChat2 [48]
4.89
60.7
VideoL.LaVA [49]
4.85
62.5
Method
starting division Ending division
GPT-4o [33]
4.69
65.5
VGCM (wlo
1.12
L.44
VAR [10]
4.96
57.3
VGCM
1.12
0.93
Multi-modal
Videobert [50]
4.95
60.9
(VIT-L1))) [51]
4.77
62.9
Table 4: Illusory temporal causality exper- iment. w/o F indicates without front-door adjustment and Ave indicates average.
Fine-tuned
VideoChat2 [48]
4.77
66.9
VLLM Base
Vide [49]
4.73
67.1
VGCM VGCM
4.51
67.0
Method
Yo Acc
FM-1 Ace Ave Ace
Ours
4.19
71.2
VAR
[10]
(33.5
54.6 (33.7)
57.3
VGCM (wlo F)
6-.6
63.7 (-3.2)
66.9
Humans
Deductive Reasoning
2.05
87.2
VGCM
(-0.7)
68.4 (-0.3)
68.7
5 Experiments
5.1 Main results
Implementation details. including the pretraining process, detailed architecture of VGCM, and hyper-parameters settings can be found in Appendix Sec. A due to space constraints.
Baselines. We mainly compared our model with basic mulli--modl models such bascline model Vidcobert [50] and widely used CLIP-L [51] and the most similar reasoning model VAR [10]. Besides, also conduct experiments powerful LLM, including GPT-4 [33] and Gemini-Pro [44]. VLLM utilized for comparison includes widely accepted GPT4-o [33], VideoLLaVA [49], MiniGPT-4 [46], Video-Ilama [47], VideoChat2 [48] and MeiGGPT4-videe [45]. Specifically, LLMs and VLLMS are conducted under the few shot setting (In- Ln-Context Learning) following the causal discovery tasks in NLP [52-54), additionally, we reported the performance of fine-tuned VideoLLa and VideoChat2.
Metrics. We utilize the t aceuracy of the output causal relation chains with respect to the final event to evaluate the model's capability in causal discovery. Although our VGCM is designed to discover the causal relations leading to the final event, when truncating the video during inference and redefining the final event as the new result, VGCM can generate comprehensive causal diagram for the entire video without introducing additional training. Consequently, in addition to the primary metric accuracy, we introduce Structural Hamming Distance (SHD) [55, 56] supplementary metric. SHD measures the degree of matching between comprehensive causal graphs by summing the number of incorrect causal relations. In the MECD test set, the average number of causal relations in video causal graphs is 12.31, and a lower Ave SHD value of the test set indicates better performance.
We report the quantitative results in Tab. 1. Our VGCM without causal inference reaches an accuracy of 66.9%, demonstrating basic reasoning capabilities. Furtherly, the complete VGCM reaches a better performance with accuracy of 71.2%, outperforming the GPT-4, GPT4-o, fine- tuned VideoLLa VA [49] by 11.6%, 5.7%, and 4.1% Additionally, we explored the effect of altering the input format of the two modalities in Appendix Sec. C. indicating that VGCM is not dependent on the input format. The results compared with the two metrics indicate that for most models, accuracy is already adequate to represent their causal discovery capabilities. However, the additional metric Ave SHD indicates that Gemini and GPT-4 exhibit a superior overall capacity for discovering complete relations. An example of the output complete causal diagram is visualized in Figure 7.
Results.
GPT-4 [33] stands out as one of the most advanced L.LM models, however, we found that even being provided with sufficient shw-shot examples (detailed in Appendix Sec. C.2), its accuracy remains at
8

---

74
And Bresbaurent toling bitting the foor.
TOP- Ascuracy
Rundom Flipped Data
Duta Volume
86
Brushhiigg
Ihrosinng Bi serdwichh LALDAN
0.1
0.2
0.3
0.4
11.5 Pencenaage
0.6
0.7
0.8
0.9
Figure 5: Dataset robustness. Accuracy decreases slightly when increasing noise, and increases slowly when increasing the training data.
People: A, B,
Center
causality
Two emilattes division
1.2
1.15
VGCM VOCM CW
Figure 7: Complete causal diagram.
1.1
1.05
Table S: Open-set ability VGCM.
0.95
Method
TOP-1 Accuracy
Training Fpoch
5
VAR [10]
54.8
VGCM
59.2
Figure 6: Causality discovered analysis. The simi- larity of masking causal premise events is obviously lower through counterfactual intervention.
VGCM
64.4
only 59.6%. Possible explanations may be due to task contamination [57], GPT-4 mainly performs well on datasets released before the training date, while our task is novel. Moreover, other reasons may include the causal hallucination problem of establishing a threshold for differentiating between scenarios with and without causality [58]. For further insights into GPT-4's failure cases, refer to Appendix Sec. B.2.
As illustrated in Tab. 6, we have assessed the inference speed of various models, with VGCM achieving swift 0.76 seconds per sample. The proposed method incurs an overhead of only 8.57% over the Videobert baseline. It is noteworthy that our inference speed is 3 to 6 times faster than that of all Video LLMs. The inference speed experiments were conducted on NVIDIA A6000 GPU,
5.2 Ablation Study
Video Granger Causality Model design We designed causal discovery model based on the Granger Causality, three auxiliary losses are applied. The performances in Tab. indicate that our VGCM benefits from the design of Lv and Lc, for they support our method of inferring causal relations by facilitating the model with event prediction ability. Ls also benefits our model by supervising the causal feature similarity of eN with and without non-causal event ex masked.
Front-door adjustment with chain of thoughts candidate The method does improve reasoning ability in Tab. 2. We conduct an experiment in Tab, 4 for further proof. Since events closer to the result event are higher as the cause, the model likely learns these biased time-domain tendencies. So compare the accuracy of VGCM without front-door adjustment with chain of thought candidate and VGCM in determining the first relation r1 and the last relation r ~- The results demonstrate that temporal illusory causality is greatly mitigated, visualization can be found in Fig. 8 Example
Counterfactual intervention with existence-only descriptions The performance in Tab. 2 shows that counterfactual intervention with existence-only descriptions does facilitate the model with powerful reasoning ability. We dive into further analysis on the basis that when a non-causal event masked, the causal feature F fed into the causal relation head should be similar to the unmasked feature FP, instead, a bigger gap appears when masking a causal event. For stronger proof, we measure the difference in feature similarity in Tab. 3 and Fig. 6. We define the similarities division as the quotient of the similarity F F²) with a non-causal é masked over with a causal éş masked. In the experiment, we find that the similarity division is always above 1 without the counterfactual intervention, however, the existence illusory is solved with counterfactual intervention for the reason that the division is below 1 of VGCM, example visualization can be found in Fig. 8 Example 2.
5.3 Robustness Analysis
Model Robustness To prove our model's robust reasoning ability, we split the MECD dataset into five categories, and conduct an experiment similar to the open-set setting with cross- validatio...... VGCM

---

without causality with causality
Illusory temporal causality removed by causal inference
Example 1 pregaration
instruction
sharpens one side points to the knife another side
throw clean
Example
Illusory existence causality removed by existence description
interviews riders race information
wait to take off
man falls
another falls off
interviows two men
Existence-only description: There is interviewer and racer.
Figure 8: Successful abduction examples of our VGCM. Results indicate that after utilizing causal inference methods, illusory causality is suppressed and robust abduction ability is facilitated.
Table 6: Infernce speed. Our VGCM is 3-6 times faster than all Video LLMs while
Table 7: Model's generalizability test. Higher VQA Ace and VQA Score are reached when prompted with causal relations from our VGCM.
slightly slower than the baseline.
Model
Inference Speed
Output Causal Relations
VQA Acc
VQA Score
Videobert [50]
0.70
w/o (Standard QA setting for VL.LMs)
43.17
2.82
Our VGCM
0.76
Gemini-Pro [44]
49.10
2,90
VideoL.LaVA [49]
2.12
GPT-4 [33]
49.36
2.89
VideoChat2 [48]
2.96
VideoChat2 [48]
51.01
2,95
MiniGPT4-video [45]
3,98
Videol.LaVA [49]
51.88
2.93
MiniGPT-4 [46]
4.72
Our VGCM
62.21
3.12
reaches an average accuracy of 64.4%, outperforms VGCM without causal inference and VAR by 5.2% and 9.6%, details can be found in Appendix Sec. D.1.
Moreover, to further validate the generalization capabilities of our model, we evaluate the quality of output causal relations on related and representative video reasoning task: Video Question Answering (VQA) as shown in Tab. 7. Specifically, during inference on the multi-event subset of ActivityNet-QA [59] (The part that overlaps with the MECD test set), we prompted MiniGPT4- video [45] with additional causal relations outputs alongside the standard question inputs. This paradigm facilitates the VLLMs in considering the task from causal perspective. As shown in the table below, when prompted with these additional causal relations, the answering accuracy of MiniGPT4-video [45] improved by our VGCM surpasses other strong VLLMs like VideoChat2 [48]. These findings confirm that our model can provide accurate causal perception for videos, significantly improving performance related video reasoning tasks.
Dataset Robustness We study the subjectivity and data volume of our proposed MECD dataset, which is shown in Tab. 5. In the experiments of increasing the ratio of randomly flipped annotated causal relations (flipping only one relation of the whole causal relations of video), the accuracy decreases slightly, demonstrating the small amount of subjectivity in labeling does not have a serious impact. Besides, we analyze the scale data, the increment from 600 examples to 806 examples yields very modest improvement, indicating the adequacy of our dataset.
6 Conclusion
We proposed a novel task, multi-event video causal discovery (MECD), which focuses on event- level causal discovery in long-term videos. Besides, we built the MECD dataset with long-term daily life video datasets with causal relations to support this task and proposed the first video events causal discovery framework VGCM in the principles of Granger Causality. Additionally, our proposed VGCM was facilitated with deeper reasoning ability through causal inference with the chain of thoughts and existence-only descriptions. Our VGCM significantly outperforms GPT-4o and VideoLLaVA by 5.7% and 4.1 %6 respectively, demonstrating its robust reasoning ability.
10

---

7 Acknowledgement
The paper is supported in part by the National Natural Science Foundation of China (No. 62325109, U2 U21B2013) and the Lenovo Academic Collaboration Project.
References
[1] Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou. Mist: Multi- modal iterative ppatial-tempore transformer for long-form video question answering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14773- 2023.
[2] Ruoyue Shen, Nakamasa Inoue, and Koichi Shinoda. Text-guided object detector for multi- modal video question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1032-1042, 2023.
[3] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained Image-languae model for video localization and question answering. arXiv preprint 2.05::2300,066888, 2023,
[4] Tianwen Qian, Ran Cui, Jingjing Chen, Pai Peng, Xiaowei Guo, and Yu-Gang Jiang. Locate before answering: Answer guided question localization for video question answering. IEEE Transactions on Multimedia, 2023.
[5] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
[6] Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33:9180-911 9192, 2020.
[7] Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. arXiv preprint arXiv:1910.0474: 2019.
[8] Tao Zhuo, Zhiyong Cheng, Peng Zhang. Yongkang Wong, and Mohan Kankanhalli. Explainable video action reasoning via prior knowledge and state transitions. In Proceedings of the 27th aem international conference on multimedia, pages 521-529, 2019.
[9] Yang Jin, Linchao Zhu, and Yadong Mu. Complex video action reasoning via learnable markov logic network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3242-3251, 2022.
[10] Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang. Visual abductive reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15565-15575, 2022.
[11] Clement Tan, Chai Kiat Yeo, Cheston Tan, and Basura Fernando. Abductive action inference. arXiv preprint arXiv:2210.1332 139944 2022.
[12] Anil Seth. Granger causality. Scholarpedia, 2(7):1667, 2007.
[13] Mariusz Maziarz. A review of the granger-causality fallacy. The journal of philosophical economics: Reflections on economic and social issues, 8(2):86-00) 105, 2015.
[14] Ali Shojaie and Emily B Fox. Granger causality: A review and recent advances. Annual Review of Statistics and Its Application, 9:289-319, 2022.
[15] Judea Pearl. Causal inference. Causality: objectives and assessment, pages 39-58, 2010.
[16] Xu Yang, Hanwang Zhang, and Jianfei Cai. Deconfounded image captioning: A causal retrospect. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 13010, 2021.

---

[17] Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision- language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 9847-9857, 2021. 2021.
[18] Long Chen, Yuhang Zheng. Yulei Niu, Hanwang Zhang, and Jun Xiao. Counterfactual samples synthesizing and training for robust visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[19] Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. Clicks can be cheating: Counterfactual recommendation for mitigating clickbait issue. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288-1297 2021.
[20] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
[21] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, Advances in neural information processing systems, 01122199-222111 2022.
[22] Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309. 15402, 2023.
[23] Andreas Gerhardus and Jakob Runge. High-recall causal discovery for autocorrelated time series with latent confounders. Advances in Neural Information Processing Systems, 33: 13:15555-, 2020.
[24] Charles K Assaad, Emilie Devijver, and Eric Gaussier. Discovery of extended summary graphs in time series. In Uncertainty in Artificial Intelligence, pages 106. PMLR, 2022.
[25] Joris M Mooij, Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. The Journal of Machine Learning Research, 21(1):3919-400 2020.
[26] Xiangyu Sun, Oliver Schulte. Guiliang Liu, and Pascal Poupart. Nts-notears: Learning nonpara- metric dbns with prior knowledge. arXiv preprint arXiv:2109.0420 2021.
[27] Roxana Pamfil, Nisara Stiwattanaworachhii, Shaan Desai. Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beaumont, and Bryon Aragam. Dynotears: Structure learning from time-series data. In International Conference on Artificial Intelligence and Statistics, pages 1595-1605. PMLR, 2020.
[28] Tian Gao, Debarun Bhattacharjya, Elliot Nelson, Miao Liu, and Yue Yu. Idyno: Learning nonparametric dags from interventional dynamic data. In International Conference on Machine Learning, pages 6988-7001. PMLR, 2022.
[29] Ruichu Cai, Siyu Wu, Jie Qiao, Zhifeng Hao, Keli Zhang, and Xi Zhang. Thp: Topolog- ical hawkes processes for learning granger causality event sequences. arXiv preprint arXiv:2105.10884, 2021.
[30] Wei Chen, Jibin Chen, Ruichu Cai, Yuequn Liu, and Zhifeng Hao. Learning granger causality for non-stationary hawkes processes. Neurocomputing, 468:22-32, 2022.
[31] Tsuyoshi Idé, Georgios Kollias, Dzung Phan, and Naoki Abe. rardinality-rogullaiieed hawkes- granger model. Advances in Neural Information Processing Systems, 34:2682-2694, 2021.
[32] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[33] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arX1v:2303.00874.00 2023.
12

---

[34] David Lopez-Paz, Krikamol Muandet, and Benjamin Recht. The randomized causation coeffi- cient. Mach. Learn. Res., 16:2901-2900 2015.
[35] Jean-François Ton, Dino Sejdinovic, and Kenji Fukumizu. Meta learning for causal direction. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9897-9905, 2021.
[36] Hebi Li, Qi Xiao, and Jin Tian. Supervised whole dag causal discovery. arXiv preprint arXiv:2006. 04697, 2020.
[37] Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara L Berg, and Mohit Bansal. Mart: Memory- augmented recurrent transformer for coherent video paragraph captioning. arXiv preprint arXiv:2005.05402, 2020.
[38] Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6847-6857, 2021.
[39] Ziqi Zhang. Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zueng-Jun Zha. Object relational graph with macher-recommendel learning for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13278-13288, 2020.
[40] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
[41] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
[42] Judea Pearl. Causality. Cambridge university press, 2009.
[43] Yang Liu, Guanbin Li, and Liang Lin. Cross-modal causal relational reasoning for event-level visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[44] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
[45] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny, Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. arXiv preprini arXiv:2404 40440311.,,, 2024.
[46] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprins arXiv:2304. 10592, 2023.
[47] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tur sonnneeetttt-teeeddddee audio-visual language model for video understanding. arXiv preprint arXiv:2306.02883 2023.
[48] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 22195 22195-22206, 22206, 2024.
[49] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 2023.
[50] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning, In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473, 2019.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
13

---

[52] Xinyi Wang. Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
[53] Kun Luo, Tong Zhou, Yubo Chen, Jun Zhao, and Kang Liu. Open event causality extraction by the assistance of Ilm in task annotation, dataset, and method. In Proceedings of the Workshop: Bridging Neurons and Symbols for Natural Language Processing and Knowledge Graphs Reasoning [NeusymBridge)@ BREC-COLING-2024, pages 33-44, 2024.
[54] Aniket Vashishtha, Abbavaram Gowtham Reddy, Abhinav Kumar, Saketh Bachu, Vineeth N Balasubramanian, and Amit Sharma. Causal inference using Ilm-guided discovery. arXiv preprint arXiv:2310.1511 151.17 2023.
[55] Verónica Rodríguez-LLpez and Luis Enrique Sucar. Knowledge transfer for causal discovery. International Journal of Approximate Reasoning, 143:1-25, 2022.
[56] Konstantina Biza, Ioannis Tsamardinos, and Sofia Triantafillou. Tuning causal discovery algorithms. In International Conference Probabilistic Graphical Models, pages 17-28. PMLR, 2020.
[57] Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few-shot anymore. arXiv preprint arXiv:23/2./6337, 2023.
[58] SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313 2024.
[59] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134, 2019.
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[61] Chang Gong, Di Yao, Chuzhe Zhang, Wenbin Li, and Jingping Bi. Causal discovery from temporal data: An overview and new perspectives. arXiv preprint arXiv:2303. (tti:2303.101111 12, 2023.
[62] Blai Meléndez Catalán, Emilio Molina, and Emilia Gómez Gutiérrez. Bat: An open-source, web-based audio events annotation tool. 2017.
[63] Weiyao Lin, Huabin Liu, Shizhan Liu, Yuxi Li, Hongkai Xiong, Guojun Qi, and Nicu Sebe. Hieve: A large-scale benchmark for human-centric video analysis in complex events. Interna- tional Journal of Computer Vision, 131(1 (1(11):29943301 2023.
[64] Renáta Németh, Domonkos Sik, and Fanni Máté. Machine learning of concepts hard even for humans: The case of online depression forums. International Journal of Qualitative Methods, 99:16994069299993338, 2020.
[65] Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Ex- ploring llm-based general bug reproduction. In 2023 IEEF/ACM 45th International Conference on Software Engineering (ICSE), pages 2312-2323, IEEE, 2023.
[66] Toufique Ahmed and Premkumar Devanbu. Few-shot training Ilms for project- specific code- summarization. Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1-5, 2022.
14

---

Appendix
A Implementation details
Pretraining process For each video event, visual features are extracted using ActivityNet pretrained ResNet200 [60], following [10, 37-39]. Prior domain knowledge could benefit the Granger Causality Causal discovery method [61], we fully pre-trained our model for the dense video captioning task on a 3.1k ActivityNet Captioning video dataset, each video sample contains more than 4 events. conducted on 1 NVIDIA A40 GPU. We train our model for 20
Training set All the experiments epochs with a learning rate of 16e-5 about 6 hours. Our optimizer is consistent with BertAdam [50] optimizer, with 3 epochs of warm-up. The open-set experiment set can be found in Appendix Sec. D.I. We report the average results during all experiments under three random seeds (2023, 2024, 2025). The ablation of two modalities can be found in Appendix Sec. C.I.
Model details Our encoder Ency, Encc, and multi-modal video decoder Dec are built upon Videobert [50], joint model for video and language representation learning. The details of the GPT-4 API prompt can be found in Sec. D.2 in the Appendix.
Hyperparameters Ac An. Av. Äs are set be 1.0, 4.0, 0.25, 0.05. Maximum input lengths of the caption, the chain of thoughts, and the existence-onl descriptions are set to 50.
Implementation of VAR We migrate the VAR our task through an effective method: We mask any event ex. (k<N), and then utilize the fully trained VAR to perform event prediction of ex. If the prediction results é is obviously various from ex, is considered that the event ex is non-causal. Then r& is labeled as 0; in the opposite case, is labeled as 1. We also report the average results of VAR under three random seeds (2023, 2024, 2025).
Implementation of LLMs As for GPT-4 and Gemini-Pro, We report the average results of three calls.
Implementation of VLLMs We report the average results of VLLMs under three random seeds (2023, 2024, 2025). When VLLMs do not output r in the required format, we order them to re-answer until the outputs match the format to measure their best performance.
B Additional Visualization
B.1 Successful abduction examples of our VGCM
In Fig. 9, additional examples are presented to showcase the performance of our VGCM, particularly excelling in complex abduction scenarios. The first example successfully discovers that there is no causal relation between We see the targets in front of a backdrop." and The instructor walks over to the targets.", despite the backdrop being necessary object of the result event. This abduction avoids the illusory existence causality.
The second example successfully discovers that there is causal relation between The video shows different cricket matches taking place where Sri Lanka is playing against teams from different countries." and The stadium is filled with spectators cheering for the cricketers." ", despite the spectators' cheering often happening after the game playing. This abduction avoids the illusory temporal causality. Both instances align with the foundational principles motivating method design.
The third example shows the 83.3% accuracy of video causal relations abduction. Notably, it correctly discerns the most complex causal relations, however, it fails to realize that person B doesn't hit the tennis ball can contribute to the anticipation of the result event of continuous hitting. This indicates that VGCM might still require refinement in understanding causality within higher-level semantics, especially in the mining of some obscure mental emotional influences. We will strive to explore further solutions in the follow-up work.
B.2 Failure abduction examples GPT-4
While examining the causal discovery results of GPT-4, we encountered some intriguing observations. In the example presented in Fig. 10, the GPT-4 API incorrectly infers that all premise events have a causal relation with the result event of the team winning. However, the initial appearance of the team
15

---

without causality causality
First Leads another
backdrop
Backdrop collapses
People laughing
Instructor walks
Unprofessional but persistent
Collapse linked to the check
Attracting instructor's attention
cricket play
competitive game Home team blue Batsmon scores four Different game
Spectators cheering
L Game attracts spectators
etmosphere
K Directly contributes to the final outcome
hits ball
hits a ball
presares to misses hittins
hits asgin
hits back
B&C Continue hitting
Creating an anticipation
Expecting subsequent attempts
Fail discover
Creating anticipation discover)
① E Emphasizing the reciprocal nature
Figure More successful abduction examples our proposed VGCM. The relation which reveals our method of eliminating illusory causality is marked by red five-pointed star The failure case is annotated red dotted line
does not directly lead to their victory, and the subsequent celebrations also lack any causal links with the outcome. Indeed, the false discovery by the GPT-4 API could stem from the illusion of causality, where the team's mere presence is perceived the illusion of temporal causality may also play a role, as statistics indicate that celebrations often occur before the announcement of the competition winner. These cognitive biases could contribute to the erroneous causal inference made by the GPT-4 API this scenario.
necessary condition for the outcome. Additionally,
When we request detailed explanation from the GPT-4 API regarding the discovered causal relation between the result event and the initial appearance of the team, the response is "Setting up the motive for the last event." Obviously, the GPT-4 confuses causality with the illusion of existence causality. In contrast, our VGCM makes a correct inference this scenario. Furthermore, when we seek detailed reasons from the GPT-4 API for the discovered causal relations between the result event and the celebrations, the answer is "Indicating their satisfaction and confidence in their performance, implying they believe they have good chance to win." Here, the GPT-4 API misinterprets causality by associating it with the expression subjective emotions unrelated to the events in question. may mistake the display of subjective emotions for the presence of objectively implied causality.
B.3 Annotation pipeline of MECD dataset
To improve the accuracy and mitigate subjective biases in annotating causal relations, we employ cross-annotation strategy [62-64]. The interactive interface used by annotators during the labeling process of our MECD dataset is illustrated in Fig. 1.. Each video example is endowed with
16

---

without causality with causality
Attempts flips
Another follows
Team dances
Team celebrates
Members
False discovery,
False discovery,
Team wins
False discovery
Figure 10: Failure abduction examples GPT-4. Many failure cases GPT's causal reasoning are due to confusion with illusions and the conflation of subjective emotions with objective laws.
Annotator 4's view:
Videos:
Captions:
1. "man is sitting on table eating a sandwich in a rstaurant talking to other man.",
2." man is watching the man while eats and talking to him.",
3." both men stands and play rock paper scissors and man hits the man sand this falls to the floor."
4. man sitting on chair and talks the on thefloor and throw him the sandwich.", "
5. anoher man siting front talk to man in desk.",
6. man stands and talk to the man on the floor and hits him again and the man spit blood."
Annotated reference:
Tips:
GPT-4 Relation: 00111
Annotator Relation; 01111
1.lllusory existence causality and illusory temporal causality should avoided.
Annotator Relation: 01111
Annotator Relation: 01111
2. Click
to play the video.
Annotator Relation: 11111
Your Annotation:
(0 for without causality. with causality)
Figure 11: Annotation pipeline of MECD dataset. Illustration of the interactive interface used by annotators during the labeling process of our MECD dataset. Key information is provided during annotation.
"relation" attribute. First, GPT-4 [33] provides an initial annotation of attribution, which is then further refined by five human annotators. Ground truth labels are determined based on the majority choices of the annotators regarding causal relations. This methodology ensures the creation of more reliable and objective dataset.
B.4 Annotation examples of MECD
Annotation examples MECD are shown in Fig. 12, our MECD dataset is carefully annotated to support the challenging task proposed with complete premise information.
17

---

with causalty
"Relations"
01111
"enttncees"
1.Tnah la sitting table coting sandwich in rsturant talking to otherman"",
watching the
him."
and
this full floor",
sitting
theloor
man sting 6. mon stands and talk the man on the floor and hits him again and the man spit blood."
desk.".
"COT"
Initallyy introduces the key characters, interactive iniionsipp
which enjostng sondeich
setting communication confrentation
dynamic sets ecosyatum 2 The event shows observer-man's who interest and engagement with the eating man, Hustre ng reletionship that could potentially vully be comflicity hence leading to physical interaction in the end."
might
3 "his sudden aggressive action shifts the dynamic between the two men, escaloting the shustion from prestously pesceful meai one with physical confrontation. This change interaction seis the stage for the violens unding." ", a This
throwing sandwich the the flocr further escalates the houility the interaction. Dionegert for other dignily apparent, depicts
5:
man on protongs conflict between main characters, pushing aituation towards rs bloody anding."
lanoring
"Existence"
1 There a man sitting a table, resturant, sandwich another man...
2 There a man watching and talking.".
3: There are two men, rock paper schoors game and sand."
4. There is man stting chait another the floor and sandwich"" desk,
"hhre
another onother the desk."
the florr"
There
without cauzaltty caualityy
"Relations"
!
11010
"Sentences"
Th cricket toam Sri Lanka is plaving against anchher
2. the cricecters are playing competitive me the field.".
The Sii Lanka team reprisenced the blue unilerm..
The batsman scores four the bowler throus overhand ball",
video different matches
Lanka
countries.",
stadium
spectators cheering
"COT"
1 cricket
another
competitive, noturally
expectations, leading competitvve reaction
the game implies spectations.
every
signfficntly.
3, "hh uniform identifies the teams and creates sence belonging, unlty and rhoiry omong the teoms and the spectators, sol solidifying sides spectators will cheer fox".
a The act of sooring runs in cricket match is significam event directly contribune buttes the firal outoome the marthh and dinaws strong responses from the crowd, the the the louder the cheers".
5,
motches featuring Lanka culminatin stadium cheering
builds the
the
"xistence"
1. There a cricket team of lanbe and another country.".
2 Thore are cridkcters and fiels""
3 There a Larkka team blue uniform...
4 T'Thrre
overhand ball."
Figure 12: Annotation examples of MECD. Annotation examples of MECD are shown. Newly annotated attributes "Relations", "COT", "Existence" and the existing caption attribute "Sentences" are shown along with the video frames.
C Additional Experiments
C.1 Modalities analysis of causality discovering
The MECD task employs both video input and corresponding captions to uncover causality. Our objective in this experiment is to evaluate the degree of reliance these two modalities in causal discovery. Typically, each event in our MECD task consists of a textual input with an average of 13.5 words caption and visual input of 50 frames.
To investigate the influence the text modality, we employ masking strategy for the input caption of the premise event, gradually increasing the masking ratio from 10% to 80%. The results presented
18

---

Table 8: VGCM performance with masked premise event caption input. indicates frames masked at the same time.
Table 9: VGCM performance with masked premise event visual input. indicates 10 words masked at the same time.
Num of words masked
Accuracy
Num of frames masked
Accuracy
non man-md
71.2
non-masked
71.2
2 per event
70.2
per
70.3
5 per event
69.7
15 per event
69.0
8 per event
69.2
20 per event
68.3
8 per event"
67.4
20 per event"
67.1
11 per event
68.9
40 per event
67.9
TOP-1 Aecuracy
57
56
55
54
53
0
2
3
5
7
Number of examples
Figure 13: The trend chart of inference accuracy as the number of examples changes under the In Context Learning paradigm. Accuracy increases slightly when increasing the number of few-shot examples, when the number of examples > 3, the accuracy tends to remain constant.
in Tab. indicate that our VGCM does not rely on the textual modality input; VGCM can also conduct causal discovery for videos without any captions.
In contrast, the experimental results suggest a more obvious performance decrease towards less visual modality input in the causality discovery task, as shown in Tab. 9. However, even with 80% masking of either modality, the results consistently outperform our strong baseline model, VideoLLaVA, underscoring the robust causal discovery capability of VGCM.
Furthermore, we conducted experiments involving simultancous masking of both modalities of information. Interestingly, observe noticeable decrease in aceuracy compared to when only one modality is masked. This observation highlights the importance jointly considering both modalities in the causality discovery task.
C.2 Adequacy the prompts provided to GPT-4
To delve deeper into the limitations of the straightforward baseline approach of prompting GPT-4, we examined the correlation between its accuracy and the number of video examples provided in the few-shot prompts. The findings, illustrated in Fig. 13, suggest that increasing the number examples shown to GPT-4 does not effectively enhance its accuracy. This suggests that the limitation of the GPT-4 baseline is not strongly correlated with the number of presented examples but rather is more attributable its intrinsic limitation in understanding complex causal relationships solely through text modality.
D Experiments details
D.1 Details of causality discovery experiment
In the open-set experiment of exploring reasoning ability, the five categories mainly consist of the activities below, demonstrating the colorful daily activities included in our dataset.
Sports: Arm wrestling, BMX, Beach soccer, Blow-drying hair, Capoeira, Croquet, Futsal, Ice fishing, Kite flying, Playing beach volleyball
19

---

Assembling bicycle, Baking cookies, Building sandcastles, Carving jack- o-lanterns, Decorating the Christmas tree, Hanging wallpaper, Making a cake, Making an omelet, Painting fence, Putting in contact lenses
Creating & Making:
Daily Activities:
Changing car wheel, Cleaning sink, Drinking coffee, Eating ice cream, Gargling mouthwash, Hanging wallpaper, Kneeling, Peeling potatoes, Putting on shoes, Washing face
Performing: Baton twirling, Bullfighting. Drum corps, Fun sliding down, Hula hoop, Playing congas, Playing drums, Playing rubik cube, Playing saxophone, Tumbling
Socializing: Beer pong, Playing blackjack, Playing field hockey, Playing harmonica, Playing piano, Playing squash, Playing water polo, Rock climbing, Smoking hookah, Belly dance
D.2 Prompts to generate auxiliary premise information
In this section, introduce the detailed method of prompting GPT-4 [33] to generate more premise information. Firstly, we prompt the GPT-4 with the following prompts to generate the description-only sentences.
Task: Each input consists of sentences, and the text description of each sentence has been given correspondingly (separated by " ",) You need to offer the existence description of each sentence.
Besides the task description, we further append the few-shot paradigm (In-Context Learning) intro- duced [52-54, 65, 66). Similarly, we prompt the GPT-4 [33] with the following instructions to generate the chain-of-thouggh candidate sentences in the same few-shot paradigm.
# Task: Each video consists of events and the text description of each event has been given correspondingly separated by ",). First n-1 events might be the cause of the last event. You need to offer the chain of thoughts you derive that causes the last event.
D.3 Chain of thoughts examples
In this section, we present an example of the chain of thoughts prompted, the corresponding premise event and result event descriptions are also shown below:
{"premise event sentence": "He continues sharpening the knife, turn it again to further sharpen the other side and wipe it with paper towel."
"result event sentence"; "Throws the old and dirty paper towel and reach the roll of paper towel and clean the knife."
"COT": "The repeated action of sharpening and wiping the knife underscores the importance of both the knife's sharpness and cleaners, leading directly to the final action of disposing of the used paper towel and getting new one to ensure the knife is thoroughly clean"}
The chain of thoughts shown above provides a logical causal chain between the event of the cleaning of the knife and the subsequent throwing of the dirty paper towel. The reasoning initiates by considering the heightened need for sharp and pristine knives achieved through sharpening. This causal chain is then expanded by suggesting that this demand could have led the replacement of the paper towel. The chain of thoughts generated from GPT-4 serves candidate in the process of correct reasoning, contributing to the exploration of potential causal relations.
E
Limitations and future works
The video we input for causal discovery needs to provide timestamps, we encourage future work to realize causal discovery with weakly annotated inputs.
20

---

2. VGCM might still require refinement in understanding causality within higher-level semantics, especially in the mining of some obscure mental or emotional influences according the failure cases analysis in Appendix Sec. B.1.
3. VGCM is based the supervised paradigm of causal discovery, subsequent works may be able extend to the unsupervised paradigm.
4. The causal graphs proposed by the MECD may also enhance other video understanding tasks, such as video dense captioning and video event prediction, or could introduced to other reasoning tasks, including text reasoning and mathematical reasoning tasks.
The evaluation results of VLLMs and LLMs on the MECD task also help researchers study language models' current issues and limitations in complex reasoning.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions and scope are summarized in Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Sec. the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification: We have already provided the proof of theoretical results in Sec. 5.1, Sec. 5.2 and Sec. 5.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Implementation details are provided in Sec. A in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Codes and datasets are released on the GitHub Page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wctpublicc///cccccccccccccc...le gwwcc//ggu/////eee (CodeSummissionPor
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are provided in Sec. A in the Appendix.
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We report the average results under three random seeds (2023, 2024, 2025).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Implementation details are provided in Sec. the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrrp.ee/ee
Answer: [Yes]
Justification: We conducted the research in the paper conform, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: ActivityNet Captions Dataset is with no license needed.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Implementation details and limitations are provided in Sec. A and Sec. E in the Appendix.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "rOlvIvbLnU", "reviewer": "Reviewer_9my4", "summary": "The paper introduces the Multi-Event Causal Discovery (MECD) task, aiming to uncover causal relationships in videos with multiple events. It presents a novel framework inspired by the Granger Causality method, utilizing a mask-based event prediction model to perform causal inference. The paper also introduces a new dataset for training and evaluation, demonstrating the framework's effectiveness in outperforming existing models like GPT-4 and Video-LLaVA.", "questions": "I appreciate that the authors proposed a more novel and interesting setting. However, given the missing details and overly complicated pipeline, I tend to reject this paper for now. And I hope the authors can provide more details about the training and architecture. This will help us reviewers to make a fair evaluation.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novelty and Relevance: The paper addresses a significant gap in video reasoning tasks by focusing on multi-event causal discovery, which is more reflective of real-world scenarios.\n- Framework Design: The use of Granger Causality, combined with advanced causal inference techniques like front-door adjustment and counterfactual inference, is innovative and well-justified.\n- Empirical Validation: The framework demonstrates substantial performance improvements over state-of-the-art models, indicating the robustness and efficacy of the proposed approach.\n- Dataset Contribution: The creation of the MECD dataset, with detailed annotations and diverse scenarios, is a valuable contribution to the field.", "weaknesses": "- Complexity of Implementation: The framework's reliance on multiple advanced techniques may pose challenges for implementation and replication by other researchers. Besides, does VGCM run slower than those VLLM baselines?\n- Missing Details: What's the architecture of different encoders? How do the authors pretrain them? How do different loss design? The authors should give more details about fair comparisons.\n- Some Typos:\n  - In Figure 3, the box colors for the masked and unmasked input should be green and orange.\n  - In Line 166, it should be $Env_{V}$ and $Env_{C}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions. We’re glad that you found our work interesting and novel. We address your concerns below.\n\n**Q1. Complexity of implementation**\n\nWe have the following points for this question:\n\n1. *The main pipeline of our framework is clear and not overly complex in its implementation*:\n\n- Our basic pipeline (Sec. 4.1) follows a straightforward dual-branch architecture for processing visual and textual inputs. Semantic and causal information interact between two branches, culminating in causality estimation via a masked-based prediction task in both modalities.\n\n- While the causal inference component (Sec. 4.2) may require some background knowledge to fully grasp theoretically, its practical implementation is neat and clear. In our final manuscript, we will provide pseudocode to further clarify these aspects.\n\n- Contrary to the reviewer's concerns, our framework does not rely on numerous advanced techniques or require additional large-scale pretraining. It only requires pretrained visual and textual feature encoders, which are fundamental and readily accessible in the multi-modality understanding community.\n\n- Our framework is also highly flexible, allowing researchers to replace or modify specific modules for targeted improvements. Options include selecting different visual/textual encoders, altering interaction methods between modalities, or adjusting the similarity evaluation criteria for the prediction task.\n\n2. *The simplicity of our implementation is further evidenced by its parameters and inference speed*:\n\n- Our model contains only 144M parameters, significantly smaller than even the lightest VLLM models (e.g., Video-LLaVA, VideoChat2), which have 7B parameters (~50 times more than ours).\n\n- Our method is fast (0.76s/sample) in inference speed. The overall method introduces only 8.57% overheads to the VideoBERT baseline. Notably, our inference speed is **3-6 times faster** than all VLLMs (as shown in Table 1). Our inference speed experiment is conducted on 1 NVIDIA A6000 GPU.\n\n3. *To facilitate replication and further research, we will release our code along with the dataset as soon as possible. This will enable other researchers to reproduce our results and build upon our work*.\n\n| Model| Inference Speed (seconds/sample) |\n| ----------------------------------- | :-------------------: |\n| VideoBERT|0.70 |\n| *Our VGCM ( built upon VideoBERT )* |**0.76**|\n| Video-LLaVA|2.12|\n| VideoChat2|2.96|\n| Minigpt4-Video|3.98|\n| Minigpt4|4.72|\n\n*[Table 1. Efficiency of different models. Reported with average inference speed of each sample.]*\n\n**Q2-1. Details of encoders**\n\nWe do have provided more details about the architecture of encoders in our supplementary (Sec. B lines 510-511). Specifically, our encoder  $\\text{Enc}_v$ and $\\text{Enc}_c$ are built upon VideoBERT, a joint model for video and language representation learning. \n\nBesides, as also provided in our supplementary (Sec. B lines 501-502), the architecture of our pretrained encoder $\\Phi_{pre}$ is ResNet-200, following the setting of the Dense Video Captioning task [1-2].\n\n**Q2-2. Details of pretraining**\n\nWe have provided the detailed pre-training procedures in our supplementary (Sec. B, lines 501-504). For $\\text{Enc}_v$ and $\\text{Enc}_c$, our pre-training approach is relatively modest. We conducted a video dense captioning task on 3.1K samples from the ActivityNet Caption dataset to warm up the model. It requires a low resource footprint and time cost (2 NVIDIA A6000 GPUs for 8 hours), while leading to a significant improvement in final performance from 69.5 to 71.2.\n\n**Q2-3. Details of loss design**\n\nOverall, the causal relation loss $L_R$ provides direct supervision for output causal relations using standard Cross-Entropy loss, while $L_C$, $L_V$, and $L_S$ aim to further enhance the causal discovery capability from different aspects.\n\nSpecifically, $L_C$ and $L_V$ introduce auxiliary caption and reconstruction losses to facilitate event prediction, implemented by Label Smoothing loss and MSE loss respectively. They are incorporated because the Granger Causality Method determines whether an earlier event aids in predicting a subsequent one.\n\n$L_S$, the similarity loss (implemented by InfoNCE), prompting causal relation discovery by comparing output causal feature similarity when a premise event is masked. The rationale behind this loss is that masking a non-causal event should result in a prediction of the result event similar to that of the unmasked stream.\n\nIn our manuscript, we have thoroughly examined the contribution of each loss term in Tab. 2 and Sec. 5.2 (lines 285-289). Furthermore, we have detailed the balance between these losses in Sec. 4.1 (lines 188-200), and outlined the specific hyperparameter settings in our supplementary (Sec. B lines 513-514).\n\n**Q2-4. Details about the training**\n\nDetails about the training procedures have been provided in our supplementary (cf. Sec. B, lines 505-509). \n\n**Q2-5. Details about fair comparisons**\n\nTo make a fair comparison, for multi-modal backbones VAR and VideoBERT, we reported their results after being fully trained on the MECD dataset. For all LLM-based models, we reported the results under the few-shot paradigm (cf. supplementary Sec. I, lines 601-606), where three representative examples are provided during inference, similar to the paradigm used in causal discovery in NLP tasks [3-4]. \n\n**Q3. Typos**\n\nThank you for your corrections. We will correct them in our final version.\n\n**References**\n\n[1] Visual abductive reasoning. CVPR 2022.\n\n[2] End-to-end dense video captioning with parallel decoding. CVPR 2021.\n\n[3] Reasoning subevent relation over heterogeneous event graph. KIS 2024.\n\n[4] Is chatgpt a good causal reasoner? a comprehensive evaluation. EMNLP 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Ac5pKXgZ9", "reviewer": "Reviewer_eXFX", "summary": "The paper introduces a new task and dataset, Multi-Event Causal Discovery (MECD) to better understand long videos in a casual perspective. Inspired by the Granger Causality method, the authors devise a framework, dubbed VGCM, to perform the Event Granger Test. Also, VGCM is combined with front-door adjustment and counterfactual inference to tackle the issues of causality confounding and illusory causality. Experiments show that VGCM outperforms GPT-4 and Video-LLaVA in providing causal relationships in multi-event videos.", "questions": "- The authors employ Accuracy as the primary performance metric. Could the authors consider or design some other metrics that are more suitable for Multi-Event Causal Discovery (MECD)?\n- It would be highly beneficial if the dataset and associated code could be made publicly available at the earliest opportunity.\n- Regarding lines 154-155, I suspect that the confounding factor in the spurious causal relationship could introduce significant discrepancies when comparing the output features with and without the factor. Could the authors provide further analysis on this matter?", "limitations": "The model depends largely on video captions, which makes it possibly incapable of processing video datasets without captions or descriptions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To advance comprehensive and structured causality analysis for videos with multiple events, the authors introduce a benchmark for the multi-event causal discovery task.\n- The authors have crafted an innovative model framework that integrates the Event Granger Test with various causal inference techniques.\n- Experiments demonstrate the efficacy of the proposed framework in providing causal relationships within multi-event videos.", "weaknesses": "- To enhance the demonstration of the model's generalizability, it is recommended to conduct experiments on a variety of related datasets.\n- In Figures 1(c) and 1(e), the events occurring within the video frames are not readily discernible. Additional verbal descriptions are recommended to facilitate a clearer understanding of the figures' intended message.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and for recognizing the novelty and contribution of our work. Please see the responses to your comments.\n\n**Q1. Model's generalizability**\n\nTo the best of our knowledge, our benchmark is currently the first and only one for the video causal discovery task. To further validate the generalization capabilities of our model, we evaluated the quality of output causal relations on a related and representative video reasoning task: Video Question Answering (VQA). \n\nSpecifically, during inference on the ActivityNet-QA dataset, we prompted Minigpt4-video with additional causal relations outputs alongside the standard question inputs. This paradigm facilitates the VLLMs to consider the task from a causal perspective. As shown in the table below, when prompted with these additional causal relations, the answering accuracy of Minigpt4-video improved by our VGCM surpasses other strong VLLMs like VideoChat2. These findings confirm that our model can provide accurate causal perception for videos, significantly improving performance on related video reasoning tasks.\n\n| Output Causal Relations             | &nbsp;&nbsp;VQA Acc | VQA Score |\n| ----------------------------------- | :-----------------: | :-------: |\n| w/o (Standard QA setting for VLLMs) |        43.17        |   2.82    |\n| w Gemini-Pro                        |        49.10        |   2.90    |\n| w GPT-4                             |        49.36        |   2.89    |\n| w VideoChat2                        |        51.01        | **2.95**  |\n| w VideoLLaVA                        |      **51.88**      |   2.93    |\n| **w Our VGCM**                      |      **62.21**      | **3.12**  |\n\n**Q2. Improve the display of Fig.1**\n\nThanks for your advice, in the final version, we will appropriately explain the key elements in Fig.1 and the revised version of Fig.1 can be found in the attached PDF file in our rebuttal.\n\n**Q3. Additional metrics**\n\nAs mentioned in our main manuscript (Sec. 2 lines 100-104 & Sec. 5.4 lines 318-321), our model can output a complete causal diagram, consequently, we can introduce Structural Hamming Distance (SHD) [1-2] as a supplementary metric. SHD measures the degree of matching between causal graphs by summing the number of incorrect causal relations. Compared to the Acc metric, it focus more on the global causal relationships for all events in each video.\n\nIn our MECD test set, the average number of causal relations in video causal graphs is 12.31. We report the average SHD for incomprehension in the following table, a lower Ave SHD value indicates better performance.\n\n| | Models | Ave SHD (12.31) $\\downarrow$ | Acc $\\uparrow$ |\n| :------------------: | ------------------- | :--------------------------: | :------------: |\n| LLM | Gemini-1.5-Pro      |             4.91             |      59.3      |\n| | GPT-4 |             4.92             |      59.6      |\n| Video LLM | Minigpt4-Video      |             5.16             |      56.8      |\n|  | Minigpt-4 |             5.14             |      57.5      |\n| | VideoChat2 |             4.89             |      60.7      |\n| | VideoLLaVA |           **4.85**           |    **62.5**    |\n| Multi-Modal Backbone | VAR                 |             4.96             |      57.3      |\n| | Videobert  |             4.95             |      60.9      |\n| **Ours** | **VGCM(Videobert)** |           **4.19**           |    **71.2**    |\n\nThe results also indicate that for a majority of the models, the current metric, accuracy in identifying causal relations leading to the result event, is already adequate to represent their causal discovery capabilities. However, Gemini and GPT-4 exhibit a superior overall capacity for discovering complete causal relations.  \n\n**Q4. Dataset & Code available**\n\nThanks for your suggestion. We confirm that our code and dataset will be released as soon as possible.\n\n**Q5. Confounding factor**\n\nAs you have thought, the confounding factor is unfavorable in causal relation discovery. For a further analysis of lines 205-226: When $e_k$ is masked for comparison, the causal relations between $e_k$'s adjacent events and the last event $e_N$ are affected, leading to a confounding of causal effects. \n\nIn lines 154-155, we conduct the Event Causality Test to compare the predictions of the two streams. Consequently, under the concept of control variables, as we want to evaluate the causal effect between the current event $e_k$ and $e_N$, we must prevent the causal effects between the adjacent events and $e_N$ from being affected by confounding factors. Confounding factors result in a redundant or missing causal effect. Therefore, we introduce causal inference to cut off the redundant causal effect between $e_{k+1}$ and $e_N$ by counterfactual intervention. Likewise, by reconstructing $e_k$'s causal effect through front-door adjustment, we introduce the distinction after reconstruction in the causal effect between $e_{k-1}$ and $e_N$ as the compensation.\n\n**Q6. Limitation: Captions dependency**\n\nCurrently, with the rapid development of VLLMs, we believe that generating accurate captions for videos will be much easier. \n\nMoreover, in our supplementary (Sec. E, lines 554-567), we have experimented with the dependency of the caption and video inputs, respectively. Specifically, we explored the effect of altering the input format of the two modalities and found that when we masked out 80% of the captions, the accuracy only dropped by 2.3%, proving that we can also conduct tests for videos without any captions. \n\nThanks for pointing this and we'll delve deeper into this issue in our future work.\n\n**References**\n\n[1] Knowledge transfer for causal discovery. IJAR 2022.\n\n[2] Tuning causal discovery algorithms. PMLR 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CvJ8uZ9Zwt", "reviewer": "Reviewer_2Vef", "summary": "This paper introduces a new task called Multi-Event Causal Discovery (MECD). Given a video which comprises multiple  temporal events that are chronologically organized,  the goal is to predict if any previous event has a causal effect on the last event in the sequence. The MECD dataset is filtered and curated from the existing ActivityNet v1.3 video dataset. Additionally, the paper also proposes the Video Granger Causality Model (VGCM) approach, which leverages an event prediction model inspired by the Granger Causality method to perform an Event Granger Test. This test estimates causality by comparing predicted result events with masked versus unmasked premise events. Furthermore, the front-door adjustment and counterfactual inference techniques are also incorporated into VCGM. The authors demonstrate the benefits of their proposed VCGM approach by comparing it to state-of-the-art LLMs and VLLMs on the MECD benchmark, where it outperforms the latter by a significant margin.", "questions": "This is a minor note but it will be helpful for readers to define the term ‘premise event’ at the beginning. Also, please look at the above-mentioned limitations.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Overall, the paper addresses a significant problem in video understanding, especially with the recent interest in understanding the capabilities of visual LLMs (VLLMs). In contrast to how existing visual-language foundation models are pretrained and evaluated, the proposed MECD benchmark and the Video Granger Causality Model are aimed at understanding the causal relationships between multiple temporal events in videos instead of simply describing them.\n\n2) The intuition and theoretical reasoning underlying the proposed  Video Granger Causality Model is sound. In particular, the two main problems of Causality confounding and Illusory Causality in video causal discovery are well-motivated. Additionally, the mathematical formulation for integrating the front-door adjustment and counterfactual inference methods to resolve the listed issues is insightful.\n\n3) The model figures are informative and especially helpful in helping the reader to understand the different stages of the data curation process as well as the intuition behind each stage. The paper is also well-organized and well-written.", "weaknesses": "1) While the introduced Multi Events Causal Discovery dataset is a nice contribution as a benchmark, it is relatively limited in size with 1107 data samples. For example, there are only 808 and 299 video samples for training and evaluation, respectively. With such a small number of evaluation samples, it is challenging to evaluate these trained LMMs comprehensively. It may be beneficial to source videos from a wider variety of sources such as long and instructional videos from Youtube. Additionally, why is a validation set not included as part of the splits?\n\n2) The results in Table 1 provide some interesting comparisons between language-only baselines like Gemini 1.5 Pro and VLLM baselines including Minigpt4-video and Video-LLaVA. However, there are some other state-of-the-art video-language models that are not compared to in the paper such as Video-Llama [1]. Additionally, it has been demonstrated before that the image variants of VLLMs actually outperform their video counterparts on some video understanding tasks. It may be beneficial to include comparisons to such models such as InstructBLIP [2] and Llava to make the analysis more comprehensive.\n\n3) In section 4.1, it would be helpful to include a brief overview of the Granger Causality method, which forms the basis of the proposed Video Granger Causality Model.  Additionally, there are some sections, especially those that briefly describe the Event Granger Test and the causal inference techniques, could benefit from additional technical depth and explanations.\n\n4) Another concern lies in the training and evaluation setup. The VGCM approach is built off the VideoBert model and it will be helpful to include the results obtained by the base VideoBert model. Furthermore, the other baselines, that are being compared to, are evaluated under a zero-shot setting. However, the proposed model appears to be trained and evaluated under a strongly-supervised setting with the annotations from the MECD dataset.\n\n[1] Hang Zhang et al. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. EMNLP 2023 demo track.\n\n[2] Wenliang Dai et al. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments. We’re glad to hear that you found our task interesting and that you see our methods as insightful. Please see below for responses to your comments and questions.\n\n**Q1. Dataset Scale**\n\nWe have the following points to this issue:\n\n1. We envision MECD as a long-term, continuously maintained benchmark.  We will progressively expand its scope by incorporating a wider range of video types and increasing data volumes. In the final version, we plan to include an additional validation set as we have collected and annotated more videos.  Nonetheless, it's important to note that the current MECD is already effective in assessing the capabilities of LLMs for video causal discovery.  As demonstrated in Tab. 1 of our main manuscript, even with a limited number of test samples, LLMs struggle to accurately identify and discover causal relations between video events.\n   \n2. In causal discovery tasks, the evaluation scale is more accurately reflected by the number of event pairs rather than just the number of video samples. This is analogous to similar NLP tasks focused on causal discovery within texts or documents. Frequently utilized datasets in this domain, such as SeRI [1-2], Causal-TimeBank [3-4], HiEve [5], and EventStoryLine [6], comprise a limited number of test samples. For instance, Causal-TimeBank, HiEve, and EventStoryLine datasets have only 95, 100, and 318 articles or documents available for testing, respectively. However, as the evaluation is conducted between event pairs within documents, these datasets contain 1,475, 2,282, 2,725, and 4,027 event pairs, respectively. Similarly, our MECD provides 3,681 event pairs during the test phase, ensuring a comprehensive and convincing evaluation.\n\n3. Finally, as suggested by Reviewer-eXFX, we also try to incorporate additional metrics to provide a more comprehensive evaluation. For example, we newly report the *Ave SHD* metric for different baselines to reveal the ability of complete causal graph discovery for the whole video (cf. our response to Q3 of Reviewer-eXFX).\n\n**Q2. More comparison**\n\nThank you for your suggestion. In the table below, we included results for Video-Llama and two image-based VLLMs (Instruct BLIP and LLaVA) for a comprehensive comparison. The results demonstrate that image-based LLaVA does not outperform its video-based counterpart Video-LLaVA in our task. This finding further confirms the challenging nature of our task and highlights the necessity of long-term temporal understanding to accurately identify causal relationships between video events.\n\n| LLM          | Acc  |\n| ------------ | ---- |\n| Video-Llama  | 60.6 |\n| InstructBLIP | 59.7 |\n| LLaVA        | 60.2 |\n| Video-LLaVA  | 62.5 |\n\n**Q3. Overview of causal concepts**\n\nThanks for your valuable advice. We will add an overview of the causal concepts to the final version.\n\n**Q4-1. Base VideoBERT performance**\n\nWe have also conducted experiments on our baseline model VideoBERT, achieving an accuracy of 60.9%. Our VGCM model (71.2%) has made a significant improvement compared to the baseline model.\n\n**Q4-2. Training and evaluation under strongly-supervised setting**\n\nFor base multi-modal models VAR and Videobert, we report results from models fully trained on the MECD dataset. All LLM-based models are evaluated under a few-shot setting rather than zero-shot (cf. supplementary Sec. I, lines 600-606). Specifically, following the approach in causal discovery for NLP tasks [2, 7], three representative examples are provided during inference. We also investigated the impact of varying the number of few-shot examples, demonstrating the adequacy of our prompt (cf. supplementary Sec. G, lines 572-578).\n\nFurthermore, as you suggested, to ensure a fairer comparison, we conducted a strongly-supervised experiment on the advanced VLLM method Video-LLaVA. Specifically, we fine-tuned its LLM and encoder components using LoRA under its official implementation on our entire MECD training set. As shown in the table below, Video-LLaVA gains a 4.6% improvement from fine-tuning on our dataset. However, it still falls behind our proposed method. Given that fine-tuning LLM-based baselines is time-consuming, we will include more results of VLLMs with strongly-supervised settings for a comprehensive comparison in our final version.\n\n| Setting                  | Acc      |\n| ------------------------ | -------- |\n| Video-LLaVA (few-shot)   | 62.5     |\n| Video-LLaVA (fine-tuned) | 67.1     |\n| Ours                     | **71.2** |\n\n**Q5. Definition of premise event**\n\nThanks for your advice, the premise event is the event that happens earlier than the result event in the same video, and we will add the definition to the final version.\n\n**References**\n\n[1] Seri: A dataset for sub-event relation inference from an encyclopedia. NLPCC 2018.\n\n[2] Reasoning subevent relation over heterogeneous event graph. KIS 2024.\n\n[3] Annotating causality in the TempEval-3 corpus. EACL 2014.\n\n[4] Learning to teach large language models logical reasoning. arXiv 2023.\n\n[5] HiEve: A corpus for extracting event hierarchies from news stories. LREC 2014.\n\n[6] Neural granger causality. TPAMI 2021.\n\n[7] Is chatgpt a good causal reasoner? a comprehensive evaluation. EMNLP 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gqLY3fcIVk", "reviewer": "Reviewer_17Ce", "summary": "The paper studies causal reasoning in video, specifically, causal diagrams in long, multi-event videos. To do this, the authors attempt to define MEDC, a new task for discovering the complete causal relation diagram in multi-event chronological videos (~ 2 minutes). For example, analyzing events from traffic surveillance videos across different times, to identify the causes of an accident. Coupled with the task, they create a dataset of 1107 lifestyle videos from ActivityNet with multiple events in them. To facilitate the task, they propose Video Granger Causality Model, a framework inspired by the Granger Causality Method, and show it surpasses existing models (GPT-4, Gemini-1.5 Pro, Video-LLava).", "questions": "N/A", "limitations": "yes", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper introduces a new, challenging, and important task in video understanding\n* The methodology section was interesting to read, with the approach being mostly easy to follow.\n* There is a wealth of analyses and ablation work both in the main paper and the supplementary, which cover many aspects of the paper, such as the task itself, the trained model, and the baselines the model is compared to.", "weaknesses": "No major weaknesses come to mind.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. \n\nWe fully agree that the proposed MECD represents a novel, challenging, and significant task in video understanding. \n\nWe also appreciate the acknowledgment of the interest and clarity provided by the VGCM method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. We are glad to find that reviewers generally recognized our contributions:

- Novelty and contribution of MECD task (Reviewer-9my4, eXFX, 2Vef, 17Ce)
- Innovative and well-motivated framework design of VGCM (Reviewer-9my4, eXFX, 2Vef, 17Ce)
- Strong experimental results (Reviewer-9my4, eXFX, 17Ce)
- Well-organized presentation, clear description (Reviewer-2Vef, 17Ce)

Regarding the concerns proposed by reviewers, our main responses can be summarized as follows:

- **More details about our framework and training** (cf. response to Q2 of Reviewer-9my4)

- **Model's generalizability on related VQA task** (cf. response to Q1 of Reviewer-eXFX)
- **New evaluation metric** (cf. response to Q3 of Reviewer-eXFX)
- **Results of VLLM under strongly-supervised setting in our dataset** (cf. response to Q4 of Reviewer-2Vef)

All the reviewers can refer to the attached PDF for the task definition (Fig.1, Reviewer-eXFX) and a clear display of our pipeline (Fig.2, Reviewer-9my4).

We hope our responses could address your concerns. Please let us know if any clarification or additional experiments would further strengthen the paper. We would be happy to incorporate all these suggestions in the final version.

Meta-review: This submission targets the problem of causal relationships from video with multiple events. The authors proposed a method with a mask-based event prediction model and to capture causal inference. This submission is well-written, and easy to follow. All the reviews give positive ratings to this submission. The AC recommend to accept this submission as a spotlight paper.
Decision: Accept (spotlight)