MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning

Authors: tieyuan  chen
huabin  liu
tianyao  he
yihang  chen
chaofan  gan
xiao  ma
cheng  zhong
yang  zhang
yingxue  wang
hui  lin
weiyao  lin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 04:23:20

Abstract: Video causal reasoning aims to achieve a high-level understanding of video content from a causal perspective. However, current video reasoning tasks are limited in scope, primarily executed in a question-answering paradigm and focusing on short videos containing only a single event and simple causal relationships, lacking comprehensive and structured causality analysis for videos with multiple events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relationships between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD requires identifying the causal associations between these events to derive a comprehensive, structured event-level video causal diagram explaining why and how the final result event occurred. To address MECD, we devise a novel framework inspired by the Granger Causality method, using an efficient mask-based event prediction model to perform an Event Granger Test, which estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to address challenges in MECD like causality confounding and illusory causality. Experiments validate the effectiveness of our framework in providing causal relationships in multi-event videos, outperforming GPT-4o and VideoLLaVA by 5.7% and 4.1%, respectively.

Main paper: 
 introduction: 
 Video causal reasoning aims to achieve a high-level understanding and analysis of video content from a causal perspective. Video Question Answering (VQA) [1] [2] [3] [4] [5] represents one of the most prominent tasks in causal reasoning, where models are tested on their causal ability to understand video content through causal questions such as explanations, predictions, and counterfactual assumptions. Recently, some studies have sought to move beyond the single QA task, attempting to construct more complex and challenging video reasoning tasks and methodologies. For example, CLEVRER [5] , V-CDN [6] and CATER [7] explored more difficult causal reasoning tasks in virtual scenes by constructing object-aware features or using graph neural networks. Neural-symbolic paradigm AAR [8] and LMLN [9] extended to derive inference rules by symbolizing data. VAR [10] and BiGED [11] extended to daily video causal reasoning by introducing causality during prediction.However, current video causal reasoning tasks are still limited in scope (primarily QA-based) and mainly focus on short videos containing only a single event or a few events. Most importantly, they cannot provide a comprehensive and structured causal representation for multi-event video reasoning, which is typically required in real-world scenarios. For instance, in traffic surveillance videos, it is necessary to cross-analyze events happening at different times to determine which events, or combinations of events, led to the final traffic accident event.To address this gap, we set up a new task: Multi-Event Causal Discovery (MECD), which aims to uncover causal relationships among events that distribute chronologically in long videos. As illustrated in Fig. 1 , given multiple chronologically arranged event segments in a video along with their corresponding textual descriptions (Fig. 1(a) ), MECD requires identifying causal associations between these events to derive a comprehensive and structured event-level causal diagram (Fig. 1(b) ), indicating why and how the final result event happens. Meanwhile, we contribute a new dataset for the training and evaluation of MECD by collecting long-form videos involving multiple events and manually annotating real causal relations between events for them. However, to our knowledge, no available solutions can directly comprehend causal relationships at the event level, necessitating the development of a new framework to tackle this complex task.To this end, we draw inspiration from the Granger Causality Method [12] [13] [14] for solution, which is widely used in traditional causal discovery for low-dimensional time-series data (e.g., stock prices, weather patterns). The main idea is that temporal causality can be effectively estimated by predictive ability. Specifically, applied to videos, if Event A occurs prior to Event B, we consider A to be a cause of B only if A could facilitate the prediction of B. We term this criterion the Event Causality Test. However, compared to simple low-dimensional data, the inputs of MECD involve much more complex modalities, including both visual and textual content, which may introduce bias in the estimation of causality using such a predictive paradigm. Specifically, we observe that directly applying Event Causality Test to video causal discovery presents two main problems:(1) Causality confounding indicates that the original causal relationships between events are disrupted or interfered with by other relay or adjacent events. Such confounding stems from the fact that many causal relationships flow through an intermediary event that acts as a bridge. As shown in Fig. 1(c ), event "submitting the paper" serves as a necessary bridge between "taking the test" and "obtaining a grade." In this case, this bridge event might be mistakenly regarded as the only cause of the result event, while another cause, "taking the test," is overlooked. However, the bridge event can only occur if "taking the test" happens first. Therefore, we cannot identify the real causality between events that linked by such bridges following a simple predictive criterion, and eliminating such confounding is thus crucial for an accurate causal discovery.(2) Illusory Causality, which includes illusory temporal and existence causality. Illusory temporal causality exists when events exhibit a close correlation in temporal distribution. Such correlation may mislead the test of real causality. As shown in Fig. 1(d) , the event "adding oil when cooking" often occurs before "adding vegetables to stir-fry," but there is no real causality between them. As for illusory existence causality, it occurs when some objects in early events may serve as necessary existence conditions of a later event. For instance (Fig. 1(e) ), consider determining the causal relation between "a large brown dog enters the room" (at the start of the video) and "the dog runs towards the camera." (at the end of the video). Although the presence of the dog in the former event is a prerequisite for the subsequent event, it does not directly cause the dog to rush towards the camera.Building upon the preceding discussion, we introduce a novel framework to tackle MECD. This framework executes the Event Granger Test via an efficient mask-based event prediction model. It deduces the causality of a premise event by comparing the predicted features of the result event when the premise is either masked or unmasked. Furthermore, to mitigate the challenges of causality confounding and illusory causality discussed earlier, we integrate two additional causal inference techniques-front-door adjustment [15] [16] [17] and counterfactual inference [15, 18, 19] -into our framework. Specifically, these techniques compensate for or remove the causal effects of previous or subsequent adjacent bridge events to eliminate confounding. Simultaneously, they address the issue of illusory causality through the incorporation of an extra chain of thought [20] [21] [22] and existence-only descriptions during inference. Extensive experiments validate the effectiveness of our proposed framework in predicting structured causal relationships for given long-form videos. 
related work: 
 Video causal reasoning Many tasks in the past have tried to carry out causal reasoning in videos. Among these, the most common task is Video Question Answering (VQA) [1] [2] [3] [4] , aiming to give a reasonable answer to the question, methods such as SeViLA and LocAns [3, 4] made abductions based on the result, they grounded a single reason in previous time. However, VQA does not extend to abduct multiple reasons, merely creating a single causal link from reason to result.Many tasks were based on VQA task for further causal reasoning attempts. CLEVRER [5] , CATER [7] and V-CDN [6] explored causal reasoning based on physics and other basic laws in virtual scenes. However, these tasks haven't been committed to extending to the general video causal reasoning. AAR [8] and LMLN [9] symbolized data and derived inference rules using external knowledge. However, they can only reason within a defined symbol domain. The most similar VAR [10] predicted explanation events with premise events, and the causality was introduced during its prediction process. However, firstly it hasn't been committed to discovering the complete causal diagram. Besides, there is no explicit utilization of causal methods which constrains its ability.All tasks above are for causal reasoning in short videos, while ours aims to handle long-duration videos. Besides, most of these are coarse video-level tasks, ours is more fine-grained event-level reasoning. Additionally, we want to establish a whole causal diagram rather than a single causal link. In conclusion, all these tasks haven't been committed to discovering causality among complex multi-event videos. Consequently, there exists a necessity need for a more comprehensive task.Causal discovery in low-dimensional temporal data Traditional causal discovery methods of simple temporal data are mainly divided into three categories. Constraint-based methods use conditional independence tests to identify causal relations [23] [24] [25] . Score-based methods search through the space of all possible causal structures to optimize a specified metric [26] [27] [28] . The Granger Causality method discovers causal relations by calculating the degree to which the earlier occurred event contributes to the prediction of the latter occurred event [29] [30] [31] . The constraint-based and score-based methods require stringent assumptions about data distribution, making them less suitable for video data. The Granger Causality methods are more suitable yet face challenges when applied directly to video data, our method reaches better performance by utilizing causal inference methods.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations are discussed in Sec. E in the Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
mecd task settings: 
 Our Multi-Event Causal Discovery (MECD) task is designed to test the ability of causal discovery in multi-event videos. Given a video E that contains chronologically organized N events, E := {e 1 , . . . , e N }, the task aims at determining whether any previous event e n (n < N ) has a causal The word cloud is also summarized for video types. In (a2), the left chart indicates the impact of positions of events on their causality where we find the second last event tends to be more significant; while the right chart plots the number of events in videos.relation with the last one (i.e., e N ). Specifically, an event e n = {v n , c n } consists of a video clip v n and the corresponding caption c n . Without loss of generality, relations of previous events to the last one can be expressed as r = [r 1 , . . . , r N -1 ], where r k (k < N ) is set to "1" to indicate the existence of e k 's causal relation with e N , and "0" otherwise. Notably, this setting is generalizable to causal relations of any of two events as long as we cut off the video and treat the latter one as the last event.Data Source The Multi Events Causal Discovery (MECD) task contains videos with multiple events and intricate causal relationships. The ActivityNet Captions dataset [32] is built on ActivityNet v1.3 which includes 20k 120-second YouTube untrimmed videos. We carefully reorganize the ActivityNet Captions dataset and select 1,105 lifestyle videos that span diverse scenarios. We call this new dataset as MECD dataset, where 806 and 299 videos are randomly split for training and testing, respectively. Specifically, each video in the MECD dataset contains 4 to 11 events, with a minimum of 2 premise events exhibiting causal relations with the last one. Fig. Dataset Annotation The annotations of MECD dataset include 4 attributes. The "duration", "sentence", and "timestamps" attributes in annotations remain the same as the ActivityNet Captions dataset. Specifically, in the context of our task, a new attribute "relation" is introduced. To obtain this attribute, relations among events are firstly annotated by GPT-4 API [33] , and subsequently refined by five human annotators. Through a cross-annotation process, gt labels are determined by the majority of the annotators' causal relation choices, thus mitigating potential inaccuracies and subjective biases to a certain extent. We also present the impact of positions of events on their causality and number of events in videos in Fig. 2 (a2), annotation pipeline is in Appendix Sec. B.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: Codes and datasets are released on the GitHub Page.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
methodology: 
 In this section, we present our Video Granger Causality Model (VGCM), as shown in Fig. 3 . This model establishes the global connections across all events, and deduces the causality of a premise event by comparing the output features when it is masked or not, under the concept of the Event Causality Test. However, masking out an event may lead to the problem of confounding and illusion.In this context, we further utilize causal inference methods to address these by compensating or removing the effect of previous or subsequent causal events to mitigate the confounding meanwhile during inference the extra chain of thoughts and existence-only descriptions relieve the illusion.Building upon the Granger Causality method introduced in [34] [35] [36] , our core motivation for constructing VGCM is Event Causality Test: To compare the prediction result of the last event using all the premise events with or without a certain event in it. If the results exhibit obvious divergence, it indicates that the current premise event is causally related to the result event. Two data streams V p and V m k serve as input, video and text embeddings are concatenated after being separately embedded. The VGCM incorporates two classifiers, the caption head takes the unmasked stream to accomplish the event-predicting task, while the relation head discovers the causal relations with two embedding streams.We design VGCM to take in both the video clips and the captions to maximize information utilization. As illustrated in Fig. 3 , our proposed VGCM is a multi-modal transformer-based structure with a video encoder and caption encoder, and a multi-modal decoder with causal relation head to discover causal relations through the predicting process and the comparison of predicting results.Based on this, we denote E p as the set of all the premise events E p := E \ e N , and E m k := E p \ e k as the event set where the premise event e k (k < N ) is masked. Notably, we mask the event e k by setting all zeros to its video clip v k and assign constant characters to the caption c k .Following [10, [37] [38] [39] , we firstly pretrain a video encoder Φ pre under an action recognition task to extract the features of the video clips. We essentially create two paths, one for the unmasked event set E p (orange path in Fig. 3 ) while the other for the set with one event (i.e., e k ) masked E m k (green path in Fig. 3 ). The video clips and captions are first separately encoded using Enc V and Enc C to obtain compact features, then their features are sent to a multi-modal decoder Dec that shares weights for both paths to fuse the information. Afterward, several model heads are employed for feature comparison and loss measurement. V p and C p are the video clip and caption matrix concatenated from all premise events set E p , similarly,EQUATIONwhere Enc V and Enc T represent the encoder module of video clips and captions, respectively. Dec is a multi-modal decoder that shares weights for both paths. Cat indicates the concatenate operation, and [-] N -1 indicates the (N -1)-th slice at dimension 0. F p and F m k are features after encoding, and O p and O m k are the output features, which are then used for comparison of difference. Incorporating both visual and linguistic representations, the decoder conducts cross-modal reasoning and leverages the context from the unmasked premise events to posit a meaningful representation of the most likely explanatory result event.Subsequently, the feature O p deduced from the unmasked events is sent to the caption head for prediction. Additionally, in order to compare the difference of the prediction result, O p , O m k are directed to the relation head for causal relation discovery. The result event e N is encoded the same way as e k to get featureF N = Enc V (Φ pre (v N )) and the output O N = Dec(Cat(F N , Enc C (C N )),O N is aggregated for reasoning (red path in Fig. 3 ). The relation head consists of a semantic query module and a self-enhancement module, where outputs are concatenated and then passed through the cross-reasoning layer g r for further interaction. Last but not least, the auxiliary similarity is measured between O p and O m k as a supplement to the output information of the relation head. After the reasoning process, the prediction output of the causal relation rk can be represented by:rk = g r (Cat(Φ C att (Cat(O m k , O N ), Cat(O p , O N )), Φ I att (Cat(O m k , O N ))))(2) where Φ C att represents cross-attention, Φ I att represents self-attention, g r represents linear layer. The training objective consists of two main directions as previously discussed:To reconstruct the textual and visual representation of the result event e N , we introduce caption loss and reconstruction loss, respectively. Caption loss L C ensures an accurate prediction of the result caption ĉN given all the premise events E p . Simultaneously, visual reconstruction loss L V forces the encoder to "imagine" a representation of the result video clip vN that better aligns with the original representation v N . These losses allow the model to predict visual and textual representations that are close to the original representations, which better supports our method of inferring causal relations by comparing the results of the two-stream predictions.For the objective of causal discovery, we introduce causal relation loss and an auxiliary semantics similarity loss. Causal relation loss L R supervised the output relations rk . Meanwhile, the semantics similarity loss L S is introduced to guarantee the semantics similarity of result event prediction under the presence or absence of a causal-relation-free premise event. The complete loss function is:EQUATIONwhere λ R , λ V , and λ S are weights for trade off. L C and L R are the cross-entropy losses, L V and L S are the mse losses, F p N is the Nth slice of F p , which represents the feature of e N .In Sec. 4.1, we employ the concept of Granger Causality to design our VGCM model under the principle of Event Causality Test which may, however, introduce causality confounding and illusory. Below we introduce these issues in detail, as well as how we manage to solve the problems.Causality confounding is a phenomenon where the original causal relations across events are impacted due to modification (i.e., masking) of some intermediate events (i.e., e k ). Existing disentangled representation learning works [40, 41] disentangled different attributes of a variable by supervising high-order distribution under strict assumptions but failed in disentangling different variables.Specifically, when e k is masked for the comparison in VGCM, the causal relations between e k 's adjacent events and the last event are impacted, leading to a confounding of causal effects. Notably, for brevity, we only employ e k 's previous one event e k-1 and its subsequent one event e k+1 for analysis, but the same analysis also applies to all the previous or subsequent events. To be specific, there exist two distinct kinds of confounding when e k is absent: 1) Causal effects of e k-1 to e N may be lost, as its connection to e N is built upon e k , (green path in Fig. 4 (a1)). 2) Causal effects of e k+1 to e N may be redundant, as e k must be a necessary prior of its causality, (red path in Fig. 4 (a1)).Illusory causality is another issue that may lead to some spatial or temporal misunderstandings, including illusory temporal and existence causality. 1) Illusory temporal causality is the situation that events could have tight temporal ordering, but they in fact have no causal relations. 2) Additionally, illusory existence causality occurs when an object introduced in the premise event is a necessary condition for the result, but the premise event does not semantically serve as a reason. Notably, we find that illusory in multi-event videos is much more significant than two independent events, which also tends to be exacerbated by causality confounding.Overall, causality confounding and illusory causality both bring difficulties for relation modeling of events in videos. Notably, these two issues are coupled in that causality confounding tends to exacerbate illusory causality by misallocating attention to temporal ordering and causal effect. Therefore, illusory causality can be partially relieved by solving the problem of causality confounding.When considering taking the illusory causality, the chain of thoughts [20] [21] [22] has been shown in LLMs to lead the model to logical thinking which is similar to human thought process, the chain of thoughts T cot[e k-1 :e N ] provides a step-by-step process of reasoning the e N from e k-1 . Specifically, T cot[e k-1 :e N ] is obtained using GPT-4 API [33] by feeding it with e k-1 , e N along with a prompt asking it to provide the probable reasoning chain. We consider utilizing it in causal inference to eliminate the attention bias on temporal correlations introduced by non-causal temporal knowledge.Besides, as the illusory existence causality is caused by the objects' correlation between the events, we address this influence by keeping objects in the green path in Fig. 3 the same as those in the orange path. We introduce an alternative event e 0 k = {v 0 k , c 0 k } of e k to briefly recaps the objects in e k . Specifically, c 0 k is obtained using GPT-4 API [33] by feeding it with c k along with a prompt asking it to extract the objects from c k and organize them as the sentence such as "There are objects A, B and C.". Consequently, we opt to employ c 0 k to approximate e 0 k in our VGCM model while omitting v 0 k , as c 0 k is sufficient already to convey the information of objects. By providing e 0 k , all the necessary objects are still available in this path, thus effectively mitigating the illusory existence causality, facilitating the model to focus more on essential and causality-related semantic information.To tackle the issues above, we introduce two causal inference methods: the front-door adjustment [42] for the missing causal effect of e k-1 and counterfactual inference [42] for the redundant causal effect of e k+1 . Meanwhile, the chain of thoughts T cot[e k-1 :e N ] and the descriptions of existence c 0 k are also provided to carefully address illusory causality, which in turn mitigates confounding.We establish a causality diagram in Fig. 4 (a2) for an improved elaboration. On masking e k , the causality confounding that requires compensation F C or removement F R can be expressed as:EQUATIONwhere P (e N |e k ) and P (e N |e k+1 )represents the process by which we predict e N from e k and e k+1 in the orange path in Fig. 3 , and do(•) represents do-operation in causal inference [15] that cuts off the causal relation between the event and its causes.We aggregate the subsequent events e k+1 , the current event e k and the chain of thoughts T cot[e k-1 :e N ] using a linear layer g do for aggregation and the cross-attention and self-attention, according to the study in [43, 17] , P (e N |do(e k )) can be implemented as:EQUATIONHere, we re-use the cross-attention Φ C att and the self-attention Φ I att modules as in (2) to cut off the causal effect from e k-1 to e k through do-operation, e k only interacts with subsequent events in predicting e N . Then the missing causal effect F C can be compensated since the causal-view operation and illusory temporal causality can be suppressed at the same time with the introduction of the chain of thoughts. Similarly, the redundant causal effect F R can be removed by applying counterfactual intervention, then P (e N |do(e k+1 )) can be represented by:EQUATIONP (e N |do(e k+1 )) effectively cuts off the redundant causal effect between e k+1 and e N for the reason that the causes of e k+1 are replaced with counterfactual description e 0 k , then the illusory existence causality can be suppressed at the same time.To refine the originally decoded feature O m k from the path with premise events masking:EQUATIONwhere O ′m k is the refined feature that replaces O m k for further deduction of the model. With the refinement feature O ′m k , our VGCM model effectively compensates the connections between e k-1 and e N that were originally lost due to the removal of e k , and effectively removes the redundant causal effect between e k+1 and e N as well. 5 Experiments 
main results: 
 Implementation details. including the pretraining process, detailed architecture of VGCM, and hyper-parameters settings can be found in Appendix Sec. A due to space constraints.Baselines. We mainly compared our model with basic multi-modal models such as baseline model Videobert [50] and widely used CLIP-L [51] and the most similar reasoning model VAR [10] . Besides, we also conduct experiments on powerful LLM, including GPT-4 [33] and Gemini-Pro [44] . VLLM utilized for comparison includes widely accepted GPT4-o [33] , VideoLLaVA [49] , MiniGPT-4 [46] , Video-llama [47] , VideoChat2 [48] and MiniGPT4-video [45] . Specifically, LLMs and VLLMs are conducted under the few shot setting (In-Context Learning) following the causal discovery tasks in NLP [52] [53] [54] , additionally, we reported the performance of fine-tuned VideoLLaVA and VideoChat2.Metrics. We utilize the top-1 accuracy of the output causal relation chains with respect to the final event to evaluate the model's capability in causal discovery. Although our VGCM is designed to discover the causal relations leading to the final event, when truncating the video during inference and redefining the final event as the new result, VGCM can generate a comprehensive causal diagram for the entire video without introducing additional training. Consequently, in addition to the primary metric accuracy, we introduce Structural Hamming Distance (SHD) [55, 56] as a supplementary metric. SHD measures the degree of matching between comprehensive causal graphs by summing the number of incorrect causal relations. In the MECD test set, the average number of causal relations in video causal graphs is 12.31, and a lower Ave SHD value of the test set indicates better performance.Results. We report the quantitative results in Tab. 1. Our VGCM without causal inference reaches an accuracy of 66.9%, demonstrating basic reasoning capabilities. Furtherly, the complete VGCM reaches a better performance with an accuracy of 71.2%, outperforming the GPT-4, GPT4-o, finetuned VideoLLaVA [49] by 11.6%, 5.7%, and 4.1%. Additionally, we explored the effect of altering the input format of the two modalities in Appendix Sec. C.1, indicating that VGCM is not dependent on the input format. The results compared with the two metrics indicate that for most models, accuracy is already adequate to represent their causal discovery capabilities. However, the additional metric Ave SHD indicates that Gemini and GPT-4 exhibit a superior overall capacity for discovering complete relations. An example of the output complete causal diagram is visualized in Figure 7 .GPT-4 [33] stands out as one of the most advanced LLM models, however, we found that even being provided with sufficient few-shot examples (detailed in Appendix Sec. C.2), its accuracy remains at As illustrated in Tab. 6, we have assessed the inference speed of various models, with our VGCM achieving a swift 0.76 seconds per sample. The proposed method incurs an overhead of only 8.57% over the Videobert baseline. It is noteworthy that our inference speed is 3 to 6 times faster than that of all Video LLMs. The inference speed experiments were conducted on 1 NVIDIA A6000 GPU.Video Granger Causality Model design We designed our causal discovery model based on the Granger Causality, three auxiliary losses are applied. The performances in Tab. 2 indicate that our VGCM benefits from the design of L V and L C , for they support our method of inferring causal relations by facilitating the model with event prediction ability. L S also benefits our model by supervising the causal feature similarity of e N with and without non-causal event e k masked.Front-door adjustment with chain of thoughts candidate The method does improve reasoning ability in Tab. 2. We conduct an experiment in Tab. 4 for further proof. Since events closer to the result event are higher as the cause, the model likely learns these biased time-domain tendencies. So we compare the accuracy of VGCM without front-door adjustment with chain of thought candidate and VGCM in determining the first relation r 1 and the last relation r N -1 . The results demonstrate that temporal illusory causality is greatly mitigated, visualization can be found in Fig. 8 Example 1.The performance in Tab. 2 shows that counterfactual intervention with existence-only descriptions does facilitate the model with powerful reasoning ability. We dive into further analysis on the basis that when a non-causal event is masked, the causal feature F m k fed into the causal relation head should be similar to the unmasked feature F p , instead, a bigger gap appears when masking a causal event. For stronger proof, we measure the difference in feature similarity in Tab. 3 and Fig. 6 . We define the similarities division as the quotient of the similarity(F m k , F p ) with a non-causal e k masked over with a causal e k masked. In the experiment, we find that the similarity division is always above 1 without the counterfactual intervention, however, the existence illusory is solved with counterfactual intervention for the reason that the division is below 1 of VGCM, example visualization can be found in Fig. 8 Example 2.Model Robustness To prove our model's robust reasoning ability, we split the MECD dataset into five categories, and conduct an experiment similar to the open-set setting with cross-validation. VGCM interviews riders race information wait to take off a man falls off interviews two men another falls offExistence-only description: There is an interviewer and a racer. 2.12 VideoChat2 [48] 2.96 MiniGPT4-video [45] 3.98 MiniGPT-4 [46] 4.72 reaches an average accuracy of 64.4%, outperforms VGCM without causal inference and VAR by 5.2% and 9.6%, details can be found in Appendix Sec. D.1.Moreover, to further validate the generalization capabilities of our model, we evaluate the quality of output causal relations on a related and representative video reasoning task: Video Question Answering (VQA) as shown in Tab. 7. Specifically, during inference on the multi-event subset of ActivityNet-QA [59] (The part that overlaps with the MECD test set), we prompted MiniGPT4video [45] with additional causal relations outputs alongside the standard question inputs. This paradigm facilitates the VLLMs in considering the task from a causal perspective. As shown in the table below, when prompted with these additional causal relations, the answering accuracy of MiniGPT4-video [45] improved by our VGCM surpasses other strong VLLMs like VideoChat2 [48] . These findings confirm that our model can provide accurate causal perception for videos, significantly improving performance on related video reasoning tasks.We study the subjectivity and data volume of our proposed MECD dataset, which is shown in Tab. 5. In the experiments of increasing the ratio of randomly flipped annotated causal relations (flipping only one relation of the whole causal relations of video), the accuracy decreases slightly, demonstrating the small amount of subjectivity in labeling does not have a serious impact. Besides, we analyze the scale of data, the increment from 600 examples to 806 examples yields a very modest improvement, indicating the adequacy of our dataset. 
conclusion: 
 We proposed a novel task, multi-event video causal discovery (MECD), which focuses on eventlevel causal discovery in long-term videos. Besides, we built the MECD dataset with long-term daily life video datasets with causal relations to support this task and proposed the first video events causal discovery framework VGCM in the principles of Granger Causality. Additionally, our proposed VGCM was facilitated with deeper reasoning ability through causal inference with the chain of thoughts and existence-only descriptions. Our VGCM significantly outperforms GPT-4o and VideoLLaVA by 5.7% and 4.1%, respectively, demonstrating its robust reasoning ability. does not directly lead to their victory, and the subsequent celebrations also lack any causal links with the outcome. Indeed, the false discovery by the GPT-4 API could stem from the illusion of causality, where the team's mere presence is perceived as a necessary condition for the outcome. Additionally, the illusion of temporal causality may also play a role, as statistics indicate that celebrations often occur before the announcement of the competition winner. These cognitive biases could contribute to the erroneous causal inference made by the GPT-4 API in this scenario.When we request a detailed explanation from the GPT-4 API regarding the discovered causal relation between the result event and the initial appearance of the team, the response is "Setting up the motive for the last event." Obviously, the GPT-4 confuses causality with the illusion of existence causality. In contrast, our VGCM makes a correct inference in this scenario. Furthermore, when we seek detailed reasons from the GPT-4 API for the discovered causal relations between the result event and the celebrations, the answer is "Indicating their satisfaction and confidence in their performance, implying they believe they have a good chance to win." Here, the GPT-4 API misinterprets causality by associating it with the expression of subjective emotions unrelated to the events in question. It may mistake the display of subjective emotions for the presence of objectively implied causality.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Implementation details are provided in Sec. A in the Appendix.• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We report the average results under three random seeds (2023, 2024, 2025).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Implementation details are provided in Sec. A in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: ActivityNet Captions Dataset is with no license needed.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To improve the accuracy and mitigate subjective biases in annotating causal relations, we employ a cross-annotation strategy [62] [63] [64] . The interactive interface used by annotators during the labeling process of our MECD dataset is illustrated in Fig. 11 . Each video example is endowed with a without causality with causalityTeam comes out Attempts flips Another girl follows Team dances Your Annotation： (0 for without causality, 1 for with causality) "relation" attribute. First, GPT-4 [33] provides an initial annotation of attribution, which is then further refined by five human annotators. Ground truth labels are determined based on the majority choices of the annotators regarding causal relations. This methodology ensures the creation of a more reliable and objective dataset.Annotation examples of MECD are shown in Fig. 12 , our MECD dataset is carefully annotated to support the challenging task proposed with complete premise information."Existence" "Sentences"1."man is sitting on a table eating a sandwich in a rstaurant talking to other man.", 2." man is watching the man while eats and talking to him.", 3." both men stands and play rock paper scissors and man hits the man sand this falls to the floor.", 4." man is sitting on the chair and talks to the man on thefloor and throw him the sandwich.", 5." anoher man siting in front talk to the man in the desk.", 6." man stands and talk to the man on the floor and hits him again and the man spit blood.""COT"1."This event initially introduces the two key characters, one of which enjoying a sandwich in a public setting and communication with the other. This dynamic sets the ecosystem for an interactive relationship, which might include confrontation leading to the end scene.", 2."This event shows the observer-man's active interest and engagement with the eating man, illustrating a relationship that could potentially be conflicly, hence leading to a physical interaction in the end.", 3."This sudden aggressive action shifts the dynamic between the two men, escalating the situation from a previously peaceful meal to one with physical confrontation. This change in interaction sets the stage for the violent ending.", 4."This gesture of throwing the sandwich at the man on the floor further escalates the hostility in the interaction. Disregard for the other man's dignity is apparent, indicating potential for further violent interactions.", 5."This event depicts an interaction between a third man and the man on the desk, ignoring the man on the floor. The lack of intervention prolongs the conflict between the two main characters, pushing the situation towards the bloody ending."1."There is a man sitting, a table, a restaurant, a sandwich and another man.", 2."There is a man watching and talking.", 3."There are two men, rock paper scissors game and a sand.", 4."There is a man sitting on a chair, another man on the floor and a sandwich.", 5."There is a man, a desk, a computer, a water, and another man laying on the floor.", 6."There is a man sitting in front and another man on the desk.""Relations" 01111 without causality with causality "Existence" "Sentences" "COT"1."A cricket match is ongoing with Sri Lanka playing against another team, the atmosphere is competitive, which naturally increases pressure and expectations, leading to intense crowd reactions.", 2."The competitive nature of the game implies that every action, every run scored, matters significantly, which in turn can cause intense reactions from the spectators." 3."The uniform identifies the teams and creates a sense of belonging, unity, and rivalry among the teams and the spectators, solidifying the sides spectators will cheer for.", 4."The act of scoring runs in a cricket match is a significant event -it directly contributes to the final outcome of the match and draws strong responses from the crowd; the more the runs, the louder the cheers.", 5."The video shows not just one, but several cricket matches featuring the Sri Lanka team. This series of events builds up the hype, intensity, and anticipation among the spectators, culminating in a stadium filled with cheering crowds."1."There is a cricket team of Sri Lanka and another country.", 2."There are cricketers and a field.", 3."There is a Sri Lanka team in a blue uniform.", 4."There is a batsman, a bowler and an overhand ball.", 5."There are different cricket matches of Sri Lanka, and teams from different countries.""Relations" 11010 without causality with causality The MECD task employs both video input and corresponding captions to uncover causality. Our objective in this experiment is to evaluate the degree of reliance on these two modalities in causal discovery. Typically, each event in our MECD task consists of a textual input with an average of 13.5 words caption and a visual input of 50 frames.To investigate the influence of the text modality, we employ a masking strategy for the input caption of the premise event, gradually increasing the masking ratio from 10% to 80%. The results presented In contrast, the experimental results suggest a more obvious performance decrease towards less visual modality input in the causality discovery task, as shown in Tab. 9. However, even with 80% masking of either modality, the results consistently outperform our strong baseline model, VideoLLaVA, underscoring the robust causal discovery capability of VGCM. Furthermore, we conducted experiments involving simultaneous masking of both modalities of information. Interestingly, we observe a noticeable decrease in accuracy compared to when only one modality is masked. This observation highlights the importance of jointly considering both modalities in the causality discovery task.To delve deeper into the limitations of the straightforward baseline approach of prompting GPT-4, we examined the correlation between its accuracy and the number of video examples provided in the few-shot prompts. The findings, illustrated in Fig. 13 , suggest that increasing the number of examples shown to GPT-4 does not effectively enhance its accuracy. This suggests that the limitation of the GPT-4 baseline is not strongly correlated with the number of presented examples but rather is more attributable to its intrinsic limitation in understanding complex causal relationships solely through text modality.In the open-set experiment of exploring reasoning ability, the five categories mainly consist of the activities below, demonstrating the colorful daily activities included in our dataset. In this section, we introduce the detailed method of prompting GPT-4 [33] to generate more premise information. Firstly, we prompt the GPT-4 with the following prompts to generate the description-only sentences.# Task: Each input consists of n sentences, and the text description of each sentence has been given correspondingly (separated by " ",). You need to offer the existence description of each sentence.Besides the task description, we further append the few-shot paradigm (In-Context Learning) introduced in [52-54, 65, 66] . Similarly, we prompt the GPT-4 [33] with the following instructions to generate the chain-of-thoughts candidate sentences in the same few-shot paradigm.# Task: Each video consists of n events and the text description of each event has been given correspondingly separated by " ",). First n-1 events might be the cause of the last event. You need to offer the chain of thoughts you derive that causes the last event.In this section, we present an example of the chain of thoughts prompted, the corresponding premise event and result event descriptions are also shown below:{"premise event sentence": "He continues sharpening the knife, turn it again to further sharpen the other side and wipe it with paper towel." "result event sentence": "Throws the old and dirty paper towel and reach the roll of paper towel and clean the knife.""COT": "The repeated action of sharpening and wiping the knife underscores the importance of both the knife's sharpness and cleaners, leading directly to the final action of disposing of the used paper towel and getting a new one to ensure the knife is thoroughly clean"} The chain of thoughts shown above provides a logical causal chain between the event of the cleaning of the knife and the subsequent throwing of the dirty paper towel. The reasoning initiates by considering the heightened need for sharp and pristine knives achieved through sharpening. This causal chain is then expanded by suggesting that this demand could have led to the replacement of the paper towel. The chain of thoughts generated from GPT-4 serves as a candidate in the process of correct reasoning, contributing to the exploration of potential causal relations.1. The video we input for causal discovery needs to provide timestamps, we encourage future work to realize causal discovery with weakly annotated inputs.2. VGCM might still require refinement in understanding causality within higher-level semantics, especially in the mining of some obscure mental or emotional influences according to the failure cases analysis in Appendix Sec. B.1.3. VGCM is based on the supervised paradigm of causal discovery, subsequent works may be able to extend to the unsupervised paradigm. • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We conducted the research in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This submission targets the problem of causal relationships from video with multiple events. The authors proposed a method with a mask-based event prediction model and to capture causal inference. This submission is well-written, and easy to follow. All the reviews give positive ratings to this submission. The AC recommend to accept this submission as a spotlight paper.

Reviews: review: paper_summary: paper_summary: The paper introduces the Multi-Event Causal Discovery (MECD) task, aiming to uncover causal relationships in videos with multiple events. It presents a novel framework inspired by the Granger Causality method, utilizing a mask-based event prediction model to perform causal inference. The paper also introduces a new dataset for training and evaluation, demonstrating the framework's effectiveness in outperforming existing models like GPT-4 and Video-LLaVA.
strength_weakness: strength_weakness: - Complexity of Implementation: The framework's reliance on multiple advanced techniques may pose challenges for implementation and replication by other researchers. Besides, does VGCM run slower than those VLLM baselines?
- Missing Details: What's the architecture of different encoders? How do the authors pretrain them? How do different loss design? The authors should give more details about fair comparisons.
- Some Typos:
  - In Figure 3, the box colors for the masked and unmasked input should be green and orange.
  - In Line 166, it should be $Env_{V}$ and $Env_{C}$.
questions: questions: I appreciate that the authors proposed a more novel and interesting setting. However, given the missing details and overly complicated pipeline, I tend to reject this paper for now. And I hope the authors can provide more details about the training and architecture. This will help us reviewers to make a fair evaluation.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces a new task and dataset, Multi-Event Causal Discovery (MECD) to better understand long videos in a casual perspective. Inspired by the Granger Causality method, the authors devise a framework, dubbed VGCM, to perform the Event Granger Test. Also, VGCM is combined with front-door adjustment and counterfactual inference to tackle the issues of causality confounding and illusory causality. Experiments show that VGCM outperforms GPT-4 and Video-LLaVA in providing causal relationships in multi-event videos.
strength_weakness: strength_weakness: - To enhance the demonstration of the model's generalizability, it is recommended to conduct experiments on a variety of related datasets.
- In Figures 1(c) and 1(e), the events occurring within the video frames are not readily discernible. Additional verbal descriptions are recommended to facilitate a clearer understanding of the figures' intended message.
questions: questions: - The authors employ Accuracy as the primary performance metric. Could the authors consider or design some other metrics that are more suitable for Multi-Event Causal Discovery (MECD)?
- It would be highly beneficial if the dataset and associated code could be made publicly available at the earliest opportunity.
- Regarding lines 154-155, I suspect that the confounding factor in the spurious causal relationship could introduce significant discrepancies when comparing the output features with and without the factor. Could the authors provide further analysis on this matter?
limitations: limitations: The model depends largely on video captions, which makes it possibly incapable of processing video datasets without captions or descriptions.
paper_summary: paper_summary: This paper introduces a new task called Multi-Event Causal Discovery (MECD). Given a video which comprises multiple  temporal events that are chronologically organized,  the goal is to predict if any previous event has a causal effect on the last event in the sequence. The MECD dataset is filtered and curated from the existing ActivityNet v1.3 video dataset. Additionally, the paper also proposes the Video Granger Causality Model (VGCM) approach, which leverages an event prediction model inspired by the Granger Causality method to perform an Event Granger Test. This test estimates causality by comparing predicted result events with masked versus unmasked premise events. Furthermore, the front-door adjustment and counterfactual inference techniques are also incorporated into VCGM. The authors demonstrate the benefits of their proposed VCGM approach by comparing it to state-of-the-art LLMs and VLLMs on the MECD benchmark, where it outperforms the latter by a significant margin.
strength_weakness: strength_weakness: 1) While the introduced Multi Events Causal Discovery dataset is a nice contribution as a benchmark, it is relatively limited in size with 1107 data samples. For example, there are only 808 and 299 video samples for training and evaluation, respectively. With such a small number of evaluation samples, it is challenging to evaluate these trained LMMs comprehensively. It may be beneficial to source videos from a wider variety of sources such as long and instructional videos from Youtube. Additionally, why is a validation set not included as part of the splits?

2) The results in Table 1 provide some interesting comparisons between language-only baselines like Gemini 1.5 Pro and VLLM baselines including Minigpt4-video and Video-LLaVA. However, there are some other state-of-the-art video-language models that are not compared to in the paper such as Video-Llama [1]. Additionally, it has been demonstrated before that the image variants of VLLMs actually outperform their video counterparts on some video understanding tasks. It may be beneficial to include comparisons to such models such as InstructBLIP [2] and Llava to make the analysis more comprehensive.

3) In section 4.1, it would be helpful to include a brief overview of the Granger Causality method, which forms the basis of the proposed Video Granger Causality Model.  Additionally, there are some sections, especially those that briefly describe the Event Granger Test and the causal inference techniques, could benefit from additional technical depth and explanations.

4) Another concern lies in the training and evaluation setup. The VGCM approach is built off the VideoBert model and it will be helpful to include the results obtained by the base VideoBert model. Furthermore, the other baselines, that are being compared to, are evaluated under a zero-shot setting. However, the proposed model appears to be trained and evaluated under a strongly-supervised setting with the annotations from the MECD dataset.

[1] Hang Zhang et al. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. EMNLP 2023 demo track.

[2] Wenliang Dai et al. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. NeurIPS 2023.
questions: questions: This is a minor note but it will be helpful for readers to define the term ‘premise event’ at the beginning. Also, please look at the above-mentioned limitations.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper studies causal reasoning in video, specifically, causal diagrams in long, multi-event videos. To do this, the authors attempt to define MEDC, a new task for discovering the complete causal relation diagram in multi-event chronological videos (~ 2 minutes). For example, analyzing events from traffic surveillance videos across different times, to identify the causes of an accident. Coupled with the task, they create a dataset of 1107 lifestyle videos from ActivityNet with multiple events in them. To facilitate the task, they propose Video Granger Causality Model, a framework inspired by the Granger Causality Method, and show it surpasses existing models (GPT-4, Gemini-1.5 Pro, Video-LLava).
strength_weakness: strength_weakness: No major weaknesses come to mind.
questions: questions: N/A
limitations: limitations: yes

Comments: title: Response to reviewer-9my4
Response to authors
Response to reviewer-2Vef
Response to reviewer-9my4
Response to authors
Kind Request for Discussions and Feedback for Paper 4887
comment: Thanks for your response.

We really appreciate your acknowledgment that our further claim effectively addressed your concerns. We will provide the additional results and all the details in the final version of the manuscript following your suggestion. Besides, we will make the complete dataset and codes available as soon as the manuscript is accepted.

Once again, we would like to express our sincere gratitude for your positive feedback and for contributing to our manuscript.
Thanks for the further claim, and I slightly raise the rating. I suggest adding these additional results and details in the final version to make the whole paper more reliable and reproducible.
Thank you for your response. 

We appreciate your acknowledgment of the usefulness of our rebuttal. We confirm that we will include the additional SHD metric and improve the comparison methods as you suggested in the final version of the manuscript. 


Once again, we are grateful for your valuable feedback and acknowledgment of our work.
Thank you very much for your detailed responses to my questions. In particular, I find the addition of the new Ave SHD metric to be helpful and especially insightful for understanding the limitations of existing large multimodal models. Consequently, I will retain my initial rating.
Thanks for your great efforts and time in reviewing our paper. 

### 1. *Results of fine-tuned Video-LLaVA*

Firstly, we want to state that the few-shot evaluation (In-Context Learning) for LLMs is widely recognized as a strong baseline for reasoning and causal discovery tasks [1-6], effectively reflecting their performance on downstream tasks. Therefore, comparing LLMs in a few-shot setting is a common and accepted practice in the field. 

Secondly, neither GPT-4 nor Gemini-Pro offer interfaces for fine-tuning, consequently, we reported the few-shot results of LLMs and investigated the impact of varying the number of few-shot examples, demonstrating the adequacy of our prompt (cf. supplementary Sec. G, lines 572-578). 

Furthermore, as you and reviewer-2Vef suggested, to ensure a fairer comparison, we conducted a strongly-supervised experiment on the open-source method Video-LLaVA. Specifically, we fine-tuned Video-LLaVA using LoRA under its official implementation on our MECD training set. As shown in the table, Video-LLaVA gains a 4.6% improvement from fine-tuning on our MECD. **However, it still falls behind our proposed method**. Since fine-tuning LLM-based baselines is time-consuming, we will include more results of fine-tuning VLLMs for a comprehensive comparison in our final version. 

|Setting| Acc|
|------------------------ | -------- |
|Video-LLaVA (few-shot)| 62.5|
|Video-LLaVA (fine-tuned) |67.1|
|Ours|**71.2**|

[1] Language models are few-shot learners. NeurIPS 2020.

[2] Rethinking the role of demonstrations: what makes in-context learning work? EMNLP 2022.

[3] Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. NeurIPS 2023.

[4] CLADDER: assessing causal reasoning in language models. NeurIPS 2023.

[5] Causal inference using llm-guided discovery. AAAI 2024.

[6] Open Event Causality Extraction by the Assistance of LLM in Task Annotation, Dataset, and Method. ACL 2024.


### 2. *More details*

We're happy to provide more details here to clarify your concerns. However, we need to state that **these details can be also found in the supplementary in our initial submission**. To further relieve your concerns about our implementation and reproducibility, we provide an **anonymous GitHub project** (https://anonymous.4open.science/r/NeurIPS-4887-MECD), which contains our *data annotation*,  *training and evaluation codes*, and details about how we evaluate LLMs and how we fine-tuning the Video-LLaVA. Additional details can be found in the README.md file.

- **Training procedure**: All the experiments are conducted on 1 NVIDIA A40 GPU. We train our model for 20 epochs with a learning rate of 16e-5 about 6 hours. Our optimizer is consistent with the BertAdam optimizer, with 3 epochs of warm-up. We report the average results during all experiments under three random seeds (2023, 2024, 2025). 
- **Hyperparameters**: $\lambda_C$, $\lambda_R$, $\lambda_V$,$\lambda_S$ are set to be $1.0, 4.0, 0.25, 0.05$.
- **Encoder & Decoder architecture**: Our encoder $Enc_{v}$, $Enc_{c}$, and multi-modal video decoder $Dec$ are built upon videoBERT, a joint model for video and language representation learning. 
- **Evaluation of LLMs**: We prompt the GPT-4 with the following few-shot prompts to conduct evaluation (Details can be found in the anonymous GitHub project):

```
# Task: Each video consists of n events, and the text description of each event has been given correspondingly (separated by " ",). You need to judge whether the first n-1 events in the video are the cause of the last event, and the probability of the cause 0(uncausal) or 1(causal) is expressed as the output, Let's think step by step through the chain of thought.
Here are several examples of judging whether the first n-1 events in the video are the cause of the last event:
<start>
First example:
Text description of n events:
...
The probability output should be:
...

Second example:
Text description of n events:
...
The probability output should be:
...

Third example:
Text description of n events:
...
The probability output should be:
...
<end>
```


As a long-term maintenance benchmark, we confirm that the dataset and code will be public after being accepted.

We hope our responses can address your concerns and raise your rating of our paper.
The authors solve part of my issues. However, some key issues are not clear, like the details to reproduce the whole results. It will be better if the authors can provide detailed training hyperparameters, architecture, data in clear tables, and even the code.

Besides, the LLM-based models are not fine-tuned with the proposed MECD, which is unfair. Can the authors provide more fair results with training on VideoChat or Video-LLaVA. And the authors should clearly claim they use `few-shot paradigm` in the main text. 

Considering the problems, I maintain my original rating.
Thanks for your response. 

We greatly appreciate your acknowledgment that our rebuttal has effectively addressed your concerns. We have duly noted your suggestion regarding the model's generalizability and the additional metric in the final version of the manuscript. We will ensure that this change is made to improve our work.

Once again, we would like to express our gratitude for your valuable feedback and for contributing to the improvement of our manuscript.
Thank you for your detailed response. My concerns have been addressed, and after considering the feedback from other reviewers, I will maintain my current rating.
Dear Reviewers,

Firstly, we'd like to express our sincere gratitude for taking the time to review our paper and for providing invaluable feedback. 

Since the deadline for discussion is just around the corner, if you still have any concerns about our paper after reading our response, please feel free to contact us by adding comments.

Your engagement during this discussion period is crucial for the improvement of our work. We genuinely value your insights and look forward to your continued feedback.

Thank you for your time and consideration.

Best regards,

Authors of Paper 4887