LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer
Authors: Guangyi Chen
Yuke Li
Xiao Liu
Zijian Li
Eman Al Suradi
Donglai Wei
Kun Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Video Question Answer
Visual Reasoning
Causal Represetation Learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Current approaches to Video Question Answering (VideoQA) primarily focus on cross-modality matching, which is limited by the requirement for extensive data annotations and the insufficient capacity for causal reasoning (e.g. attributing accidents). To address these challenges, we introduce a causal framework for video reasoning, termed Learning Latent Causal Processes (LLCP). At the heart of LLCP lies a multivariate generative model designed to analyze the spatial-temporal dynamics of objects within events. Leveraging the inherent modularity of causal mechanisms, we train the model through self-supervised local auto-regression eliminating the need for annotated question-answer pairs. During inference, the model is applied to answer two types of reasoning questions: accident attribution, which infers the cause from observed effects, and counterfactual prediction, which predicts the effects of counterfactual conditions given the factual evidence. In the first scenario, we identify variables that deviate from the established distribution by the learned model, signifying the root cause of accidents. In the second scenario, we replace embeddings of previous variables with counterfactual ones, enabling us to forecast potential developments. Once we have identified these cause/effect variables, natural language answers are derived through a combination of grammatical parsing and a pre-trained vision-language model. We assess the efficacy of LLCP on both synthetic and real-world data, demonstrating comparable performance to supervised methods despite our framework using no paired textual annotations.

Published as a conference paper at ICLR 2024
LLCP: LEARNING LATENT CAUSAL PROCESSES FOR REASONING-BASED VIDEO QUESTION ANSWER
Guangyi Chen' Yuke Li Xiao Liu", Zijian Li", Eman Al Suradi", Donglai Wei', Kun Zhang" "Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Yuke
CCameggi Mellon University, Pittsburgh PA, USA *Boston College. Massachusetts, USA
ABSTRACT
Current approaches to Video Question Answering (VideoQA) primarily focus on cross-modality matching, which is limited by the requirement for extensive data annotations and the insufficient capacity for causal reasoning (e.g. attributing accidents). To address these challenges, we introduce causal framework for video reasoning, termed Learning Latent Causal Processes (LLCP). At the heart of LLCP lies multivariate generative model designed to analyze the spatial-temporal dynamics of objects within events. Leveraging the inherent modularity of causal mechanisms, we train the model through self-supervised local auto-regression eliminating the need for annotated question-answer pairs. During inference, the model is applied to answer two types of reasoning questions: accident attribution, which infers the cause from observed effects, and counterfactual prediction, which predicts the effects of counterfactual conditions given the factual evidence. In the first scenario, we identify variables that deviate from the established distribution by the learned model, signifying the root cause of accidents. In the second, we replace embeddings of previous variables with counterfactual ones, enabling us forecast potential developments. Once we have identified these cause/effect variables, natural language answers are derived through a combination of grammatical parsing and a pre-trained vision-language model. We assess the efficacy of LLCP on both synthetic and real-worl world data, demonstrating comparable performance to supervised methods despite our framework using no paired textual annotations. The code is available at ht ttsps: /ggtttb...... CoonCHENNY122 2/LLCP.
INTRODUCTION
Video Question Answering (VideoQA) aims to comprehend and analyze video content, subsequently providing responses to queries posed in natural language. Leveraging its capability of visual dynamics understanding and cross cross-modality alignment, it serves as pivotal component in the field of interactive AI, garnrring significant attention in recent years.
Current methods Chen al. (2023); Cong al. (2021); Feng al. (2021); Xu al. (2021); Le et al. (2020) for VideoQA often treat it as a cross-modality matching task to learn temporal dynamics with annotations, shown the part (a) of Figure 1. For example, HCRN Le et al. (2020) enables video understanding by matching the visual and textual latent embeddings which are learned with annotated natural language guestion-answer pairs. However, these methods utilize the likelihood of cross-modal matching to learn the apatial-temport relations, which may fail to capture the underlying causality of the temporal dynamics, leading to serious issues for the performance of video reasoning systems.
To address this challenge, this paper introduces novel framework for video reasoning, named LLCP (Learning Latent Causal Processes), which utilizes a causality- based approach to uncover the latent causal relations and enhance the VideoQA task. LLCP employs temporal multivariate generative model (TMGM) to understand the underlying causal patterns of the agents the video events. As illustrated on the right side (b) of Figure 1, we show example of using our framework for traffic
"Equal contribution.

---

Published as a conference paper at ICLR 2024
Causai Processes
Video
Video Encoder
Video Content
Local Agents
Self Supersision
Match
Label
Reasoning
Text
Test
Local
Rensoning Keywords
Question
Encoder
Agents
Identity Cause Variable
[a] Conventional Method
LLCP
Figure I: Comparisons between conventional VideoQA methods and LLCP. Conventional meth- ods directly match the video content and texts, where the optimization heavily relies on textual annotations. While LLCP learns the latent causal processes with a temporal multivariate generative model by self-supervision (without paired data). Taking accident attribution as an example, LLCP can answer the reasoning question by matching the identified cause agent with parsed keywords. The red node on the right side denotes the root cause (the agent causing the accident).
accident analysis. We apply the causal mechanisms to model the interactions among agents, and leverage the trained model to conduct causal inference to answer questions.
Specifically, to learn temporal multivariate generative model (TMGM), we first localize the possible objects in the video event. Following the current video reasoning methods Cong et al. (2021); Ji et al. (2020), we utilize a pre-trained tracker object detector to identify and monitor agents. Subsequently, we employ TMGM to identify the underlying causal mechanisms that guide the behaviors these agents. Relying on the inherent modularity of causal mechanisms, the TMGM focuses solely on the parent variables from the last time step for each variable. These variables include the agent's historical state, interactions the vicinity, and environmental information. This generative model is optimized by maximizing the Evidence Lower Bound (ELBO) through self sufe--iurviiion (without textual annotations).
In the inference, we apply this self-supervised model to answer two types of reasoning questions, including accident attribution, which infers the cause from observed effects, and counterfactual prediction, which predicts the effects of specific counterfactual assumption. For accident attribution, we detect causes by identifying changing factors that do not follow the regular causal processes, i.c., the agents that cannot fit the causal mechanism well, to explain the accident. Specifically, the reconstruction accuracy to check whether an agent aligned well and then identify the root cause with low reconstruction accuracy. For counterfactual prediction, with factual evidence. Then we replace the embeddings of the cause variables with the ones in the counterfactual condition, and predict the potential changes with the learned model. After identifying the cause/effect variables, we apply them to answer the neural language questions. We first implement grammar parse which uses logical rules identifyy the key objects from the question-anse -nnswer (QA) pairs. Then, leveraging pre-trained vision-language model, we match the recognized visual objects with the extracted textual concept, thereby answering basic reasoning questions effectively. We evaluated our framework on two simulation datasets and two real-world VideoQA datasets. It consists of - Xu et al. (2021) for accident attribution and Causal VidQA Li et al. (2022a) for counterfactual prediction. We show that LLCP using no annotated textual question-answet pairs exhibits comparable performance with those methods that train with full supervision.
apply
first abduct the intra- charrccersssers
Contributions: 1) We propose framework, LLCP, which introduces a causal perspective of video
reasoning to reduce annotation reliance and enhance reasoning ability. 2) We propose TMGM to learn the local latent causal process and apply it for both cause-based and effect-based reasoning tasks. 3) We show that LLCP achieves comparable performance with supervised methods without using the paired Question-Answerdde annotations for the model training.
2
METHOD
In this section, we first introduce the TMGM model and then describe how we can leverage this model the inference for both cause-based and effect-based video reasoning tasks.
2.1 TEMPORAL MULTIVARIATE GENERATIVE MODEL
Causal Processes. Without loss of generality, assume the given video X consists of T frames and N agents as X (x For each variable, we introduce our data-generating pro-
2

---

Published as a conference paper at ICLR 2024
cess shown in Figure 2. In this figure, provide closer look using 2,-1.3 the example to explain the causal processes. The gray shade of nodes indicates that the variable is observable. Here assume that the observed objects x have the latent variables 2. Then the spatial-temporal transitions of the latent variables are divided into three parts: historical states (2t,3), neighborhood interactions (Zt.1 and Zr,2), and environmental information (e:). Specifically, For a given variable X its his- torical state is the variable with the same identity but in the past frame, i.e., xt,t
14+3,3
Neighborhoods
ALS
t. Similarly, its neighborhoods denote other variables (not experimental variables) in the previous frame, such as jl
ALS
أاا & i, The environment variables denote the environmental context, such as traffic lights, zebra crossing, and traffic signals. Mathematically,
Figure 2: The multivariate causal generation process, in which the historical state Zt,3, neighbor- hoods Zt,1, 2r,2, and environment variables e, are variables in the temporal dynamic system
Zt,i = (12 (e-ra- E Pa(zt,)),..s)) X1.i g(Ze,)).
(1)
6,, Per.,
where the observed object x generated from latent vari- able Zr,; with nonlinear function g. We apply a pre-trained multiple object tracking model Pang et al. (2021); Contributors (2020) detect the objects, which indicates part of g and the tracking labels serve as a bridge to connect objects and words. The latent variables 2 have stationary time-delayed (r denotes the time lag) causal relations which are formulated by where Pa(z,,)) are the parent variables of Zr.i, i..., ,a(z,,) denotes the set of latent factors that directly cause Zz,i. In addition to variables, To simplify, and W.L.O.G, in the following sections,
parent variables, Z4,i is also influenced by a noise term 6t,i, for different the noise terms ç, are we utilize the traffic events as an example to show the model and consider the setting with r
In this scenario, we further fi- nalize the formulation of parent variables Pa(z),)) as Pa(z).)
(3--1,.N(Z(-11)) (,)--1,),,,--1 et-r) where Z denotes the historical state of the current agent. (((--1.)) denotes the neighborhood variables in the t- - time step, which can be easily obtained by the distance on spatial locations. e is the historical environmental state.
X
Encoder
fn
fruse
Decoder
Based on the aforementioned data gener- ation process, we further provide the the- oretical analysis show that the Granger Causal relationship can be captured. We begin with the definition Granger non- causality of multi-type object sequences.
Figure 3: Network architecture of the temporal mul- tivariate generative model. We provide an example to show how the temporal multivariate generative model learns the latent causal processes of Xr+ 3. Blue denotes the neighborhood objects, green denotes the historical state, gray denotes no-related variables, and orange de- notes the environment variable. fH. fN. and fg are the sub-networks to learn the causal processes from histori- cal states, neighborhood interactions, and environmental clues, respectively. SLuse is sub-network to fuse them.
Definition (Granger non-causality of multi-type object sequences) Suppose that the object sequences X are gener- ated according to Equation 1, we can determine the Granger non-causality of the object x; with respect to object x, as follows: For all
Xt- 1at-,,1 Xt 111-1t--22 11t-,,2"" ... :x--ittT,,V and the same variable with different values Xt-1: 1--1--12 7, o X 1 if the following condition holds:
Xt-1:t- Xt- =
that is, Xt.j is invariant to Xt-1it-14 with d;.
Sequentially, we provide that the Granger Causality in the estimated function fy is identifiable. Proof of the theoretical results are shown in Appendix A2.
Proposition 1 (Identification of Granger Causality the estimated functiio fy.) Suppose that the estimated function f, well models the relationship between x; and x from the training data. Given
3

---

Published as a conference paper at ICLR 2024
Train
Env
t+1
Temporal Multivariate
Nbrhd
Encoder
Generative Model
Decoder
LELBO
Hist
(Paa
Normal video variables
Model causal process - Optimization by self-supervision
Inference
min Lyacon
Question: might be the reason for this
White bus made an illegal
Met1 ((aa(a))
CLIP
max 1ed
Answer: White bus made an illegui turn
Xx,1
Sicident-TTracke Generate with TMGM-+ Identify cause
Answer
Parsing for keyword QA-pairs
Figure 4: Overall framework of LLCP. In the training, we first obtain agents from regular videos by tracking model, then apply these agents fit temporal multivariate generative model learn the causal processes. Note that "Hist", "Nbrhd", "Env" denote history, neighborhood, and environment, respectively. This model is learned by the ELBO loss with self-supervision. first apply the learned generative model to process the test data, and then identify the cause variable (changing factor) which doesn't t follow the regular causal processes. With identified cause variables and language keywords parsed from the reasoning task, with pre-trained CLIP models.
the inference, we
give the answer by matching them
the ground truth Granger Causal structure G(V, Ev) with the maximum time lag where V and Ey denote the nodes and edges, respectively. Assume that the data are generated by Equation (1), 0.
then x, + +x & Ey if and only if
0x1.1 Ux.-1.1
Network. Given the features of observed variables extracted from frozen CL.IP image encoder, we model TMGM as an extension of Conditional Variational Auto-Encoders Sohn et (2015) (CVAE) with tailored modules to enforce the causal processes as Eq. 1. As shown in Figure 3, TMGM consists of three modules including learnable encoder network to map the features of observed variables into the latent space, a decoder to generate the observed variables with the latent variables, and transition module modeling the latent causal dynamics. In the transition module, we respectively learn three bebbneewwrks fN fH. and fE. to learn the transitions from historical states (green), neighborhood interaction (blue), and environment information (orange). Then, we apply another sub-network to fuse the information that comes from different sources. All sub-networks built by the Multilayer Perceptron (MLP).
sub-netwond
Optimization. To train the TMGM, formulate it as auto-regression task that predicts the future state with past information. Specifically, it is optimized by the classical ELBO objective:
N
CELBO
CRocon (X5,,, xc, - Blgg, log q (Zr, X4, 22 (P((Z..), (3)
1=1 1=1
where the CReeo to calculate the distance between the generation and the ground truth is imple- mented by the binary cross-entropy function, and 3 hyperparameter to balance two loss terms. The KL divergence LRLD between the prior distribution p(ž,,, Pa((,,) and posterior distribution (er,/Xx... Pa(2..)). The implementation details can be found in Appendix A4.
2.2 IDENTIFYING RooT CAUSE OF ACCIDENT
Definition of Root Cause. In temporal dynamic system, the root causes of accidents are defined as the variables that significantly diverge from the established local causal mechanisms Sheet (2016); Yang et al. (2022b). Basically, these alterations occur exclusively from the local causal mechanism, unaffected by other extraneous variables. To illustrate, consider traffic accident scenario where the
4

---

Published as a conference paper at ICLR 2024
root causes can clearly be attributed to individuals or vehicles that disregard traffic rules, rather than the agents impacted by these variables.
Implementation. We apply the TMGM method to learn the stationary local causal mechanisms of each variable, by analyzing transitions from historical states to the present states under normal events. When variable serves the root cause, notable shift in its local causal mechanisms becomes evident. To identify the changing factors, we contrast the observed transitions of each variable in the query sample against the causal mechanisms established through TMGM. The variable demonstrating deviation from the learned transitions potentially stands under the influence of altering factors. Specifically, we propose two criteria to examine whether the query variable the root cause, including the Maximum Reconstruction Error (MRE) and the Outlier Reconstruction Noise (ORN).
MRE serves as a direct method to confirm whether variable aligns with the local causal mechanisms by analyzing the reconstruction error. Typically, variables exhibiting larger reconstruction error are more likely to contain changing factors. Given query video, we first detect the variables X' {x(,;
Maximum Reconstruction Error.
T.N == in the same manner as the training stage. For
each variable X we can generate its predictions with our learned TMGM model using its parent variables. The process for detecting the root variable using MRE is as follows:
arg max ||x - x(,ills
where X,,i is the expected prediction ((Zi-rj ---pi------ E Pa(zi))),
obtained by the learned TMGM
model
Outlier Reconstruction Noise. An alternative manner is detect outliers by analyzing noise variables-with a fixed functional form of causal influence, we consider the instance as an outlier if the value of the noise is outlier. Specifically, we generate set of predictions (G,) with different samples ε사 of the noise variable, and find the noise value with the best prediction (the one with minimal predictive errors)
6.5 = arg min ||x|, - x<,,(ex))l
(5)
ck
Given the prior distribution of noise variables can be touched in Eq. i.e., a standard Gaussian distribution OO,, I), we can conduct the statistical test to verify whether (or how) & follows the prior distribution. Variables that demonstrate notable deviations in best-reconstruution noise from prior expectations are identified as root causes.
2.3 COUNTERFACTUAL PREDICTION
Definition. The counterfactual prediction in the causal dynamic system aims to reason the potential
development if some historical actions are changed. Mathematically, let Zr and Ze denote the sets of history and future latent variables, respectively. The counterfactual prediction Zr(uZZz = 2n) denotes that "Zr would be zy. given the situation u, if had Zn been z,."
Implementation. Given the factual evidence (existing event X), this counterfactual prediction ZF Z zi.) can be estimated with the following steps Pearl (2009):
Abduction: Estimate the situation u using existing factual event X as P(u[X). We employ the learned TMGM model given by f ((zi-r.jl Zt-r.j E Pa(zr,)) to represent the local causal mechanism. The variable represents the situation variable that requires estimation. For given query video X', we initially produce set of predictions by introducing different noises, represented as (.)). We then select the prediction with the optimal reconstruction as the estimated û, as outlined in Equation (5).
Action: Take the action do = 2%) to modify the causal process into counterfactual world. Based on the textual description of the counterfactual condition, we derive the textual embedding through an encoder, treating it as the estimated visual counterfactual condition 2n.
Use the modified causal process to predict counterfactuals as ZF ZF(ulZ zn). With the estimated situation ú and counterfactual condition 2n at hand, adjust the relevant variables in the TMGM model to forecast possible effects. For these counterfactual conditions, modify only the changing variables, retaining the rest in their original state.
Prediction:
5

---

Published as a conference paper at ICLR 2024
2.4 ANSWER REASONING QUESTION
In this section, we demonstrate how the derived causal/effect variables can be applied to effectively answer fundamental reasoning questions. Prior to this, we first outline a fundamental characteristic of reasoning questions that allows them to be differentiated from reasoning-free free questions. At their core, reasoning questions necessitate understanding variable "changes" and the corresponding impacts. To clarify, we provide several representative examples of reasoning questions, drawing attention to the notion of variable "changes". Consider the question Q1: "What is the reason for the accident?", It could be rephrased as: "Which variable change led to the accident?". Similarly, the question Q2: "What would happen if the rope broke?", translates to: "What would occur if the rope's 's status changes from intact to broken?". Lastly, Q3: "Would the accident still happen if the blue car did not run red light?", can be understood as: "Would the accident still happen when the state of variable 'blue car' is changed to 'did not run red light"?"
This suggests that the reasoning questions center on the variable "changes", evident in both questions (Q2, Q3) and answers (Q1). Furthermore, these questions can be segmented into two types: inferring causes (Q1, Q3) and predicting counterfactual effects Q2. Specifically, Q1 and Q2 represent attribu- tion and counterfactual prediction respectively. For Q3, though it also follows the sentence pattern of Q2 such as happen if ...", it is a true-or -a--or--hhe or-false question but not an open- question. Such a question can be answered via attribution reasoning. For instance, if the established cause is not 'run red light', the answer to this question is "no""
Technically, for open-choice attribution questions like Q1, we measure the distance between the identified root cause and answer candidates, choosing the most fitting one as the answer. This distance calculation can be facilitated by a pretrained vision-language model, such as CLIP Radford et al. (2021). In the case of open-choice counterfactual prediction questions like Q2, we begin by estimating potential counterfactual effects with counterfactull inference using the trained model, followed by comparing the effect to textual candidates. For true-or-false questions like Q3, initially identify the key object of the question, such as 'blue car'. It is accomplished via language parsing, particularly by examining conditional adverbial clauses. (Additional parsing outcomes available in the Appendix A6.4.) We then determine if this subject aligns with the identified root causal variable by gauging its cross-matching similarity against set threshold. This match indicates that modifying this element might prevent the incident (answer yes) or not (answer no).
3 EXPERIMENTS
We executed the experiments in two synthetic and two real-world VideoQA datasets. In the simulation tests, we assess the capability of LLCP to identify the root cause and predict counterfactual effects, utilizing the ground truth as a benchmark. For the real-world tests, we evaluate LLCP with reasoning- based VideoQA tasks related to cause identification and counterfactual prediction, respectively.
3.1 SIMULATION EXPERIMENTS
In this section, we conducted these types of simulation experiments to evaluate whether LLCP can learn the causal process and how it can help downstream inferences. The first one focuses on root cause identification, which aims to attribute the cause of accidents. The second one is counterfactual inference, predicting the possible state with the given factual evidence and counterfactual conditions. We further conduct simulation experiments to demonstrate the proposed LLCP can uncover underlying causal relationships, whose details are shown in Appendix A5.3.
Root cause identification. In this experiment, we examine two potential root causes: function change (where the accident occurs during the generation process) and structure change (where the accident pertains to the causal structure). We construct a random causal structure, G. comprising 10 nodes, and introduce both function and structure changes to simulate accidents. Our dataset comprises a training set of 700 regular samples, and test set of 1243 regular samples and 543 accident samples. For assessment, we employ Recall and F1 F1-score as our metrics, marking an evaluation as false either metric is not met.
Table 1 displays the experimental results. We have considered both VAE and LSTM- VAE as baseline methods. For each method, we employed five random seeds and reported the average outcomes.

---

Published as a conference paper at ICLR 2024
Additionally, conducted the Wilcoxon signed-rank test on these results. LLCP distinctly surpasses the baselines, with p-value threshold set at 0.05. Some observations are summarized below. Firstly, in terms of recall performance, the proposed approach markedly excels over other methods across all datasets, indicating LLCP 's effectiveness in accurately identifying true causal variables with minimal misses. Furthermore, LLCP also demonstrates considerable advantages in the F1-Score metric, signifying its efficiency in minimizing false detections retections-paaticllaty crucial in scenarios with sparse accident samples. Delve deeper into data generation specifies and extended experimental analysis in Appendix A5.
Counterfactual inference. For sim- ulating counterfactual inference, we generate paired validation data com- prising both factual evidence and counterfactual events (which share identical noise). During the infer- ence phase, we examine the factual evidence and forecast the counterfac- tual effects under counterfactual con-
Table I: the simulation datasets.
Performance comparison (Recall and F1 Scores) on
Function change
Structure change
Methods
Recall
F1 Score
Recall
FT Score
LSTM-VAE
52.11
27.81
37.75
20.78
VAE
51.92
27.33
31.08
19.35
LLCP
63.34
63.01
45.04
44.19
ditions, using the trained model. Our dataset consists of 32,000 training samples and 16,800 paired samples (both factual and counterfactual) for evaluation. The RMSE of the prediction serves as our evaluation metric.
Table displays the experimental results. Predic- tions evaluated under conditions with and with- out counterfactual inference. The tables provided are averages derived from three distinct experiments. Furthermore, after employing the Wilcoxon signed- rank test on these findings, it's evident our approach markedly surpasses the baselines, given p-value threshold set at 0.05. It's worth emphasizing that the model that incorporates counterfactual inference consistently demonstrates superior performance compared to its counterpart. Further elaboration can be found in Appendix A5.
Table 2:
Performance comparison (RMSE) on the simulation datasets.
w/ Counterfactual Inference
w/o Counterfactual Inference
0.0936
0.0967
3.2 REAL-WORLD EXPERIMENTS: SUTD-TRAFFICQA
We conduct experiments on the SUTD TTT--iiaficQA Xu et al. (2021) dataset to evaluate the efficacy of LLCP in terms of traffic accident reasoning. The questions in SUTD-TrafficO) focus on the analysis of accidents, i.e., identifying the root cause of accidents.
Dataset and experimental settings. SUTD-TrafficQA consists of 10,080 traffic video sequences depicting vari- ous scenarios, along with over 60,000
Table 3: More Ablation study on SUTD-TraffiQAA. We
focus on the reasoning tasks including C: "Counterfac- tual inference". I: "Introspection" and A: "Attribution" CLIP* denotes using the template to refine answer candi- dates in the unsupervised way as Chen et al. (2023).
QA sets, where 56,460 QA sets are used for training and the rest 6,075
are used for testing. It includes 6 tasks, including "Basic Understanding". "Event Forecasting". "Reverse Reason- ing". "Counterfactual Inference". "Intro- spection". and "Attribution". For our study, we primarily focus on causality- based question types, such as "Attribu- tion". "Introspection", and "Counterfac- tual", as prediction-basee problems can be solved using conventional likelihood- based methods. Specifically, we selected approximately 2,000 questions for our experiments. To ensure that the ques- tions were challenging and causal-related, we removed questions that focused on basic prediction and videos without any tracked variables. For detailed list of the selected questions, please refer
SUTD- TrafficQA
Methods
C
A
Avg
CLIP Radford et al. (2021)
32.6
23.5
27.7
27.7
CLIP* Chen et al. (2023)
43.4
34.8
30.1
31.2
w/o fN w/o fE
49.7
45.6
30.1
31.4
48.8
46.1
30.4
31.8
random JN
49.1
44.6
30.0
32.2
random fE
45.6
45.6
30.2
32.2
w/ Order
47.8
47.3
30.7
32.4
LLCP (ORN) LLCP (MRE)
49.6
48.2
30.8
33.2
50.4
46.4
31.3
33.5
7

---

Published as a conference paper at ICLR 2024
to the supplementary materials. All tasks were formulated multiple-choiee questions, with no limitations on the number of candidate answers. The evaluation metric we used was the accuracy of the multiple-choice responses.
Ablation study. Here we compare LLCP with the bascline and other ablation versions on SUTD- TraffiQQ. The ablation versions consist of "w/o fx", "w/o fE". and "w/ Order", where "wlo fx" and "w/o fE" denote that we remove the fN or fE part. "w/ Order" denotes that we don't use the shared network for different neighborhoods but build sub-network for each of them, where use the spatial distance to calculate the order of neighborhoods to avoid misalignment.
Table 3 summarizes the performance of all versions of LLCP and the baseline unsupervised CLIP Rad- ford et al. (2021) method. We can draw the following conclusions from the comparison. First, comparing the CLIP baseline with our LLCP we observe that learning causal process with multi- variate generative model can obtain overall significant improvement. Second, removing fN and fE hurts the performance, which demonstrates both these two modules help learn the causal generation process. Beyond "w/o fx", "w/o fE", we added "random fN" and "random fx='. We found that the parameters also influence the performance since the performance of using the random feature is a bit better than the total removal. Second, using random fx and LE hurts the performance, which demonstrates both these two modules help learn the causal generation process. Third, when we use different sub-networks for different neighborhoods and introduce the order, the performance drops. It reasonable since the calculation of orders might be inaccurate since the spatial distance in the image affected by 3D-to-2D projection and the quality of tracking models. Besides, also compare two kinds of methods (ORN and MRE) to identify the cause variable. We find that both of them achieve good performance and LLCP (MRE) is little higher.
Compared with supervised SOTAs. We also compare our LLCP with other state-of- the-art supervised methods. The results are summarized in Table 4. We apply LLCP to reasoning-basee based questions and employ the un- supervised CLIP model for the other reasoning- free questions, which directly matches the video and the declarative version QA pairs. Though without using the QA annotation (only using whether the video is regular or not). LLCP achieves promising results com- pared with other supervised methods. Specif- ically, outperform some baseline meth- ods such as CNN+LSTM. 13D+LSTM, and VIS+LSTM Ren al. (2015a). Besides, com- pared to the recent method ATP Buch et al. (2022), which also used the CLIP and further learned an adapter module with the annotated supervision signals, LLCP can still achieve comparable performance.
Table 4: Comparison with the state-of-the- art methods on SUTD-TrafficQA. QA denotes whether the QA pair annotation is used for training.
Methods
QA
Acc
Avgpooling
30.5
CNN+LSTM
30.8
13D+LSTM
33.2
VIS+LSTM Ren et al. (2015a)
29.9
BERT-VQA Yang et al. (2020)
Yes
33.7
TVQA Lei et al. (2018)
35.2
HCRN Le al. (2020)
36.5
Eclipse Xu et al. (2021)
37.1
Tem-Adapter Chen et al. (2023)
46.0
ATP + CL.P Buch et al. (2022)
35.6
CLIP Radford et al. (2021)
27.5
No
LLCP + CLIP
33.3
3.3 REAL- WORLD EXPERIMENTS: CAUSAL-VIDQA
To further assess the capability of LLCP for counterfactual prediction tasks, we have performed experiments on the Causal- VidQA Li et al. (2022a) dataset. We have chosen not to include the NexIQA dataset in this study as they possess identical question types.
Dataset and experimental settings. Questions in the Causal-VidQA dataset are divided into four types: description, explanatory, prediction, and counterfactual. In this study, we only focus on reasoning-basse questions, specifically those of counterfactuals. We conduct evaluations on both validation and test sets. We follow the experimental protocol in Li et al. (2022a) to report the results.
Results and Analysis. As shown in Table 5, we compared LLCP with other state-of-the-ar methods. Every method was re-implemented using the official code, uniformly employing the same CLIP feature. Compared with the CLIP baseline, LLCP registers improvement of nearly 10 percent on average. Remarkably, even when compared against other supervised learning approaches, our method yields results that are comparable, despite the absence of annotated data for training.
8

---

Published as a conference paper at ICLR 2024
Table 5: Comparison with baseline methods on Causal-VidQA. We re-train all methods using the same CLIP features using the officially released code and report the results both validation and test datasets related to reasoning tasks. More results are in the Appendix.
Validation
Test
Methods
QA
Avg
Q -
Q -
R
Q
AR
Q
A
Q
R
AR
EVQA Antol et al. (2015)
28.27
28.05
9.54
28.05
28.05
10.09
22.00
CoMem Gao al. (2018)
47.61
47.01
25.60
45.70
47.60
25.44
39.83
HME Fan al. (2019)
46.09
47.12
25.53
45.48
46.51
24.74
39.25
Yes
HCRN Le et al. (2020)
45.64
46.01
24.79
44.26
45.64
24.35
38.49
HGA Jiang & Han (2020)
45.57
45.75
24.38
45.28
46.80
24.81
38.77
B2A Park et al. (2021)
48.83
48.98
27.68
47.41
48.74
27.39
41.51
CLIP Radford et al. (2021)
31.61
28.50
11.32
29.95
29.51
11.48
23.73
No
LLCP
38.36
38.91
17.93
39.07
38.46
19.03
32.13
Q: What types of vehicles that if removed from the videos, won't be accident?
Q: Could the accident be prevented if the involved vehicles change property?
E Bicycle tricycle or non non-ootorrno vehidles.
Counterfactual (success)
Introspection (success)
@: What might the reason which accident? AI The white sedan did an illegal lane changing.
Could involved vehicles stop time the accident?
(c) Attribution (success)
(d) Introspection (fail)
Figure 5: Visualization results of LLCP on SUTD-TrafficQA. We show positive examples of different question types including (a) Counterfactual, (b) Introspection, and (c) Attribution. Besides, we also provide a failure case (d) Introspection.
3.4 QUALITATIVE RESULTS AND DISCUSSION
In this section, provide some qualitative results the SUTD-TraffiQA dataset in Figure 5, to analyze our framework by case. We provide both success and failure cases on different question types, such as (a) Counterfactual, (b) Introspection, and (c) Attribution, understand the methods and explore the boundary. We provide the question, our answer, and our learned causal process to show why LLCP can work fail. Taking (a) Counterfactual an example, LLCP identifies the root cause as "bicycle", i.c., the bicycle doesn't 't follow the normal causal mechanism (fall down). For (c) Auribution, LLCP can identify the abnormal action of the white car (the illegal lane changing) given its neighborhoods and environmental information. For the failure case (d), though the model can identify the cause variable, the understanding of the current variable's 's state is incorrect, which leads to the failure.
CONCLUSION
In this paper, we introduce novel framework, LLCP, designed to tackle reasoning-based VideoQA through causality lens. In this framework, we exploit the modularity of causal mechanisms to develop temporal multivariate generative model tailored for local causal processes. During the inference phase, LLCP can attribute the accidents by identifying the root cause, or forecast the potential outcomes under counterfactual scenarios. Furthermore, aligning the visual and textual domains with CLIP allows us to respond to queries using neural language. the experiments. our framework has proven effective both synthetic and real-world datasets.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This material is based upon work supported by the Al Research Institutes Program funded by the National Science Foundation under Al Institute for Societal Decision Making (AI-SDM), Award No. 2229881. The project is also partially supported by the National Institutes of Health (NIH) under Contract R0IHL159805, and grants from Apple Inc., KDDI Research Inc., Quris AI, and Infinite Brain Technology.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pp. 2425-2433, 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 01300010.,,,....000011 2023.
Max Bain, Arsha Nagrani, Güil Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. ICCV, pp. 1728-1738, 2021.
Nimrod Berman, Ilan Naiman, and Omri Azencot. Multifactor sequential disentanglement via structured koopman autoencoders. arXiv preprint arXiv:2303. /7264, 2023.
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" in video-language understanding. In CVPR, pp. 2917-2927, 2022.
Guangyi Chen, Xiao Liu, Guangrun Wang, Kun Zhang, Philip HS Torr, Xiao-Ping Zhang, and Yansong Tang. ada-trr Adapting image-text pretraining for video question answer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13945-13955, 2023.
Kai-Wei Cheng, Yie-Tarng Chen, and Wen-Hsien Fang. Video anomaly detection and localization using hierarchical feature representation and gaussian process regression. In CVPR, pp. 2909-2917, 2015.
Yang Cong, Junsong Yuan, and Ji Liu. Sparse reconstruction cost for abnormal event detection. In CVPR, pp.
2011.
Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial- temporal transformer for dynamic scene graph generation. In ICCV, pp. 16372-16382, 2021.
MMTracking Contributors. MMTracking: OpenMMLab video perception toolbox and benchmark.
https: /ttp::////thubbb// com
cow/open open mml aatpennnmmmmb/mmmmmmmm.a mabmmimtrccking 2020.
Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A
large-scale benchmark for tracking any object. In ECCV, pp. 436-454. Springer, 2020.
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heteroge- neous memory enhanced multimodal attention model for video question answering. In CVPR, pp. 1999-2007, 2019.
Shengyu Feng, Subarna Tripathi, Hesham Mostafa, Marcel Nassar, and Somdeb Majumdar. Exploiting long-term dependencies for generating dynamic scene graphs. arXiv preprint arXiv:2112.09826 2021.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision- languaelanguage pre- training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 14(3-4): 2022.
10

---

Published as a conference paper at ICLR 2024
Jiyang Gao, Runzhou Ge. Kan Chen, and Ram Nevatia. Motion appeearapee co-memory networks for video question answering. In CVPR. pp. 6576-6585. 2018.
Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K Roy- Bay-Chowdhury, and Larry S Davis. Learning temporal regularity in video sequences. In CVPR, pp. 733-742, 2016.
Ryota Hinami, Tao Mei, and Shin' ichi Satoh. Joint detection and recounting of abnormal events by learning deep generic knowledge. In ICCV, pp. 3619-3627, 2017.
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR, pp. 10236-10247, 2020.
Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In AAAI, volume 34, pp. 11109-11116, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint UrXiv:1412.69880 2014.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint wwXiv:2007.10930 2020.
Gouthaman Kv and Anurag Mittal. Reducing language biases in visual question answering with question encoder. In Computer Exion-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 18-34. Springer, 2020.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering, In CVPR, pp. 9972-99811 2020.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video 2018.
question answering. arXiv preprint arXiv:/809.0//
Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. Tvgat: Spatio-temporal grounding for video question answering. arXiv preprint arXiv:/904.1 1574, 2019.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-langusae lannaudd learning via sparse sampling. In CVPR, pp. 7331-7341, 2021.
Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-ansen answering.. CVPR, pp. 21273-21282, 2022a.
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, pp. 2928-2937, 2022b.
Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a new baseline. In CVPR, pp. 6536-6545, 2018.
Cewu Lu, Jianping Shi, and Jiaya Jia. Abnormal event detection at 150 fps matlab. ICCV, pp. 2720-2727, 2013.
Weixin Luo, Wen Liu, and Shenghua Gao. A revisit of sparse coding based anomaly detection in stacked rnn framework. ICCV, pp. 341-349, 2017.
Ričards Marcinkevičs and Julia E Vogt. Interpretable models for granger causality using self- explaining neural networks. arXiv preprint arXiv:2101.07600, 2021.
Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In CVPR, pp. 164-173. 173, 2021.
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Bridge to answer: Structure-ave aware graph interaction network for video question answering. In CVPR, pp. 15526-15533 2021.
Judea Pearl. Causality. Cambridge university press, 2009.

---

Published as a conference paper at ICLR 2024
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763. PMLR, 2021.
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Advances in neural information processing systems, 28, 2015a.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, pp. 91-99, 2015b.
Mohammad Sabokrou, Mohsen Fayyaz, Mahmood Fathy, and Reinhard Klette. Deep-cascade: Cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. TIP, 2694): 2017.
OSHA Fact Sheet. The importance of root cause analysis during incident investigation. URL https://www: osha. gov/Publications/OSS/AAAS pdf. 2016.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Hung-Ting Su, Yulei Niu, Xudong Lin, Winston H Hsu, and Shih-Fu Chang. Language models are causal knowledge extractors for zero-shot video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4950-4959, 2023.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, pp. 4631-4640, 2016.
Yushen Wei, Yang Liu, Hong Yan, Guanbin Li, and Liang Lin. Visual causal scene refinement for video question answering. In ACM MM, 2023.
Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. Debiased visual question answering from feature and sample perspectives. Advances in Neural Information Processing Systems, 01377843779, 2021.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, pp. 9777-9786, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pp.
2016.
Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and efficient network for vidco reasoning over traffic events. In CVPR, pp. 9878-9888, 2021.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. NeurIPS, 2022a.
Wenzhuo Yang, Kun Zhang, and Steven Hoi. A causal approach to detecting multivariate time-series anomalies and root causes. arXiv preprint arXiv:2206. /5033, 2022b.
Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. Bert representations for video question answering. In WACV, pp. 1556-1565, 2020.
Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. arXiv preprint S.XXv:2210.1364/ 2022a.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. In ICLR, 2022b.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenhaum. Clevrer: Collision events for video representation and reasoning. In ICLR, 2020.
12

---

Published as a conference paper at ICLR 2024
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, volume 33, pp. 9127-9134, 2019. 2019.
Desen Yuan. Language bias in visual question answering: A survey and taxonomy. arXiv preprint arXiv:2111.08531 2021.
Chuanqi Zang, Hanqing Wang, Mingtao Pei, and Wei Liang. Discovering the real association: Multimodal causal reasoning in video question answering. In CVPR, pp. 19027-19036, 2023.
13

---

Published as a conference paper at ICLR 2024
Appendix for
"LLCP:
Learning Latent Causal Processes for Reasoning-based Video
Question Answer"
Appendix organization:
A1 Related Works
15
A2 Theoretical Analysis
15
A3 Further Analysis and Dissusions of LLCP
16
A3.1 Assumptions of LLCP
16
A3.2 Limitations
16
A3.3 Rationality and Motivation for Training without Textual Annotations
17
A4 Implementation Details of LLCP
17
AS More Details of Simulation Experiments
18
A5.1 Root Causal Identification
18
A5.2 Counterfactual Prediction
19
A5.3 Granger Causality Discovery
19
A6 More experimental analysis
20
A6.1 Parameters Analysis
20
A6.2 Compared with LLMs-based methods
20
A6.3 More Experimental Results Causal VidQA
21
A6.4 Parsing Results
14

---

Published as a conference paper at ICLR 2024
Table Al: A summary and classification of VideoQA datasets.
Datasets
MSVD-QA AISVITE-OA OAiiviyNee-OA
(2016)Xa (2016) (2019) Lei (2018)Taparwi
TVQA
MovieQA
CLEVRER
(2020)Xioo (2021)
NEXT-QA
TrathoOA
Cansal-VidOA
Reasoning
Multimodal
Real Videos
A1 RELATED WORKS
Video Question Answering. Depending upon the nature of the questions, VideoQA tasks can be categorized into two distinct sectors: perception- based VideoQA and reasoning-based VideoQA, shown in Table A1. The former, evidenced in studies such as Xu et al. (2016); Yu et al. (2019); Tapaswi et al. (2016); Lei et al. (2018; 2019), focuses on deciphering video content to respond natural language inquiries concerning visual facts such location (where), time (when), object (who), attributes (e.g.. color or quantity), and manner (how). A core hurdle here is achieving seamless coos-mmd matching. On the other hand, reasoning-based VideoQA, documented in Xu et al. (2021); Yi et al. (2020); ; Li et al. (2022a); Xiao et al. (2021), concentrates on analyzing temporal dynamics and discerning the causality of video events, addressing 'why' or 'what if" types of questions. This necessitates the model's comprehension of the causal relations among the agents. Conventional methods for reasoning-basee based VideoQA often delineate the task through the lens of cross-modality matching Ji et al. (2020); Cong et al. (2021); Feng et al. (2021); Le et al. (2020); Gao al. (2018); Lei al. (2021); I); Bain al. (2021); Yang al. (2022a). Yet, these strategies are heavily dependent on extensive paired annotated data, and may be misled by spurious correlations present in the training sets. Recently, several methods Wei et al. (2023); Zang et al. (2023); Li et al. (2022b) have incorporated causal inference techniques to mitigate bias and enhance resilience. However, these approaches necessitate prior understanding of established causal frameworks, alongside paired annotations. In this study, we propose groundbreaking framework, LLCP, which eliminates the need for paired data and prior knowledge of causal structures.
Abnormal Event Detection. Abnormal event detection methods often learn familiarity model to distinguish the outlier samples from the pool of regular videos. To build the regular distribution, existing methods apply dictionary learning methods Cong et al. (2011); Lu al. (2013); Cheng et al. (2015) or deep neural networks including deep generic knowledge Hinami et al. (2017), stacked RNN Luo al. (2017), C3D Sabokrou et al. (2017), and plug-and and-play CNNs Hasan et al. (2016); Liu et al. (2018). Unlike abnormal event detection which only focuses on the distribution of the whole event, we go further to understand the underlying causal processes of the event and find the reason why this event is abnormal.
Causal Representation Learning. Recently, causal representation learning methods have demon- strated remarkable success in identifying the latent variables from time-series data by learning the causal latent processes in an unsupervised manner Yao et al. (2022b;a). Unlike these methods iden- tifying latent variables, in this paper, focus more on how to apply the learned causal mechanism to discover changed factors and analyze the video event.
A2 THEORETICAL ANALYSIS
In this section, we will theoretically analyze the identification of the proposed model. To achieve this, we begin with the definition of the Granger non-causality of multi-type object sequences as follows.
Definition 1 (Granger non-causality of multi-type object sequences) Suppose that the object se- quences X are generated according to Equation I, we can determine the Granger non-causality of the object x, with respect object xj as follows: For all X, '"" .Xt-1:t- and the same variable with different values X Xt-lit-ri + x/-100-,p
the following condition holds:
(x Xt--it-rds"" Xt- Qy (xt-11t-r,11"" X4-1:t-7,5" X Sr-1st-rae
that is, Xt,j is invariant X1-1---,3 with dj-
Proposition 1 (Identification of Granger Causality in the estimated functionn fy.) Suppose that the estimated function fú well models the relationship between x; and x, from the training data. Given the ground truth Granger Causal structure GVV Ev) with the maximum time lag of I, where V and
15

---

Published as a conference paper at ICLR 2024
Ey denote the nodes and edges, respectively. Assume that the data generated by Equation (1), then x; X3 & Ey if and only if 0xx,1 = O.
0x1-1.1
Proof 1 = If,; X2 & Ey (there is no Granger Causality between x; and x; in the ground truth process), then according to Definition I, there must be X; = x for all the different values of X2. If @x1,1 iix-111 0, then when the input values are different, i.e, X; + X,, the outputs f, are also different, i.e., X3 # x.. which results in contradictions.
T: Suppose x, + X3 Ev (there Granger Causality between x; and x; in the ground truth process), for any Xj and X3 in Definition 1, if X3 + X3y there must be different input x; 4 X,. If 0x 0, then when x # X3y there might be x; = X,, which results in contradictions.
8x;
A3
FURTHER ANALYSIS AND DISSUSIONS OF LLCP
A3.1
ASSUMPTIONS OF LLCP
To well illustrate our method, we intuitively discuss the assumptions inherent in LLCP, which are crucial for understanding the causality in the temporal dynamic system. To strengthen the credibility and rationality of our outlined assumptions, we also delve deeper into their practical applications and justifications:
1. Granger Causality Principle: It is assumed that the past values of one variable predict the future values of another. This implies a directional relationship between variables in time-series context. It also indicates there are no instantaneous effects the temporal dynamics, i.e., the variables 2, are independent given the historical state Pa(z)) This assumption aligns seamlessly with the data generation process, as described in Eq. Implementation: Implementationaally, we employed auto-regression to capitalize on the concept of Granger Causality. This method predicates future events based on historical data, thereby validating our assumption about the predictability of one variable's 's future values based on another's past values. We provided a proposition and corresponding proof to show the identification of the proposed model in Section A2.
2. Shared Feature Spaces in Visual and Textual Domains: feature spaces for both visual and textual domains are shared, allowing for matching the same variables across these domains. Implementation: To validate this assumption about the shared feature spaces visual and textual domains, we incorporated the pre-trained CLIP model. This model acts as feature extractor for both video objects and textual elements (questions and answers). Its application ensures that both visual and textual data are represented in congruent feature spaces, thereby affirming the feasibility of matching variables across these domains.
The assumption here is that the
A3.2 LIMITATIONS
One notable limitation in our approach is the reliance on the Granger Causality assumption, which can be considered somewhat robust. For the sake of simplicity, our current model does account for more complex scenarios, such as the presence of latent confounders or non-stationary conditions. These factors, while critical in certain contexts, are beyond the scope of our initial assumptions. Despite this, our application of Granger Causality serves as foundation for potential framework. This framework instrumental in understanding and leveraging causal relationships within temporal dynamics for subsequent reasoning tasks. Looking ahead, aim to explore the identifiability of causal variables under less stringent assumptions. This progression will enable a more nuanced understanding of causal relationships in diverse and complex environments.
Another aspect of our methodology is its reliance on pre- pre-trained models, notably CLIP and tracking models. The primary motivation behind employing these models is to minimize the necessity of textual annotations. We leave it as the future work to remove these assumptions and learn the latent variable from seratch with textual annotations.
16

---

Published as a conference paper at ICLR 2024
A3.3 RATIONALITY AND MOTIVATION FOR TRAINING WITHOUT TEXTUAL ANNOTATIONS
Please note that the usages of textual information in training and inference are different. We provide the discussions on usage proposals in both stages below.
Training Without Textual Pairs. would be really good if we could always have enough annotated question-answering pairs as the supervision. It would help the learning of the reasoning model. However. the annotated question-answering pairs are not easy to obtain, especially for tasks requiring domain knowledge such traffic accident attribution. For humans, learning to reason can be effectively achieved without cross-modality supervision. relying solely on observation induction within shared feature spaces. This concept aligns with causality theories Yao et al. (2022b;a), suggesting the feasibility of uncovering latent variables and causal relationships unsupervised. This approach is particularly relevant in real-world applications for several reasons. Firstly, the acquisition of annotated textual pairs is more challenging and costly compared to collecting unannotated videos. Additionally, models trained with textual annotations are susceptible language biases, where they might learn shortcuts based on linguistic patterns rather than understanding the content. For instance, if all questions in the training set beginning with "What is the reason" " have similar answers, the model might generalize inaccurately, associating any question with this format to a specific answer, such as "White car speeding". regardless of context. This phenomenon, known as language bias, has been extensively discussed in question-answert research Yuan (2021); Kv & Mittal (2020); Wen et al. (2021).
Use of Text Queries in the Inference Phase. Despite the avoidance of textual pairs in training, text queries play crucial role during the inference phase. Our model, trained exclusively on visual data, can identify the visual causal relations of variables. During inference, textual queries are employed to first identify the local causal mechanisms we are interested in. Then, we leverage the causal relations to answer the detailed question. For example, if two accidents occur in the same video, our model will find causal mechanisms with two root causes. Then the query questions can help to identify the root cause we are more interested in.
A4 IMPLEMENTATION DETAILS OF LLCP
In this section, we provide the implementation details of our approach for better reproduction. Models and Network Architecture. In our experiments, we track the variables with QDTrack Pang et al. (2021) pre-trained on the TAO Dave et al. (2020) dataset. We select only traffic-related categories, such as "bicycle", "bus", "car", "motorcycle," and so on. After obtaining the bounding box, we extract the feature of each agent by the ROI Ren et al. (2015b) pool from the CLIP Radford et al. (2021) (RN50) image encoder. For the environment variable, we use the feature of the whole image and thus do not need to build specific causal processes for environment variables. Given the dimension of features from CLIP is 1024, the encoder network we used for CVAE is Linear( 1024, 256). Correspondingly, the decoder Linear(256. 1024). For the sub-networks fH. fN. and fÆ to learn the causal process from historical states, neighborhood interactions, and environmental clues, are built with Linear( (1024, 256), respectively. the experiments for SUTD-TraffcQAA we set the number of neighborhoods as 2, and they share the same fN. Then we connect 4 local features and use a Linear( 1024, 16) for information fusion. The dimension of the generative noise is set as 10. In the inference, the features of textual keywords are extracted by CLIP text encoder.
Hyperparameters. We follow the video frames sampling strategy with Le et al. (2020). We first uniformly sample 8 frames from the video sequence, then with these frames the center, we select 4 frames around each center. Finally, we obtained 32 frames for each video. For the loss function, we set the B = 0.5 to balance the reconstruction and KLD.
Optimization Details, We apply Adam Kingma& & Ba (2014) as the optimizer with the initial learning rate 1e-4 and batch size as 16. The learning rate decay is set to 0.5 to reduce the learning rate to half in every 10 epochs. The number of total epochs is set as 50. The model is implemented in PyTorch 1.13.1 and trained a single Testa V100 GPU.
17

---

Published as a conference paper at ICLR 2024
A5
MORE DETAILS OF SIMULATION EXPERIMENTS
A5.1 ROOT CAUSAL IDENTIFICATION
Dataset and experimental settings.To evaluate whether LLCP can learn the causal process and identify the cause variables, we design series of simulation experiments based on a random causal structure with a given sample size and variable size. To simulate the ground-trun circumstance, we assume that the training dataset does not contain any accident videos and that the test dataset contains regular videos and accident videos. The simulated datasets are generated in the following three steps.
First, we randomly generate causal structure G with the number of 10 nodes for the training data. Specif- ically, we let G.g be the value in the i-th row and j-th column of G. Therefore, if G., & O, thej-th variable becomes the direct cause of i-th variable, i.e. 4-115 + Zt.i- Since the causal structure might be assume that each node most
((Pa(t 1a)) - ((Pa(x aal (((((((()))))))))) ((...))))
Z++1a
Zz,1
1++1,1
Zu2
2e+12
Zr,2
Zt+1,2
sparse in reality, has three causes. Second, we recursively generate the simulated data with the nonlinear function:
7
Lt3
7,+
(a) Function change (b) Structure change
Z411,1 Sigmoid( Σ
EGss
Gas x Zt, + Ex) c++1
Figure Al: The examples of two types of change factors. Function change denotes that changes are caused by the generation function. For example, the generation function f (Pa(z)42.) (+111) changes to f' (Pa(zı+1.a) c++111). (b) shows the changes come from the causal structures, e.g. the edge žt,2 Z441,1 is blocked due to some accidents.
/=0
X4+1,1 = o(z++1,),
where denotes the latent variables and denotes the observed variables; <(.) is a linear transformation from to x; F, sin(t) denotes the environment variables and c++1 denotes the noise variables. Third, to simulate accidents, we consider two types of changes, i.e., the function change and the structure change, as shown in Figure A1. Function changes act on the nonlinear functions while structure changes structure. The function changes are implemented as the direct value change of the latent variables:
the causal
4+1.i = a (Z+1, Z1.t,) + Zist,é
in where Zi:t; denotes the average value of z, from 1-st timestamp to t-th timestamp. For the structure change, block some edges (e.g., 2,2 - 24+-1.1 1) from G and obtain intervened causal graph G², and generate the accident value by replacing G with G' in Equation (7). Using these two types of accidents, we build test set that contains 1243 regular samples and 543 accident samples.
Evaluation metric. Based on the aforementioned simulation dataset, we evaluate LLCP with two tasks including abnormal event detection classify where the data contains accidents and changed factor identification to find the changed variable. We apply both Recall and Fl-score as our evaluation metrics and we mark false when either one of them is wrong.
Quantitative results.
Experiment results on the simula- tion dataset are shown in Table A2, where the VAE, LSTM-VAE, Slow- VAE Klindt et al. (2020). and SKD Berman al. (2023) are consid- ered compared to baseline meth- ods. For all the methods, we try five different random seeds and report the average results. We also conducted the Wilcoxon signed-rank test on the results, our method significantly out- performs the baselines, with p- p-value threshold of 0.05.
Table A2: Performance comparison (Recall and Fl Scores) on the simulation datasets.
Function change
Structure change
Methods
Recall
Fl Score
Recall
FI Score
LSTM-VAE
52.11
27.81
37.75
20.78
VAE
51.92
27.33
31.08
19.35
SlowVAE
54.41
29.03
38.12
21.39
SKD
58.93
34.76
35.72
19.79
LLCP
63.34
63.01
45.04
44.19
According to the experiment results, we have the following observations. First, compared with the performance of recall, we find that the performance of the proposed approach outperforms the other
18

---

Published as a conference paper at ICLR 2024
methods on all the datasets with large gap, reflecting that LLCP identify true cause variables with low missed detections. This because our method leverages causal mechanisms to infer the accident with the local causal process instead of detecting the variables with the abnormal distribution. Moreover, we find LLCP also enjoy more advantages in the metric of FI-Score, reflecting that obtains low false detections and thus works well for the situation where accident samples are rare. We found that even compared with the advanced SKD Berman et al. (2023) model, LLCP can achieve significant improvement.
A5.2 COUNTERFACTUMN PREDICTION
Dataset and experiment settings.
To evaluate whether the proposed method can address the counterfactual inference problem, devise the counterfactual prediction simulation experiment.
The generation processes of factual and counterfactual data are shown in Figure A2. As for the factual datasets, we randomly generate a causal structure G with the number of 10 nodes. Specifically, we let G be the value in the i-th row and j-column of G. Similar to the simulated data in the function change scenario 1, we assume that each node most has three causes and the factual simulated datasets are recursively generated via Equation (7). As for the coun- terfactual datasets, we employ the same data genera- tion process but different historical variables, which is shown in Equation(9))
1e+1 f(2
f f(ff)
zfA
42
xf2
1+,,; Sigmoid(
D GiD (ziz)) E1) c141
2,3
1E413
z
j=0
Factual Data
Counterfactual Data
Xi+1, (Zitt1,))
(9)
Figure A2: The examples of factual and counterfactual data. Both factual and counterfactual data are generated via the same causal process. The counterfactual data are generated with perturbed input.
where z denotes the counterfactual latent variables and x" denotes the observed counterfactual variables; y is nonlinear function. We use the historical observation Z input and predict the future observation Z There are 32000 samples in the training dataset and 16800 pairs of factual and counterfactual samples in the test dataset for counterfactual inference.
Evaluation metric. Based on the above simulation dataset, we employ Root Mean Square Error (RMSE) evaluate our counterfactual prediction performance, the lower values of RMSE reflect better performance of counterfactual inference.
Quantitative result. Experiment results on the simulation datasets are shown in Table A3. The w/o Counterfactual Inference denotes the model of standard variational autoencoder (VAE) and the w/ Counterfactual Inference denotes the VAE model with noise estimation like Pearl (2009). For all the methods, we try three different random seeds
Table A3: Performance comparison (RMSE) the simulation datasets.
w/ Counterfactual Inference 0.0936
w/o Counterfactual Inference 0.0967
and report the average results. We also conducted the Wilcoxon signed-rank test on the results, our method significantly outperforms the baselines, with paluue threshold of 0.05. According the experiment results, we find that the model with noise estimation achieves a better performance than the normal VAE, showing that our method can address the counterfactual prediction problem.
A5.3 GRANGER CAUSALITY DISCOVERY
To demonstrate the proposed LLCP can uncover underlying causal relations as opposed to merely correlational patterns, we further evaluate our method on the Granger Causal-
ity task. To achieve this, we follow the setting in Marcinkeviës & Vogt (2021) and evaluate the proposed methods on the simulated FMRI time series benchmark.
19

---

Published as a conference paper at ICLR 2024
Ratio distance
Latent dimension
Middle layer di mension
23.0
22.
22.0
Acc(%)
Acc(%)
40-4
00.7
10.7
22.0
32.5
Ratio
Dimension
Dimension
Figure A3: Parameter analysis of LLCP. We analyze three main hyperparameters including the distance ratio to fuse the local distance and global distance, the dimension of the latent space, and the dimension of the local sub-networks (fH-fN-fE). which show in the left, middle, and right, respectively.
To make our method generate the Granger Causality explicitly, we employ the conditional-VAR architecture and consider the Granger causal structures as the latent variables. As for the HCRN model, we obtain the estimated structure by an inner product between the extracted features from the HCRN model. We further consider four different metrics, i.e., accuracy (ACC), balanced accuracy (BA) scores, areas under receiver operating characteristic (AUROC), and precision-recall (AUPRC) curves. Experiment results shown in Table A4. According to the experiment results, we can find that the proposed LLCP can capture the underlying causal relations.
Table A4: Experiments results the capability to uncover Granger Causality.
Models
ACC
BA
AUROC
AUPRC
LLCP HCRN
0.788
0.632
0.657
0.289
0.632
0.491
0,504
0.095
A6
MORE EXPERIMENTAL ANALYSIS
A6.1 PARAMETERS ANALYSIS
In this section, we re-trained our model with different hyperparameters and provided the experimental results the SUTD- -UTD-TraffiQAA dataset to investigate their effects. Specifically, analyzed three main hyperparameters including the distance ratio to fuse the local distance and global distance, the dimension of the latent space, and the dimension of the local sub-networks (fff.).fE)
The local distance is calculated by the detected variable and language keywords and the global distance is calculated by the global visual features (the average of features of all frames) and textual description. We fuse these two complementary distances for better reasoning, where the local one can identify the separated causal variable while the global one focuses
on overall event perception. The left one of Figure A3 shows the results on the SUTD-Traffice dataset with different ratios. We found that our model achieves the optimal results with the ratio of 1/6 between local and global distances.
As shown in the middle and right parts of Figure A3, we provide the performance of the models with different latent and middle layer dimensions. We observe that LLCP achieves better results when we increase the latent dimensions and middle layer dimensions, since the representation abilities increase with more parameters. However, we also find that this increase will converge when parameters are enough. Balancing the accuracy and cost, we set the latent dimension as 16 and the middle layer dimensions as 256, respectively.
A6.2 COMPARED WITH LLMS-BASED METHOOSS
The LLMs-basee methods Alayrac et al. (2022); Gan et al. (2022); Awadalla et al. (2023); Su et al. (2023) also don't require the reasoning QA pair for the training. To make more complete comparison, we also compare our method with these LLMs-based methods, such as OpenFlamingo Awadalla et al. (2023). We evaluate all the models the SUTD-TrafficQA dataset with zero-shot setting. For
20

---

Published as a conference paper at ICLR 2024
OpenFlamingo, used the 2/4 textual demo examples (no visual data due to zero-shot) guide the outputs. Here is an illustrative Python code of how to use the demo example in the prompt.
As shown in Table A5, summarize the average accuracy of OpenFlamingo, CLIP, and LLCP. We found that the LLM models don't work as well as we expected since no particular knowledge used. It is interesting and may inspire the following research.
[aaguage==Pyhon masicstyle=httfamme
breaklinesztrue,
showstri showstringsspces=-fllss
for i in range(len questions )):
token www.iinutssppppp.
Question: What might have happened moments ago?
Candidate 1: The blue truck hit the white sedan from the back.
Candidate 2: The white sedan crashed into the blue truck.
Candidate 3: The blue truck did emergency brake.
Candidate 4: The white sedan lost its control.
Answer: The blue truck did an emergency brake. <lendofchunkl>
Question: What might be the reason which led to this accident? Candidatel: The black sedan exceeded the speed limit Candidate2: The white sedan did an illegal lane changing Candidate3: Extreme weather condition
Candidate4: The blue truck violated the traffic lights Answer: The white sedan did an illegal lane changing. <lendofchunkl>
<image>
Question: (} Candidatel :0) Candidate2: () Candidate3: candidate4: (
Answer: " format
questions
ans ans_candidates [O][i].
ans ans cas_candidates [i][i],
ans ans_ ans_candidates [i][
ans ans_candidates [i]
Table A5: Comparison with LLM-based methods on SUTD-TraffiQQA
OpenFlamingo (2 examples)
OpenFlamingo (4 examples)
CL.IP
LLCP
Accuracy
28.7
29.1
27.7
33.7
A6.3 MORE EXPERIMENTAL RESULTS ON CAUSALVIDQA
We have included the results reported in the original Causa paper al. (2022a). As detailed in table A6, we present the results of all baselines using the pre-trained 300-dimensional GloVe Pennington al. (2014) word embeddings. It is evident that the CLIP features outperform the Glove features consistently. In our approach, we solely utilize the CLIP feature and eschew the Glove feature, owing the pre-alignment of the visual and textual CLIP features. Notably, our method does not leverage annotated data and, thus, cannot learn cross-modality matching from scratch.
A6.4 PARSING RESULTS
To explain our process of grammar parsing, which used to extract key variables from the question- answers, share detailed parsing results varied question types. In Table A7, we enumerate the type reasoning question, the original question itself. our logic rules employed for parsing, and the resulting keywords. Consider, for instance, Open-choice Counterfactual Prediction
21

---

Published as a conference paper at ICLR 2024
Table A6: Comparison with baseline methods on Causal-VidQA. We re-train all methods using the same CLIP features using the officially released code. We also provide the results that use the glove features. 'f" indicates the result re-implemented by the official code.
Test_G
Methods
QA
Features
Avg
Q A
Q R
Q -
AR
EVQA Antol et al. (2015)
Glove
27.72
27.57
10.63
21.97
EVQAT Antol et al. (2015)
CLIP
28.05
28.05
10.09
22.00
CoMem Gao et al. (2018)
Glove
42.97
42.24
22.25
35.82
CoMem i
Gao et al. (2018)
CLIP
45.70
47.60
25.44
39.83
HME Fan et al. (2019)
Glove
35.29
34.19
15.34
28.27
HME Fan et al. (2019)
CLIP
45.48
46.51
24.74
39.25
Yes
HCRN Le et al. (2020)
Glove
43.69
43,47
22.75
36.64
HCRNí Le et al. (2020)
CLIP
44.26
45.64
24.35
38.49
HGA Jiang & Han (2020)
Glove
44.00
44.04
23.63
37.22
HGA Jiang & Han (2020)
CLIP
45.28
46.80
24.81
38.77
B2A Park et al. (2021)
Glove
45.12
44.99
25.29
38.47
B2A Park et al. (2021)
CLIP
47.41
48.74
27.39
41.51
CLIP Radford et al. (2021)
CLIP
29.95
29.51
11.48
23.73
No
LLCP
CLIP
39.07
38.46
19.03
32.13
question:" Whaa will happen if [person_5] has ingrown hairs cuts". Initially, we classify its question type and determine the pertinent question word. Following this, we identify the conditional clause by pinpointing the word if", This conditional clause then serves as the keywords necessary for responding to counterfactual prediction questions. Delving further, we can pinpoint key objects that as the subject within the clause.
22

---

Published as a conference paper at ICLR 2024
Table A7: Examples of parsing reasoning questions on SUTD-TraffiQAA and Causal-VidQA datasets.
Reasoning Question Type
Open-choice Attribution
Question
What could possibly cause this accident?
Identify the "What could" questions and Attrihution question type. Identify the effect is "the accident".
Rules
It means that the keyword is the answer candidates.
Keywords
In the answer candidates
Reasoning Question Type
Open-choice Attribution
Question
Which might be the reason for this accident?
Identify the "Which might be" questions and Attribution question type. Identify the effect is "the accident".
Rules
It means that the keyword is the answer candidates.
Keywords
the answer candidates
Reasoning Question Type
Open-choice Counterfactual Prediction
Question
What would happen if [person_22 2] dropped the plates?
Identify the "What would happen questions and Counterfactual question types.
Rules
Parse the question and find the conditional clause introduced by "if"..
Keywords
[person 2] dropped the plates
Reasoning Question Type
Open- choice Counterfactual Prediction
Question
What will happen [person 5] has ingrown hairs or cuts?
Identify the "What will happen" questions and Counterfactual question types. Parse the question and find the conditional clause introduced by "if".
Rules
Keywords
[person S] has ingrown hairs
Reasoning Question Type
Counterfactual True-or-false
Question
Will an accident happen if the vehicle in front suddenly stop?
Identify the "Will happen if" questions and Truc-or-falss question types. Parse the question and find the pattern "... happen .f." Parse the conditional clause and find the noun "vehicle".
Rules
Keywords
Vehicle
Reasoning Question Type
Counterfactual True-or-false
Question
Would accident happen vehicles drive their correct lane?
Identify the "Would happen" questions and True-or-false question types. Parse the question and find the pattern " ... happen if... Parse the conditional clause and find the nouns "vehicles" and "lane".
Rules
Keywords
Vehicles and Lane
Reasoning Question Type
Introspection True-or-false
Question
Could the accident be prevented if all vehicles keep a safe distance away from one another?
Identify the "Could ... prevented if" questions and Truc-or-false question types. Parse the question and find the pattern ... prevented if ..."
Rules
Parse the conditional clause and find the nouns "vehicles" and "distance". Align with the logic: cause - accident. Got "vehicles" and "distance"
Keywords
Vehicles and Distance
23


Reviews:
[{"review_id": "3koYwBUYt0", "reviewer": "Reviewer_6hfp", "summary": "The paper introduces a novel causal framework, LLCP, that advances the field of VideoQA by focusing on self-supervised learning of spatial-temporal dynamics without the need for annotated data. The model is adept at reasoning about video content through accident attribution and counterfactual predictions, leveraging a generative model and natural language processing to generate answers. This approach demonstrates comparable performance to supervised methods, showcasing its potential to reduce reliance on large annotated datasets and enhance the AI's understanding of causality in videos.", "questions": "Could the authors provide a comprehensive list of the limitations and assumptions inherent in LLCP, and discuss how these might affect the generalizability and applicability of the model?\nCould you provide additional empirical evidence or case studies that demonstrate LLCP's specific capability to uncover underlying causal relations as opposed to merely correlational patterns?\nIs there a quantitative evaluation comparing LLCP's causal reasoning capacity with that of existing approaches, and if so, what benchmarks or metrics were used?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper in question exhibits a commendable level of originality by shifting the focus of Video Question Answering (VideoQA) from pattern recognition to causal reasoning, an approach that has not been extensively explored in this field. The introduction of the Learning Latent Causal Processes (LLCP) framework marks a creative synthesis of self-supervised learning, generative modeling, and natural language processing, tailored to decipher the causal dynamics of video content without relying on annotated question-answer pairs. This methodological innovation reflects the paper's high quality, as it seemingly adheres to rigorous empirical standards and offers a robust validation on both synthetic and real-world datasets. In terms of clarity, the paper articulates its contributions and methodologies with precision, making the novel concepts accessible and understandable, which is indicative of the authors' commitment to effective communication of complex ideas. The significance of this work is multifold, promising to reduce the need for labor-intensive labeled data in VideoQA, enhance the interpretive and interactive capabilities of AI systems with causal reasoning, and potentially influencing a range of applications where understanding the underlying causal relationships in visual data is crucial. Overall, this paper appears to make a substantial and meaningful contribution to the literature, potentially setting a new course for future research in the AI domain, with implications that extend beyond the immediate scope of VideoQA.", "weaknesses": "In the paper, potential areas for improvement include enhancing the model's robustness to spurious correlations and label noise, more explicitly demonstrating LLCP's ability to capture causal relations through additional experiments, and benchmarking its causal reasoning against current approaches. The paper could also benefit from a more detailed discussion of the challenges in video reasoning it aims to address and a clearer explanation of its operation independent of established causal frameworks and annotations. Finally, a dedicated section that explicitly outlines the paper's limitations and assumptions would add transparency and guide future research directions. Addressing these points could strengthen the paper's contributions and its value to the VideoQA field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8l59fdwInP", "reviewer": "Reviewer_dMTZ", "summary": "The Learning Latent Causal Processes (LLCP) framework introduces a novel approach to Video Question Answering (VideoQA) by focusing on causal reasoning rather than traditional cross-modality matching. LLCP utilizes a multivariate generative model to analyze spatial-temporal dynamics and trains through self-supervised local auto-regression, thus eliminating the need for annotated question-answer pairs. It adeptly handles accident attribution and counterfactual prediction tasks, identifying root causes and forecasting potential outcomes through modifications in variable embeddings.", "questions": "Please check weakness for details.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of learning causality from video is interesting.\n2. The paper is overall presented clearly.\n3, The authors made efforts on providing fair comparisons with existing methods.", "weaknesses": "1. The main concern is that the current setting is too far from realistic settings. The current evaluation setting is really more like hacking parts of existing datasets. The reviewer encourage the authors to make this work more complete.\n\na. It is too constraint to only evaluate the proposed method when it has no access to QA labels. If the obtain model really captures the casual relationship in videos, plugging it into existing methods in the supervised training setting can show a much more broad application of the proposed method.\n\nb. It is also not realistic to exclude the text query from the training process since the fusion between visual and textual input is also the crucial design to really solve the VQA problem. For example, if there are two accidents going on in the video, the current framework will have systematical flaw as the model is not conditioned on the question text which specifies which accident it is about.\n\nc. Despite the effort of re-training many existing methods, it is not well-justified why it is necessary to discard important features like motion or object as used in Causal-Vid-QA, which brings all the models to a low-performance scheme.\n\nd. There is no proper comparison with methods that do not require QA data like but not including to [a,b,c]. The authors should also acknowledge and at least provide comparison with some of these relevant methods to really provide the audience a correct and comprehensive understanding of the relevant solution to this setting.\n\ne. Once a and b are done, it is also necessary to provide additional comparison on broader VideoQA datasets to understand the importance of causal learning process in videos for broader videoQA tasks, which is really beneficial for the community.\n\n\n\n[a] 🦩Flamingo: a Visual Language Model for Few-Shot Learning\n[b] Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners\n[c] Language Models are Causal Knowledge Extractors for Zero-shot Video Question Answering\n\n\nMinor:\n1. Title, related work: Video Question Answer -> Video Question Answering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Hxdm3kJch", "reviewer": "Reviewer_naVu", "summary": "This paper focused on the task of reasoning-based video QA. The challenge is the lack of causal relations in the cross-modal matching pipelines. This work employs a temporal multivariate generative model to understand the latent causal patterns, which can be used to answer caused-based and effect-based questions about videos. The experiments are conducted on two simulation datasets and two real-world VideoQA datasets.", "questions": "1. What does the red node in Figure 1 (b) mean? What it is red during test pipeline but not highlighted during training pipeline?\n\n2. The abstract mentioned that the proposed LLCP employs a temporal multivariate generative model to understand the causal patterns. I am not aware of how the *temporal* model can discover *causal* relations. I am wondering how to guarantee that the learned pattern are causal relations rather than temporal correlations using the so-called temporal model? In the method part, I am not aware of how the learned pattern are causality rather than correlation.\n\n3. I didn't find strict definitions of historical state, neighborhoods, and environment variables. Could the authors provide a precise and accurate definition, or give examples of these three variables? For example, are they features of a single frame, an object, or a set of tracked frames? How does the object tracking model obtain these three variables in temporal and spatial dimensions? It seems that Figure 4 provide an example, but it appears a bit late when I was reading Sec. 3.1 but didn't find the examples.\n\n4. In Ex. (2), is x_{t,i} an image (or region of interest) or feature vector?\n\n5. According to Figure 4, the question seems to be the input of text side rather than video side. In this case, how can we determine the object of interest when extract the visual information and make the prediction of visual states? Is it reasonable to extract question-independent visual feature to answer the question?\n\n6. What does the arrows mean in Figure 5? Are they drew manually or automatically estimated by the model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ The proposed model can achieve comparable performance to supervised methods while no paired textual annotations are used. The proposed framework can answer both cause-based (i.e., accident attribution) and effect-based (i.e., counterfactual prediction) reasoning questions. It shows the potential of self supervision.\n\n+ The motivation of explore latent causal relations for video QA makes sense and is interesting.", "weaknesses": "- The presentation is not satisfying enough. It is hard for me to figure out the connection between motivation and implementation details, especially how the method can guarantee that the learned relation is causal rather than temporal, and the terminologies of variables. Please see \"questions\" for detailed comments.\n\n- Another main concern is whether the model is causality-based or just capture temporal relations. In addition to the concept of causal relation, I didn't see how the implementation reflect the tools of causal inference or causal reasoning. Therefore, I am wondering whether the causal understanding ability is over-claimed. Experimental results can verify the ability empirically, but theoretical explanations or guarantees are missing.\n\n- The ablation studies in Table 4 mainly demonstrate the contributions of sub-networks rather than the role of historical states and environment. What if we replace the historical states and environment with wrong ones? What is the performance then? That would show that the performance drop is due to the lack of visual information rather than fewer parameters.\n\n\n---\n\nAfter rebuttal: the authors' responses addressed my concerns including causal theory, ablation studies, and clarification on terminologies. I didn't carefully check the details of Granger Causality as it is out of my knowledge but assume that the usage of Granger Causality is correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62p9FdGMsQ", "reviewer": "Reviewer_DRzn", "summary": "This paper introduces a video reasoning framework called Learning Latent Causal Processes (LLCP). LLCP first makes use of a conditional VAE framework to extract the spatio-temporal dynamics of objects in videos in a self-supervised manner. These extracted dynamics then serve as weak supervision to analyze the relationships between events during inference time, facilitating the identification of root cause of accidents and counterfactual effects. Tested on both synthetic and real-world benchmarks, the proposed method demonstrates advantage over baselines while still underperforming supervised methods.", "questions": "Apart from my concerns in the Weakness section. I have some other questions:\n\n- Regarding the first task of root cause analysis, is there any reference for the definition of root causes in Sec. 3.2? The provided definition seems to be applicable with only anomaly/outlier detection.\n\n- In Sec. 3.4, what happens if we formulate counterfactual questions in an open-ended format, i.e. “what if”?\n\n- The outlier analysis at inference times relies on extracted features from pretrained models (i.e., CLIP) which can be brittle if tested data is outside of the trained distribution. In addition, using language parser to parse user queries is not ideal as well. Have you thought of an alternative solution towards this?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- It is well-motivated and a desired but missing capability in AI systems to model causal relationships between objects/events in video understanding.\n\n- The implementation seems straightforward.\n\n- The proposed method demonstrates promising results when compared with some baseline methods and its supervised counterparts on both synthetic and real-world benchmarks.", "weaknesses": "- The novelty of the paper is limited. While the paper draws motivation from causal modeling, both the theoretical analysis and the implementation of TMGM using conditional VAEs do not adequately realize the causal processes in Eq. 1. \n\n- It is not clear how the authors enforce the independence constraints, both spatially and temporally, on the noise variables by the three subnetworks f_N, f_H and f_E. I would say the fusion of these three networks encodes spatio-temporal dynamics across objects (agents) but it is unclear if this has any bearing on causal relations.\n\n- Experiments only compare LLCP with weaker variants of VAEs which offer little to no  temporal modeling. In addition, the proposed method largely underperforms supervised methods as indicated by Table 5 and Table 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wGe6OyYe3w", "reviewer": "Reviewer_U5ds", "summary": "This work introduces a weakly-supervised approach for reasoning about accidental events, leveraging causal representation learning combined with experimental tests on the derived causal variables. These causal variables, when integrated with CLIP, are aligned into natural language, forming an explainable AI (XAI) system. The proposed method undergoes rigorous analysis on simulated data and demonstrates comparable, performance to recent supervised techniques when tested on two real-world datasets.", "questions": "See the weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-structured, appealing to a broad readership.\n2. It effectively applies causal learning to a real-world scenario, using a robust implementation with statistical tests on TMGM-derived causal variables.\n3. The technique innovatively connects causal variables and natural language through CLIP, potentially reshaping standards in explainable AI (XAI).\n4. Evaluation is thorough, including simulations and comparisons with supervised methods.\n5. Impressively, the weakly-supervised approach matches the performance of supervised methods.", "weaknesses": "The authors primarily use a generative model for potential causes and treat language as a secondary filter. This approach however neglects language cues. Is that possible to have the lanuage involved for the genertive process? While the experiments show good improvements, could the author clarify the motivation of unsupervised generative approach over the supervised ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)