Year: 2021

Abstract:
Causal discovery, i.e., inferring underlying cause-effect relationships from observations of a scene or system, is an inherent mechanism in human cognition, but has been shown to be highly challenging to automate. The majority of approaches in the literature aiming for this task consider constrained scenarios with fully observed variables or data from stationary time-series. In this work we aim for causal discovery in a more general class of scenarios, scenes with non-stationary behavior over time. For our purposes we here regard a scene as a composition objects interacting with each other over time. Nonstationarity is modeled as stationarity conditioned on an underlying variable, a state, which can be of varying dimension, more or less hidden given observations of the scene, and also depend more or less directly on these observations. We propose a probabilistic deep learning approach called State-Dependent Causal Inference (SDCI) for causal discovery in such conditionally stationary time-series data. Results in two different synthetic scenarios show that this method is able to recover the underlying causal dependencies with high accuracy even in cases with hidden states.

text: Causal discovery, i.e., inferring underlying cause-effect relationships from observations of a scene or system, is an inherent mechanism in human cognition, but has been shown to be highly challenging to automate. The majority of approaches in the literature aiming for this task consider constrained scenarios with fully observed variables or data from stationary time-series. In this work we aim for causal discovery in a more general class of scenarios, scenes with non-stationary behavior over time. For our purposes we here regard a scene as a composition objects interacting with each other over time. Nonstationarity is modeled as stationarity conditioned on an underlying variable, a state, which can be of varying dimension, more or less hidden given observations of the scene, and also depend more or less directly on these observations. We propose a probabilistic deep learning approach called State-Dependent Causal Inference (SDCI) for causal discovery in such conditionally stationary time-series data. Results in two different synthetic scenarios show that this method is able to recover the underlying causal dependencies with high accuracy even in cases with hidden states.

heading: 1 INTRODUCTION
text: The ability of deep learning approaches to discover and reason about causal relationships in data has become a prominent direction of work over the recent years (Yi et al., 2020; Girdhar & Ramanan, 2020; Sauer & Geiger, 2021). Despite the recent success of deep learning methods in related tasks such as classification, localization, and segmentation, causal discovery and reasoning, an inherent mechanism in human cognition (Spelke & Kinzler, 2007) allowing reasoning about counterfactuals and understanding the reasons of events, still poses an considerable challenge.
Causal discovery involves uncovering the underlying logic, temporal and causal structure of the observed processes in the data. Current approaches (see Section 2) commonly address quite constrained scenarios with a stationary behavior over time. In the present paper, we extend the current work by addressing scenarios with conditional stationarity, where the dynamics of the observed system changes with the value of underlying variables. This is the case in almost all real-world scenarios, e.g. with people who behave differently and take different decisions depending on underlying factors such as mood, previous experience, and the actions of other agents. We propose a method (see Section 3) for causal discovery from time-series observations of systems where the underlying causal graph changes depending on a state variable.
The causal discovery task from such conditionally stationary time-series poses different challenges depending on the observability of the underlying state variable. Four scenario classes can be seen:
1. The first class concerns a simplified version of the problem, where the state variable is observed and not dependent on other observed time-series data.
2. In the second class of scenarios, the state is not directly observed, but directly dependent on and continuously inferable from an observed variable. A real-life example is a traffic scenario where taxis (visually distinguishable by the sign on their roof) follow slightly different rules than normal cars, i.e. are allowed to drive in bus lanes.
3. A more challenging scenario class is when the state depends on earlier events, and thus is not continuously observable. A real-life example is a chain of events in a football game, where the action of one player is triggered by an earlier action by another player.
4. Finally, a large share of scenarios in the real world are governed by underlying state variables that are not fully inferable from the observations from the scenario over time. In such scenarios, the state is an unknown confounder to the observed time-series, and causal discovery from such scenarios is inherently ill-defined.
We evaluate the method (see Section 4) in two different synthetic scenarios, where we vary the complexity of system dynamics and observability of the underlying state variable covering the first three scenario classes above. Finally we conclude and discuss directions for future work (see Section 5).

heading: 2 RELATED WORK
text: Causal discovery approaches aim to identify causal relationships over a set of variables from observational data. These methods can basically be classified into three different types (Glymour et al., 2019): 1) Constraint-based; 2) Score-based; 3) Functional causal model based methods.
Constraint-based methods rely on conditional independence tests to recover the underlying DAG structure of the data, such as the PC algorithm (Spirtes et al., 2000), which assumes faithfulness and causal Markov condition and considers i.i.d. sampling and no latent confounders. There exists a great variety of variations of PC. One of them is the Fast Causal Inference (FCI) (Spirtes, 2001), which is able to cope with the unknown confounders and selection bias; furthermore, it can be adapted for time-series data, such as such as tsFCI (Entner & Hoyer, 2010).
Score-based methods define score functions of causal graph structures and then optimize score functions by performing a search to identify the underlying causal structure, such as the Greedy Equivalence Search (GES) (Chickering, 2002). Notice that searching in the graph space poses a combinatorial optimization problem. Recent approaches try to avoid this by reformulating it as a continuous optimization problem which introduces a score function h for measuring the acyclicity of the graph (Zheng et al., 2018). Regarding time-series data, these methods are reformulated as learning dynamic Bayesian Networks (DBNs) from data (Murphy et al., 2002). Among these algorithms we recently find DYNOTEARS (Pamfil et al., 2020), which aims to simultaneously estimate instantaneous and time-lagged relationships between variables in a time-series.
Functional causal model-based methods represent the effect as a function of its direct causes and their independent immeasurable noise (Glymour et al., 2019). For non-temporal data, there are linear non-Gaussian acyclic models (Shimizu et al., 2006), additive noise models (Peters et al., 2014), post-nonlinear models (Zhang & Hyvärinen, 2009), etc. For temporal data, these approaches fit a dynamics model, which is often regularized in terms of its sparsity, and its form is analyzed to identify the underlying causal connections in the data. Granger causal analysis falls into this category, since we first model the dynamics and some analysis is performed to extract the latent causal structure (Granger, 1969).
Causal discovery is, in general, a challenging task and its study arises a great amount of practical issues. The problem is ill-posed when considering linearity and Gaussian disturbances, since it can be proved that the underlying causal model is not identifiable, while under proper assumptions, such as non-Gaussianity, it becomes identifiable (Shimizu et al., 2006). When considering non-linear transformations, the symmetry between observed variables disappears, allowing the identification of causal relations in the context of Gaussian disturbances (Hoyer et al., 2008). Other practical issues consist on the existence of latent confounders (Ranganath & Perotte, 2018), the presence of measurement error (Zhang et al., 2017) or considering observations with missing data (Tu et al., 2019). In order to avoid these common problems, simplifications of the problem need to be applied. In fact, the assumptions we make in this work are: (i) all the instances belonging to the causal graph are observed, (ii) we have no missing data selection bias, and (iii) no latent confounders exist.
The work most related to ours are the approaches by (Löwe et al., 2020; Li et al., 2020); we extend these by allowing the causal model of the underlying process to vary depending on a state variable. Our method can in the future be applied to a wider class of non-stationary visual scenarios where the
interacting objects are only partially and noisily observed as semantically segmented visual regions, or by tracked image keypoints (Löwe et al., 2020). This would allow addressing challenging tasks such as scene understanding, counterfactual reasoning, etc. The recent work by (Sauer & Geiger, 2021) also uses the concept of causality for a similar task, generation of counterfactual images.

heading: 3 STATE-DEPENDENT CAUSAL INFERENCE
text: In this section, we introduce our formulation to extract causal graphs from time-series data where their dynamics are altered by means of a categorical variable, referred to as their state. We refer to our method as State-Dependent Causal Inference (SDCI).

heading: 3.1 PROBLEM FORMULATION
text: The input consists of a set of N time-series which not only obey some dynamics that might change over time but also undergo different states along the sequence. These states are responsible for the changes in the dynamics of the sample. We observe the sequence for a total of T frames and we denote the sample x as
x = {{x1i }Ni=1, ..., {xTi }Ni=1}, xti = {pti, sti}, (1)
where {s1:Ti }Ni=1 represents the hidden states and {p1:Ti }Ni=1 are the observed quantities of interest. For simplicity, we drop the subscript when referring to all the elements in a single time-step (e.g. xt, pt, st, etc). In a causal graph, the observed quantities are represented by the vertices and the edge of the causal graph represents the interaction type between vertices. We denote the amount of possible interaction types by n .
Assumptions. In this work, we assume that the data generation process obeys a structural causal model (SCM) (Pearl, 2009), G1:T , and that the model satisfies the first-order Markov property. Moreover, according to the definitions of causality (Granger, 1969), we assume that edges of a causal graph cannot go back in time. The first assumption follows related works concerning samples where the generative process also follows an SCM (Löwe et al., 2020; Li et al., 2020). Although we assume the first-order Markov property, one can extend to the more general p − th order Markov property in a more complex scenario.
State-dependent causal inference. Based on the assumptions we mainly focus on the nonstationary causal graph, which means that we can find different edge-types at different times. As for an edge between two vertices, the edge-type interaction between two vertices changes according to the state of the variable which is the source of the interaction. The main focus of our method consists on extracting a causal summary graph G (also denoted as such by Li et al. (2020) and Löwe et al. (2020)). Previous approaches aiming for this task assume stationary time-series data and therefore, this causal summary graph is constant. Nonetheless, since we condition the stationarity of the samples on the states, our causal summary graph is expressed by means of this categorical variable. In other words, our method will extract K summary graphs, one per state considered. The edge-type interaction can be then queried at each time-step t as follows:
ztij = Gij(s t i) (2)
where ztij ∈ {0, ..., n − 1} denotes the edge-type interaction from i to j at time-step t. Figure 1 illustrates this task. In general, this causal summary graph is specific to the input sample and hidden from the model. Therefore, not only we require to design a parametrizable function to infer the latent causal structure, but also to evaluate how this inference fits to the actual dynamics observed in the input sequence.
Let us denote the first step of extracting the latent causal structure
G(s) = fφ(p1:T , s1:T ) (3)
where fφ denotes a parameterizable function that receives all the observed sequence as input. The next step is to fit this extracted latent sturcture into our assumed first-order Markov dynamics.
(p̃t+1, s̃t+1) = fψ(xt, G(st)) (4)
where the parameterizable function fψ represents a one-step computation of the dynamics starting from the observed values at time-step t. In this expression, we have defined fψ to predict the value of the states at the next time-step as well. However, in our experiments we will also consider settings where the states are observed at all times and modelling the dynamics is only performed with respect to the quantity p1:T . To allow for this setting, one only needs to exclude the state variable from supervision. We provide more details in the next section.
Objective. Both the causal inference and dynamics modelling modules can be optimized by minimizing some objective defined for the parameterizable functions fφ and fψ .
min φ,ψ T−1∑ t=1 L(xt+1, fψ(xt, G(st))) +R(G(·)), G(st) = fφ(x) (5)
where R(·) is a regularizer on the extracted graph structure, which can be applied to enforce a preferred interaction type.

heading: 3.2 IMPLEMENTATION
text: Since the interactions that are considered in the scene are complex, we introduce uncertainty in our approach. Therefore, we are interested in computing the probability distribution over the edge type given two instances and the state of the source instance, p(zij |si,pi,pj). This implementation gathers inspiration from related works concerning this type of tasks (Li et al., 2020; Löwe et al., 2020; Kipf et al., 2018).
The structure of the implementation is very similar to a VAE (Kingma & Welling, 2014). In this case, the latent space is represented by the set of edges zij which are discrete variables. The latent edges are conditioned to the state of the variable which is the source of the interaction, si. We allow for a total of K states. Let zij be an edge from instance i to instance j. This value represents the type of causal interaction that there is between i and j and will depend on the state of i. ztij ∈ {0, ..., n −1}, where n is the amount of inter-object interactions considered in our setting. ztij = 0 means that component i is not influenced by component j at time t. Any other value models a different type of interaction. Since we condition the edge type on the state variable, our objective is more close to the one defined in CVAE (Sohn et al., 2015).
L = Eqφ(z|s,x) [ log pψ(x|z) ] +KL ( qφ(z|s, x)||p(z|s) ) (6)
where p(z|s) is a prior defined over the edge types conditioned on each state, which acts as a regularizer over the inferred edge-type distribution. We use z to denote all the edges represented in the latent space. In our settings, we set this prior to enforce a uniform distribution no matter what the state is. We might find applications where this prior could be leveraged to encourage sparsity in the extracted graph structure.
Encoder. Following related approaches (Löwe et al., 2020), we use graph neural networks as the encoder. The model receives all the information from the sample available x1:T . We concatenate p1:T with a one-hot representation of the state variable s1:T . We aim to extract an embedding that represents the causal interaction conditioned on the state for every pair of elements present in the sample:
φij = fφ(x1:T )ij (7)
where φij ∈ Rn ×K denotes the embedding for every pair i −→ j. The approximate posterior distribution conditioned on the state variable qφ(zij |k, x) is calculated as follows:
qφ(zij |k, x) = Θ(φijk/τ) (8)
where k is the state to which the edge-type distribution of i −→ j is conditioned and Θ denotes a softmax activation with temperature τ . We use φijk ∈ Rn to the note the embedding for i −→ j at state k. Since this formulation yields a discrete latent space, we relax the resulting categorical distribution using the Gumble-softmax (Maddison et al., 2017) technique to enable back-propagation.
Once we have inferred the approximate posterior distribution over the edge-types, we can sample the interactions at every time-step t conditioning on st:
ztij ∼ qφ(zij |sti, x) (9)
Notice that this requires sampling at each time-step t, which might pose great computational expenses for large sequences. In practice, we sample right after determining the distribution of the edge-types. Then, the interactions are queried for each i −→ j depending on sti:
wijk ∼ qφ(zij |k, x), ztij = wijk′ , k′ = sti (10)
where we have stored the sampled values using wijk ∈ Rn , which denotes the edge-type using a one-hot representation. In our experiments, we refer to SDCI-Static or SDCI-Temporal for implementations based on MLPs or CNNs respectively. For more details, see the supplementary material.
Decoder. Let us describe the decoder pψ(x|z), which models the dynamics of the generative process. Consider the dynamics modelling for element j. At each time t, we first query the edge-types depending on the states of the source variable i from the sampled quantities wijk (see equation 10).
ztij = wijk, k = s t i (11)
The information along the predicted edge-type interactions is then retrieved as follows gtij = ∑ e>0 1(ztij=e)fe(x t i, x t j) (12)
{fe}n −1e=1 is a family of parametrizable functions, one defined for each edge type excluding the no-edge interaction.
The inter-object interactions are finally integrated to model the dynamics of each variable. The updates of the variable pt+1j
p̃t+1j = p t j + fp (∑ i 6=j gtij , x t j ) (13)
are predicted by means of fp, where it provides information about the update between the actual and the next time-step. The supervision of the decoder considering p1:T uses the negative log-likelihood of the following probability distribution
pψ(pt+1j |x t, zt) = N (p̃t+1j , σ2I) (14)
where σ is the variance of the resulting Gaussian distribution.
We have previously introduced the idea of considering the state variable s1:T as being independent from the dynamics of the sample, i.e. p1:T . If this happens, our method will have information about the states of the samples at all times and consequently, no supervision upon s1:T will be required. However, we might find practical to consider the setting where the state is in fact dependent on p1:T , and therefore our method will require to model its behavior and perform supervision. In this case, we present a straightforward approach to compute the estimation of the next state:
s̃t+1j = fs (∑ i6=j gtij , x t i ) (15)
and the corresponding categorical distribution can be computed as follows
pψ(s t+1 j |x t, zt) = Θ(s̃t+1j ) (16)
where Θ is a softmax activation.
In summary, the probability distribution of the decoder can be expressed as follows:
pψ(x|z) = T−1∏ t=1 pψ(xt+1|xt, zt), pψ(xt+1|xt, zt) = pψ(pt+1|xt, zt)pψ(st+1|xt, zt) (17)
Notice that with this formulation, p and s can be expressed as separate factors. We find this result useful, since in practice we calculate the log-likelihood of both probability distributions separately and balance the resulting loss term
L = Eqφ(z|s,x) [ log pψ(p|z) ] + λEqφ(z|s,x) [ log pψ(s|z) ] +KL ( qφ(z|s, x)||p(z|s) ) (18)
where we use λ to balance the contribution of both terms to the optimization process.
Hidden state regime. When considering the state as a variable hidden from the input data, we can still sample from the estimated posterior distribution as follows:
ztij ∼ K∑ k=1 qφ(zij |k, x)p(k|x), sti ∼ p(k|x) (19)
where p(k|x) is the probability distribution which models the states at each time-step. In this setting, we modify Eq. 16 and condition solely on the object dynamic variables.
pψ(s t i|xt) = Θ(s̃ti/γ), s̃ti = fs′(xti) (20)
γ < 1 is a temperature factor which increases the confidence of the model predictions. Since we should be sampling from Eq. 19 and we consider p(k|x) = pψ(k|xt), in practice we have
ztij = K∑ k=1 wijk · pψ(k|xt) (21)

heading: 4 EXPERIMENTS
text: We now present the experiments that have been performed to evaluate the method and compare it to the baseline method, amortized causal discovery (ACD) (Löwe et al., 2020). All models have been implemented using Pytorch (Paszke et al., 2019) and all training and test processes have been carried out on NVIDIA RTX 2080Ti GPUs.

heading: 4.1 EXPERIMENTS ON LINEAR DATA
text: We start with a simple scenario with linear message passing operations between a number of different time-series, where each time-step t in series i is a one-dimensional continuous variable pti ∈ R. The variables are connected by edges of n different types. Each edge-type represents a different causal effect that one variable can perform to another. This is denoted by {βk}n −1k=1 . β0 represents no connection, i.e., no causal interaction between a pair of variables. At each time-step t, each element in the sample performs the following operation to update pi:
pt+1i = αp t i + N∑ i 6=j βkptj (22)
where α ∈ R controls the self-connection, and βk ∈ R represents the edge-type connection between j and i, of type k.
We carry out experiments in this scenario to compare the results obtained by the two proposed architectures and the baseline method ACD Löwe et al. (2020). We are interested in observing whether our formulation is capable of recovering the underlying parameters of the linear message passing mechanism and identify the causal interactions in each sample in different types of scenarios (see Section 1). In this case, we experiment with observed states that are independent from the observations, and hidden states (first and third scenario classes). Unless noted otherwise, we set K = 2 states.

heading: 4.1.1 DATA GENERATION
text: The procedure for generating this dataset is as follows. First, we set the edge-type interactions. In our experiments we set α = 1 and since we experiment with 2 edge-types, we set β1 = 0.05. To generate each sample, we need to sample the initial values of the continuous variable for each element p0i and the underlying causal structure dependent on the state, G(s). At each time-step, it suffices to query the edge-type k for each pair of variables and apply the corresponding causal effect βk following Equation 22. The edge-type is k = G(stj)ji, where G(s)ji denotes the causal effect from j to i, which has been defined at the beginning of the sequence. For all our experiments with this dataset, we simulate N = 3 variables for T = 40 time-steps. We decide to keep N and the interaction values (βk) low because the generated samples are unstable, which imply that the data is not restricted within a certain range and could be problematic for higher values of N and T .
When considering hidden states, the state is updated as follows: sti = 1(pti<0).

heading: 4.1.2 TRAINING SPECIFICATIONS
text: All the models participating in the experiments of this section have been trained following the same training scheme, including ACD (Löwe et al., 2020).
Customized decoder. One of our objectives is to recover the underlying world parameters βk. Thus, we implement a decoder which imitates the message passing operation presented in Equation 22, which allows us to initialize the decoder using the underlying world parameters and analyse the performance of the encoder as a separate entity from the whole model.
Model parameters. Following Kipf et al. (2018), the models have been trained using ADAM optimizer (Kingma & Ba, 2015). The learning rate of the encoder is 5 · 10−4, the learning rate of the decoder is 1 · 10−3, and both are decayed by a factor of 0.5 every 200 epochs. We train for 1000 epochs using a batch size of 128. We use teacher forcing every 10 time-steps during training. This implies that the decoder receives the ground-truth as input every 10 time-steps, otherwise it uses its previous output. The temperature τ is set to 0.5 and the variance of the Gaussian distribution of the decoder for p is σ2 = 5 · 10−5. When considering the setting where we make the state dependent on the dynamics of the objects, we set λ = 103. For hidden states, we set the temperature γ = 0.1.

heading: 4.1.3 RESULTS
text: We firstly consider two edge-types and only one state to search for suitable learning rates for the encoder and the decoder. The edge-type accuracy of our method considering SDCI-Static for this setting is 94.88% on training data and 94.87% on test data. Since SDCI-Static is equivalent to ACD (Löwe et al., 2020) in this setting, we use this value as a reference for the following experiments.
We now proceed to comparing the performance between the two architecture choices SCDI-Static and SCDI-Temporal with the baseline ACD, and evaluate the effect of explicitly modeling the underlying state. Furthermore, we compare the two proposed methods SCDI-Static and SCDI-Temporal with a a variant where the decoder is fixed and uses the ground-truth parameters, βk. We will in the following compare all these models considering two edge-types.
Table 1 shows the edge-type identification accuracy, the distance to the world parameters, and the reconstruction MSE for three variables, two states, and two edge-types (no-edge and β1). As we can see, our method SDCI-Static successfully performs the task of identifying the state-dependent causal interactions. Furthermore, it recovers the underlying parameters of the generative process
with great accuracy. On the other hand, our SDCI-Temporal is not able to identify properly the causal interactions nor recover the underlying parameters with enough accuracy. However, when fixing the decoder using the true underlying parameters, it achieves decent performance. Regarding our predecessor (ACD), we observe that its performance is considerably lower than our proposed architectures. One should note that this formulation can still recover the underlying world parameters, but less accurately compared to our formulation with SDCI-Static.
We also experimented with SDCI-Static when considering the state variable hidden from the input data. The results show an edge-type identification accuracy of 92.25% and 91.96% in training and test data respectively. The distance to the world parameters is 1.06·10−4. To assess the effectiveness of the model to identify different behaviors (states) when considering this setting, we can inspect the estimation of the latent state probability distribution. The state decoder accuracy of the latent state function is 99.10% and 98.97% in training and test data respectively. We can observe that our formulation allows considering hidden state variables. SDCI-Static is able to successfully identify the causal interactions between the elements and successfully decomposes the different behaviors into the actual underlying states of the generative process.

heading: 4.2 EXPERIMENTS ON SPRING DATA
text: In the second experiment setting, we evaluate our methods on data similar to that used in recent work (Kipf et al., 2018; Löwe et al., 2020), consisting of particles (or small balls) connected by springs with directed impact - meaning that e.g. particle i could affect particle j with a force through a connecting spring, but leaving particle i unaffected by this spring force.
The following experiments focus solely on evaluating the performance of our SDCI-Static and comparing it with its predecessor (ACD) under the three first scenario classes introduced in the Introduction. The data generation process follows the description of Kipf et al. (2018). The only difference is the addition of the state variable to the generative process, which affects the edge-type at each time-step as in the previous case. For more details, see the supplementary material. As in the first experimental setting, the experiments regarding this data always consider 2 edge-type connections (presence/absence of directed spring) and a sequence length of T = 80.

heading: 4.2.1 TRAINING SPECIFICATIONS
text: We use the same training scheme for all the models present in the experiments of this section. In this case, the configuration is identical to the one used by Kipf et al. (2018). Thus, the experiments have been trained using ADAM optimizer (Kingma & Ba, 2015). The learning rate of both the encoder and decoder is 5 · 10−4 and decayed by a factor of 0.5 every 200 epochs. We train for 500 epochs using a batch size of 128. We also use teacher forcing every 10 time-steps during training. The temperature τ is set to 0.5 and the variance of the Gaussian distribution of the decoder for p is σ2 = 5 · 10−5. When considering the setting where we make the state dependent on the dynamics of the objects, we set λ = 103. For hidden states, we set γ = 0.05.

heading: 4.2.2 RESULTS
text: Let us consider the first scenario class, where the state is known and independent from the observations. The state transitions incrementally into the next one every 10 time-steps, and alternative settings with up to 8 states are explored. Table 2 shows the corresponding results, where for each for state (from 1 to 8) a new dataset is generated and used to train the SDCI-Static method. We observe that with one state, the original results reported by Löwe et al. (2020) are obtained since this case corresponds to stationary time-series. As the number of states increase, the accuracy in edgetype identification decreases and the reconstruction MSE becomes larger. However, our SDCI-Static is able to maintain promising results achieving 74.87% accuracy in edge-type identification when having as much as 8 states.
We now proceed to the scenario class number 3, where the state of a particle transitions when it collides with the wall of the box where it is contained. For simplicity, we only consider two states that transition alternatively on wall collision. Our proposed method SDCI-Static achieves an edgetype identification accuracy of 85.13% on training data and 79.21% on test data. The reconstrucion MSE is 1.32 · 10−4 and 1.38 · 10−3 on training and test data respectively. Since now our decoder
requires modelling the conditions for state transition, we can evaluate its performance in identifying state changes. The state accuracy of the decoder is 99.73% on training data and 98.53% on test data, which shows that our method is able to detect the effect that the event of a particle colliding with the wall will have on the whole sample. If we consider ACD, it achieves an edge-type accuracy of 69.87% on training data and 68.61% on test data. The reconstruction MSE is 4.45 · 10−4 and 1.46 · 10−3 on training and test data respectively, and the state accuracy of the decoder is 99.13% on training data and 98.21% on test data. As we can see, the limitation of ACD of considering only stationary time-series data limits the performance of the causal graph inference task. However, the results for the reconstruction MSE are comparable to the ones obtained with our formulation (SDCIStatic), which indicates that the method still makes decent predictions although it fails in identifying the edge-type interactions.
Finally we address a scenario of class 2 as listed in the Introduction. In this scenario, the underlying state of a particle changes depending its location in the box; thus, the state is not directly observable but indeed directly dependent on an observable variable, the position of the particle. Our proposed method SDCI-Static shows an edge-type identification accuracy of 85.35% and 80.82% on training and test data respectively. The reconstruction MSE is 1.44 · 10−4 and 1.19 · 10−3 on training and test data respectively. Notice that in this case, we also find accuracy levels that are similar to the second and third layout. If we perform inspection on the estimation of the latent state variable distribution, we observe a state decoder accuracy of 98.96% in training data and 99.41% in test data. As we can see, our formulation allows to learn the dynamics of the state without any supervision. Regarding ACD, it achieves an edge-type accuracy of 71.06% in training data and 69.60% in test data. The reconstruction MSE is 4.11 ·10−4 and 1.31 ·10−3 in training and test data respectively. As before, ACD is restricted since it considers a stationary generative process. However, the behavior of the sequences in this data regime is non-stationary as the state variable is hidden. Our SDCI-Static successfully decomposes the non-stationary dynamics into the true underlying conditional stationary ones and is able to identify the causal links between particles successfully enough.

heading: 5 CONCLUSIONS
text: In this work we propose a method, SDCI, for performing causal discovery in scenes with multiple objects interacting over time, and where the dynamics depend on the value of underlying states. SDCI allows recovering the underlying causal structure of non-stationary time-series data by conditioning its stationarity on categorical state variables. Furthermore, we provide a deep probabilistic implementation of this method and perform an empirical study on two synthetic scenarios. Our results show the effectiveness of our formulation in modelling conditional time-series data, in comparison to the state-of-the-art approaches that consider stationary time-series data.

heading: 5.1 FUTURE WORK
text: This work can be regarded as a preliminary but necessary contribution towards causal discovery in video. In order to approach this goal, we propose to add a visual front-end as a pre-processing step of the SDGI module. This front-end would operate on the video and output probabilistic segmentation of video into object hypotheses, pose extraction of humans in the scene, detection and recognition of known object classes, etc.
This automatical extraction of causal abstractions of natural scenes would enable taking steps towards challenging tasks such as high-level scene understanding or counterfactual reasoning.

Review summaries:
This paper proposes a new method for discovering the causal graph from time-series data when the time-series are generated by a non-stationary process. The method relies on previous work from Lowe et al, 2020 and proposes to condition the causal summary graph driving the (causal) edge generation between variables by a categorical state variable. The method is defined within a variational inference framework, where edges are state-dependent latent variables, based on which one can generate/reconstruct future observations and/or state variables. Results are shared on two synthetic datasets with promising performances.

This paper aims to solve the problem of causal summary graph extraction and time series reconstruction at the same time. They propose a conditional VAE based model. The model is conditioned on state variables $s$, which makes it different from a normal VAE. Experiments on two datasets show the method outperforms ACD on linear data and its performance drops as number of states increase with spring data.